PLearn 0.1
Public Member Functions | Static Public Member Functions | Static Public Attributes | Protected Member Functions | Protected Attributes | Private Types
PLearn::CompressedVMatrix Class Reference

#include <CompressedVMatrix.h>

Inheritance diagram for PLearn::CompressedVMatrix:
Inheritance graph
[legend]
Collaboration diagram for PLearn::CompressedVMatrix:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 CompressedVMatrix ()
 default constructor (for automatic deserialization)
 CompressedVMatrix (int the_max_length, int the_width, size_t memory_alloc=0)
 CompressedVMatrix (VMat v, size_t memory_alloc=0)
virtual string classname () const
virtual OptionListgetOptionList () const
virtual OptionMapgetOptionMap () const
virtual RemoteMethodMapgetRemoteMethodMap () const
virtual CompressedVMatrixdeepCopy (CopiesMap &copies) const
virtual ~CompressedVMatrix ()
virtual void appendRow (Vec v)
 This method must be implemented for matrices that are allowed to grow.
size_t memoryAllocated ()
 returns the number of bytes allocated for the data
size_t memoryUsed ()
 returns the memory actually used for the data
virtual void compacify ()
 This will reallocate the data chunk to be exactly of the needed size, so that no memory will be wasted.

Static Public Member Functions

static string _classname_ ()
 RowBufferedVMatrix.
static OptionList_getOptionList_ ()
static RemoteMethodMap_getRemoteMethodMap_ ()
static Object_new_instance_for_typemap_ ()
static bool _isa_ (const Object *o)
static void _static_initialize_ ()
static const PPathdeclaringFile ()

Static Public Attributes

static StaticInitializer _static_initializer_

Protected Member Functions

void init (int the_max_length, int the_width, size_t memory_alloc)
virtual void getNewRow (int i, const Vec &v) const
 This is the only method requiring implementation in subclasses.

Protected Attributes

int max_length
 maximum number of rows that can be appended to the initially empty matrix
signed char * data
signed char ** rowstarts
signed char * dataend
 data+memory_alloc
signed char * curpos
 next writing position in data chunk

Private Types

typedef RowBufferedVMatrix inherited

Detailed Description

Like MemoryVMatrix this class holds the data in memory. But it is designed to keep a compact representation. Each row is compressed/decompressed through the methods of VecCompressor (it's the same encoding of row vectors that is used in the DiskVMatrix) This encodes only non zero values, using one byte for small integers, and 4-byte floating points for all other values. This representation should be reasonably good for both sparse matrices, and matrices containing categorical data (represented by small integers) possibly with one hot representations.

Definition at line 63 of file CompressedVMatrix.h.


Member Typedef Documentation

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 65 of file CompressedVMatrix.h.


Constructor & Destructor Documentation

PLearn::CompressedVMatrix::CompressedVMatrix ( )

default constructor (for automatic deserialization)

Definition at line 53 of file CompressedVMatrix.cc.

    : data(0), rowstarts(0), dataend(0), curpos(0)
{
}
PLearn::CompressedVMatrix::CompressedVMatrix ( int  the_max_length,
int  the_width,
size_t  memory_alloc = 0 
)

This initializes the matrix with a length of 0 (but up to the_max_length rows can be appended) If no memory_alloc value is given, a sufficient default value will be used initially. You can always call reallocateCompactMemory() later to use the minimum memory

Definition at line 58 of file CompressedVMatrix.cc.

References init(), and PLearn::VecCompressor::worstCaseSize().

{ init(the_max_length, the_width,
       memory_alloc!=0 ?memory_alloc :the_max_length*VecCompressor::worstCaseSize(the_width)); }

Here is the call graph for this function:

PLearn::CompressedVMatrix::CompressedVMatrix ( VMat  v,
size_t  memory_alloc = 0 
)

This initializes a matrix from the data of another. If no memory_alloc value is given, a sufficient default value will be used initially. You can always call reallocateCompactMemory() later to use the minimum memory

Definition at line 73 of file CompressedVMatrix.cc.

References appendRow(), i, init(), PLearn::VMat::length(), PLearn::VMat::width(), and PLearn::VecCompressor::worstCaseSize().

{
    if(memory_alloc==0)
        init(m.length(), m.width(), m.length()*VecCompressor::worstCaseSize(m.width()));
    Vec v(m.width());
    for(int i=0; i<m.length(); i++)
    {
        m->getRow(i,v);
        appendRow(v);
    }
}

Here is the call graph for this function:

PLearn::CompressedVMatrix::~CompressedVMatrix ( ) [virtual]

Definition at line 125 of file CompressedVMatrix.cc.

References data, and rowstarts.

{
    if(data)
        delete[] data;
    if(rowstarts)
        delete[] rowstarts;
}

Member Function Documentation

string PLearn::CompressedVMatrix::_classname_ ( ) [static]

RowBufferedVMatrix.

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 51 of file CompressedVMatrix.cc.

OptionList & PLearn::CompressedVMatrix::_getOptionList_ ( ) [static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 51 of file CompressedVMatrix.cc.

RemoteMethodMap & PLearn::CompressedVMatrix::_getRemoteMethodMap_ ( ) [static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 51 of file CompressedVMatrix.cc.

bool PLearn::CompressedVMatrix::_isa_ ( const Object o) [static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 51 of file CompressedVMatrix.cc.

Object * PLearn::CompressedVMatrix::_new_instance_for_typemap_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 51 of file CompressedVMatrix.cc.

StaticInitializer CompressedVMatrix::_static_initializer_ & PLearn::CompressedVMatrix::_static_initialize_ ( ) [static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 51 of file CompressedVMatrix.cc.

void PLearn::CompressedVMatrix::appendRow ( Vec  v) [virtual]

This method must be implemented for matrices that are allowed to grow.

Reimplemented from PLearn::VMatrix.

Definition at line 96 of file CompressedVMatrix.cc.

References PLearn::VecCompressor::compressVec(), curpos, dataend, PLearn::VMatrix::length_, max_length, PLERROR, and rowstarts.

Referenced by CompressedVMatrix().

{
    if(length_>=max_length)
        PLERROR("In CompressedVMatrix::appendRow, max_length exceeded");
    rowstarts[length_] = curpos;
    curpos = VecCompressor::compressVec(v,curpos);
    if(curpos>dataend)
        PLERROR("In CompressedVMatrix::appendRow not enough space reserved for data");
    ++length_;
}

Here is the call graph for this function:

Here is the caller graph for this function:

string PLearn::CompressedVMatrix::classname ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 51 of file CompressedVMatrix.cc.

void PLearn::CompressedVMatrix::compacify ( ) [virtual]

This will reallocate the data chunk to be exactly of the needed size, so that no memory will be wasted.

Reimplemented from PLearn::VMatrix.

Definition at line 107 of file CompressedVMatrix.cc.

References curpos, data, dataend, i, PLearn::VMatrix::length_, max_length, and rowstarts.

{
    size_t datasize = curpos-data;
    signed char* old_data = data;
    signed char** old_rowstarts = rowstarts;
    data = new signed char[datasize];
    dataend = data+datasize;
    curpos = dataend;
    rowstarts = new signed char*[length_];
    memcpy(data, old_data, datasize);

    for(int i=0; i<length_; i++)
        rowstarts[i] = data + (old_rowstarts[i]-old_data);
    max_length = length_;
    delete[] old_data;
    delete[] old_rowstarts;
}
static const PPath& PLearn::CompressedVMatrix::declaringFile ( ) [inline, static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 101 of file CompressedVMatrix.h.

:

CompressedVMatrix * PLearn::CompressedVMatrix::deepCopy ( CopiesMap copies) const [virtual]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 51 of file CompressedVMatrix.cc.

void PLearn::CompressedVMatrix::getNewRow ( int  i,
const Vec v 
) const [protected, virtual]

This is the only method requiring implementation in subclasses.

Implements PLearn::RowBufferedVMatrix.

Definition at line 85 of file CompressedVMatrix.cc.

References PLearn::TVec< T >::length(), PLearn::VMatrix::length_, PLERROR, rowstarts, PLearn::VecCompressor::uncompressVec(), and PLearn::VMatrix::width_.

{
#ifdef BOUNDCHECK
    if(v.length() != width_)
        PLERROR("In CompressedVMatrix::getNewRow length of v and width of matrix do not match");
    if(i<0 || i>=length_)
        PLERROR("In CompressedVMatrix::getNewRow OUT OF BOUNDS row index");
#endif
    VecCompressor::uncompressVec(rowstarts[i],v);
}

Here is the call graph for this function:

OptionList & PLearn::CompressedVMatrix::getOptionList ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 51 of file CompressedVMatrix.cc.

OptionMap & PLearn::CompressedVMatrix::getOptionMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 51 of file CompressedVMatrix.cc.

RemoteMethodMap & PLearn::CompressedVMatrix::getRemoteMethodMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 51 of file CompressedVMatrix.cc.

void PLearn::CompressedVMatrix::init ( int  the_max_length,
int  the_width,
size_t  memory_alloc 
) [protected]

This initializes the matrix with a length of 0 (but up to the_max_length rows can be appended) A memory_alloc of length*(2 + 4*width + width/128) should be enough in every case (and much less is needed for matrices containing a lot of 0 or small integers)

Definition at line 62 of file CompressedVMatrix.cc.

References curpos, data, dataend, PLearn::VMatrix::length_, max_length, rowstarts, and PLearn::VMatrix::width_.

Referenced by CompressedVMatrix().

{
    length_ = 0;
    width_ = the_width;
    max_length = the_max_length;
    data = new signed char[memory_alloc];
    dataend = data+memory_alloc;
    curpos = data;
    rowstarts = new signed char*[max_length];
}

Here is the caller graph for this function:

size_t PLearn::CompressedVMatrix::memoryAllocated ( ) [inline]

returns the number of bytes allocated for the data

Definition at line 113 of file CompressedVMatrix.h.

{ return dataend-data; }
size_t PLearn::CompressedVMatrix::memoryUsed ( ) [inline]

returns the memory actually used for the data

Definition at line 116 of file CompressedVMatrix.h.

{ return curpos-data; }

Member Data Documentation

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 101 of file CompressedVMatrix.h.

signed char* PLearn::CompressedVMatrix::curpos [protected]

next writing position in data chunk

Definition at line 75 of file CompressedVMatrix.h.

Referenced by appendRow(), compacify(), and init().

signed char* PLearn::CompressedVMatrix::data [protected]

Definition at line 70 of file CompressedVMatrix.h.

Referenced by compacify(), init(), and ~CompressedVMatrix().

signed char* PLearn::CompressedVMatrix::dataend [protected]

data+memory_alloc

Definition at line 72 of file CompressedVMatrix.h.

Referenced by appendRow(), compacify(), and init().

maximum number of rows that can be appended to the initially empty matrix

Definition at line 68 of file CompressedVMatrix.h.

Referenced by appendRow(), compacify(), and init().

signed char** PLearn::CompressedVMatrix::rowstarts [protected]

Definition at line 71 of file CompressedVMatrix.h.

Referenced by appendRow(), compacify(), getNewRow(), init(), and ~CompressedVMatrix().


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines