PLearn 0.1
|
#include <CompressedVMatrix.h>
Public Member Functions | |
CompressedVMatrix () | |
default constructor (for automatic deserialization) | |
CompressedVMatrix (int the_max_length, int the_width, size_t memory_alloc=0) | |
CompressedVMatrix (VMat v, size_t memory_alloc=0) | |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual CompressedVMatrix * | deepCopy (CopiesMap &copies) const |
virtual | ~CompressedVMatrix () |
virtual void | appendRow (Vec v) |
This method must be implemented for matrices that are allowed to grow. | |
size_t | memoryAllocated () |
returns the number of bytes allocated for the data | |
size_t | memoryUsed () |
returns the memory actually used for the data | |
virtual void | compacify () |
This will reallocate the data chunk to be exactly of the needed size, so that no memory will be wasted. | |
Static Public Member Functions | |
static string | _classname_ () |
RowBufferedVMatrix. | |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Protected Member Functions | |
void | init (int the_max_length, int the_width, size_t memory_alloc) |
virtual void | getNewRow (int i, const Vec &v) const |
This is the only method requiring implementation in subclasses. | |
Protected Attributes | |
int | max_length |
maximum number of rows that can be appended to the initially empty matrix | |
signed char * | data |
signed char ** | rowstarts |
signed char * | dataend |
data+memory_alloc | |
signed char * | curpos |
next writing position in data chunk | |
Private Types | |
typedef RowBufferedVMatrix | inherited |
Like MemoryVMatrix this class holds the data in memory. But it is designed to keep a compact representation. Each row is compressed/decompressed through the methods of VecCompressor (it's the same encoding of row vectors that is used in the DiskVMatrix) This encodes only non zero values, using one byte for small integers, and 4-byte floating points for all other values. This representation should be reasonably good for both sparse matrices, and matrices containing categorical data (represented by small integers) possibly with one hot representations.
Definition at line 63 of file CompressedVMatrix.h.
typedef RowBufferedVMatrix PLearn::CompressedVMatrix::inherited [private] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 65 of file CompressedVMatrix.h.
PLearn::CompressedVMatrix::CompressedVMatrix | ( | ) |
PLearn::CompressedVMatrix::CompressedVMatrix | ( | int | the_max_length, |
int | the_width, | ||
size_t | memory_alloc = 0 |
||
) |
This initializes the matrix with a length of 0 (but up to the_max_length rows can be appended) If no memory_alloc value is given, a sufficient default value will be used initially. You can always call reallocateCompactMemory() later to use the minimum memory
Definition at line 58 of file CompressedVMatrix.cc.
References init(), and PLearn::VecCompressor::worstCaseSize().
{ init(the_max_length, the_width, memory_alloc!=0 ?memory_alloc :the_max_length*VecCompressor::worstCaseSize(the_width)); }
PLearn::CompressedVMatrix::CompressedVMatrix | ( | VMat | v, |
size_t | memory_alloc = 0 |
||
) |
This initializes a matrix from the data of another. If no memory_alloc value is given, a sufficient default value will be used initially. You can always call reallocateCompactMemory() later to use the minimum memory
Definition at line 73 of file CompressedVMatrix.cc.
References appendRow(), i, init(), PLearn::VMat::length(), PLearn::VMat::width(), and PLearn::VecCompressor::worstCaseSize().
{ if(memory_alloc==0) init(m.length(), m.width(), m.length()*VecCompressor::worstCaseSize(m.width())); Vec v(m.width()); for(int i=0; i<m.length(); i++) { m->getRow(i,v); appendRow(v); } }
PLearn::CompressedVMatrix::~CompressedVMatrix | ( | ) | [virtual] |
string PLearn::CompressedVMatrix::_classname_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 51 of file CompressedVMatrix.cc.
OptionList & PLearn::CompressedVMatrix::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 51 of file CompressedVMatrix.cc.
RemoteMethodMap & PLearn::CompressedVMatrix::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 51 of file CompressedVMatrix.cc.
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 51 of file CompressedVMatrix.cc.
Object * PLearn::CompressedVMatrix::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 51 of file CompressedVMatrix.cc.
StaticInitializer CompressedVMatrix::_static_initializer_ & PLearn::CompressedVMatrix::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 51 of file CompressedVMatrix.cc.
void PLearn::CompressedVMatrix::appendRow | ( | Vec | v | ) | [virtual] |
This method must be implemented for matrices that are allowed to grow.
Reimplemented from PLearn::VMatrix.
Definition at line 96 of file CompressedVMatrix.cc.
References PLearn::VecCompressor::compressVec(), curpos, dataend, PLearn::VMatrix::length_, max_length, PLERROR, and rowstarts.
Referenced by CompressedVMatrix().
{ if(length_>=max_length) PLERROR("In CompressedVMatrix::appendRow, max_length exceeded"); rowstarts[length_] = curpos; curpos = VecCompressor::compressVec(v,curpos); if(curpos>dataend) PLERROR("In CompressedVMatrix::appendRow not enough space reserved for data"); ++length_; }
string PLearn::CompressedVMatrix::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 51 of file CompressedVMatrix.cc.
void PLearn::CompressedVMatrix::compacify | ( | ) | [virtual] |
This will reallocate the data chunk to be exactly of the needed size, so that no memory will be wasted.
Reimplemented from PLearn::VMatrix.
Definition at line 107 of file CompressedVMatrix.cc.
References curpos, data, dataend, i, PLearn::VMatrix::length_, max_length, and rowstarts.
{ size_t datasize = curpos-data; signed char* old_data = data; signed char** old_rowstarts = rowstarts; data = new signed char[datasize]; dataend = data+datasize; curpos = dataend; rowstarts = new signed char*[length_]; memcpy(data, old_data, datasize); for(int i=0; i<length_; i++) rowstarts[i] = data + (old_rowstarts[i]-old_data); max_length = length_; delete[] old_data; delete[] old_rowstarts; }
static const PPath& PLearn::CompressedVMatrix::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 101 of file CompressedVMatrix.h.
:
CompressedVMatrix * PLearn::CompressedVMatrix::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 51 of file CompressedVMatrix.cc.
This is the only method requiring implementation in subclasses.
Implements PLearn::RowBufferedVMatrix.
Definition at line 85 of file CompressedVMatrix.cc.
References PLearn::TVec< T >::length(), PLearn::VMatrix::length_, PLERROR, rowstarts, PLearn::VecCompressor::uncompressVec(), and PLearn::VMatrix::width_.
{ #ifdef BOUNDCHECK if(v.length() != width_) PLERROR("In CompressedVMatrix::getNewRow length of v and width of matrix do not match"); if(i<0 || i>=length_) PLERROR("In CompressedVMatrix::getNewRow OUT OF BOUNDS row index"); #endif VecCompressor::uncompressVec(rowstarts[i],v); }
OptionList & PLearn::CompressedVMatrix::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 51 of file CompressedVMatrix.cc.
OptionMap & PLearn::CompressedVMatrix::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 51 of file CompressedVMatrix.cc.
RemoteMethodMap & PLearn::CompressedVMatrix::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 51 of file CompressedVMatrix.cc.
void PLearn::CompressedVMatrix::init | ( | int | the_max_length, |
int | the_width, | ||
size_t | memory_alloc | ||
) | [protected] |
This initializes the matrix with a length of 0 (but up to the_max_length rows can be appended) A memory_alloc of length*(2 + 4*width + width/128) should be enough in every case (and much less is needed for matrices containing a lot of 0 or small integers)
Definition at line 62 of file CompressedVMatrix.cc.
References curpos, data, dataend, PLearn::VMatrix::length_, max_length, rowstarts, and PLearn::VMatrix::width_.
Referenced by CompressedVMatrix().
{ length_ = 0; width_ = the_width; max_length = the_max_length; data = new signed char[memory_alloc]; dataend = data+memory_alloc; curpos = data; rowstarts = new signed char*[max_length]; }
size_t PLearn::CompressedVMatrix::memoryAllocated | ( | ) | [inline] |
returns the number of bytes allocated for the data
Definition at line 113 of file CompressedVMatrix.h.
size_t PLearn::CompressedVMatrix::memoryUsed | ( | ) | [inline] |
returns the memory actually used for the data
Definition at line 116 of file CompressedVMatrix.h.
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 101 of file CompressedVMatrix.h.
signed char* PLearn::CompressedVMatrix::curpos [protected] |
next writing position in data chunk
Definition at line 75 of file CompressedVMatrix.h.
Referenced by appendRow(), compacify(), and init().
signed char* PLearn::CompressedVMatrix::data [protected] |
Definition at line 70 of file CompressedVMatrix.h.
Referenced by compacify(), init(), and ~CompressedVMatrix().
signed char* PLearn::CompressedVMatrix::dataend [protected] |
data+memory_alloc
Definition at line 72 of file CompressedVMatrix.h.
Referenced by appendRow(), compacify(), and init().
int PLearn::CompressedVMatrix::max_length [protected] |
maximum number of rows that can be appended to the initially empty matrix
Definition at line 68 of file CompressedVMatrix.h.
Referenced by appendRow(), compacify(), and init().
signed char** PLearn::CompressedVMatrix::rowstarts [protected] |
Definition at line 71 of file CompressedVMatrix.h.
Referenced by appendRow(), compacify(), getNewRow(), init(), and ~CompressedVMatrix().