PLearn 0.1
|
A VMatrix whose (compressed) data resides in a directory and can span several files. More...
#include <DiskVMatrix.h>
Public Member Functions | |
DiskVMatrix () | |
DiskVMatrix (const PPath &the_dirname, bool readwrite=false, bool call_build_=true) | |
DiskVMatrix (const PPath &the_dirname, int the_width, bool write_double_as_float=false) | |
virtual void | putRow (int i, Vec v) |
virtual void | appendRow (Vec v) |
This method must be implemented for matrices that are allowed to grow. | |
virtual void | flush () |
For matrices stored on disk, this should flush all pending buffered write operations. | |
virtual void | build () |
Simply calls inherited::build() then build_(). | |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Transform a shallow copy into a deep copy. | |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual DiskVMatrix * | deepCopy (CopiesMap &copies) const |
virtual | ~DiskVMatrix () |
Static Public Member Functions | |
static string | _classname_ () |
DiskVMatrix. | |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
PPath | dirname |
double | tolerance |
Static Public Attributes | |
static const int | NO_LENGTH_SENTINEL = -42 |
static const int | HEADER_OFFSET_LENGTH = 4 |
static StaticInitializer | _static_initializer_ |
Protected Member Functions | |
virtual void | getNewRow (int i, const Vec &v) const |
This is the only method requiring implementation in subclasses. | |
virtual void | closeCurrentFiles () |
Close all current files opened by this DiskVMatrix. | |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declare this class' options. | |
Protected Attributes | |
FILE * | indexf |
TVec< FILE * > | dataf |
bool | freshnewfile |
bool | old_format |
bool | swap_endians |
bool | last_op_was_append |
Private Types | |
typedef RowBufferedVMatrix | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. |
A VMatrix whose (compressed) data resides in a directory and can span several files.
Each row is compressed/decompressed through the methods of VecCompressor
Definition at line 57 of file DiskVMatrix.h.
typedef RowBufferedVMatrix PLearn::DiskVMatrix::inherited [private] |
Reimplemented from PLearn::RowBufferedVMatrix.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 59 of file DiskVMatrix.h.
PLearn::DiskVMatrix::DiskVMatrix | ( | ) |
Definition at line 110 of file DiskVMatrix.cc.
References PLearn::VMatrix::writable.
: indexf (0), freshnewfile(false), old_format (false), swap_endians(false), last_op_was_append(false), tolerance (1e-6) { writable = false; }
PLearn::DiskVMatrix::DiskVMatrix | ( | const PPath & | the_dirname, |
bool | readwrite = false , |
||
bool | call_build_ = true |
||
) |
Opens an existing one. If directory does not exist or has missing files, an error is issued. If readwrite is true, then the files are opened in read/write mode and appendRow can be called. If readwrite is false (the default), then the files are opened in read only mode, and calling appendRow will issue an error.
Definition at line 121 of file DiskVMatrix.cc.
References build_(), dirname, PLearn::PPath::removeTrailingSlash(), and PLearn::VMatrix::writable.
: inherited (call_build_), indexf (0), freshnewfile(false), old_format (false), swap_endians(false), last_op_was_append(false), dirname (the_dirname), tolerance (1e-6) { writable = readwrite; dirname.removeTrailingSlash(); // For safety. if (call_build_) build_(); }
PLearn::DiskVMatrix::DiskVMatrix | ( | const PPath & | the_dirname, |
int | the_width, | ||
bool | write_double_as_float = false |
||
) |
Create a new one. If directory already exist an error is issued (you may consider calling force_rmdir prior to this.) Howver if it is a file then the file is erased and replaced by a new directory (this was to allow TmpFilenames to be used with this class). Files are opened in read/write mode so appendRow can be called.
Definition at line 138 of file DiskVMatrix.cc.
References build_(), dirname, PLearn::PPath::removeTrailingSlash(), and PLearn::VMatrix::writable.
: inherited (0, the_width, true), indexf (0), freshnewfile(true), old_format (false), swap_endians(false), last_op_was_append(false), dirname (the_dirname), tolerance (1e-6) { writable = true; dirname.removeTrailingSlash(); // For safety. build_(); }
PLearn::DiskVMatrix::~DiskVMatrix | ( | ) | [virtual] |
Definition at line 446 of file DiskVMatrix.cc.
References closeCurrentFiles(), PLearn::VMatrix::hasMetaDataDir(), and PLearn::VMatrix::saveFieldInfos().
{ if (hasMetaDataDir()) saveFieldInfos(); DiskVMatrix::closeCurrentFiles(); }
string PLearn::DiskVMatrix::_classname_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 103 of file DiskVMatrix.cc.
OptionList & PLearn::DiskVMatrix::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 103 of file DiskVMatrix.cc.
RemoteMethodMap & PLearn::DiskVMatrix::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 103 of file DiskVMatrix.cc.
Reimplemented from PLearn::RowBufferedVMatrix.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 103 of file DiskVMatrix.cc.
Object * PLearn::DiskVMatrix::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 103 of file DiskVMatrix.cc.
StaticInitializer DiskVMatrix::_static_initializer_ & PLearn::DiskVMatrix::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 103 of file DiskVMatrix.cc.
void PLearn::DiskVMatrix::appendRow | ( | Vec | v | ) | [virtual] |
This method must be implemented for matrices that are allowed to grow.
Reimplemented from PLearn::VMatrix.
Definition at line 374 of file DiskVMatrix.cc.
References PLearn::TVec< T >::append(), PLearn::binwrite_compressed(), PLearn::TVec< T >::data(), dataf, dirname, indexf, last_op_was_append, PLearn::TVec< T >::length(), PLearn::VMatrix::length_, PLearn::new_write_compressed(), old_format, PLERROR, PLearn::TVec< T >::size(), swap_endians, tolerance, PLearn::tostring(), PLearn::VMatrix::width(), and PLearn::VMatrix::writable.
Referenced by PLearn::VMatrix::saveDMAT().
{ if(!writable) PLERROR("In DiskVMatrix::appendRow cannot append row in read only mode, set readwrite parameter to true when calling the constructor"); if(v.length() != width()) PLERROR("In DiskVMatrix::appendRow, length of v (%d) does not match matrix width (%d)",v.length(),width()); int filenum = dataf.size()-1; FILE* f = dataf[filenum]; if(!last_op_was_append) //otherwise, we are already at end fseek(f,0,SEEK_END); unsigned int position = (unsigned int)ftell(f); if(position>500000000L) { fflush(f); filenum++; string filename = dirname / (tostring(filenum) + ".data"); f = fopen(filename.c_str(), "w+b"); dataf.append(f); position = 0; } if(old_format) binwrite_compressed(f,v.data(),v.length()); else new_write_compressed(f, v.data(),v.length(), tolerance, swap_endians); if(!last_op_was_append) //otherwise, we are already at end fseek(indexf,0,SEEK_END); fputc(filenum,indexf); fwrite((char*)&position,sizeof(unsigned int),1,indexf); length_++; /* DO NOT write length in header; wait 'till file is closed fseek(indexf,sizeof(int),SEEK_SET); int le = length_; if(swap_endians) endianswap(&le); fwrite(&le,sizeof(int),1,indexf); */ last_op_was_append= true; }
void PLearn::DiskVMatrix::build | ( | ) | [virtual] |
Simply calls inherited::build() then build_().
Reimplemented from PLearn::VMatrix.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 154 of file DiskVMatrix.cc.
References PLearn::VMatrix::build(), and build_().
Referenced by PLearn::TemporaryDiskVMatrix::build(), and makeDeepCopyFromShallowCopy().
{ inherited::build(); build_(); }
void PLearn::DiskVMatrix::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::VMatrix.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 160 of file DiskVMatrix.cc.
References PLearn::TVec< T >::append(), PLearn::byte_order(), closeCurrentFiles(), PLearn::RowBufferedVMatrix::current_row, PLearn::RowBufferedVMatrix::current_row_index, dataf, dirname, PLearn::endianswap(), PLearn::force_mkdir(), freshnewfile, PLearn::VMatrix::getFieldInfos(), HEADER_OFFSET_LENGTH, indexf, PLearn::isdir(), PLearn::isfile(), last_op_was_append, PLearn::VMatrix::length_, PLearn::VMatrix::loadAllStringMappings(), PLearn::VMatrix::map_rs, PLearn::VMatrix::map_sr, NO_LENGTH_SENTINEL, old_format, PLearn::RowBufferedVMatrix::other_row, PLearn::RowBufferedVMatrix::other_row_index, PLASSERT_MSG, PLERROR, PLearn::TVec< T >::resize(), PLearn::VMatrix::setMetaDataDir(), swap_endians, PLearn::tostring(), PLearn::VMatrix::updateMtime(), PLearn::VMatrix::width_, and PLearn::VMatrix::writable.
Referenced by build(), and DiskVMatrix().
{ closeCurrentFiles(); // All file pointers will be re-created. if(writable && width_ > 0 && !isdir(dirname)) freshnewfile= true; if(!freshnewfile) { if(!isdir(dirname)) PLERROR("In DiskVMatrix constructor, directory '%s' could not be found",dirname.c_str()); setMetaDataDir(dirname + ".metadata"); updateMtime( dirname/"indexfile" ); string omode; if(writable) omode = "r+b"; else // read-only omode = "rb"; string indexfname = dirname/"indexfile"; indexf = fopen(indexfname.c_str(), omode.c_str()); if(!indexf) PLERROR("In DiskVMatrix constructor, could not open file %s in specified mode", indexfname.c_str()); unsigned char header[4]; fread(header,1,4,indexf); if(header[0]=='L' || header[0]=='B') { // New format old_format = false; swap_endians = (header[0]!=byte_order()); } else if(header[0]==16) { // Old format old_format = true; if(byte_order()!='L') PLERROR("Old format DiskVMatrix can only be read from a little-endian machine.\n" "Convert it to a new format on a little-endian machine prior to attempt\n" "using it from a big endian machine.\n"); swap_endians = false; } else { PLERROR("Wrong header byte in index file %s: ascii code %d\n" "(should be 'L' or 'B' or '...')\n", indexfname.c_str(), header[0]); } fread(&length_,sizeof(int),1,indexf); fread(&width_,sizeof(int),1,indexf); if(swap_endians) { endianswap(&length_); endianswap(&width_); } // Calc. length of VMat from indexfile length fseek(indexf, 0, SEEK_END); int pos= ftell(indexf); // pos less 12 header bytes s/b a multiple of 5 since each index // entry is 5-byte long i.e. pos%5==2 PLASSERT_MSG(pos%5 == 2, "Length of DiskVMatrix is absent from header and length" " of index file is " + tostring(pos) + ", not \\eqv 2 (mod 5)" ); int calculated_length= (pos-12) / 5; //nb. lines in matrix if(length_ == NO_LENGTH_SENTINEL) // check after endianswap { PLASSERT_MSG(!writable, // already open for writing? not good... "Trying to open a DiskVMatrix for writing but length in header is " "NO_LENGTH_SENTINEL (="+tostring(NO_LENGTH_SENTINEL)+"); " "Is it already being written to by another process? " "dirname='" + dirname + "'."); length_= calculated_length; } else { PLASSERT_MSG(length_ == calculated_length, //length mismatch? not good... "Length in DiskVMatrix's header does not match the indexfile's length " "and is not NO_LENGTH_SENTINEL (="+tostring(NO_LENGTH_SENTINEL)+"); " "indexfile='" + indexfname + "', " "header len=" + tostring(length_) + ", " "calc. len=" + tostring(calculated_length) + "."); if(writable) {// put NO_LENGTH_SENTINEL in header to indicate that we are appending to this file. fseek(indexf, HEADER_OFFSET_LENGTH, SEEK_SET); fwrite(&NO_LENGTH_SENTINEL, 4, 1, indexf); // will seek to end before end of build_ } } int k=0; string fname = dirname/tostring(k)+".data"; while(isfile(fname)) { FILE* f = fopen(fname.c_str(), omode.c_str()); if(!f) PLERROR("In DiskVMatrix constructor, could not open file %s in specified mode", fname.c_str()); dataf.append(f); fname = dirname/(tostring(++k)+".data"); } // Stuff related to RowBufferedVMatrix, for consistency current_row_index = -1; current_row.resize(width_); other_row_index = -1; other_row.resize(width_); //resize the string mappings map_sr = TVec<map<string,real> >(width_); map_rs = TVec<map<real,string> >(width_); getFieldInfos(); if (writable) fseek(indexf, 0, SEEK_END); } else { if(isdir(dirname)) PLERROR("In DiskVMatrix constructor (with specified width), directory %s already exists",dirname.c_str()); setMetaDataDir(dirname + ".metadata"); updateMtime( dirname/"indexfile" ); if(isfile(dirname)) // patch for running mkstemp (TmpFilenames) unlink(dirname.c_str()); if(!force_mkdir(dirname)) // force directory creation PLERROR("In DiskVMatrix constructor (with specified width), could not create directory %s Error was: %s",dirname.c_str(), strerror(errno)); string indexfname = dirname/"indexfile"; indexf = fopen(indexfname.c_str(),"w+b"); char header[4]; header[0] = byte_order(); header[1] = ' '; header[2] = ' '; header[3] = ' '; fwrite(header,1,4,indexf); //no more length in header while opened [was fwrite((char*)&length_,sizeof(int),1,indexf);] fwrite(&NO_LENGTH_SENTINEL, sizeof(int), 1, indexf); fwrite((char*)&width_,sizeof(int),1,indexf); length_= 0; string fname = dirname/"0.data"; FILE* f = fopen(fname.c_str(), "w+b"); dataf.append(f); } freshnewfile=false; loadAllStringMappings();//make sure string mappings are loaded after width is set last_op_was_append= false; }
string PLearn::DiskVMatrix::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 103 of file DiskVMatrix.cc.
void PLearn::DiskVMatrix::closeCurrentFiles | ( | ) | [protected, virtual] |
Close all current files opened by this DiskVMatrix.
As a side effect, empty the 'dataf' vector and set 'indexf' to 0.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 315 of file DiskVMatrix.cc.
References dataf, PLearn::endianswap(), HEADER_OFFSET_LENGTH, i, indexf, last_op_was_append, PLearn::TVec< T >::length(), PLearn::VMatrix::length_, PLearn::TVec< T >::resize(), and swap_endians.
Referenced by build_(), PLearn::TemporaryDiskVMatrix::closeCurrentFiles(), and ~DiskVMatrix().
{ for (int i = 0; i < dataf.length(); i++) if(dataf[i]) { fclose(dataf[i]); dataf[i] = 0; } dataf.resize(0); if (indexf) { // write final length to indexfile before closing fseek(indexf, HEADER_OFFSET_LENGTH, SEEK_SET); // it is and will always be 4 bytes int le= length_; if(swap_endians) endianswap(&le); fwrite(&le,sizeof(int),1,indexf); fclose(indexf); indexf = 0; } last_op_was_append= false; }
void PLearn::DiskVMatrix::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declare this class' options.
Reimplemented from PLearn::VMatrix.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 305 of file DiskVMatrix.cc.
References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::VMatrix::declareOptions(), dirname, and tolerance.
Referenced by PLearn::TemporaryDiskVMatrix::declareOptions().
{ declareOption(ol, "dirname", &DiskVMatrix::dirname, OptionBase::buildoption, "Directory name of the.dmat"); declareOption(ol, "tolerance", &DiskVMatrix::tolerance, OptionBase::buildoption, "The absolute error tolerance for storing doubles as floats"); inherited::declareOptions(ol); }
static const PPath& PLearn::DiskVMatrix::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 116 of file DiskVMatrix.h.
: void build_();
DiskVMatrix * PLearn::DiskVMatrix::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::RowBufferedVMatrix.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 103 of file DiskVMatrix.cc.
void PLearn::DiskVMatrix::flush | ( | ) | [virtual] |
For matrices stored on disk, this should flush all pending buffered write operations.
Reimplemented from PLearn::VMatrix.
Definition at line 418 of file DiskVMatrix.cc.
References dataf, indexf, last_op_was_append, and PLearn::TVec< T >::size().
{ int filenum = dataf.size()-1; FILE* f = dataf[filenum]; fflush(f); fflush(indexf); last_op_was_append= false; }
This is the only method requiring implementation in subclasses.
Implements PLearn::RowBufferedVMatrix.
Definition at line 342 of file DiskVMatrix.cc.
References PLearn::binread_compressed(), PLearn::TVec< T >::data(), dataf, PLearn::endianswap(), indexf, last_op_was_append, PLearn::TVec< T >::length(), PLearn::VMatrix::length(), PLearn::new_read_compressed(), old_format, PLASSERT, PLERROR, swap_endians, and PLearn::VMatrix::width().
{ #ifdef BOUNDCHECK if(i<0 || i>length()) PLERROR("In DiskVMatrix::getNewRow, bad row number %d",i); if(v.length() != width()) PLERROR("In DiskVMatrix::getNewRow, length of v (%d) does not match matrix width (%d)",v.length(),width()); #endif unsigned char filenum; unsigned int position; fseek(indexf,3*sizeof(int) + i*(sizeof(unsigned char)+sizeof(unsigned int)), SEEK_SET); fread(&filenum,sizeof(unsigned char),1,indexf); fread(&position,sizeof(unsigned int),1,indexf); if(swap_endians) endianswap(&position); FILE* f = dataf[int(filenum)]; PLASSERT( f ); fseek(f,position,SEEK_SET); if(old_format) binread_compressed(f,v.data(),v.length()); else new_read_compressed(f, v.data(), v.length(), swap_endians); last_op_was_append= false; }
OptionList & PLearn::DiskVMatrix::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 103 of file DiskVMatrix.cc.
OptionMap & PLearn::DiskVMatrix::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 103 of file DiskVMatrix.cc.
RemoteMethodMap & PLearn::DiskVMatrix::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 103 of file DiskVMatrix.cc.
void PLearn::DiskVMatrix::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transform a shallow copy into a deep copy.
Reimplemented from PLearn::RowBufferedVMatrix.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 430 of file DiskVMatrix.cc.
References build(), dataf, PLearn::deepCopyField(), PLearn::TVec< T >::fill(), indexf, PLearn::RowBufferedVMatrix::makeDeepCopyFromShallowCopy(), and PLearn::TVec< T >::resize().
Referenced by PLearn::TemporaryDiskVMatrix::makeDeepCopyFromShallowCopy().
{ inherited::makeDeepCopyFromShallowCopy(copies); deepCopyField(dataf, copies); // Set pointers to zero, because we will open again the file (file pointers // should not be shared between copied objects). indexf = 0; dataf.fill(0); dataf.resize(0); build(); // Open again the dataset. }
Reimplemented from PLearn::VMatrix.
Definition at line 368 of file DiskVMatrix.cc.
References PLERROR.
{ PLERROR("putRow cannot in general be correctly and efficiently implemented for a DiskVMatrix.\n" "Use appendRow if you wish to write more rows."); }
Reimplemented from PLearn::RowBufferedVMatrix.
Reimplemented in PLearn::TemporaryDiskVMatrix.
Definition at line 116 of file DiskVMatrix.h.
TVec<FILE*> PLearn::DiskVMatrix::dataf [mutable, protected] |
Definition at line 63 of file DiskVMatrix.h.
Referenced by appendRow(), PLearn::TemporaryDiskVMatrix::build_(), build_(), closeCurrentFiles(), flush(), getNewRow(), and makeDeepCopyFromShallowCopy().
Definition at line 73 of file DiskVMatrix.h.
Referenced by appendRow(), PLearn::TemporaryDiskVMatrix::build_(), build_(), declareOptions(), DiskVMatrix(), and PLearn::TemporaryDiskVMatrix::~TemporaryDiskVMatrix().
bool PLearn::DiskVMatrix::freshnewfile [protected] |
Definition at line 64 of file DiskVMatrix.h.
Referenced by build_().
const int PLearn::DiskVMatrix::HEADER_OFFSET_LENGTH = 4 [static] |
Definition at line 70 of file DiskVMatrix.h.
Referenced by build_(), and closeCurrentFiles().
FILE* PLearn::DiskVMatrix::indexf [mutable, protected] |
Definition at line 62 of file DiskVMatrix.h.
Referenced by appendRow(), PLearn::TemporaryDiskVMatrix::build_(), build_(), closeCurrentFiles(), flush(), getNewRow(), and makeDeepCopyFromShallowCopy().
bool PLearn::DiskVMatrix::last_op_was_append [mutable, protected] |
Definition at line 67 of file DiskVMatrix.h.
Referenced by appendRow(), build_(), closeCurrentFiles(), flush(), and getNewRow().
const int PLearn::DiskVMatrix::NO_LENGTH_SENTINEL = -42 [static] |
Definition at line 69 of file DiskVMatrix.h.
Referenced by build_().
bool PLearn::DiskVMatrix::old_format [protected] |
Definition at line 65 of file DiskVMatrix.h.
Referenced by appendRow(), build_(), and getNewRow().
bool PLearn::DiskVMatrix::swap_endians [protected] |
Definition at line 66 of file DiskVMatrix.h.
Referenced by appendRow(), build_(), closeCurrentFiles(), and getNewRow().
Definition at line 75 of file DiskVMatrix.h.
Referenced by appendRow(), and declareOptions().