PLearn 0.1
|
#include <TextFilesVMatrix.h>
Public Types | |
typedef RowBufferedVMatrix | inherited |
Public Member Functions | |
TextFilesVMatrix () | |
virtual | ~TextFilesVMatrix () |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual TextFilesVMatrix * | deepCopy (CopiesMap &copies) const |
real | getMapping (int fieldnum, const string &strval) const |
returns the mapped value for field fieldnum corresponding to strval If there was no such mapping present, and auto_extend_map is true, then a new mapping is appended for that field (with value -1000-size_of_map). | |
string | getMapFilePath (int fieldnum) const |
string | getTextRow (int i) const |
Returns the raw text rows. | |
int | nTextFields () const |
Returns the number of text fields in each row (this may be different from the width() of the resulting VMatrix, as some fields may be skipped, and others may be processed into several columns. | |
TVec< string > | splitIntoFields (const string &raw_row) const |
Split the passed string into fields given the delimiter. | |
TVec< string > | getTextFields (int i) const |
Returns the split raw text rows. | |
int | getIndexOfTextField (const string &fieldname) const |
Returns the index in a split text row of the given named text field (this is the position of that named field in fieldspec ) | |
virtual void | setMetaDataDir (const PPath &the_metadatadir) |
This should be called by the build method of every VMatrix that has a metadatadir. | |
virtual void | transformStringToValue (int k, string strval, Vec dest) const |
Transform field-k value strval according to its fieldtype into one ore more reals and write those into dest (which should be of appropriate size: colrange[k].second) | |
virtual void | getNewRow (int i, const Vec &v) const |
This is the only method requiring implementation in subclasses. | |
virtual void | build () |
Simply calls inherited::build() then build_(). | |
void | buildVMatrixStringMapping () |
Builds standard VMatrix string mapping from this txtmat's specific mapping saveAllStringMappings() can then be called to save them (for later reference from another VMatrix for ex.) | |
void | generateMapCounts () |
Will generate a counts/ directory in the metadatadir and record, for each field with a mapping, how many times a particular mapped string was encountered. | |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Transforms a shallow copy into a deep copy. | |
Static Public Member Functions | |
static string | _classname_ () |
RowBufferedVMatrix. | |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
TVec< pair< int, int > > | colrange |
list of startpos : width that associate a column range in the resulting VMat to each initial field of the fieldspecs This is built by the setColumnNamesAndWidth method | |
PPath | metadatapath |
The path to the .metadata directory. | |
TVec< PPath > | txtfilenames |
A list of paths to raw text files containing the records. | |
string | delimiter |
Delimiter to use to split the fields. | |
string | quote_delimiter |
The escate caractere where the delimiter is not considered. | |
TVec< int > | skipheader |
An (optional) list of integers, one for each of the txtfilenames indicating the number of header lines at the top of the file to be skipped. | |
TVec< pair< string, string > > | fieldspec |
Specification of field names and types. | |
bool | auto_build_map |
If true, all mappings will be automatically computed at build time if they do not exist yet. | |
bool | auto_extend_map |
If true, new strings for fields of type AUTO will automatically appended to the mapping (in the metadata/mappings/fieldname.map file) | |
bool | build_vmatrix_stringmap |
If true, standard vmatrix stringmap will be built from the txtmat specific stringmap. | |
bool | reorder_fieldspec_from_headers |
bool | partial_match |
string | default_spec |
If their is no fieldspec, we will use this value reorder_fieldspec_from_headers must be true. | |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Protected Member Functions | |
void | autoBuildMappings () |
If no existing mapping has been loaded, will automatically create all mappings. | |
void | loadMappings () |
virtual real | getPostalEncoding (const string &strval, bool display_warning=true) const |
virtual bool | isValidNonSkipFieldType (const string &ftype) const |
Return true iff 'ftype' is a valid type that does not need to be skipped. | |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declares this class' options. | |
Protected Attributes | |
FILE * | idxfile |
TVec< FILE * > | txtfiles |
TVec< hash_map< string, real > > | mapping |
a possible string->real mapping for each field We keep our own mapping in addition to the standard VMatrix string mapping because we want to handle the case where different strings map to the same value. | |
TVec< FILE * > | mapfiles |
Private Member Functions | |
void | build_ () |
This does the actual building. | |
void | setColumnNamesAndWidth () |
void | getFileAndPos (int i, unsigned char &fileno, int &pos) const |
void | buildIdx () |
void | closeCurrentFile () |
Static Private Member Functions | |
static void | readAndCheckOptionName (PStream &in, const string &optionname, char buf[]) |
FORMAT DESCRIPTION
This type of vmatrix is described in a basename.txtmat file and has an associated metadata/ directory
The metadata directory will contain the following files:
The txtmat.idx file is a binary file structured as follows
Definition at line 68 of file TextFilesVMatrix.h.
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 183 of file TextFilesVMatrix.h.
PLearn::TextFilesVMatrix::TextFilesVMatrix | ( | ) |
Definition at line 53 of file TextFilesVMatrix.cc.
: idxfile(0), delimiter("\t"), quote_delimiter(""), auto_build_map(false), auto_extend_map(true), build_vmatrix_stringmap(false), reorder_fieldspec_from_headers(false), partial_match(false) {}
PLearn::TextFilesVMatrix::~TextFilesVMatrix | ( | ) | [virtual] |
Definition at line 1062 of file TextFilesVMatrix.cc.
References idxfile, PLearn::TVec< T >::length(), mapfiles, PLearn::TVec< T >::size(), and txtfiles.
{ if(idxfile) fclose(idxfile); for(int k=0; k<txtfiles.length(); k++) fclose(txtfiles[k]); for(int k=0; k<mapfiles.size(); k++) { if(mapfiles[k]) fclose(mapfiles[k]); } }
string PLearn::TextFilesVMatrix::_classname_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 82 of file TextFilesVMatrix.cc.
OptionList & PLearn::TextFilesVMatrix::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 82 of file TextFilesVMatrix.cc.
RemoteMethodMap & PLearn::TextFilesVMatrix::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 82 of file TextFilesVMatrix.cc.
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 82 of file TextFilesVMatrix.cc.
Object * PLearn::TextFilesVMatrix::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 82 of file TextFilesVMatrix.cc.
StaticInitializer TextFilesVMatrix::_static_initializer_ & PLearn::TextFilesVMatrix::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 82 of file TextFilesVMatrix.cc.
void PLearn::TextFilesVMatrix::autoBuildMappings | ( | ) | [protected] |
If no existing mapping has been loaded, will automatically create all mappings.
Definition at line 555 of file TextFilesVMatrix.cc.
References auto_extend_map, fieldspec, PLearn::RowBufferedVMatrix::getRow(), i, PLearn::VMatrix::length(), PLearn::TVec< T >::length(), mapping, PLWARNING, PLearn::ProgressBar::update(), and PLearn::VMatrix::width().
Referenced by setMetaDataDir().
{ // TODO We should somehow check the date of existing mappings to see if they need to be built. // For now we just create them if they do not exist yet. // First make sure there is no existing mapping. int nb_already_exist = 0; int nb_type_no_mapping = 0; for (int i = 0; i < mapping.length(); i++) { if (!mapping[i].empty()) nb_already_exist++; else if(fieldspec[i].second!="char")//should add auto when it is char that are selected nb_type_no_mapping++; } if(nb_already_exist == 0){ // Mappings need to be built. // We do this by reading the whole data. Vec row(width()); bool auto_extend_map_backup = auto_extend_map; auto_extend_map = true; ProgressBar pb("Building mappings", length()); for (int i = 0; i < length(); i++) { getRow(i, row); pb.update(i + 1); } auto_extend_map = auto_extend_map_backup; }else if (nb_already_exist+nb_type_no_mapping < mapping.length()) { for (int i = 0; i < mapping.length(); i++) if(fieldspec[i].second=="char" && mapping[i].empty())//should add auto when it is char that are selected PLWARNING("In TextFilesVMatrix::autoBuildMappings - mapping already existing but not for field %d (%s)",i,fieldspec[i].first.c_str()); PLWARNING("In TextFilesVMatrix::autoBuildMappings - The existing " "mapping is not complete! There are %d fields with build " "mapping and there are %d fields that do not need mapping " "in a total of %d fields. Erase the mapping directory in " "the metadatadir to have it regenerated next time!", nb_already_exist,nb_type_no_mapping,mapping.length()); }//else already build }
void PLearn::TextFilesVMatrix::build | ( | ) | [virtual] |
Simply calls inherited::build() then build_().
Reimplemented from PLearn::VMatrix.
Definition at line 1056 of file TextFilesVMatrix.cc.
References PLearn::VMatrix::build(), and build_().
Referenced by makeDeepCopyFromShallowCopy().
{ inherited::build(); build_(); }
void PLearn::TextFilesVMatrix::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::VMatrix.
Definition at line 438 of file TextFilesVMatrix.cc.
References default_spec, PLearn::VMatrix::metadatadir, metadatapath, PLERROR, PLWARNING, reorder_fieldspec_from_headers, and setMetaDataDir().
Referenced by build().
{ if (!default_spec.empty() && !reorder_fieldspec_from_headers) PLERROR("In TextFilesVMatrix::build_() when the option default_spec is used, reorder_fieldspec_from_headers must be true"); if (metadatapath != "") { PLWARNING("In TextFilesVMatrix::build_() metadatapath option is deprecated. " "You should use metadatadir instead.\n"); metadatadir = metadatapath; setMetaDataDir(metadatapath); } }
void PLearn::TextFilesVMatrix::buildIdx | ( | ) | [private] |
Definition at line 105 of file TextFilesVMatrix.cc.
References PLearn::byte_order(), PLearn::endl(), fieldspec, PLearn::VMatrix::getMetaDataDir(), idxfile, PLearn::isBlank(), PLearn::TVec< T >::isEmpty(), PLearn::TVec< T >::length(), PLearn::VMatrix::length_, PLearn::mvforce(), PLearn::perr, PLERROR, PLWARNING, PLearn::TVec< T >::size(), skipheader, splitIntoFields(), PLearn::strlen(), and txtfiles.
Referenced by setMetaDataDir().
{ perr << "Building the index file. Please be patient..." << endl; if(idxfile) fclose(idxfile); PPath ft=getMetaDataDir()/"txtmat.idx.tmp"; PPath f=getMetaDataDir()/"txtmat.idx"; idxfile = fopen(ft.c_str(),"wb"); FILE* logfile = fopen((getMetaDataDir()/"txtmat.idx.log").c_str(),"a"); if (! idxfile) PLERROR("TextFilesVMatrix::buildIdx: could not open index file '%s'", ( getMetaDataDir()/"txtmat.idx").c_str()); if (! logfile) PLERROR("TextFilesVMatrix::buildIdx: could not open log file '%s'", (getMetaDataDir()/"txtmat.idx.log").c_str()); // write endianness fputc(byte_order(), idxfile); // We don't know length yet, length_ = 0; fwrite(&length_, 4, 1, idxfile); TVec<string> fields; char buf[50000]; int lineno = 0; for(unsigned char fileno=0; fileno<txtfiles.length(); fileno++) { FILE* fi = txtfiles[(int)fileno]; fseek(fi,0,SEEK_SET); int nskip = 0; // number of header lines to skip if(!skipheader.isEmpty()) nskip = skipheader[int(fileno)]; // read the data rows and build the index for(;;) { long pos_long = ftell(fi); if (pos_long > INT_MAX) PLERROR("In TextFilesVMatrix::buildIdx - 'pos_long' cannot be " "more than %d", INT_MAX); int pos = int(pos_long); if(!fgets(buf, sizeof(buf), fi)) break; #ifdef CYGWIN_FGETS_BUGFIX // Bugfix for CYGWIN carriage return bug. // Should be safe to enable in all case, but need to be tester more widely. long new_pos = ftell(fi); long lbuf = long(strlen(buf)); if (lbuf+pos != new_pos){ if(lbuf+1+pos==new_pos && buf[lbuf-1]=='\n' && buf[lbuf-2]!='\r') { //bug under windows. fgets return the good string if unix end of lines, but //change the position suppossing the use of \r\n as carrige return. //So if their is only a \n, we are a caractere too far. //if dos end of lines, return \n as end of lines in the strings and put the pos correctly. fseek(fi,-1,SEEK_CUR); //if unix end of lines if(fgetc(fi)!='\n') fseek(fi,-1,SEEK_CUR); } //in the eof case? else if(lbuf-1+pos==new_pos && buf[lbuf-1]=='\n' && buf[lbuf-2]!='\r') fseek(fi,+1,SEEK_CUR); else PLERROR("In TextFilesVMatrix::buildId - The number of characters read " "does not match the position in the file."); } #endif buf[sizeof(buf)-1] = '\0'; // ensure null-terminated lineno++; if(nskip>0) --nskip; else if(!isBlank(buf)) { fields = splitIntoFields(buf); int nf = fields.length(); if(nf!=fieldspec.size()){ fprintf(logfile, "ERROR In file %d line %d: Found %d fields (should be %d):\n %s",fileno,lineno,nf,fieldspec.size(),buf); PLWARNING("In file %d line %d: Found %d fields (should be %d):\n %s",fileno,lineno,nf,fieldspec.size(),buf); } else // Row OK! append it to index { fputc(fileno, idxfile); fwrite(&pos, 4, 1, idxfile); length_++; } } else PLWARNING("In TextFilesVMatrix::buildIdx() - The line %d is blank",lineno); } // end of loop over lines of file } // end of loop over files // Write true length and width fseek(idxfile, 1, SEEK_SET); fwrite(&length_, 4, 1, idxfile); // close files fclose(logfile); fclose(idxfile); mvforce(ft,f); perr << "Index file built." << endl; }
void PLearn::TextFilesVMatrix::buildVMatrixStringMapping | ( | ) |
Builds standard VMatrix string mapping from this txtmat's specific mapping saveAllStringMappings() can then be called to save them (for later reference from another VMatrix for ex.)
Definition at line 636 of file TextFilesVMatrix.cc.
References PLearn::VMatrix::addStringMapping(), PLearn::TVec< T >::begin(), colrange, PLearn::TVec< T >::end(), fieldspec, PLearn::TVec< T >::first(), j, mapping, n, and PLearn::TVec< T >::size().
Referenced by setMetaDataDir().
{ int n = fieldspec.size(); for(int k=0; k<n; k++) { if(mapping[k].size()>0) { // get the corresponding VMatrix column range and add the VMatrix mapping int colstart = colrange[k].first; int ncols = colrange[k].second; hash_map<string,real>::const_iterator it = mapping[k].begin(); hash_map<string,real>::const_iterator itend = mapping[k].end(); while(it!=itend) { for(int j=colstart; j<colstart+ncols; j++) addStringMapping(j, it->first, it->second); ++it; } } } }
string PLearn::TextFilesVMatrix::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 82 of file TextFilesVMatrix.cc.
void PLearn::TextFilesVMatrix::closeCurrentFile | ( | ) | [private] |
void PLearn::TextFilesVMatrix::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declares this class' options.
Reimplemented from PLearn::VMatrix.
Definition at line 959 of file TextFilesVMatrix.cc.
References auto_build_map, auto_extend_map, build_vmatrix_stringmap, PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::VMatrix::declareOptions(), default_spec, delimiter, fieldspec, metadatapath, partial_match, quote_delimiter, reorder_fieldspec_from_headers, skipheader, and txtfilenames.
{ declareOption(ol, "metadatapath", &TextFilesVMatrix::metadatapath, OptionBase::buildoption, "Path of the metadata directory (in which to store the index, ...)\n" "DEPRECATED: use metadatadir instead.\n"); declareOption(ol, "txtfilenames", &TextFilesVMatrix::txtfilenames, OptionBase::buildoption, "A list of paths to raw text files containing the records"); declareOption(ol, "delimiter", &TextFilesVMatrix::delimiter, OptionBase::buildoption, "Delimiter to use to split the fields. Common delimiters are:\n" "- \"\\t\" : used for SAS files (the default)\n" "- \",\" : used for CSV files\n" "- \";\" : used for a variant of CSV files\n" "If more then 1 delimiter, we will select one based on the" " first two line\n"); declareOption(ol, "quote_delimiter", &TextFilesVMatrix::quote_delimiter, OptionBase::buildoption, "The escape character to indicate the delimiter is not considered.\n" "For instance, '\"' is frequently used."); declareOption(ol, "skipheader", &TextFilesVMatrix::skipheader, OptionBase::buildoption, "An (optional) list of integers, one for each of the txtfilenames,\n" "indicating the number of header lines at the top of the file to be skipped."); declareOption(ol, "fieldspec", &TextFilesVMatrix::fieldspec, OptionBase::buildoption, "Specification of field names and types (type indicates how the text field is to be mapped to one or more reals)\n" "Currently supported types: \n" "- skip : Ignore the content of the field, won't be inserted in the resulting VMat\n" "- auto : If a numeric value, keep it as is, if not, look it up in the mapping (possibly inserting a new mapping if it's not there) \n" "- auto-num : take any float value with the decimal separator as the dot. If there is a $\n" " at the start or end it is removed. If there are commas they are removed.\n" "- num : numeric value, keep as is\n" "- num-comma : numeric value where thousands are separeted by comma\n" "- char : look it up in the mapping (possibly inserting a new mapping if it's not there)\n" "- date : date of the form 25DEC2003 or 25-dec-2003 or 2003/12/25 or 20031225, will be mapped to float date format 1031225\n" "- jdate : date of the form 25DEC2003 or 25-dec-2003 or 2003/12/25 or 20031225, will be mapped to *julian* date format\n" "- sas_date : date used by SAS = number of days since Jan. 1st, 1960 (with 0 = missing)\n" "- YYYYMM : date of the form YYYYMM (e.g: 200312), will be mapped to the julian date of the corresponding month. Everthing "\ " other than a number or lower than 197000 is considered as nan\n" "- postal : canadian postal code \n" "- dollar : strangely formatted field with dollar amount. Format is sth like '$12 003'\n" "- dollar-comma : strangely formatted field with dollar amount. Format is sth like '$12,003'\n" "- bell_range : a range like \"A: $0- 250\", replaced by the average of the two bounds;\n" " if the \"Negative Value\" string is found, it is replaced by -100\n" ); declareOption(ol, "auto_extend_map", &TextFilesVMatrix::auto_extend_map, OptionBase::buildoption, "If true, new strings for fields of type AUTO will automatically appended to the mapping (in the metadata/mappings/fieldname.map file)"); declareOption(ol, "auto_build_map", &TextFilesVMatrix::auto_build_map, OptionBase::buildoption, "If true, all mappings will be automatically computed at build time if they do not exist yet\n"); declareOption(ol, "build_vmatrix_stringmap", &TextFilesVMatrix::build_vmatrix_stringmap, OptionBase::buildoption, "If true, standard vmatrix stringmap will be built from the txtmat specific stringmap"); declareOption(ol, "reorder_fieldspec_from_headers", &TextFilesVMatrix::reorder_fieldspec_from_headers, OptionBase::buildoption, "If true, will reorder the fieldspec in the order given " "by the field names taken from txtfilenames."); declareOption(ol, "partial_match", &TextFilesVMatrix::partial_match, OptionBase::buildoption, "If true, will repeatedly expand all fieldspec name ending " "with * to the full name from header." "The expansion is equivalent to the regex 'field_spec_name*'." "The option reorder_fieldspec_from_headers must be true"); declareOption(ol, "default_spec", &TextFilesVMatrix::default_spec, OptionBase::buildoption, "If there is no fieldspec for a fieldname, we will use this" "value. reorder_fieldspec_from_headers must be true."); // Now call the parent class' declareOptions inherited::declareOptions(ol); }
static const PPath& PLearn::TextFilesVMatrix::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 184 of file TextFilesVMatrix.h.
{ return getMetaDataDir()/"mappings"/fieldspec[fieldnum].first+".map"; }
TextFilesVMatrix * PLearn::TextFilesVMatrix::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 82 of file TextFilesVMatrix.cc.
void PLearn::TextFilesVMatrix::generateMapCounts | ( | ) |
Will generate a counts/ directory in the metadatadir and record, for each field with a mapping, how many times a particular mapped string was encountered.
Definition at line 594 of file TextFilesVMatrix.cc.
References PLearn::TVec< T >::begin(), PLearn::TVec< T >::end(), fieldspec, PLearn::VMatrix::find(), PLearn::VMatrix::getMetaDataDir(), getTextFields(), i, PLearn::VMatrix::length(), PLearn::TVec< T >::length(), mapping, n, PLearn::VMatrix::save(), and PLearn::TVec< T >::size().
{ int n = fieldspec.size(); TVec< hash_map<string, int> > counts(n); for(int k=0; k<n; k++) { if(!mapping[k].empty()) { hash_map<string, real>& mapping_k = mapping[k]; hash_map<string, int>& counts_k = counts[k]; hash_map<string, real>::const_iterator it = mapping_k.begin(); hash_map<string, real>::const_iterator itend = mapping_k.end(); while(it!=itend) { counts_k[ it->first ] = 0; ++it; } } } int l = length(); ProgressBar pg("Generating counts of mappings",l); for(int i=0; i<l; i++) { TVec<string> fields = getTextFields(i); for(int k=0; k<fields.length(); k++) { if(mapping[k].find(fields[k])!=mapping[k].end()) ++counts[k][fields[k]]; } pg(i); } // Save the counts for(int k=0; k<n; k++) { if(!counts[k].empty()) PLearn::save( getMetaDataDir() / "counts" / fieldspec[k].first+".count", counts[k] ); } }
void PLearn::TextFilesVMatrix::getFileAndPos | ( | int | i, |
unsigned char & | fileno, | ||
int & | pos | ||
) | const [private] |
Definition at line 85 of file TextFilesVMatrix.cc.
References idxfile, PLearn::VMatrix::length(), PLASSERT, and PLERROR.
Referenced by getTextRow().
{ PLASSERT(idxfile!=0); if(i<0 || i>=length()) PLERROR("TextFilesVMatrix::getFileAndPos out of range row %d (only %d rows)", i, length()); fseek(idxfile, 5+i*5, SEEK_SET); fileno = fgetc(idxfile); fread(&pos, sizeof(int), 1, idxfile); }
int PLearn::TextFilesVMatrix::getIndexOfTextField | ( | const string & | fieldname | ) | const |
Returns the index in a split text row of the given named text field (this is the position of that named field in fieldspec )
Definition at line 95 of file TextFilesVMatrix.cc.
References fieldspec, i, n, PLERROR, and PLearn::TVec< T >::size().
{ int n = fieldspec.size(); for(int i=0; i<n; i++) if(fieldspec[i].first==fieldname) return i; PLERROR("In TextFilesVMatrix::getIndexOfTextField unknown field %s",fieldname.c_str()); return -1; // to make the compiler happy }
string PLearn::TextFilesVMatrix::getMapFilePath | ( | int | fieldnum | ) | const [inline] |
Definition at line 192 of file TextFilesVMatrix.h.
Referenced by getMapping(), and loadMappings().
{ return getMetaDataDir()/"mappings"/fieldspec[fieldnum].first+".map"; }
returns the mapped value for field fieldnum corresponding to strval If there was no such mapping present, and auto_extend_map is true, then a new mapping is appended for that field (with value -1000-size_of_map).
If not a PLERROR is thrown.
Definition at line 658 of file TextFilesVMatrix.cc.
References auto_extend_map, fieldspec, PLearn::TVec< T >::find(), PLearn::force_mkdir_for_file(), getMapFilePath(), m, mapfiles, mapping, and PLERROR.
Referenced by transformStringToValue().
{ hash_map<string, real>& m = mapping[fieldnum]; hash_map<string, real>::const_iterator found = m.find(strval); if(found!=m.end()) // found it! return found->second; // strval not found if(!auto_extend_map) PLERROR("In TextFilesVMatrix::getMapping - No mapping found for field %d (%s) string-value \"%s\" ", fieldnum, fieldspec[fieldnum].first.c_str(), strval.c_str()); // OK, let's extend the mapping... real val = real(-1000 - int(m.size())); m[strval] = val; if(!mapfiles[fieldnum]) { string fname = getMapFilePath(fieldnum); force_mkdir_for_file(fname); mapfiles[fieldnum] = fopen(fname.c_str(),"a"); if(!mapfiles[fieldnum]) PLERROR("In TextFilesVMatrix::getMapping - Could not open map file %s\n for appending\n",fname.c_str()); } fprintf(mapfiles[fieldnum],"\n\"%s\" %f", strval.c_str(), val); return val; }
This is the only method requiring implementation in subclasses.
Implements PLearn::RowBufferedVMatrix.
Definition at line 937 of file TextFilesVMatrix.cc.
References colrange, fieldspec, PLearn::TVec< T >::first(), getTextFields(), i, PLearn::PLearnError::message(), n, PLERROR, PLearn::TVec< T >::size(), PLearn::TVec< T >::subVec(), and transformStringToValue().
{ TVec<string> fields = getTextFields(i); int n = fields.size(); for(int k=0; k<n; k++) { string fieldname = fieldspec[k].first; string fieldtype = fieldspec[k].second; string strval = fields[k]; Vec dest = v.subVec(colrange[k].first, colrange[k].second); try { transformStringToValue(k, strval, dest); } catch(const PLearnError& e) { PLERROR("In TextFilesVMatrix, while parsing field %d (%s) of row %d: \n%s", k,fieldname.c_str(),i,e.message().c_str()); } } }
OptionList & PLearn::TextFilesVMatrix::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 82 of file TextFilesVMatrix.cc.
OptionMap & PLearn::TextFilesVMatrix::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 82 of file TextFilesVMatrix.cc.
real PLearn::TextFilesVMatrix::getPostalEncoding | ( | const string & | strval, |
bool | display_warning = true |
||
) | const [protected, virtual] |
Definition at line 702 of file TextFilesVMatrix.cc.
References MISSING_VALUE, and PLWARNING.
Referenced by transformStringToValue().
{ if(strval=="") return MISSING_VALUE; char first_char = strval[0]; int second_digit = strval[1]; real val = 0; if(first_char=='A') val = 30 + second_digit; else if(first_char=='B') val = 40 + second_digit; else if(first_char=='C') val = 50 + second_digit; else if(first_char=='E') val = 60 + second_digit; else if(first_char=='G') val = 0 + second_digit; else if(first_char=='H') val = 10 + second_digit; else if(first_char=='J') val = 20 + second_digit; else if(first_char=='K') val = 70 + second_digit; else if(first_char=='L') val = 80 + second_digit; else if(first_char=='M') val = 90 + second_digit; else if(first_char=='N') val = 100 + second_digit; else if(first_char=='P') val = 110 + second_digit; else if(first_char=='R') val = 120 + second_digit; else if(first_char=='S') val = 130 + second_digit; else if(first_char=='T') val = 140 + second_digit; else if(first_char=='V') val = 150 + second_digit; else if(first_char=='W') val = 160 + second_digit; else if(first_char=='X') val = 170 + second_digit; else if(first_char=='Y') val = 180 + second_digit; else if(first_char=='0' || first_char=='1' || first_char=='2' || first_char=='3' || first_char=='4' || first_char=='5' || first_char=='6' || first_char=='7' || first_char=='8' || first_char=='9') { // That would be a C.P. int first_digit = strval[0]; val = 260 + first_digit * 10 + second_digit; } else { //http://en.wikipedia.org/wiki/Canadian_postal_code //No postal code includes the letters D, F, I, O, Q, or U, //as the OCR equipment used in automated sorting could easily //confuse them with other letters and digits, if (display_warning) { string errmsg; if(first_char=='D' ||first_char=='F' ||first_char=='I' || first_char=='O' ||first_char=='Q' ||first_char=='U') errmsg = "Postal code don't use letters D, F, I, O, Q, or U: "; else errmsg = "Currently only some postal codes are supported: "; errmsg += "can't process " + strval + ", value will be set to 0."; PLWARNING(errmsg.c_str()); } val = 0; } return val; }
RemoteMethodMap & PLearn::TextFilesVMatrix::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 82 of file TextFilesVMatrix.cc.
Returns the split raw text rows.
Definition at line 691 of file TextFilesVMatrix.cc.
References fieldspec, getTextRow(), PLERROR, PLearn::removeblanks(), PLearn::TVec< T >::size(), and splitIntoFields().
Referenced by generateMapCounts(), and getNewRow().
{ string rowi = getTextRow(i); TVec<string> fields = splitIntoFields(rowi); if(fields.size() != fieldspec.size()) PLERROR("In TextFilesVMatrix::getTextFields - In getting fields of row %d, wrong number of fields: %d (should be %d):\n%s\n",i,fields.size(),fieldspec.size(),rowi.c_str()); for(int k=0; k<fields.size(); k++) fields[k] = removeblanks(fields[k]); return fields; }
string PLearn::TextFilesVMatrix::getTextRow | ( | int | i | ) | const |
Returns the raw text rows.
Definition at line 514 of file TextFilesVMatrix.cc.
References getFileAndPos(), PLERROR, PLearn::removenewline(), and txtfiles.
Referenced by getTextFields().
{ unsigned char fileno; int pos; getFileAndPos(i, fileno, pos); FILE* f = txtfiles[(int)fileno]; fseek(f,pos,SEEK_SET); char buf[50000]; if(!fgets(buf, sizeof(buf), f)) PLERROR("In TextFilesVMatrix::getTextRow - fgets for row %d returned NULL",i); return removenewline(buf); }
bool PLearn::TextFilesVMatrix::isValidNonSkipFieldType | ( | const string & | ftype | ) | const [protected, virtual] |
Return true iff 'ftype' is a valid type that does not need to be skipped.
Definition at line 219 of file TextFilesVMatrix.cc.
Referenced by setColumnNamesAndWidth().
{ return (ftype=="auto" || ftype=="num" || ftype=="date" || ftype=="jdate" || ftype=="postal" || ftype=="dollar" || ftype=="dollar-comma" || ftype=="YYYYMM" || ftype=="sas_date" || ftype == "bell_range" || ftype == "char" || ftype=="num-comma" || ftype=="auto-num"); }
void PLearn::TextFilesVMatrix::loadMappings | ( | ) | [protected] |
Definition at line 528 of file TextFilesVMatrix.cc.
References fieldspec, getMapFilePath(), PLearn::getNonBlankLines(), i, PLearn::isfile(), PLearn::loadFileAsString(), mapping, n, PLearn::pl_isnumber(), PLERROR, PLearn::TVec< T >::size(), and PLearn::VMatrix::updateMtime().
Referenced by setMetaDataDir().
{ int n = fieldspec.size(); for(int k=0; k<n; k++) { string fname = getMapFilePath(k); if (isfile(fname)) { updateMtime(fname); vector<string> all_lines = getNonBlankLines(loadFileAsString(fname)); for (size_t i = 0; i < all_lines.size(); i++) { string map_line = all_lines[i]; size_t start_of_string = map_line.find('"'); size_t end_of_string = map_line.rfind('"'); string strval = map_line.substr(start_of_string + 1, end_of_string - start_of_string - 1); string real_val_str = map_line.substr(end_of_string + 1); real real_val; if (!pl_isnumber(real_val_str, &real_val)) PLERROR("In TextFilesVMatrix::loadMappings - Found a mapping to something that is not a number (%s) in file %s at non-black line %ld", map_line.c_str(), fname.c_str(), long(i)); mapping[k][strval] = real_val; } } } }
void PLearn::TextFilesVMatrix::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 1076 of file TextFilesVMatrix.cc.
References auto_build_map, build(), idxfile, PLearn::RowBufferedVMatrix::makeDeepCopyFromShallowCopy(), PLearn::TVec< T >::resize(), and txtfiles.
{ inherited::makeDeepCopyFromShallowCopy(copies); idxfile=0; txtfiles.resize(0); //the map should be already build. auto_build_map=false; build(); }
int PLearn::TextFilesVMatrix::nTextFields | ( | ) | const [inline] |
Returns the number of text fields in each row (this may be different from the width() of the resulting VMatrix, as some fields may be skipped, and others may be processed into several columns.
Definition at line 202 of file TextFilesVMatrix.h.
{ return fieldspec.length(); }
void PLearn::TextFilesVMatrix::readAndCheckOptionName | ( | PStream & | in, |
const string & | optionname, | ||
char | buf[] | ||
) | [static, private] |
Definition at line 1040 of file TextFilesVMatrix.cc.
References PLearn::PStream::get(), PLERROR, PLearn::PStream::readUntil(), PLearn::removeblanks(), and PLearn::PStream::skipBlanksAndComments().
{ in.skipBlanksAndComments(); in.readUntil(buf, sizeof(buf), "= "); string option = removeblanks(buf); in.skipBlanksAndComments(); char eq = in.get(); if(option!=optionname || eq!='=') PLERROR("In TextFilesVMatrix::readAndCheckOptionName - " "Bad syntax in .txtmat file.\n" "Expected option %s = ...\n" "Read %s %c\n", optionname.c_str(), option.c_str(), eq); }
void PLearn::TextFilesVMatrix::setColumnNamesAndWidth | ( | ) | [private] |
Definition at line 226 of file TextFilesVMatrix.cc.
References PLearn::TVec< T >::append(), colrange, PLearn::VMatrix::declareField(), default_spec, delimiter, PLearn::endl(), fieldspec, PLearn::TVec< T >::first(), i, isValidNonSkipFieldType(), j, PLearn::TVec< T >::length(), n, NORMAL_LOG, partial_match, PLCHECK, PLCHECK_MSG, PLERROR, PLWARNING, PLearn::TVec< T >::pop(), PLearn::TVec< T >::push_back(), PLearn::removeblanks(), reorder_fieldspec_from_headers, PLearn::TVec< T >::size(), splitIntoFields(), PLearn::string_begins_with(), PLearn::tostring(), txtfilenames, txtfiles, and PLearn::VMatrix::width_.
Referenced by setMetaDataDir().
{ width_ = 0; TVec<string> fnames; TVec<string> fnames_header;//field names take in the header of source file char buf[50000]; //select witch delimiter we will use for all the files. if(delimiter.size()>1){ FILE* f = txtfiles[0]; fseek(f,0,SEEK_SET); if(!fgets(buf, sizeof(buf), f)) PLERROR("In TextFilesVMatrix::setColumnNamesAndWidth() - " "Couldn't read the fields names from file '%s'", txtfilenames[0].c_str()); string s1 = string(buf); if(!fgets(buf, sizeof(buf), f)) PLERROR("In TextFilesVMatrix::setColumnNamesAndWidth() - " "Couldn't read the fields names from file '%s'", txtfilenames[0].c_str()); string s2 = string(buf); TVec<int> nbs1(delimiter.size()); TVec<int> nbs2(delimiter.size()); string old_delimiter = delimiter; for(uint i=0;i<old_delimiter.size();i++){ delimiter = old_delimiter[i]; TVec<string> fields1 = splitIntoFields(s1); TVec<string> fields2 = splitIntoFields(s2); nbs1[i]=fields1.size(); nbs2[i]=fields2.size(); } delimiter=old_delimiter; for(uint i=0;i<old_delimiter.size();i++){ if(nbs1[i]==nbs2[i]&& nbs1[i]>0){ delimiter = old_delimiter[i]; } } MODULE_LOG << "Selected delimiter: <" << delimiter << ">" << endl; if(delimiter.size()!=1){ PLERROR("In TextFilesVMatrix::setColumnNamesAndWidth() - We can't" " automatically determine the delimiter to use as the two" " first row don't have a common delimiter with the same" " number of occurence. nbs1=%s, nbs2=%s", tostring(nbs1).c_str(),tostring(nbs2).c_str()); } } PLCHECK(delimiter.size()==1); //read the fieldnames from the files. for(int i=0; i<txtfiles.size(); i++){ FILE* f = txtfiles[i]; fseek(f,0,SEEK_SET); if(!fgets(buf, sizeof(buf), f)) PLERROR("In TextFilesVMatrix::setColumnNamesAndWidth() - " "Couldn't read the fields names from file '%s'", txtfilenames[i].c_str()); fseek(f,0,SEEK_SET); TVec<string> fields = splitIntoFields(buf); //check that we have the good delimiter if(fields.size()==1 && fieldspec.size()>1) PLERROR("In TextFilesVMatrix::setColumnNamesAndWidth() -" " We found only 1 column in the first line, but" " their is %d fieldspec. Meaby the delimiter '%s'" " is not the right one. The line is %s", fieldspec.size(),delimiter.c_str(), string(buf).c_str()); if(reorder_fieldspec_from_headers || partial_match){ fields.append(removeblanks(fields.pop())); fnames_header.append(fields); } } if(partial_match) { TVec< pair<string, string> > new_fieldspec; TVec<string> no_expended_fields; PLCHECK_MSG(reorder_fieldspec_from_headers, "In TextFilesVMatrix::setColumnNamesAndWidth - " "when partial_match is true, reorder_fieldspec_from_headers" " must be true."); for(int i=0;i<fieldspec.size();i++) { bool expended = false; string fname=fieldspec[i].first; if(fname[fname.size()-1]!='*') { new_fieldspec.append(fieldspec[i]); continue; } fname.resize(fname.size()-1);//remove the last caracter (*) for(int j=0;j<fnames_header.size();j++) { if(string_begins_with(fnames_header[j],fname)) { pair<string,string> n=make_pair(fnames_header[j], fieldspec[i].second); // perr<<"expanding "<<fieldspec[i] << " to " << n <<endl; new_fieldspec.append(n); expended = true; } } if(!expended) no_expended_fields.append(fieldspec[i].first); } if(no_expended_fields.length()>0){ NORMAL_LOG<<"In TextFilesVMatrix::setColumnNamesAndWidth - " <<"Did not find any partial match for " <<no_expended_fields.length()<<" spec:"; for(int i=0;i<no_expended_fields.length();i++) NORMAL_LOG<<" "<<no_expended_fields[i]; NORMAL_LOG<<endl; } fieldspec = new_fieldspec; } if(reorder_fieldspec_from_headers) { //check that all field names from the header have a spec TVec<string> not_used_fn; for(int i=0;i<fnames_header.size();i++) { string name=fnames_header[i]; int j=0; for(;j<fieldspec.size();j++) if(fieldspec[j].first==name) break; if(j>=fieldspec.size()){ if(default_spec!=""){ fieldspec.append(make_pair(name,default_spec)); }else not_used_fn.append(name); } } //check that all fieldspec names are also in the header TVec<string> not_used_fs; for(int i=0;i<fieldspec.size();i++) { string name=fieldspec[i].first; int j=0; for(;j<fnames_header.size();j++) if(fnames_header[j]==name) break; if(j>=fnames_header.size()) not_used_fs.append(name); } //check that we have the good number of fieldspec //if partial match is true, we don't want to generate the warning everytime if(fnames_header.size()!=fieldspec.size() && !partial_match) { PLWARNING("In TextFilesVMatrix::setColumnNamesAndWidth() - " "We read %d field names from the header but have %d" "fieldspec",fnames_header.size(),fieldspec.size()); } if(not_used_fs.size()!=0) PLWARNING("TextFilesVMatrix::setColumnNamesAndWidth() - " "%d fieldspecs exists for field(s) that are not in the source: %s\n" "They will be skipped.", not_used_fs.length(), tostring(not_used_fs).c_str()); if(not_used_fn.size()!=0) PLWARNING("TextFilesVMatrix::setColumnNamesAndWidth() - " "%d fieldnames in source that don't have fieldspec: %s\n" "They will be skipped.", not_used_fn.length(), tostring(not_used_fn).c_str()); //the new order for fieldspecs TVec< pair<string, string> > fs(fnames_header.size()); for(int i=0;i<fnames_header.size();i++) { string name=fnames_header[i]; int j=0; for(;j<fieldspec.size();j++) if(fieldspec[j].first==name) break; if(j>=fieldspec.size()) fs[i]=pair<string,string>(name,"skip"); else fs[i]=fieldspec[j]; } fieldspec=fs; } for(int k=0; k<fieldspec.length(); k++) { string fname = fieldspec[k].first; string ftype = fieldspec[k].second; if(isValidNonSkipFieldType(ftype)) { // declare the column name fnames.push_back(fname); colrange.push_back( pair<int,int>(width_,1) ); ++width_; } else if(ftype=="skip") { colrange.push_back( pair<int,int>(width_,0) ); } else PLERROR("In TextFilesVMatrix::setColumnNamesAndWidth, Invalid field type specification for field %s: %s",fname.c_str(), ftype.c_str()); } for(int j=0; j<width_; j++) declareField(j, fnames[j]); }
void PLearn::TextFilesVMatrix::setMetaDataDir | ( | const PPath & | the_metadatadir | ) | [virtual] |
This should be called by the build method of every VMatrix that has a metadatadir.
It will create said directory if it doesn's already exist. Throws a PLERROR if called with an empty string.
Reimplemented from PLearn::VMatrix.
Definition at line 453 of file TextFilesVMatrix.cc.
References auto_build_map, autoBuildMappings(), build_vmatrix_stringmap, buildIdx(), buildVMatrixStringMapping(), PLearn::byte_order(), fieldspec, PLearn::TVec< T >::fill(), PLearn::force_mkdir(), PLearn::VMatrix::getMetaDataDir(), i, idxfile, PLearn::isemptyFile(), PLearn::VMatrix::isUpToDate(), PLearn::TVec< T >::length(), PLearn::VMatrix::length_, loadMappings(), mapfiles, mapping, n, PLERROR, PLearn::TVec< T >::resize(), setColumnNamesAndWidth(), PLearn::VMatrix::setMetaDataDir(), PLearn::TVec< T >::size(), txtfilenames, txtfiles, and PLearn::VMatrix::updateMtime().
Referenced by build_().
{ inherited::setMetaDataDir(the_metadatadir); if(getMetaDataDir().empty()) PLERROR("In TextFilesVMatrix::setMetaDataDir() - We need a metadatadir"); if(!force_mkdir(getMetaDataDir())) PLERROR("In TextFilesVMatrix::setMetaDataDir() - could not create" " directory '%s'", getMetaDataDir().absolute().c_str()); for(int i=0;i<txtfilenames.length();i++) updateMtime(txtfilenames[i]); PPath metadir = getMetaDataDir(); PPath idxfname = metadir/"txtmat.idx"; // Now open txtfiles int nf = txtfilenames.length(); txtfiles.resize(nf); for(int k=0; k<nf; k++) { string fnam = txtfilenames[k]; txtfiles[k] = fopen(fnam.c_str(),"r"); if(txtfiles[k]==NULL){ perror("Can't open file"); PLERROR("In TextFilesVMatrix::setMetaDataDir - Can't open file %s", fnam.c_str()); } } setColumnNamesAndWidth(); // open the index file if(!isUpToDate(idxfname) || isemptyFile(idxfname)) buildIdx(); // (re)build it first! idxfile = fopen(idxfname.c_str(),"rb"); if(fgetc(idxfile) != byte_order()) PLERROR("In TextFilesVMatrix::setMetaDataDir - Wrong endianness." " Remove the index file %s for it to be automatically rebuilt", idxfname.c_str()); fread(&length_, 4, 1, idxfile); // Initialize some sizes int n = fieldspec.size(); mapping.resize(n); mapfiles.resize(n); mapfiles.fill(0); // Handle string mapping loadMappings(); if (auto_build_map) autoBuildMappings(); if(build_vmatrix_stringmap) buildVMatrixStringMapping(); // Sanity checking }
TVec< string > PLearn::TextFilesVMatrix::splitIntoFields | ( | const string & | raw_row | ) | const |
Split the passed string into fields given the delimiter.
Definition at line 686 of file TextFilesVMatrix.cc.
References delimiter, quote_delimiter, PLearn::removeblanks(), and PLearn::split_quoted_delimiter().
Referenced by buildIdx(), getTextFields(), and setColumnNamesAndWidth().
{ return split_quoted_delimiter(removeblanks(raw_row), delimiter,quote_delimiter); }
void PLearn::TextFilesVMatrix::transformStringToValue | ( | int | k, |
string | strval, | ||
Vec | dest | ||
) | const [virtual] |
Transform field-k value strval according to its fieldtype into one ore more reals and write those into dest (which should be of appropriate size: colrange[k].second)
Definition at line 777 of file TextFilesVMatrix.cc.
References colrange, PLearn::date_to_float(), fieldspec, PLearn::TVec< T >::first(), getMapping(), getPostalEncoding(), i, PLearn::is_missing(), PLearn::TVec< T >::length(), MISSING_VALUE, PLearn::pl_isnumber(), PLERROR, PLearn::removeblanks(), PLearn::toint(), and PLearn::PDate::toJulianDay().
Referenced by getNewRow().
{ strval = removeblanks(strval); string fieldname = fieldspec[k].first; string fieldtype = fieldspec[k].second; real val; if(dest.length() != colrange[k].second) PLERROR("In TextFilesVMatrix::transformStringToValue, destination vec for field %d should be of length %d, not %d",k,colrange[k].second, dest.length()); if(fieldtype=="skip") { // do nothing, simply skip it return; } if(strval=="") // missing dest[0] = MISSING_VALUE; else if(fieldtype=="auto") { if(pl_isnumber(strval,&val)) dest[0] = real(val); else dest[0] = getMapping(k, strval); } else if(fieldtype=="auto-num") {//We suppose the decimal point is '.' string s=strval; if(strval[0]=='$') s.erase(0,1); if(strval[strval.size()-1]=='$') s.erase(s.end()); for(unsigned int pos=0; pos<strval.size(); pos++) if(s[pos]==',') s.erase(pos--,1); if(pl_isnumber(s,&val)) dest[0] = real(val); else PLERROR("In TextFilesVMatrix::transformStringToValue -" " expedted [$]number[$] as the value for field %d(%s)." " Got %s", k, fieldname.c_str(), strval.c_str()); } else if(fieldtype=="char") { dest[0] = getMapping(k, strval); } else if(fieldtype=="num") { if(pl_isnumber(strval,&val)) dest[0] = real(val); else PLERROR("In TextFilesVMatrix::transformStringToValue - expedted a number as the value for field %d(%s). Got %s",k,fieldname.c_str(),strval.c_str()); } else if(fieldtype=="date") { dest[0] = date_to_float(PDate(strval)); } else if(fieldtype=="jdate") { dest[0] = PDate(strval).toJulianDay(); } else if(fieldtype=="sas_date") { if(strval == "0") // missing dest[0] = MISSING_VALUE; else if(pl_isnumber(strval,&val)) { dest[0] = val; if (val <= 0) { PLERROR("In TextFilesVMatrix::transformStringToValue - " "I didn't know a sas_date could be negative"); } } else PLERROR("In TextFilesVMatrix::transformStringToValue - " "Error while parsing a sas_date"); } else if(fieldtype=="YYYYMM") { if(!pl_isnumber(strval) || toint(strval)<197000) dest[0] = MISSING_VALUE; else dest[0] = PDate(strval+"01").toJulianDay(); } else if(fieldtype=="postal") { dest[0] = getPostalEncoding(strval); } else if(fieldtype=="dollar" || fieldtype=="dollar-comma") { char char_torm = ' '; if(fieldtype=="dollar-comma") char_torm = ','; if(strval[0]=='$') { string s = ""; for(unsigned int pos=1; pos<strval.size(); pos++) if(strval[pos]!=char_torm) s += strval[pos]; if(pl_isnumber(s,&val)) dest[0] = real(val); else PLERROR("In TextFilesVMatrix::transformStringToValue - Goat as value '%s' while parsing field %d (%s) with fieldtype %s",strval.c_str(),k,fieldname.c_str(),fieldtype.c_str()); } else PLERROR("In TextFilesVMatrix::transformStringToValue - Got as value '%s' while expecting a value beggining with '$' while parsing field %d (%s) with fieldtype %s",strval.c_str(),k,fieldname.c_str(),fieldtype.c_str()); } else if(fieldtype=="bell_range") { if (strval == "Negative Value") { // We put an arbitrary negative value since we don't have more info. dest[0] = -100; } else { // A range of the kind "A: $0- 250". string s = ""; unsigned int pos; unsigned int end; for (pos=0; pos<strval.size() && (strval[pos] == ' ' || !pl_isnumber(strval.substr(pos,1))); pos++) {} for (end=pos; end<strval.size() && strval[end] != ' ' && pl_isnumber(strval.substr(end,1)); end++) { s += strval[end]; } real number_1,number_2; if (!pl_isnumber(s,&number_1) || is_missing(number_1)) { PLERROR(("TextFilesVMatrix::transformStringToValue: " + strval + " is not a well formatted Bell range").c_str()); } s = ""; for (pos=end; pos<strval.size() && (strval[pos] == ' ' || !pl_isnumber(strval.substr(pos,1))); pos++) {} for (end=pos; end<strval.size() && strval[end] != ' ' && pl_isnumber(strval.substr(end,1)); end++) { s += strval[end]; } if (!pl_isnumber(s,&number_2) || is_missing(number_2)) { PLERROR(("TextFilesVMatrix::transformStringToValue: " + strval + " is not a well formatted Bell range").c_str()); } dest[0] = (number_1 + number_2) / (real) 2; } } else if(fieldtype=="num-comma") { string s=""; for(uint i=0;i<strval.length();i++) { if(strval[i]!=',') s=s+strval[i]; } if(pl_isnumber(s,&val)) dest[0] = real(val); else PLERROR("In TextFilesVMatrix::transformStringToValue - expedted a number as the value for field %d(%s). Got %s",k,fieldname.c_str(),strval.c_str()); } else { PLERROR("TextFilesVMatrix::TextFilesVMatrix::transformStringToValue, Invalid field type specification for field %s: %s",fieldname.c_str(), fieldtype.c_str()); } }
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 184 of file TextFilesVMatrix.h.
If true, all mappings will be automatically computed at build time if they do not exist yet.
Definition at line 117 of file TextFilesVMatrix.h.
Referenced by declareOptions(), makeDeepCopyFromShallowCopy(), and setMetaDataDir().
If true, new strings for fields of type AUTO will automatically appended to the mapping (in the metadata/mappings/fieldname.map file)
Definition at line 121 of file TextFilesVMatrix.h.
Referenced by autoBuildMappings(), declareOptions(), and getMapping().
If true, standard vmatrix stringmap will be built from the txtmat specific stringmap.
Definition at line 125 of file TextFilesVMatrix.h.
Referenced by declareOptions(), and setMetaDataDir().
TVec< pair<int,int> > PLearn::TextFilesVMatrix::colrange |
list of startpos : width that associate a column range in the resulting VMat to each initial field of the fieldspecs This is built by the setColumnNamesAndWidth method
indicates start position and size in result vector for each field. (as a text field may be transformed into more than one real)
Definition at line 85 of file TextFilesVMatrix.h.
Referenced by buildVMatrixStringMapping(), getNewRow(), setColumnNamesAndWidth(), and transformStringToValue().
If their is no fieldspec, we will use this value reorder_fieldspec_from_headers must be true.
Definition at line 132 of file TextFilesVMatrix.h.
Referenced by build_(), declareOptions(), and setColumnNamesAndWidth().
Delimiter to use to split the fields.
Common delimiters are:
Definition at line 102 of file TextFilesVMatrix.h.
Referenced by declareOptions(), setColumnNamesAndWidth(), and splitIntoFields().
TVec< pair<string, string> > PLearn::TextFilesVMatrix::fieldspec |
Specification of field names and types.
Definition at line 113 of file TextFilesVMatrix.h.
Referenced by autoBuildMappings(), buildIdx(), buildVMatrixStringMapping(), declareOptions(), generateMapCounts(), getIndexOfTextField(), getMapping(), getNewRow(), getTextFields(), loadMappings(), setColumnNamesAndWidth(), setMetaDataDir(), and transformStringToValue().
FILE* PLearn::TextFilesVMatrix::idxfile [protected] |
Definition at line 75 of file TextFilesVMatrix.h.
Referenced by buildIdx(), getFileAndPos(), makeDeepCopyFromShallowCopy(), setMetaDataDir(), and ~TextFilesVMatrix().
TVec<FILE*> PLearn::TextFilesVMatrix::mapfiles [mutable, protected] |
Definition at line 169 of file TextFilesVMatrix.h.
Referenced by getMapping(), setMetaDataDir(), and ~TextFilesVMatrix().
TVec< hash_map<string, real> > PLearn::TextFilesVMatrix::mapping [mutable, protected] |
a possible string->real mapping for each field We keep our own mapping in addition to the standard VMatrix string mapping because we want to handle the case where different strings map to the same value.
(Standard VMatrix string mapping concept is meant to possibly map several values to the same string, i.e. that's the other way round).
Definition at line 167 of file TextFilesVMatrix.h.
Referenced by autoBuildMappings(), buildVMatrixStringMapping(), generateMapCounts(), getMapping(), loadMappings(), and setMetaDataDir().
The path to the .metadata directory.
Definition at line 93 of file TextFilesVMatrix.h.
Referenced by build_(), and declareOptions().
Definition at line 128 of file TextFilesVMatrix.h.
Referenced by declareOptions(), and setColumnNamesAndWidth().
The escate caractere where the delimiter is not considered.
Definition at line 106 of file TextFilesVMatrix.h.
Referenced by declareOptions(), and splitIntoFields().
Definition at line 127 of file TextFilesVMatrix.h.
Referenced by build_(), declareOptions(), and setColumnNamesAndWidth().
An (optional) list of integers, one for each of the txtfilenames indicating the number of header lines at the top of the file to be skipped.
Definition at line 110 of file TextFilesVMatrix.h.
Referenced by buildIdx(), and declareOptions().
A list of paths to raw text files containing the records.
Definition at line 96 of file TextFilesVMatrix.h.
Referenced by declareOptions(), setColumnNamesAndWidth(), and setMetaDataDir().
TVec<FILE*> PLearn::TextFilesVMatrix::txtfiles [protected] |
Definition at line 76 of file TextFilesVMatrix.h.
Referenced by buildIdx(), getTextRow(), makeDeepCopyFromShallowCopy(), setColumnNamesAndWidth(), setMetaDataDir(), and ~TextFilesVMatrix().