PLearn 0.1
Public Types | Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Protected Member Functions | Static Protected Member Functions | Protected Attributes | Private Member Functions | Static Private Member Functions
PLearn::TextFilesVMatrix Class Reference

#include <TextFilesVMatrix.h>

Inheritance diagram for PLearn::TextFilesVMatrix:
Inheritance graph
[legend]
Collaboration diagram for PLearn::TextFilesVMatrix:
Collaboration graph
[legend]

List of all members.

Public Types

typedef RowBufferedVMatrix inherited

Public Member Functions

 TextFilesVMatrix ()
virtual ~TextFilesVMatrix ()
virtual string classname () const
virtual OptionListgetOptionList () const
virtual OptionMapgetOptionMap () const
virtual RemoteMethodMapgetRemoteMethodMap () const
virtual TextFilesVMatrixdeepCopy (CopiesMap &copies) const
real getMapping (int fieldnum, const string &strval) const
 returns the mapped value for field fieldnum corresponding to strval If there was no such mapping present, and auto_extend_map is true, then a new mapping is appended for that field (with value -1000-size_of_map).
string getMapFilePath (int fieldnum) const
string getTextRow (int i) const
 Returns the raw text rows.
int nTextFields () const
 Returns the number of text fields in each row (this may be different from the width() of the resulting VMatrix, as some fields may be skipped, and others may be processed into several columns.
TVec< string > splitIntoFields (const string &raw_row) const
 Split the passed string into fields given the delimiter.
TVec< string > getTextFields (int i) const
 Returns the split raw text rows.
int getIndexOfTextField (const string &fieldname) const
 Returns the index in a split text row of the given named text field (this is the position of that named field in fieldspec )
virtual void setMetaDataDir (const PPath &the_metadatadir)
 This should be called by the build method of every VMatrix that has a metadatadir.
virtual void transformStringToValue (int k, string strval, Vec dest) const
 Transform field-k value strval according to its fieldtype into one ore more reals and write those into dest (which should be of appropriate size: colrange[k].second)
virtual void getNewRow (int i, const Vec &v) const
 This is the only method requiring implementation in subclasses.
virtual void build ()
 Simply calls inherited::build() then build_().
void buildVMatrixStringMapping ()
 Builds standard VMatrix string mapping from this txtmat's specific mapping saveAllStringMappings() can then be called to save them (for later reference from another VMatrix for ex.)
void generateMapCounts ()
 Will generate a counts/ directory in the metadatadir and record, for each field with a mapping, how many times a particular mapped string was encountered.
virtual void makeDeepCopyFromShallowCopy (CopiesMap &copies)
 Transforms a shallow copy into a deep copy.

Static Public Member Functions

static string _classname_ ()
 RowBufferedVMatrix.
static OptionList_getOptionList_ ()
static RemoteMethodMap_getRemoteMethodMap_ ()
static Object_new_instance_for_typemap_ ()
static bool _isa_ (const Object *o)
static void _static_initialize_ ()
static const PPathdeclaringFile ()

Public Attributes

TVec< pair< int, int > > colrange
 list of startpos : width that associate a column range in the resulting VMat to each initial field of the fieldspecs This is built by the setColumnNamesAndWidth method
PPath metadatapath
 The path to the .metadata directory.
TVec< PPathtxtfilenames
 A list of paths to raw text files containing the records.
string delimiter
 Delimiter to use to split the fields.
string quote_delimiter
 The escate caractere where the delimiter is not considered.
TVec< intskipheader
 An (optional) list of integers, one for each of the txtfilenames indicating the number of header lines at the top of the file to be skipped.
TVec< pair< string, string > > fieldspec
 Specification of field names and types.
bool auto_build_map
 If true, all mappings will be automatically computed at build time if they do not exist yet.
bool auto_extend_map
 If true, new strings for fields of type AUTO will automatically appended to the mapping (in the metadata/mappings/fieldname.map file)
bool build_vmatrix_stringmap
 If true, standard vmatrix stringmap will be built from the txtmat specific stringmap.
bool reorder_fieldspec_from_headers
bool partial_match
string default_spec
 If their is no fieldspec, we will use this value reorder_fieldspec_from_headers must be true.

Static Public Attributes

static StaticInitializer _static_initializer_

Protected Member Functions

void autoBuildMappings ()
 If no existing mapping has been loaded, will automatically create all mappings.
void loadMappings ()
virtual real getPostalEncoding (const string &strval, bool display_warning=true) const
virtual bool isValidNonSkipFieldType (const string &ftype) const
 Return true iff 'ftype' is a valid type that does not need to be skipped.

Static Protected Member Functions

static void declareOptions (OptionList &ol)
 Declares this class' options.

Protected Attributes

FILE * idxfile
TVec< FILE * > txtfiles
TVec< hash_map< string, real > > mapping
 a possible string->real mapping for each field We keep our own mapping in addition to the standard VMatrix string mapping because we want to handle the case where different strings map to the same value.
TVec< FILE * > mapfiles

Private Member Functions

void build_ ()
 This does the actual building.
void setColumnNamesAndWidth ()
void getFileAndPos (int i, unsigned char &fileno, int &pos) const
void buildIdx ()
void closeCurrentFile ()

Static Private Member Functions

static void readAndCheckOptionName (PStream &in, const string &optionname, char buf[])

Detailed Description

FORMAT DESCRIPTION

This type of vmatrix is described in a basename.txtmat file and has an associated metadata/ directory

The metadata directory will contain the following files:

The txtmat.idx file is a binary file structured as follows

Definition at line 68 of file TextFilesVMatrix.h.


Member Typedef Documentation

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 183 of file TextFilesVMatrix.h.


Constructor & Destructor Documentation

PLearn::TextFilesVMatrix::TextFilesVMatrix ( )

Definition at line 53 of file TextFilesVMatrix.cc.

PLearn::TextFilesVMatrix::~TextFilesVMatrix ( ) [virtual]

Definition at line 1062 of file TextFilesVMatrix.cc.

References idxfile, PLearn::TVec< T >::length(), mapfiles, PLearn::TVec< T >::size(), and txtfiles.

{
    if(idxfile)
        fclose(idxfile);
    for(int k=0; k<txtfiles.length(); k++)
        fclose(txtfiles[k]);

    for(int k=0; k<mapfiles.size(); k++)
    {
        if(mapfiles[k])
            fclose(mapfiles[k]);
    }
}

Here is the call graph for this function:


Member Function Documentation

string PLearn::TextFilesVMatrix::_classname_ ( ) [static]

RowBufferedVMatrix.

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 82 of file TextFilesVMatrix.cc.

OptionList & PLearn::TextFilesVMatrix::_getOptionList_ ( ) [static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 82 of file TextFilesVMatrix.cc.

RemoteMethodMap & PLearn::TextFilesVMatrix::_getRemoteMethodMap_ ( ) [static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 82 of file TextFilesVMatrix.cc.

bool PLearn::TextFilesVMatrix::_isa_ ( const Object o) [static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 82 of file TextFilesVMatrix.cc.

Object * PLearn::TextFilesVMatrix::_new_instance_for_typemap_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 82 of file TextFilesVMatrix.cc.

StaticInitializer TextFilesVMatrix::_static_initializer_ & PLearn::TextFilesVMatrix::_static_initialize_ ( ) [static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 82 of file TextFilesVMatrix.cc.

void PLearn::TextFilesVMatrix::autoBuildMappings ( ) [protected]

If no existing mapping has been loaded, will automatically create all mappings.

Definition at line 555 of file TextFilesVMatrix.cc.

References auto_extend_map, fieldspec, PLearn::RowBufferedVMatrix::getRow(), i, PLearn::VMatrix::length(), PLearn::TVec< T >::length(), mapping, PLWARNING, PLearn::ProgressBar::update(), and PLearn::VMatrix::width().

Referenced by setMetaDataDir().

                                         {
    // TODO We should somehow check the date of existing mappings to see if they need to be built.
    // For now we just create them if they do not exist yet.

    // First make sure there is no existing mapping.
    int nb_already_exist = 0;
    int nb_type_no_mapping = 0;
    for (int i = 0;  i < mapping.length(); i++) {
        if (!mapping[i].empty())
            nb_already_exist++;
        else if(fieldspec[i].second!="char")//should add auto when it is char that are selected
            nb_type_no_mapping++;
    }
    if(nb_already_exist == 0){
        // Mappings need to be built.
        // We do this by reading the whole data.
        Vec row(width());
        bool auto_extend_map_backup = auto_extend_map;
        auto_extend_map = true;
        ProgressBar pb("Building mappings", length());
        for (int i = 0; i < length(); i++) {
            getRow(i, row);
            pb.update(i + 1);
        }
        auto_extend_map = auto_extend_map_backup;
    }else if (nb_already_exist+nb_type_no_mapping < mapping.length()) {
        for (int i = 0;  i < mapping.length(); i++) 
            if(fieldspec[i].second=="char" && mapping[i].empty())//should add auto when it is char that are selected
                PLWARNING("In TextFilesVMatrix::autoBuildMappings - mapping already existing but not for field %d (%s)",i,fieldspec[i].first.c_str());

        PLWARNING("In TextFilesVMatrix::autoBuildMappings - The existing "
                "mapping is not complete! There are %d fields with build "
                "mapping and there are %d fields that do not need mapping "
                "in a total of %d fields. Erase the mapping directory in "
                "the metadatadir to have it regenerated next time!",
                nb_already_exist,nb_type_no_mapping,mapping.length());
    }//else already build
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::TextFilesVMatrix::build ( ) [virtual]

Simply calls inherited::build() then build_().

Reimplemented from PLearn::VMatrix.

Definition at line 1056 of file TextFilesVMatrix.cc.

References PLearn::VMatrix::build(), and build_().

Referenced by makeDeepCopyFromShallowCopy().

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::TextFilesVMatrix::build_ ( ) [private]

This does the actual building.

Reimplemented from PLearn::VMatrix.

Definition at line 438 of file TextFilesVMatrix.cc.

References default_spec, PLearn::VMatrix::metadatadir, metadatapath, PLERROR, PLWARNING, reorder_fieldspec_from_headers, and setMetaDataDir().

Referenced by build().

{
    if (!default_spec.empty() && !reorder_fieldspec_from_headers)
        PLERROR("In TextFilesVMatrix::build_() when the option default_spec is used, reorder_fieldspec_from_headers must be true");
    if (metadatapath != "") {
        PLWARNING("In TextFilesVMatrix::build_() metadatapath option is deprecated. "
                  "You should use metadatadir instead.\n");

        metadatadir = metadatapath;
        setMetaDataDir(metadatapath);
    }
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::TextFilesVMatrix::buildIdx ( ) [private]

Definition at line 105 of file TextFilesVMatrix.cc.

References PLearn::byte_order(), PLearn::endl(), fieldspec, PLearn::VMatrix::getMetaDataDir(), idxfile, PLearn::isBlank(), PLearn::TVec< T >::isEmpty(), PLearn::TVec< T >::length(), PLearn::VMatrix::length_, PLearn::mvforce(), PLearn::perr, PLERROR, PLWARNING, PLearn::TVec< T >::size(), skipheader, splitIntoFields(), PLearn::strlen(), and txtfiles.

Referenced by setMetaDataDir().

{
    perr << "Building the index file. Please be patient..." << endl;

    if(idxfile)
        fclose(idxfile);
    PPath ft=getMetaDataDir()/"txtmat.idx.tmp";
    PPath f=getMetaDataDir()/"txtmat.idx";
    idxfile = fopen(ft.c_str(),"wb");
    FILE* logfile = fopen((getMetaDataDir()/"txtmat.idx.log").c_str(),"a");

    if (! idxfile)
        PLERROR("TextFilesVMatrix::buildIdx: could not open index file '%s'",
                ( getMetaDataDir()/"txtmat.idx").c_str());
    if (! logfile)
        PLERROR("TextFilesVMatrix::buildIdx: could not open log file '%s'",
                (getMetaDataDir()/"txtmat.idx.log").c_str());

    // write endianness
    fputc(byte_order(), idxfile);
    // We don't know length yet,
    length_ = 0;
    fwrite(&length_, 4, 1, idxfile);

    TVec<string> fields;
    char buf[50000];

    int lineno = 0;
    for(unsigned char fileno=0; fileno<txtfiles.length(); fileno++)
    {
        FILE* fi = txtfiles[(int)fileno];
        fseek(fi,0,SEEK_SET);

        int nskip = 0; // number of header lines to skip
        if(!skipheader.isEmpty())
            nskip = skipheader[int(fileno)];

        // read the data rows and build the index
        for(;;)
        {
            long pos_long = ftell(fi);
            if (pos_long > INT_MAX)
                PLERROR("In TextFilesVMatrix::buildIdx - 'pos_long' cannot be "
                        "more than %d", INT_MAX);
            int pos = int(pos_long);
            if(!fgets(buf, sizeof(buf), fi))
                break;

#ifdef CYGWIN_FGETS_BUGFIX
            // Bugfix for CYGWIN carriage return bug.
            // Should be safe to enable in all case, but need to be tester more widely.
            long new_pos = ftell(fi);
            long lbuf = long(strlen(buf));
            if (lbuf+pos != new_pos){
                if(lbuf+1+pos==new_pos && buf[lbuf-1]=='\n' && buf[lbuf-2]!='\r')
                {
                    //bug under windows. fgets return the good string if unix end of lines, but
                    //change the position suppossing the use of \r\n as carrige return.
                    //So if their is only a \n, we are a caractere too far.
                    //if dos end of lines, return \n as end of lines in the strings and put the pos correctly.
                    
                    fseek(fi,-1,SEEK_CUR);
                    
                    //if unix end of lines
                    if(fgetc(fi)!='\n')
                        fseek(fi,-1,SEEK_CUR);
                }
                //in the eof case?
                else if(lbuf-1+pos==new_pos && buf[lbuf-1]=='\n' && buf[lbuf-2]!='\r')
                    fseek(fi,+1,SEEK_CUR);
                else
                    PLERROR("In TextFilesVMatrix::buildId - The number of characters read "
                            "does not match the position in the file.");
            }
#endif

            buf[sizeof(buf)-1] = '\0';         // ensure null-terminated
            lineno++;
            if(nskip>0)
                --nskip;
            else if(!isBlank(buf))
            {
                fields = splitIntoFields(buf);
                int nf = fields.length();
                if(nf!=fieldspec.size()){
                    fprintf(logfile, "ERROR In file %d line %d: Found %d fields (should be %d):\n %s",fileno,lineno,nf,fieldspec.size(),buf);
                    PLWARNING("In file %d line %d: Found %d fields (should be %d):\n %s",fileno,lineno,nf,fieldspec.size(),buf);
                }
                else  // Row OK! append it to index
                {
                    fputc(fileno, idxfile);
                    fwrite(&pos, 4, 1, idxfile);
                    length_++;
                }
            }
            else
                PLWARNING("In TextFilesVMatrix::buildIdx() - The line %d is blank",lineno);
        } // end of loop over lines of file
    } // end of loop over files

    // Write true length and width
    fseek(idxfile, 1, SEEK_SET);
    fwrite(&length_, 4, 1, idxfile);

    // close files
    fclose(logfile);
    fclose(idxfile);
    mvforce(ft,f);
    perr << "Index file built." << endl;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::TextFilesVMatrix::buildVMatrixStringMapping ( )

Builds standard VMatrix string mapping from this txtmat's specific mapping saveAllStringMappings() can then be called to save them (for later reference from another VMatrix for ex.)

Definition at line 636 of file TextFilesVMatrix.cc.

References PLearn::VMatrix::addStringMapping(), PLearn::TVec< T >::begin(), colrange, PLearn::TVec< T >::end(), fieldspec, PLearn::TVec< T >::first(), j, mapping, n, and PLearn::TVec< T >::size().

Referenced by setMetaDataDir().

{
    int n = fieldspec.size();
    for(int k=0; k<n; k++)
    {
        if(mapping[k].size()>0)
        {
            // get the corresponding VMatrix column range and add the VMatrix mapping
            int colstart = colrange[k].first;
            int ncols = colrange[k].second;
            hash_map<string,real>::const_iterator it = mapping[k].begin();
            hash_map<string,real>::const_iterator itend = mapping[k].end();
            while(it!=itend)
            {
                for(int j=colstart; j<colstart+ncols; j++)
                    addStringMapping(j, it->first, it->second);
                ++it;
            }
        }
    }
}

Here is the call graph for this function:

Here is the caller graph for this function:

string PLearn::TextFilesVMatrix::classname ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 82 of file TextFilesVMatrix.cc.

void PLearn::TextFilesVMatrix::closeCurrentFile ( ) [private]
void PLearn::TextFilesVMatrix::declareOptions ( OptionList ol) [static, protected]

Declares this class' options.

Reimplemented from PLearn::VMatrix.

Definition at line 959 of file TextFilesVMatrix.cc.

References auto_build_map, auto_extend_map, build_vmatrix_stringmap, PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::VMatrix::declareOptions(), default_spec, delimiter, fieldspec, metadatapath, partial_match, quote_delimiter, reorder_fieldspec_from_headers, skipheader, and txtfilenames.

{
    declareOption(ol, "metadatapath", &TextFilesVMatrix::metadatapath, OptionBase::buildoption,
                  "Path of the metadata directory (in which to store the index, ...)\n"
                  "DEPRECATED: use metadatadir instead.\n");

    declareOption(ol, "txtfilenames", &TextFilesVMatrix::txtfilenames, OptionBase::buildoption,
                  "A list of paths to raw text files containing the records");

    declareOption(ol, "delimiter", &TextFilesVMatrix::delimiter, OptionBase::buildoption,
                  "Delimiter to use to split the fields.  Common delimiters are:\n"
                  "- \"\\t\" : used for SAS files (the default)\n"
                  "- \",\"  : used for CSV files\n"
                  "- \";\"  : used for a variant of CSV files\n"
                  "If more then 1 delimiter, we will select one based on the"
                  " first two line\n");

    declareOption(ol, "quote_delimiter", &TextFilesVMatrix::quote_delimiter, OptionBase::buildoption,
        "The escape character to indicate the delimiter is not considered.\n"
        "For instance, '\"' is frequently used.");

    declareOption(ol, "skipheader", &TextFilesVMatrix::skipheader, OptionBase::buildoption,
                  "An (optional) list of integers, one for each of the txtfilenames,\n"
                  "indicating the number of header lines at the top of the file to be skipped.");

    declareOption(ol, "fieldspec", &TextFilesVMatrix::fieldspec, OptionBase::buildoption,
                  "Specification of field names and types (type indicates how the text field is to be mapped to one or more reals)\n"
                  "Currently supported types: \n"
                  "- skip       : Ignore the content of the field, won't be inserted in the resulting VMat\n"
                  "- auto       : If a numeric value, keep it as is, if not, look it up in the mapping (possibly inserting a new mapping if it's not there) \n"
                  "- auto-num   : take any float value with the decimal separator as the dot. If there is a $\n"
                  "               at the start or end it is removed. If there are commas they are removed.\n"
                  "- num        : numeric value, keep as is\n"
                  "- num-comma  : numeric value where thousands are separeted by comma\n"
                  "- char       : look it up in the mapping (possibly inserting a new mapping if it's not there)\n"
                  "- date       : date of the form 25DEC2003 or 25-dec-2003 or 2003/12/25 or 20031225, will be mapped to float date format 1031225\n"
                  "- jdate      : date of the form 25DEC2003 or 25-dec-2003 or 2003/12/25 or 20031225, will be mapped to *julian* date format\n"
                  "- sas_date   : date used by SAS = number of days since Jan. 1st, 1960 (with 0 = missing)\n"
                  "- YYYYMM     : date of the form YYYYMM (e.g: 200312), will be mapped to the julian date of the corresponding month. Everthing "\
                  "               other than a number or lower than 197000 is considered as nan\n"
                  "- postal     : canadian postal code \n"
                  "- dollar     : strangely formatted field with dollar amount. Format is sth like '$12 003'\n"
                  "- dollar-comma : strangely formatted field with dollar amount. Format is sth like '$12,003'\n"
                  "- bell_range : a range like \"A: $0- 250\", replaced by the average of the two bounds;\n"
                  "               if the \"Negative Value\" string is found, it is replaced by -100\n"
        );

    declareOption(ol, "auto_extend_map", &TextFilesVMatrix::auto_extend_map, OptionBase::buildoption,
                  "If true, new strings for fields of type AUTO will automatically appended to the mapping (in the metadata/mappings/fieldname.map file)");

    declareOption(ol, "auto_build_map", &TextFilesVMatrix::auto_build_map, OptionBase::buildoption,
                  "If true, all mappings will be automatically computed at build time if they do not exist yet\n");

    declareOption(ol, "build_vmatrix_stringmap", &TextFilesVMatrix::build_vmatrix_stringmap,
                  OptionBase::buildoption,
                  "If true, standard vmatrix stringmap will be built from the txtmat specific stringmap");

    declareOption(ol, "reorder_fieldspec_from_headers", 
                  &TextFilesVMatrix::reorder_fieldspec_from_headers,
                  OptionBase::buildoption,
                  "If true, will reorder the fieldspec in the order given "
                  "by the field names taken from txtfilenames.");

    declareOption(ol, "partial_match", 
                  &TextFilesVMatrix::partial_match,
                  OptionBase::buildoption,
                  "If true, will repeatedly expand all fieldspec name ending "
                  "with * to the full name from header."
                  "The expansion is equivalent to the regex 'field_spec_name*'."
                  "The option reorder_fieldspec_from_headers must be true");

    declareOption(ol, "default_spec", 
                  &TextFilesVMatrix::default_spec,
                  OptionBase::buildoption,
                  "If there is no fieldspec for a fieldname, we will use this"
                  "value. reorder_fieldspec_from_headers must be true.");

    // Now call the parent class' declareOptions
    inherited::declareOptions(ol);
}

Here is the call graph for this function:

static const PPath& PLearn::TextFilesVMatrix::declaringFile ( ) [inline, static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 184 of file TextFilesVMatrix.h.

{ return getMetaDataDir()/"mappings"/fieldspec[fieldnum].first+".map"; }
TextFilesVMatrix * PLearn::TextFilesVMatrix::deepCopy ( CopiesMap copies) const [virtual]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 82 of file TextFilesVMatrix.cc.

void PLearn::TextFilesVMatrix::generateMapCounts ( )

Will generate a counts/ directory in the metadatadir and record, for each field with a mapping, how many times a particular mapped string was encountered.

Definition at line 594 of file TextFilesVMatrix.cc.

References PLearn::TVec< T >::begin(), PLearn::TVec< T >::end(), fieldspec, PLearn::VMatrix::find(), PLearn::VMatrix::getMetaDataDir(), getTextFields(), i, PLearn::VMatrix::length(), PLearn::TVec< T >::length(), mapping, n, PLearn::VMatrix::save(), and PLearn::TVec< T >::size().

{
    int n = fieldspec.size();
    TVec< hash_map<string, int> > counts(n);
    for(int k=0; k<n; k++)
    {
        if(!mapping[k].empty())
        {
            hash_map<string, real>& mapping_k = mapping[k];
            hash_map<string, int>& counts_k = counts[k];
            hash_map<string, real>::const_iterator it = mapping_k.begin();
            hash_map<string, real>::const_iterator itend = mapping_k.end();
            while(it!=itend)
            {
                counts_k[ it->first ] = 0;
                ++it;
            }
        }
    }

    int l = length();
    ProgressBar pg("Generating counts of mappings",l);
    for(int i=0; i<l; i++)
    {
        TVec<string> fields = getTextFields(i);
        for(int k=0; k<fields.length(); k++)
        {
            if(mapping[k].find(fields[k])!=mapping[k].end())
                ++counts[k][fields[k]];
        }
        pg(i);
    }

    // Save the counts
    for(int k=0; k<n; k++)
    {
        if(!counts[k].empty())
            PLearn::save( getMetaDataDir() / "counts" / fieldspec[k].first+".count", counts[k] );
    }

}

Here is the call graph for this function:

void PLearn::TextFilesVMatrix::getFileAndPos ( int  i,
unsigned char &  fileno,
int pos 
) const [private]

Definition at line 85 of file TextFilesVMatrix.cc.

References idxfile, PLearn::VMatrix::length(), PLASSERT, and PLERROR.

Referenced by getTextRow().

{
    PLASSERT(idxfile!=0);
    if(i<0 || i>=length())
        PLERROR("TextFilesVMatrix::getFileAndPos out of range row %d (only %d rows)", i, length());
    fseek(idxfile, 5+i*5, SEEK_SET);
    fileno = fgetc(idxfile);
    fread(&pos, sizeof(int), 1, idxfile);
}

Here is the call graph for this function:

Here is the caller graph for this function:

int PLearn::TextFilesVMatrix::getIndexOfTextField ( const string &  fieldname) const

Returns the index in a split text row of the given named text field (this is the position of that named field in fieldspec )

Definition at line 95 of file TextFilesVMatrix.cc.

References fieldspec, i, n, PLERROR, and PLearn::TVec< T >::size().

{
    int n = fieldspec.size();
    for(int i=0; i<n; i++)
        if(fieldspec[i].first==fieldname)
            return i;
    PLERROR("In TextFilesVMatrix::getIndexOfTextField unknown field %s",fieldname.c_str());
    return -1; // to make the compiler happy
}

Here is the call graph for this function:

string PLearn::TextFilesVMatrix::getMapFilePath ( int  fieldnum) const [inline]

Definition at line 192 of file TextFilesVMatrix.h.

Referenced by getMapping(), and loadMappings().

    { return getMetaDataDir()/"mappings"/fieldspec[fieldnum].first+".map"; }

Here is the caller graph for this function:

real PLearn::TextFilesVMatrix::getMapping ( int  fieldnum,
const string &  strval 
) const

returns the mapped value for field fieldnum corresponding to strval If there was no such mapping present, and auto_extend_map is true, then a new mapping is appended for that field (with value -1000-size_of_map).

If not a PLERROR is thrown.

Definition at line 658 of file TextFilesVMatrix.cc.

References auto_extend_map, fieldspec, PLearn::TVec< T >::find(), PLearn::force_mkdir_for_file(), getMapFilePath(), m, mapfiles, mapping, and PLERROR.

Referenced by transformStringToValue().

{
    hash_map<string, real>& m = mapping[fieldnum];
    hash_map<string, real>::const_iterator found = m.find(strval);
    if(found!=m.end()) // found it!
        return found->second;

    // strval not found
    if(!auto_extend_map)
        PLERROR("In TextFilesVMatrix::getMapping - No mapping found for field %d (%s) string-value \"%s\" ", fieldnum, fieldspec[fieldnum].first.c_str(), strval.c_str());

    // OK, let's extend the mapping...
    real val = real(-1000 - int(m.size()));
    m[strval] = val;

    if(!mapfiles[fieldnum])
    {
        string fname = getMapFilePath(fieldnum);
        force_mkdir_for_file(fname);
        mapfiles[fieldnum] = fopen(fname.c_str(),"a");
        if(!mapfiles[fieldnum])
            PLERROR("In TextFilesVMatrix::getMapping - Could not open map file %s\n for appending\n",fname.c_str());
    }

    fprintf(mapfiles[fieldnum],"\n\"%s\" %f", strval.c_str(), val);
    return val;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::TextFilesVMatrix::getNewRow ( int  i,
const Vec v 
) const [virtual]

This is the only method requiring implementation in subclasses.

Implements PLearn::RowBufferedVMatrix.

Definition at line 937 of file TextFilesVMatrix.cc.

References colrange, fieldspec, PLearn::TVec< T >::first(), getTextFields(), i, PLearn::PLearnError::message(), n, PLERROR, PLearn::TVec< T >::size(), PLearn::TVec< T >::subVec(), and transformStringToValue().

{
    TVec<string> fields = getTextFields(i);
    int n = fields.size();

    for(int k=0; k<n; k++)
    {
        string fieldname = fieldspec[k].first;
        string fieldtype = fieldspec[k].second;
        string strval = fields[k];
        Vec dest = v.subVec(colrange[k].first, colrange[k].second);

        try
        { transformStringToValue(k, strval, dest); }
        catch(const PLearnError& e)
        {
            PLERROR("In TextFilesVMatrix, while parsing field %d (%s) of row %d: \n%s",
                    k,fieldname.c_str(),i,e.message().c_str());
        }
    }
}

Here is the call graph for this function:

OptionList & PLearn::TextFilesVMatrix::getOptionList ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 82 of file TextFilesVMatrix.cc.

OptionMap & PLearn::TextFilesVMatrix::getOptionMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 82 of file TextFilesVMatrix.cc.

real PLearn::TextFilesVMatrix::getPostalEncoding ( const string &  strval,
bool  display_warning = true 
) const [protected, virtual]

Definition at line 702 of file TextFilesVMatrix.cc.

References MISSING_VALUE, and PLWARNING.

Referenced by transformStringToValue().

{
    if(strval=="")
        return MISSING_VALUE;

    char first_char = strval[0];
    int second_digit = strval[1];
    real val = 0;
    if(first_char=='A')
        val = 30 + second_digit;
    else if(first_char=='B')
        val = 40 + second_digit;
    else if(first_char=='C')
        val = 50 + second_digit;
    else if(first_char=='E')
        val = 60 + second_digit;
    else if(first_char=='G')
        val = 0 + second_digit;
    else if(first_char=='H')
        val = 10 + second_digit;
    else if(first_char=='J')
        val = 20 + second_digit;
    else if(first_char=='K')
        val = 70 + second_digit;
    else if(first_char=='L')
        val = 80 + second_digit;
    else if(first_char=='M')
        val = 90 + second_digit;
    else if(first_char=='N')
        val = 100 + second_digit;
    else if(first_char=='P')
        val = 110 + second_digit;
    else if(first_char=='R')
        val = 120 + second_digit;
    else if(first_char=='S')
        val = 130 + second_digit;
    else if(first_char=='T')
        val = 140 + second_digit;
    else if(first_char=='V')
        val = 150 + second_digit;
    else if(first_char=='W')
        val = 160 + second_digit;
    else if(first_char=='X')
        val = 170 + second_digit;
    else if(first_char=='Y')
        val = 180 + second_digit;
    else if(first_char=='0' || first_char=='1' || first_char=='2' || first_char=='3' ||
            first_char=='4' || first_char=='5' || first_char=='6' || first_char=='7' ||
            first_char=='8' || first_char=='9') {
        // That would be a C.P.
        int first_digit = strval[0];
        val = 260 + first_digit * 10 + second_digit;
    }
    else {
        //http://en.wikipedia.org/wiki/Canadian_postal_code
        //No postal code includes the letters D, F, I, O, Q, or U,
        //as the OCR equipment used in automated sorting could easily
        //confuse them with other letters and digits, 
        if (display_warning) {
            string errmsg;
            if(first_char=='D' ||first_char=='F' ||first_char=='I' ||
               first_char=='O' ||first_char=='Q' ||first_char=='U')
                errmsg = "Postal code don't use letters D, F, I, O, Q, or U: ";
            else
                errmsg = "Currently only some postal codes are supported: ";

            errmsg += "can't process " + strval + ", value will be set to 0.";
            PLWARNING(errmsg.c_str());
        }
        val = 0;
    }

    return val;
}

Here is the caller graph for this function:

RemoteMethodMap & PLearn::TextFilesVMatrix::getRemoteMethodMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 82 of file TextFilesVMatrix.cc.

TVec< string > PLearn::TextFilesVMatrix::getTextFields ( int  i) const

Returns the split raw text rows.

Definition at line 691 of file TextFilesVMatrix.cc.

References fieldspec, getTextRow(), PLERROR, PLearn::removeblanks(), PLearn::TVec< T >::size(), and splitIntoFields().

Referenced by generateMapCounts(), and getNewRow().

{
    string rowi = getTextRow(i);
    TVec<string> fields =  splitIntoFields(rowi);
    if(fields.size() != fieldspec.size())
        PLERROR("In TextFilesVMatrix::getTextFields - In getting fields of row %d, wrong number of fields: %d (should be %d):\n%s\n",i,fields.size(),fieldspec.size(),rowi.c_str());
    for(int k=0; k<fields.size(); k++)
        fields[k] = removeblanks(fields[k]);
    return fields;
}

Here is the call graph for this function:

Here is the caller graph for this function:

string PLearn::TextFilesVMatrix::getTextRow ( int  i) const

Returns the raw text rows.

Definition at line 514 of file TextFilesVMatrix.cc.

References getFileAndPos(), PLERROR, PLearn::removenewline(), and txtfiles.

Referenced by getTextFields().

{
    unsigned char fileno;
    int pos;
    getFileAndPos(i, fileno, pos);
    FILE* f = txtfiles[(int)fileno];
    fseek(f,pos,SEEK_SET);
    char buf[50000];

    if(!fgets(buf, sizeof(buf), f))
        PLERROR("In TextFilesVMatrix::getTextRow - fgets for row %d returned NULL",i);
    return removenewline(buf);
}

Here is the call graph for this function:

Here is the caller graph for this function:

bool PLearn::TextFilesVMatrix::isValidNonSkipFieldType ( const string &  ftype) const [protected, virtual]

Return true iff 'ftype' is a valid type that does not need to be skipped.

Definition at line 219 of file TextFilesVMatrix.cc.

Referenced by setColumnNamesAndWidth().

                                                                        {
    return (ftype=="auto" || ftype=="num" || ftype=="date" || ftype=="jdate" ||
            ftype=="postal" || ftype=="dollar" || ftype=="dollar-comma" ||
            ftype=="YYYYMM" || ftype=="sas_date" || ftype == "bell_range" ||
            ftype == "char" || ftype=="num-comma" || ftype=="auto-num");
}

Here is the caller graph for this function:

void PLearn::TextFilesVMatrix::loadMappings ( ) [protected]

Definition at line 528 of file TextFilesVMatrix.cc.

References fieldspec, getMapFilePath(), PLearn::getNonBlankLines(), i, PLearn::isfile(), PLearn::loadFileAsString(), mapping, n, PLearn::pl_isnumber(), PLERROR, PLearn::TVec< T >::size(), and PLearn::VMatrix::updateMtime().

Referenced by setMetaDataDir().

{
    int n = fieldspec.size();
    for(int k=0; k<n; k++)
    {
        string fname = getMapFilePath(k);
        if (isfile(fname)) {
            updateMtime(fname);
            vector<string> all_lines = getNonBlankLines(loadFileAsString(fname));
            for (size_t i = 0; i < all_lines.size(); i++) {
                string map_line = all_lines[i];
                size_t start_of_string = map_line.find('"');
                size_t end_of_string = map_line.rfind('"');
                string strval = map_line.substr(start_of_string + 1, end_of_string - start_of_string - 1);
                string real_val_str = map_line.substr(end_of_string + 1);
                real real_val;
                if (!pl_isnumber(real_val_str, &real_val))
                    PLERROR("In TextFilesVMatrix::loadMappings - Found a mapping to something that is not a number (%s) in file %s at non-black line %ld", map_line.c_str(), fname.c_str(), long(i));
                mapping[k][strval] = real_val;
            }
        }
    }
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::TextFilesVMatrix::makeDeepCopyFromShallowCopy ( CopiesMap copies) [virtual]

Transforms a shallow copy into a deep copy.

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 1076 of file TextFilesVMatrix.cc.

References auto_build_map, build(), idxfile, PLearn::RowBufferedVMatrix::makeDeepCopyFromShallowCopy(), PLearn::TVec< T >::resize(), and txtfiles.

{
    inherited::makeDeepCopyFromShallowCopy(copies);
    idxfile=0;
    txtfiles.resize(0);
    //the map should be already build.
    auto_build_map=false;
    build();
}

Here is the call graph for this function:

int PLearn::TextFilesVMatrix::nTextFields ( ) const [inline]

Returns the number of text fields in each row (this may be different from the width() of the resulting VMatrix, as some fields may be skipped, and others may be processed into several columns.

Definition at line 202 of file TextFilesVMatrix.h.

    { return fieldspec.length(); }
void PLearn::TextFilesVMatrix::readAndCheckOptionName ( PStream in,
const string &  optionname,
char  buf[] 
) [static, private]

Definition at line 1040 of file TextFilesVMatrix.cc.

References PLearn::PStream::get(), PLERROR, PLearn::PStream::readUntil(), PLearn::removeblanks(), and PLearn::PStream::skipBlanksAndComments().

{
    in.skipBlanksAndComments();
    in.readUntil(buf, sizeof(buf), "= ");
    string option = removeblanks(buf);
    in.skipBlanksAndComments();
    char eq = in.get();
    if(option!=optionname || eq!='=')
        PLERROR("In TextFilesVMatrix::readAndCheckOptionName - "
                "Bad syntax in .txtmat file.\n"
                "Expected option %s = ...\n"
                "Read %s %c\n", optionname.c_str(), option.c_str(), eq);
}

Here is the call graph for this function:

void PLearn::TextFilesVMatrix::setColumnNamesAndWidth ( ) [private]

Definition at line 226 of file TextFilesVMatrix.cc.

References PLearn::TVec< T >::append(), colrange, PLearn::VMatrix::declareField(), default_spec, delimiter, PLearn::endl(), fieldspec, PLearn::TVec< T >::first(), i, isValidNonSkipFieldType(), j, PLearn::TVec< T >::length(), n, NORMAL_LOG, partial_match, PLCHECK, PLCHECK_MSG, PLERROR, PLWARNING, PLearn::TVec< T >::pop(), PLearn::TVec< T >::push_back(), PLearn::removeblanks(), reorder_fieldspec_from_headers, PLearn::TVec< T >::size(), splitIntoFields(), PLearn::string_begins_with(), PLearn::tostring(), txtfilenames, txtfiles, and PLearn::VMatrix::width_.

Referenced by setMetaDataDir().

{
    width_ = 0;
    TVec<string> fnames;
    TVec<string> fnames_header;//field names take in the header of source file
    char buf[50000];


    //select witch delimiter we will use for all the files.
    if(delimiter.size()>1){
        FILE* f = txtfiles[0];
        fseek(f,0,SEEK_SET);
        if(!fgets(buf, sizeof(buf), f))
            PLERROR("In TextFilesVMatrix::setColumnNamesAndWidth() - "
                    "Couldn't read the fields names from file '%s'",
                    txtfilenames[0].c_str());

        string s1 = string(buf);
        if(!fgets(buf, sizeof(buf), f))
            PLERROR("In TextFilesVMatrix::setColumnNamesAndWidth() - "
                    "Couldn't read the fields names from file '%s'",
                    txtfilenames[0].c_str());
        string s2 = string(buf);
        TVec<int> nbs1(delimiter.size());
        TVec<int> nbs2(delimiter.size());
        
        string old_delimiter = delimiter;
        for(uint i=0;i<old_delimiter.size();i++){
            delimiter = old_delimiter[i];
            TVec<string> fields1 = splitIntoFields(s1);
            TVec<string> fields2 = splitIntoFields(s2);
            nbs1[i]=fields1.size();
            nbs2[i]=fields2.size();
        }
        delimiter=old_delimiter;
        for(uint i=0;i<old_delimiter.size();i++){
            if(nbs1[i]==nbs2[i]&& nbs1[i]>0){
                delimiter = old_delimiter[i];
            }
        }
        MODULE_LOG << "Selected delimiter: <" << delimiter << ">" << endl;
        if(delimiter.size()!=1){
            PLERROR("In TextFilesVMatrix::setColumnNamesAndWidth() - We can't"
                    " automatically determine the delimiter to use as the two"
                    " first row don't have a common delimiter with the same"
                    " number of occurence. nbs1=%s, nbs2=%s",
                    tostring(nbs1).c_str(),tostring(nbs2).c_str());
        }
    }
    PLCHECK(delimiter.size()==1);

    //read the fieldnames from the files.
    for(int i=0; i<txtfiles.size(); i++){
        FILE* f = txtfiles[i];
        fseek(f,0,SEEK_SET);
        if(!fgets(buf, sizeof(buf), f))
            PLERROR("In TextFilesVMatrix::setColumnNamesAndWidth() - "
                    "Couldn't read the fields names from file '%s'",
                    txtfilenames[i].c_str());
        fseek(f,0,SEEK_SET);

        TVec<string> fields = splitIntoFields(buf);

        //check that we have the good delimiter
        if(fields.size()==1 && fieldspec.size()>1)
            PLERROR("In TextFilesVMatrix::setColumnNamesAndWidth() -"
                    " We found only 1 column in the first line, but"
                    " their is %d fieldspec. Meaby the delimiter '%s'"
                    " is not the right one. The line is %s",
                    fieldspec.size(),delimiter.c_str(),
                    string(buf).c_str());
        
        if(reorder_fieldspec_from_headers || partial_match){
            fields.append(removeblanks(fields.pop()));
            
            fnames_header.append(fields);
        }
    }
    if(partial_match)
    {
        TVec< pair<string, string> > new_fieldspec;
        TVec<string> no_expended_fields;
        PLCHECK_MSG(reorder_fieldspec_from_headers,
                    "In TextFilesVMatrix::setColumnNamesAndWidth - "
                    "when partial_match is true, reorder_fieldspec_from_headers"
                    " must be true.");
        for(int i=0;i<fieldspec.size();i++)
        {
            bool expended = false;
            string fname=fieldspec[i].first;
            if(fname[fname.size()-1]!='*')
            {
                new_fieldspec.append(fieldspec[i]);
                continue;
            }
            fname.resize(fname.size()-1);//remove the last caracter (*)
            for(int j=0;j<fnames_header.size();j++)
            {
                if(string_begins_with(fnames_header[j],fname))
                {
                    pair<string,string> n=make_pair(fnames_header[j],
                                                    fieldspec[i].second);
//                    perr<<"expanding "<<fieldspec[i] << " to " << n <<endl;
                    
                    new_fieldspec.append(n);
                    expended = true;
                }
            }
            if(!expended)
                no_expended_fields.append(fieldspec[i].first);
        }
        if(no_expended_fields.length()>0){
            NORMAL_LOG<<"In TextFilesVMatrix::setColumnNamesAndWidth - "
                      <<"Did not find any partial match for "
                      <<no_expended_fields.length()<<" spec:";
            for(int i=0;i<no_expended_fields.length();i++)
                NORMAL_LOG<<" "<<no_expended_fields[i];
            NORMAL_LOG<<endl;
        }
            
        fieldspec = new_fieldspec;
    }

    if(reorder_fieldspec_from_headers)
    {
        //check that all field names from the header have a spec
        TVec<string> not_used_fn;
        for(int i=0;i<fnames_header.size();i++)
        {
            string name=fnames_header[i];
            int j=0;
            for(;j<fieldspec.size();j++)
                if(fieldspec[j].first==name)
                    break;
            if(j>=fieldspec.size()){
                if(default_spec!=""){
                    fieldspec.append(make_pair(name,default_spec));
                }else
                    not_used_fn.append(name);
            }
        }
        //check that all fieldspec names are also in the header
        TVec<string> not_used_fs;
        for(int i=0;i<fieldspec.size();i++)
        {
            string name=fieldspec[i].first;
            int j=0;
            for(;j<fnames_header.size();j++)
                if(fnames_header[j]==name)
                    break;
            if(j>=fnames_header.size())
                not_used_fs.append(name);
        }
        //check that we have the good number of fieldspec
        //if partial match is true, we don't want to generate the warning everytime
        if(fnames_header.size()!=fieldspec.size() && !partial_match)
        {
            PLWARNING("In TextFilesVMatrix::setColumnNamesAndWidth() - "
                    "We read %d field names from the header but have %d"
                    "fieldspec",fnames_header.size(),fieldspec.size());
        }

        if(not_used_fs.size()!=0)
            PLWARNING("TextFilesVMatrix::setColumnNamesAndWidth() - "
                      "%d fieldspecs exists for field(s) that are not in the source: %s\n"
                      "They will be skipped.",
                      not_used_fs.length(), tostring(not_used_fs).c_str());
        if(not_used_fn.size()!=0)
            PLWARNING("TextFilesVMatrix::setColumnNamesAndWidth() - "
                      "%d fieldnames in source that don't have fieldspec: %s\n"
                      "They will be skipped.",
                      not_used_fn.length(), tostring(not_used_fn).c_str());
    

        //the new order for fieldspecs
        TVec< pair<string, string> > fs(fnames_header.size());
        for(int i=0;i<fnames_header.size();i++)
        {
            string name=fnames_header[i];
            int j=0;
            for(;j<fieldspec.size();j++)
                if(fieldspec[j].first==name)
                    break;
            if(j>=fieldspec.size())
                fs[i]=pair<string,string>(name,"skip");
            else
                fs[i]=fieldspec[j];
        }
        fieldspec=fs;
    }
    for(int k=0; k<fieldspec.length(); k++)
    {
        string fname = fieldspec[k].first;
        string ftype = fieldspec[k].second;
        if(isValidNonSkipFieldType(ftype))
        {
            // declare the column name
            fnames.push_back(fname);
            colrange.push_back( pair<int,int>(width_,1) );
            ++width_;
        }
        else if(ftype=="skip")
        {
            colrange.push_back( pair<int,int>(width_,0) );
        }
        else
            PLERROR("In TextFilesVMatrix::setColumnNamesAndWidth, Invalid field type specification for field %s: %s",fname.c_str(), ftype.c_str());
    }
    for(int j=0; j<width_; j++)
        declareField(j, fnames[j]);
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::TextFilesVMatrix::setMetaDataDir ( const PPath the_metadatadir) [virtual]

This should be called by the build method of every VMatrix that has a metadatadir.

It will create said directory if it doesn's already exist. Throws a PLERROR if called with an empty string.

Reimplemented from PLearn::VMatrix.

Definition at line 453 of file TextFilesVMatrix.cc.

References auto_build_map, autoBuildMappings(), build_vmatrix_stringmap, buildIdx(), buildVMatrixStringMapping(), PLearn::byte_order(), fieldspec, PLearn::TVec< T >::fill(), PLearn::force_mkdir(), PLearn::VMatrix::getMetaDataDir(), i, idxfile, PLearn::isemptyFile(), PLearn::VMatrix::isUpToDate(), PLearn::TVec< T >::length(), PLearn::VMatrix::length_, loadMappings(), mapfiles, mapping, n, PLERROR, PLearn::TVec< T >::resize(), setColumnNamesAndWidth(), PLearn::VMatrix::setMetaDataDir(), PLearn::TVec< T >::size(), txtfilenames, txtfiles, and PLearn::VMatrix::updateMtime().

Referenced by build_().

                                                                 {
    inherited::setMetaDataDir(the_metadatadir);

    if(getMetaDataDir().empty())
        PLERROR("In TextFilesVMatrix::setMetaDataDir() - We need a metadatadir");
    if(!force_mkdir(getMetaDataDir()))
        PLERROR("In TextFilesVMatrix::setMetaDataDir() - could not create"
                " directory '%s'",
                getMetaDataDir().absolute().c_str());

    for(int i=0;i<txtfilenames.length();i++)
        updateMtime(txtfilenames[i]);

    PPath metadir = getMetaDataDir();
    PPath idxfname = metadir/"txtmat.idx";

    // Now open txtfiles
    int nf = txtfilenames.length();
    txtfiles.resize(nf);
    for(int k=0; k<nf; k++)
    {
        string fnam = txtfilenames[k];
        txtfiles[k] = fopen(fnam.c_str(),"r");
        if(txtfiles[k]==NULL){
            perror("Can't open file");
            PLERROR("In TextFilesVMatrix::setMetaDataDir - Can't open file %s",
                    fnam.c_str());
        }
    }

    setColumnNamesAndWidth();

    // open the index file
    if(!isUpToDate(idxfname) || isemptyFile(idxfname))
        buildIdx(); // (re)build it first!
    idxfile = fopen(idxfname.c_str(),"rb");
    if(fgetc(idxfile) != byte_order())
        PLERROR("In TextFilesVMatrix::setMetaDataDir - Wrong endianness."
                " Remove the index file %s for it to be automatically rebuilt",
                idxfname.c_str());
    fread(&length_, 4, 1, idxfile);

    // Initialize some sizes
    int n = fieldspec.size();
    mapping.resize(n);
    mapfiles.resize(n);
    mapfiles.fill(0);

    // Handle string mapping
    loadMappings();

    if (auto_build_map)
        autoBuildMappings();

    if(build_vmatrix_stringmap)
        buildVMatrixStringMapping();

    // Sanity checking
}

Here is the call graph for this function:

Here is the caller graph for this function:

TVec< string > PLearn::TextFilesVMatrix::splitIntoFields ( const string &  raw_row) const

Split the passed string into fields given the delimiter.

Definition at line 686 of file TextFilesVMatrix.cc.

References delimiter, quote_delimiter, PLearn::removeblanks(), and PLearn::split_quoted_delimiter().

Referenced by buildIdx(), getTextFields(), and setColumnNamesAndWidth().

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::TextFilesVMatrix::transformStringToValue ( int  k,
string  strval,
Vec  dest 
) const [virtual]

Transform field-k value strval according to its fieldtype into one ore more reals and write those into dest (which should be of appropriate size: colrange[k].second)

Definition at line 777 of file TextFilesVMatrix.cc.

References colrange, PLearn::date_to_float(), fieldspec, PLearn::TVec< T >::first(), getMapping(), getPostalEncoding(), i, PLearn::is_missing(), PLearn::TVec< T >::length(), MISSING_VALUE, PLearn::pl_isnumber(), PLERROR, PLearn::removeblanks(), PLearn::toint(), and PLearn::PDate::toJulianDay().

Referenced by getNewRow().

{
    strval = removeblanks(strval);
    string fieldname = fieldspec[k].first;
    string fieldtype = fieldspec[k].second;
    real val;

    if(dest.length() != colrange[k].second)
        PLERROR("In TextFilesVMatrix::transformStringToValue, destination vec for field %d should be of length %d, not %d",k,colrange[k].second, dest.length());


    if(fieldtype=="skip")
    {
        // do nothing, simply skip it
        return;
    }
    
    if(strval=="")  // missing
        dest[0] = MISSING_VALUE;
    else if(fieldtype=="auto")
    {
        if(pl_isnumber(strval,&val))
            dest[0] = real(val);
        else
            dest[0] = getMapping(k, strval);
    }
    else if(fieldtype=="auto-num")
    {//We suppose the decimal point is '.'
        string s=strval;
        if(strval[0]=='$')
            s.erase(0,1);
        if(strval[strval.size()-1]=='$')
            s.erase(s.end());
        
        for(unsigned int pos=0; pos<strval.size(); pos++)
            if(s[pos]==',')
                s.erase(pos--,1);
        if(pl_isnumber(s,&val))
            dest[0] = real(val);
        else
            PLERROR("In TextFilesVMatrix::transformStringToValue -"
                    " expedted [$]number[$] as the value for field %d(%s)."
                    " Got %s", k, fieldname.c_str(), strval.c_str());
    }
    else if(fieldtype=="char")
    {
        dest[0] = getMapping(k, strval);
    }
    else if(fieldtype=="num")
    {
        if(pl_isnumber(strval,&val))
            dest[0] = real(val);
        else
            PLERROR("In TextFilesVMatrix::transformStringToValue - expedted a number as the value for field %d(%s). Got %s",k,fieldname.c_str(),strval.c_str());
                
    }
    else if(fieldtype=="date")
    {
        dest[0] = date_to_float(PDate(strval));
    }
    else if(fieldtype=="jdate")
    {
        dest[0] = PDate(strval).toJulianDay();
    }
    else if(fieldtype=="sas_date")
    {
        if(strval == "0")  // missing
            dest[0] = MISSING_VALUE;
        else if(pl_isnumber(strval,&val)) {
            dest[0] = val;
            if (val <= 0) {
                PLERROR("In TextFilesVMatrix::transformStringToValue - "
                        "I didn't know a sas_date could be negative");
            }
        }
        else
            PLERROR("In TextFilesVMatrix::transformStringToValue - "
                    "Error while parsing a sas_date");
    }
    else if(fieldtype=="YYYYMM")
    {
        if(!pl_isnumber(strval) || toint(strval)<197000)
            dest[0] = MISSING_VALUE;
        else
            dest[0] = PDate(strval+"01").toJulianDay();
    }
    else if(fieldtype=="postal")
    {
        dest[0] = getPostalEncoding(strval);
    }
    else if(fieldtype=="dollar" || fieldtype=="dollar-comma")
    {
        char char_torm = ' ';
        if(fieldtype=="dollar-comma")
            char_torm = ',';
        if(strval[0]=='$')
        {
            string s = "";
            for(unsigned int pos=1; pos<strval.size(); pos++)
                if(strval[pos]!=char_torm)
                    s += strval[pos];

            if(pl_isnumber(s,&val))
                dest[0] = real(val);
            else
                PLERROR("In TextFilesVMatrix::transformStringToValue - Goat as value '%s' while parsing field %d (%s) with fieldtype %s",strval.c_str(),k,fieldname.c_str(),fieldtype.c_str());
        }
        else
            PLERROR("In TextFilesVMatrix::transformStringToValue - Got as value '%s' while expecting a value beggining with '$' while parsing field %d (%s) with fieldtype %s",strval.c_str(),k,fieldname.c_str(),fieldtype.c_str());
    }
    else if(fieldtype=="bell_range") {
        if (strval == "Negative Value") {
            // We put an arbitrary negative value since we don't have more info.
            dest[0] = -100;
        } else {
            // A range of the kind "A: $0- 250".
            string s = "";
            unsigned int pos;
            unsigned int end;
            for (pos=0; pos<strval.size() && (strval[pos] == ' ' || !pl_isnumber(strval.substr(pos,1))); pos++) {}
            for (end=pos; end<strval.size() && strval[end] != ' ' && pl_isnumber(strval.substr(end,1)); end++) {
                s += strval[end];
            }
            real number_1,number_2;
            if (!pl_isnumber(s,&number_1) || is_missing(number_1)) {
                PLERROR(("TextFilesVMatrix::transformStringToValue: " + strval +
                         " is not a well formatted Bell range").c_str());
            }
            s = "";
            for (pos=end; pos<strval.size() && (strval[pos] == ' ' || !pl_isnumber(strval.substr(pos,1))); pos++) {}
            for (end=pos; end<strval.size() && strval[end] != ' ' && pl_isnumber(strval.substr(end,1)); end++) {
                s += strval[end];
            }
            if (!pl_isnumber(s,&number_2) || is_missing(number_2)) {
                PLERROR(("TextFilesVMatrix::transformStringToValue: " + strval +
                         " is not a well formatted Bell range").c_str());
            }
            dest[0] = (number_1 + number_2) / (real) 2;
        }
    }
    else if(fieldtype=="num-comma")
    {
        string s="";
        for(uint i=0;i<strval.length();i++)
        {
            if(strval[i]!=',')
                s=s+strval[i];
        }
        if(pl_isnumber(s,&val))
            dest[0] = real(val);
        else
            PLERROR("In TextFilesVMatrix::transformStringToValue - expedted a number as the value for field %d(%s). Got %s",k,fieldname.c_str(),strval.c_str());
                
    }
    else
    {
        PLERROR("TextFilesVMatrix::TextFilesVMatrix::transformStringToValue, Invalid field type specification for field %s: %s",fieldname.c_str(), fieldtype.c_str());
    }
}

Here is the call graph for this function:

Here is the caller graph for this function:


Member Data Documentation

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 184 of file TextFilesVMatrix.h.

If true, all mappings will be automatically computed at build time if they do not exist yet.

Definition at line 117 of file TextFilesVMatrix.h.

Referenced by declareOptions(), makeDeepCopyFromShallowCopy(), and setMetaDataDir().

If true, new strings for fields of type AUTO will automatically appended to the mapping (in the metadata/mappings/fieldname.map file)

Definition at line 121 of file TextFilesVMatrix.h.

Referenced by autoBuildMappings(), declareOptions(), and getMapping().

If true, standard vmatrix stringmap will be built from the txtmat specific stringmap.

Definition at line 125 of file TextFilesVMatrix.h.

Referenced by declareOptions(), and setMetaDataDir().

list of startpos : width that associate a column range in the resulting VMat to each initial field of the fieldspecs This is built by the setColumnNamesAndWidth method

indicates start position and size in result vector for each field. (as a text field may be transformed into more than one real)

Definition at line 85 of file TextFilesVMatrix.h.

Referenced by buildVMatrixStringMapping(), getNewRow(), setColumnNamesAndWidth(), and transformStringToValue().

If their is no fieldspec, we will use this value reorder_fieldspec_from_headers must be true.

Definition at line 132 of file TextFilesVMatrix.h.

Referenced by build_(), declareOptions(), and setColumnNamesAndWidth().

Delimiter to use to split the fields.

Common delimiters are:

  • '' : used for SAS files (the default)
  • ',' : used for CSV files
  • ';' : used for a variant of CSV files

Definition at line 102 of file TextFilesVMatrix.h.

Referenced by declareOptions(), setColumnNamesAndWidth(), and splitIntoFields().

TVec< pair<string, string> > PLearn::TextFilesVMatrix::fieldspec
TVec<FILE*> PLearn::TextFilesVMatrix::mapfiles [mutable, protected]

Definition at line 169 of file TextFilesVMatrix.h.

Referenced by getMapping(), setMetaDataDir(), and ~TextFilesVMatrix().

TVec< hash_map<string, real> > PLearn::TextFilesVMatrix::mapping [mutable, protected]

a possible string->real mapping for each field We keep our own mapping in addition to the standard VMatrix string mapping because we want to handle the case where different strings map to the same value.

(Standard VMatrix string mapping concept is meant to possibly map several values to the same string, i.e. that's the other way round).

Definition at line 167 of file TextFilesVMatrix.h.

Referenced by autoBuildMappings(), buildVMatrixStringMapping(), generateMapCounts(), getMapping(), loadMappings(), and setMetaDataDir().

The path to the .metadata directory.

Deprecated:
Use metadadir of VMatrix class instead.

Definition at line 93 of file TextFilesVMatrix.h.

Referenced by build_(), and declareOptions().

Definition at line 128 of file TextFilesVMatrix.h.

Referenced by declareOptions(), and setColumnNamesAndWidth().

The escate caractere where the delimiter is not considered.

  • '"' : used frequently

Definition at line 106 of file TextFilesVMatrix.h.

Referenced by declareOptions(), and splitIntoFields().

Definition at line 127 of file TextFilesVMatrix.h.

Referenced by build_(), declareOptions(), and setColumnNamesAndWidth().

An (optional) list of integers, one for each of the txtfilenames indicating the number of header lines at the top of the file to be skipped.

Definition at line 110 of file TextFilesVMatrix.h.

Referenced by buildIdx(), and declareOptions().

A list of paths to raw text files containing the records.

Definition at line 96 of file TextFilesVMatrix.h.

Referenced by declareOptions(), setColumnNamesAndWidth(), and setMetaDataDir().


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines