PLearn 0.1
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Protected Member Functions | Static Protected Member Functions | Protected Attributes | Private Types | Private Member Functions | Private Attributes
PLearn::DictionaryVMatrix Class Reference

VMat of text files, encoded with Dictionaries, The lines of the text files that are empty are ommited. More...

#include <DictionaryVMatrix.h>

Inheritance diagram for PLearn::DictionaryVMatrix:
Inheritance graph
[legend]
Collaboration diagram for PLearn::DictionaryVMatrix:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 DictionaryVMatrix ()
virtual real getStringVal (int col, const string &str) const
 returns value associated with a string (or MISSING_VALUE if there's no association for this string)
virtual string getValString (int col, real val) const
 Returns the string associated with value val for field# col.
virtual void getValues (int row, int col, Vec &values) const
 Returns the possible values for a certain field in the VMatrix.
virtual void getValues (const Vec &input, int col, Vec &values) const
 Gives the possible values of a certain field (column) given the input.
virtual void build ()
 Simply calls inherited::build() then build_().
virtual void makeDeepCopyFromShallowCopy (CopiesMap &copies)
 Transform a shallow copy into a deep copy.
virtual PP< DictionarygetDictionary (int col) const
 Get Dictionary from a certain column.
virtual string classname () const
virtual OptionListgetOptionList () const
virtual OptionMapgetOptionMap () const
virtual RemoteMethodMapgetRemoteMethodMap () const
virtual DictionaryVMatrixdeepCopy (CopiesMap &copies) const

Static Public Member Functions

static string _classname_ ()
 Declare name and deepCopy methods.
static OptionList_getOptionList_ ()
static RemoteMethodMap_getRemoteMethodMap_ ()
static Object_new_instance_for_typemap_ ()
static bool _isa_ (const Object *o)
static void _static_initialize_ ()
static const PPathdeclaringFile ()

Public Attributes

TVec< PPathfile_names
 The text input files which are processed with dictionaries.
TVec< PP< Dictionary > > dictionaries
 The dictionaries, one for each attributes.
TVec< TVec< int > > option_fields
 The options fields of every dictionaries.
string delimiters
 String delimiters for input file fields.
string code
 Snippet of python code that processes the text in the input files.
TVec< intminimum_frequencies
 Minimum frequency for a token to be added in a Dictionary, for the different fields.
TVec< TVec< string > > symbols_to_ignore
 Stop word list, mapped to OOV_TAG.
bool remove_rows_with_oov
 Indication that a row that has a OOV_TAG field should be ignored, i.e.

Static Public Attributes

static StaticInitializer _static_initializer_

Protected Member Functions

virtual void getNewRow (int i, const Vec &v) const
 Fill the vector 'v' with the content of the i-th row.

Static Protected Member Functions

static void declareOptions (OptionList &ol)
 Declares this class' options.

Protected Attributes

int n_attributes
 Number of attributes in the input text file (\t separated)
PP< PythonCodeSnippetpython
 Python code snippet.

Private Types

typedef RowBufferedVMatrix inherited

Private Member Functions

void build_ ()
 This does the actual building.

Private Attributes

Mat data

Detailed Description

VMat of text files, encoded with Dictionaries, The lines of the text files that are empty are ommited.

If no Dictionary objects are given by the user, then new Dictionary objects are created and updated from the text files. A Python script can be provided to preprocess each row of all files. The script must define a function called process_string_row(string_row), where string_row is a list of strings corresponding to the symbolic fields of a row in the input files. This function must return a list of processed strings, which will consist of the actual data contained by the VMatrix. Note that process_string_row can return a list that has more or less strings then the input files has fields. The length of the returned list will determine the width of the VMatrix. Here is an example of a Python code that puts the first field to lower case and does nothing to the second:

"def process_string_row(string_row): ret = string_row[:] ret[0] = string_row[0].lower() ret[1] = string_row[1] return ret "

Definition at line 81 of file DictionaryVMatrix.h.


Member Typedef Documentation

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 87 of file DictionaryVMatrix.h.


Constructor & Destructor Documentation

PLearn::DictionaryVMatrix::DictionaryVMatrix ( )

Definition at line 53 of file DictionaryVMatrix.cc.

References data.

    :python(), delimiters(" \t"), code(""), remove_rows_with_oov(false)
    /* ### Initialise all fields to their default value */
{
    data=0;
    // build_();
}

Member Function Documentation

string PLearn::DictionaryVMatrix::_classname_ ( ) [static]

Declare name and deepCopy methods.

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 83 of file DictionaryVMatrix.cc.

OptionList & PLearn::DictionaryVMatrix::_getOptionList_ ( ) [static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 83 of file DictionaryVMatrix.cc.

RemoteMethodMap & PLearn::DictionaryVMatrix::_getRemoteMethodMap_ ( ) [static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 83 of file DictionaryVMatrix.cc.

bool PLearn::DictionaryVMatrix::_isa_ ( const Object o) [static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 83 of file DictionaryVMatrix.cc.

Object * PLearn::DictionaryVMatrix::_new_instance_for_typemap_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 83 of file DictionaryVMatrix.cc.

StaticInitializer DictionaryVMatrix::_static_initializer_ & PLearn::DictionaryVMatrix::_static_initialize_ ( ) [static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 83 of file DictionaryVMatrix.cc.

void PLearn::DictionaryVMatrix::build ( ) [virtual]

Simply calls inherited::build() then build_().

Reimplemented from PLearn::VMatrix.

Definition at line 348 of file DictionaryVMatrix.cc.

References PLearn::VMatrix::build(), and build_().

Referenced by PLearn::VMatDictionaryCommand::run().

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::DictionaryVMatrix::build_ ( ) [private]

This does the actual building.

Reimplemented from PLearn::VMatrix.

Definition at line 164 of file DictionaryVMatrix.cc.

References PLearn::TMat< T >::clear(), code, data, delimiters, dictionaries, PLearn::TVec< T >::end(), PLearn::PStream::eof(), file_names, PLearn::TVec< T >::fill(), PLearn::VMatrix::find(), PLearn::PStream::getline(), i, j, PLearn::VMatrix::length(), PLearn::TVec< T >::length(), PLearn::TMat< T >::length(), PLearn::VMatrix::length_, minimum_frequencies, MISSING_VALUE, n_attributes, NO_UPDATE, PLearn::openFile(), option_fields, PLASSERT, PLERROR, python, PLearn::PStream::raw_ascii, remove_rows_with_oov, PLearn::TMat< T >::resize(), PLearn::TVec< T >::resize(), PLearn::PStream::skipBlanks(), PLearn::split(), symbols_to_ignore, UPDATE, PLearn::VMatrix::updateMtime(), and PLearn::VMatrix::width_.

Referenced by build().

{
    //if(data.length() != 0)
    //{
    //    width_ = data.width();
    //    length_ = data.length();
    //    if(option_fields.length()==0) option_fields.resize(width_);
    //    return;
    //}
    
    string line = "";
    vector<string> tokens;
    TVec<string> tokens_vec;
    TVec< hash_map<string,int> > frequencies;
    int it=0;
    int nlines = 0;
    bool row_has_oov = false;

    if (!python && code != "")
    {
        python = new PythonCodeSnippet(code);
        PLASSERT( python );
        python->build();
    }

    if(data.length()!=0) data.clear();

    length_ = 0;

    // Prepocessing of the data...
    for(int k=0; k<file_names.length(); k++)
    {
        PPath input_file = file_names[k];
        PStream input_stream = openFile(input_file, PStream::raw_ascii);
        updateMtime(input_file);
        input_stream.skipBlanks();
        while (!input_stream.eof()){
            input_stream.getline(line);
            input_stream.skipBlanks();
            tokens = split(line, delimiters);
            if(python)
            {
                tokens_vec.resize(int(tokens.size()));
                for(int i=0; i<tokens_vec.length(); i++)
                    tokens_vec[i] = tokens[i];
                tokens_vec = python->invoke("process_string_row",tokens_vec).as<TVec<string> >();
                tokens.resize(tokens_vec.length());
                for(int i=0; i<tokens_vec.length(); i++)
                    tokens[i] = tokens_vec[i];
            }

            // Set n_attributes
            if(it==0)
            {
                it++;
                n_attributes = int(tokens.size());
                if(minimum_frequencies.length() != 0 && minimum_frequencies.length() != n_attributes)
                    PLERROR("In DictionaryVMatrix::build_(): number of attributes (%d) and size of minimum_frequencies (%d) is different", n_attributes, minimum_frequencies.length());

                if(minimum_frequencies.length() == 0)
                {
                    minimum_frequencies.resize(n_attributes);
                    minimum_frequencies.fill(-1);
                }

                // If no dictionaries are specified, then create some
                if(dictionaries.length() == 0)
                {
                    dictionaries.resize(n_attributes);
                    for(int i=0; i<n_attributes; i++)
                    {
                        dictionaries[i] = new Dictionary();
                        dictionaries[i]->update_mode = UPDATE;
                        dictionaries[i]->build();
                    }

                }
                if(dictionaries.length() != n_attributes)
                    PLERROR("In DictionaryVMatrix::build_(): number of attributes (%d) and number of dictionaries (%d) is different", n_attributes, dictionaries.length());
                if(option_fields.length()==0) option_fields.resize(n_attributes);
                frequencies.resize(n_attributes);
            }

            if(symbols_to_ignore.length() != 0)
                for(int i=0; i<int(tokens.size()); i++)
                    if(symbols_to_ignore[i].find(tokens[i]) >= 0)
                        tokens[i] = dictionaries[i]->oov_symbol;

            row_has_oov = false;
            for(int i=0; i<int(tokens.size()); i++)
                if(tokens[i] == dictionaries[i]->oov_symbol)
                    row_has_oov = true;

            // Count frequencies...
            for(int j=0; j<n_attributes; j++)
            {
                if(tokens[j] != "nan") // Detect missing values
                {
                    if(frequencies[j].find(tokens[j]) == frequencies[j].end())
                        frequencies[j][tokens[j]] = 1;
                    else
                        frequencies[j][tokens[j]]++;
                }
            }

            if(!remove_rows_with_oov || !row_has_oov) length_++;
        }
    }

    it = 0;
    for(int k=0; k<file_names.length(); k++)
    {
        nlines = it;
        PPath input_file = file_names[k];
        PStream input_stream = openFile(input_file, PStream::raw_ascii);
        input_stream.skipBlanks();
        while (!input_stream.eof()){
            if(it==0) data.resize(length_,n_attributes);
            input_stream.getline(line);
            input_stream.skipBlanks();
            tokens = split(line, delimiters);

            if(python)
            {
                tokens_vec.resize(int(tokens.size()));
                for(int i=0; i<tokens_vec.length(); i++)
                    tokens_vec[i] = tokens[i];
                tokens_vec = python->invoke("process_string_row",tokens_vec).as<TVec<string> >();
                tokens.resize(tokens_vec.length());
                for(int i=0; i<tokens_vec.length(); i++)
                    tokens[i] = tokens_vec[i];
            }

            if(symbols_to_ignore.length() != 0)
                for(int i=0; i<int(tokens.size()); i++)
                    if(symbols_to_ignore[i].find(tokens[i]) >= 0)
                        tokens[i] = dictionaries[i]->oov_symbol;

            /*
            for(int i=0; i<to_lower_case.size(); i++)
                tokens[to_lower_case[i]] = lowerstring(tokens[to_lower_case[i]]);
            */

            if((int)tokens.size() != n_attributes)
                PLERROR("In DictionaryVMatrix::build_(): line %d (\"%s\") of file %s doesn't have %d attributes", it-nlines+1, line.c_str(), input_file.c_str(), n_attributes);

            // Insert symbols in dictionaries (if they can be updated)
            for(int j=0; j<n_attributes; j++)
            {
                if(tokens[j] == "nan") // Detect missing values
                    data(it,j) = MISSING_VALUE;
                else
                {
                    if(frequencies[j][tokens[j]] >= minimum_frequencies[j])
                    {
                        TVec<string> options(option_fields[j].length());
                        for(int k_it=0; k_it<options.length(); k_it++)
                            options[k_it] = tokens[option_fields[j][k_it]];
                        data(it,j) = dictionaries[j]->getId(tokens[j],options);
                    }
                    else
                        data(it,j) = dictionaries[j]->getId(dictionaries[j]->oov_symbol);
                }
            }
            row_has_oov = false;
            for(int j=0; j<n_attributes; j++)
                if(data(it,j) == dictionaries[j]->getId(dictionaries[j]->oov_symbol))
                    row_has_oov = true;

            if(!remove_rows_with_oov || !row_has_oov) it++;
        }
    }
    width_ = n_attributes;
    data.resize(it,n_attributes);
    length_ = it;

    // Making sure that the dictionaries cannot be
    // updated anymore, since they should contain
    // all the needed information about the data
    for(int i=0; i<dictionaries.length(); i++)
        dictionaries[i]->setUpdateMode(NO_UPDATE);
}

Here is the call graph for this function:

Here is the caller graph for this function:

string PLearn::DictionaryVMatrix::classname ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 83 of file DictionaryVMatrix.cc.

void PLearn::DictionaryVMatrix::declareOptions ( OptionList ol) [static, protected]

Declares this class' options.

Reimplemented from PLearn::VMatrix.

Definition at line 134 of file DictionaryVMatrix.cc.

References PLearn::OptionBase::buildoption, code, data, PLearn::declareOption(), PLearn::VMatrix::declareOptions(), delimiters, dictionaries, file_names, PLearn::OptionBase::learntoption, minimum_frequencies, option_fields, remove_rows_with_oov, and symbols_to_ignore.

{
    declareOption(ol, "file_names", &DictionaryVMatrix::file_names, OptionBase::buildoption,
                  "The text files from which we create the VMat");
    declareOption(ol, "dictionaries", &DictionaryVMatrix::dictionaries, OptionBase::buildoption,
                  "Vector of dictionaries\n");
    declareOption(ol, "option_fields", &DictionaryVMatrix::option_fields, OptionBase::buildoption,
                  "Vector of the fields corresponding to the options of the Dictionary, for every Dictionary\n");
    declareOption(ol, "data", &DictionaryVMatrix::data, OptionBase::buildoption,
                  "Encoded Matrix\n");
    declareOption(ol, "delimiters", &DictionaryVMatrix::delimiters, OptionBase::buildoption,
                  "Delimiters for file fields (or attributs)\n");
    declareOption(ol, "code", &DictionaryVMatrix::code, OptionBase::buildoption,
                  "Snippet of python code that processes the text in the input files\n");
    declareOption(ol, "minimum_frequencies", &DictionaryVMatrix::minimum_frequencies, OptionBase::buildoption,
                  "Minimum frequency for a token to be added in a Dictionary, for the different fields\n");
    declareOption(ol, "symbols_to_ignore", &DictionaryVMatrix::symbols_to_ignore, OptionBase::buildoption,
                  "Symbols that are ignored, i.e. considered as \"<oov>\" a priori when reading\n"
                  "the text files. In NLP, this is usually called a stop word list.\n");
    declareOption(ol, "remove_rows_with_oov", &DictionaryVMatrix::remove_rows_with_oov, OptionBase::buildoption,
                  "Indication that a row that has a OOV symbol field should be ignored, i.e. removed\n"
                  "of the VMatrix.\n");
    declareOption(ol, "data", &DictionaryVMatrix::data, OptionBase::learntoption,
                  "Matrix containing the concatenated and encoded text files\n");


    // Now call the parent class' declareOptions
    inherited::declareOptions(ol);
}

Here is the call graph for this function:

static const PPath& PLearn::DictionaryVMatrix::declaringFile ( ) [inline, static]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 182 of file DictionaryVMatrix.h.

DictionaryVMatrix * PLearn::DictionaryVMatrix::deepCopy ( CopiesMap copies) const [virtual]

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 83 of file DictionaryVMatrix.cc.

PP< Dictionary > PLearn::DictionaryVMatrix::getDictionary ( int  col) const [virtual]

Get Dictionary from a certain column.

Reimplemented from PLearn::VMatrix.

Definition at line 103 of file DictionaryVMatrix.cc.

References dictionaries, PLERROR, and PLearn::VMatrix::width_.

{
    if(col < 0 || col >= width_) PLERROR("In DictionaryVMatrix::getDictionary() : invalid col %d, width()=%d", col, width_);
    return  dictionaries[col];
}
void PLearn::DictionaryVMatrix::getNewRow ( int  i,
const Vec v 
) const [protected, virtual]

Fill the vector 'v' with the content of the i-th row.

v is assumed to be the right size.

Implements PLearn::RowBufferedVMatrix.

Definition at line 85 of file DictionaryVMatrix.cc.

References data.

{
    v << data(i);
}
OptionList & PLearn::DictionaryVMatrix::getOptionList ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 83 of file DictionaryVMatrix.cc.

OptionMap & PLearn::DictionaryVMatrix::getOptionMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 83 of file DictionaryVMatrix.cc.

RemoteMethodMap & PLearn::DictionaryVMatrix::getRemoteMethodMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 83 of file DictionaryVMatrix.cc.

real PLearn::DictionaryVMatrix::getStringVal ( int  col,
const string &  str 
) const [virtual]

returns value associated with a string (or MISSING_VALUE if there's no association for this string)

Reimplemented from PLearn::VMatrix.

Definition at line 90 of file DictionaryVMatrix.cc.

References dictionaries, and MISSING_VALUE.

{
    int ret = dictionaries[col]->getId(str);
    if(ret == -1) return MISSING_VALUE;
    else return ret;
}
string PLearn::DictionaryVMatrix::getValString ( int  col,
real  val 
) const [virtual]

Returns the string associated with value val for field# col.

Or returns "" if no string is associated.

Reimplemented from PLearn::VMatrix.

Definition at line 97 of file DictionaryVMatrix.cc.

References dictionaries, PLearn::is_missing(), and PLearn::tostring().

{
    if(is_missing(val))return tostring(val);
    return dictionaries[col]->getSymbol((int)val);
}

Here is the call graph for this function:

void PLearn::DictionaryVMatrix::getValues ( int  row,
int  col,
Vec values 
) const [virtual]

Returns the possible values for a certain field in the VMatrix.

For example, if "col" corresponds to the target column, this function could fill "values" with the class indices of the possible target classes for the example at row "row". The default getValues(...) function gives an empty "values"

Reimplemented from PLearn::VMatrix.

Definition at line 110 of file DictionaryVMatrix.cc.

References data, dictionaries, i, PLearn::TVec< T >::length(), PLearn::VMatrix::length(), PLearn::VMatrix::length_, option_fields, PLERROR, and PLearn::VMatrix::width_.

{
    if(row < 0 || row >= length_) PLERROR("In DictionaryVMatrix::getValues() : invalid row %d, length()=%d", row, length_);
    if(col < 0 || col >= width_) PLERROR("In DictionaryVMatrix::getValues() : invalid col %d, width()=%d", col, width_);
    TVec<string> options(option_fields[col].length());
    for(int i=0; i<options.length(); i++)
    {
        options[i] = dictionaries[option_fields[col][i]]->getSymbol((int)data(row,option_fields[col][i]));
    }
    dictionaries[col]->getValues(options, values);
}

Here is the call graph for this function:

void PLearn::DictionaryVMatrix::getValues ( const Vec input,
int  col,
Vec values 
) const [virtual]

Gives the possible values of a certain field (column) given the input.

Reimplemented from PLearn::VMatrix.

Definition at line 122 of file DictionaryVMatrix.cc.

References dictionaries, i, PLearn::TVec< T >::length(), PLearn::VMatrix::length(), option_fields, PLERROR, and PLearn::VMatrix::width_.

{
    if(col < 0 || col >= width_) PLERROR("In DictionaryVMatrix::getValues() : invalid col %d, width()=%d", col, width_);
    TVec<string> options(option_fields[col].length());
    for(int i=0; i<options.length(); i++)
    {
        options[i] = dictionaries[option_fields[col][i]]->getSymbol((int)input[option_fields[col][i]]);
    }
    dictionaries[col]->getValues(options, values);
}

Here is the call graph for this function:

void PLearn::DictionaryVMatrix::makeDeepCopyFromShallowCopy ( CopiesMap copies) [virtual]

Member Data Documentation

Reimplemented from PLearn::RowBufferedVMatrix.

Definition at line 182 of file DictionaryVMatrix.h.

Snippet of python code that processes the text in the input files.

Definition at line 122 of file DictionaryVMatrix.h.

Referenced by build_(), and declareOptions().

String delimiters for input file fields.

Definition at line 119 of file DictionaryVMatrix.h.

Referenced by build_(), declareOptions(), and PLearn::VMatDictionaryCommand::run().

The dictionaries, one for each attributes.

Definition at line 113 of file DictionaryVMatrix.h.

Referenced by build_(), declareOptions(), getDictionary(), getStringVal(), getValString(), getValues(), and makeDeepCopyFromShallowCopy().

The text input files which are processed with dictionaries.

Definition at line 110 of file DictionaryVMatrix.h.

Referenced by build_(), declareOptions(), makeDeepCopyFromShallowCopy(), and PLearn::VMatDictionaryCommand::run().

Minimum frequency for a token to be added in a Dictionary, for the different fields.

Definition at line 126 of file DictionaryVMatrix.h.

Referenced by build_(), declareOptions(), and makeDeepCopyFromShallowCopy().

Number of attributes in the input text file (\t separated)

Definition at line 98 of file DictionaryVMatrix.h.

Referenced by build_().

The options fields of every dictionaries.

Definition at line 116 of file DictionaryVMatrix.h.

Referenced by build_(), declareOptions(), getValues(), and makeDeepCopyFromShallowCopy().

Python code snippet.

Definition at line 101 of file DictionaryVMatrix.h.

Referenced by build_(), and makeDeepCopyFromShallowCopy().

Indication that a row that has a OOV_TAG field should be ignored, i.e.

removed of the VMatrix

Definition at line 133 of file DictionaryVMatrix.h.

Referenced by build_(), and declareOptions().

Stop word list, mapped to OOV_TAG.

Definition at line 129 of file DictionaryVMatrix.h.

Referenced by build_(), declareOptions(), and makeDeepCopyFromShallowCopy().


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines