PLearn 0.1
|
VMat of text files, encoded with Dictionaries, The lines of the text files that are empty are ommited. More...
#include <DictionaryVMatrix.h>
Public Member Functions | |
DictionaryVMatrix () | |
virtual real | getStringVal (int col, const string &str) const |
returns value associated with a string (or MISSING_VALUE if there's no association for this string) | |
virtual string | getValString (int col, real val) const |
Returns the string associated with value val for field# col. | |
virtual void | getValues (int row, int col, Vec &values) const |
Returns the possible values for a certain field in the VMatrix. | |
virtual void | getValues (const Vec &input, int col, Vec &values) const |
Gives the possible values of a certain field (column) given the input. | |
virtual void | build () |
Simply calls inherited::build() then build_(). | |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Transform a shallow copy into a deep copy. | |
virtual PP< Dictionary > | getDictionary (int col) const |
Get Dictionary from a certain column. | |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual DictionaryVMatrix * | deepCopy (CopiesMap &copies) const |
Static Public Member Functions | |
static string | _classname_ () |
Declare name and deepCopy methods. | |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
TVec< PPath > | file_names |
The text input files which are processed with dictionaries. | |
TVec< PP< Dictionary > > | dictionaries |
The dictionaries, one for each attributes. | |
TVec< TVec< int > > | option_fields |
The options fields of every dictionaries. | |
string | delimiters |
String delimiters for input file fields. | |
string | code |
Snippet of python code that processes the text in the input files. | |
TVec< int > | minimum_frequencies |
Minimum frequency for a token to be added in a Dictionary, for the different fields. | |
TVec< TVec< string > > | symbols_to_ignore |
Stop word list, mapped to OOV_TAG. | |
bool | remove_rows_with_oov |
Indication that a row that has a OOV_TAG field should be ignored, i.e. | |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Protected Member Functions | |
virtual void | getNewRow (int i, const Vec &v) const |
Fill the vector 'v' with the content of the i-th row. | |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declares this class' options. | |
Protected Attributes | |
int | n_attributes |
Number of attributes in the input text file (\t separated) | |
PP< PythonCodeSnippet > | python |
Python code snippet. | |
Private Types | |
typedef RowBufferedVMatrix | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. | |
Private Attributes | |
Mat | data |
VMat of text files, encoded with Dictionaries, The lines of the text files that are empty are ommited.
If no Dictionary objects are given by the user, then new Dictionary objects are created and updated from the text files. A Python script can be provided to preprocess each row of all files. The script must define a function called process_string_row(string_row), where string_row is a list of strings corresponding to the symbolic fields of a row in the input files. This function must return a list of processed strings, which will consist of the actual data contained by the VMatrix. Note that process_string_row can return a list that has more or less strings then the input files has fields. The length of the returned list will determine the width of the VMatrix. Here is an example of a Python code that puts the first field to lower case and does nothing to the second:
"def process_string_row(string_row): ret = string_row[:] ret[0] = string_row[0].lower() ret[1] = string_row[1] return ret "
Definition at line 81 of file DictionaryVMatrix.h.
typedef RowBufferedVMatrix PLearn::DictionaryVMatrix::inherited [private] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 87 of file DictionaryVMatrix.h.
PLearn::DictionaryVMatrix::DictionaryVMatrix | ( | ) |
Definition at line 53 of file DictionaryVMatrix.cc.
References data.
:python(), delimiters(" \t"), code(""), remove_rows_with_oov(false) /* ### Initialise all fields to their default value */ { data=0; // build_(); }
string PLearn::DictionaryVMatrix::_classname_ | ( | ) | [static] |
Declare name and deepCopy methods.
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 83 of file DictionaryVMatrix.cc.
OptionList & PLearn::DictionaryVMatrix::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 83 of file DictionaryVMatrix.cc.
RemoteMethodMap & PLearn::DictionaryVMatrix::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 83 of file DictionaryVMatrix.cc.
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 83 of file DictionaryVMatrix.cc.
Object * PLearn::DictionaryVMatrix::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 83 of file DictionaryVMatrix.cc.
StaticInitializer DictionaryVMatrix::_static_initializer_ & PLearn::DictionaryVMatrix::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 83 of file DictionaryVMatrix.cc.
void PLearn::DictionaryVMatrix::build | ( | ) | [virtual] |
Simply calls inherited::build() then build_().
Reimplemented from PLearn::VMatrix.
Definition at line 348 of file DictionaryVMatrix.cc.
References PLearn::VMatrix::build(), and build_().
Referenced by PLearn::VMatDictionaryCommand::run().
{ inherited::build(); build_(); }
void PLearn::DictionaryVMatrix::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::VMatrix.
Definition at line 164 of file DictionaryVMatrix.cc.
References PLearn::TMat< T >::clear(), code, data, delimiters, dictionaries, PLearn::TVec< T >::end(), PLearn::PStream::eof(), file_names, PLearn::TVec< T >::fill(), PLearn::VMatrix::find(), PLearn::PStream::getline(), i, j, PLearn::VMatrix::length(), PLearn::TVec< T >::length(), PLearn::TMat< T >::length(), PLearn::VMatrix::length_, minimum_frequencies, MISSING_VALUE, n_attributes, NO_UPDATE, PLearn::openFile(), option_fields, PLASSERT, PLERROR, python, PLearn::PStream::raw_ascii, remove_rows_with_oov, PLearn::TMat< T >::resize(), PLearn::TVec< T >::resize(), PLearn::PStream::skipBlanks(), PLearn::split(), symbols_to_ignore, UPDATE, PLearn::VMatrix::updateMtime(), and PLearn::VMatrix::width_.
Referenced by build().
{ //if(data.length() != 0) //{ // width_ = data.width(); // length_ = data.length(); // if(option_fields.length()==0) option_fields.resize(width_); // return; //} string line = ""; vector<string> tokens; TVec<string> tokens_vec; TVec< hash_map<string,int> > frequencies; int it=0; int nlines = 0; bool row_has_oov = false; if (!python && code != "") { python = new PythonCodeSnippet(code); PLASSERT( python ); python->build(); } if(data.length()!=0) data.clear(); length_ = 0; // Prepocessing of the data... for(int k=0; k<file_names.length(); k++) { PPath input_file = file_names[k]; PStream input_stream = openFile(input_file, PStream::raw_ascii); updateMtime(input_file); input_stream.skipBlanks(); while (!input_stream.eof()){ input_stream.getline(line); input_stream.skipBlanks(); tokens = split(line, delimiters); if(python) { tokens_vec.resize(int(tokens.size())); for(int i=0; i<tokens_vec.length(); i++) tokens_vec[i] = tokens[i]; tokens_vec = python->invoke("process_string_row",tokens_vec).as<TVec<string> >(); tokens.resize(tokens_vec.length()); for(int i=0; i<tokens_vec.length(); i++) tokens[i] = tokens_vec[i]; } // Set n_attributes if(it==0) { it++; n_attributes = int(tokens.size()); if(minimum_frequencies.length() != 0 && minimum_frequencies.length() != n_attributes) PLERROR("In DictionaryVMatrix::build_(): number of attributes (%d) and size of minimum_frequencies (%d) is different", n_attributes, minimum_frequencies.length()); if(minimum_frequencies.length() == 0) { minimum_frequencies.resize(n_attributes); minimum_frequencies.fill(-1); } // If no dictionaries are specified, then create some if(dictionaries.length() == 0) { dictionaries.resize(n_attributes); for(int i=0; i<n_attributes; i++) { dictionaries[i] = new Dictionary(); dictionaries[i]->update_mode = UPDATE; dictionaries[i]->build(); } } if(dictionaries.length() != n_attributes) PLERROR("In DictionaryVMatrix::build_(): number of attributes (%d) and number of dictionaries (%d) is different", n_attributes, dictionaries.length()); if(option_fields.length()==0) option_fields.resize(n_attributes); frequencies.resize(n_attributes); } if(symbols_to_ignore.length() != 0) for(int i=0; i<int(tokens.size()); i++) if(symbols_to_ignore[i].find(tokens[i]) >= 0) tokens[i] = dictionaries[i]->oov_symbol; row_has_oov = false; for(int i=0; i<int(tokens.size()); i++) if(tokens[i] == dictionaries[i]->oov_symbol) row_has_oov = true; // Count frequencies... for(int j=0; j<n_attributes; j++) { if(tokens[j] != "nan") // Detect missing values { if(frequencies[j].find(tokens[j]) == frequencies[j].end()) frequencies[j][tokens[j]] = 1; else frequencies[j][tokens[j]]++; } } if(!remove_rows_with_oov || !row_has_oov) length_++; } } it = 0; for(int k=0; k<file_names.length(); k++) { nlines = it; PPath input_file = file_names[k]; PStream input_stream = openFile(input_file, PStream::raw_ascii); input_stream.skipBlanks(); while (!input_stream.eof()){ if(it==0) data.resize(length_,n_attributes); input_stream.getline(line); input_stream.skipBlanks(); tokens = split(line, delimiters); if(python) { tokens_vec.resize(int(tokens.size())); for(int i=0; i<tokens_vec.length(); i++) tokens_vec[i] = tokens[i]; tokens_vec = python->invoke("process_string_row",tokens_vec).as<TVec<string> >(); tokens.resize(tokens_vec.length()); for(int i=0; i<tokens_vec.length(); i++) tokens[i] = tokens_vec[i]; } if(symbols_to_ignore.length() != 0) for(int i=0; i<int(tokens.size()); i++) if(symbols_to_ignore[i].find(tokens[i]) >= 0) tokens[i] = dictionaries[i]->oov_symbol; /* for(int i=0; i<to_lower_case.size(); i++) tokens[to_lower_case[i]] = lowerstring(tokens[to_lower_case[i]]); */ if((int)tokens.size() != n_attributes) PLERROR("In DictionaryVMatrix::build_(): line %d (\"%s\") of file %s doesn't have %d attributes", it-nlines+1, line.c_str(), input_file.c_str(), n_attributes); // Insert symbols in dictionaries (if they can be updated) for(int j=0; j<n_attributes; j++) { if(tokens[j] == "nan") // Detect missing values data(it,j) = MISSING_VALUE; else { if(frequencies[j][tokens[j]] >= minimum_frequencies[j]) { TVec<string> options(option_fields[j].length()); for(int k_it=0; k_it<options.length(); k_it++) options[k_it] = tokens[option_fields[j][k_it]]; data(it,j) = dictionaries[j]->getId(tokens[j],options); } else data(it,j) = dictionaries[j]->getId(dictionaries[j]->oov_symbol); } } row_has_oov = false; for(int j=0; j<n_attributes; j++) if(data(it,j) == dictionaries[j]->getId(dictionaries[j]->oov_symbol)) row_has_oov = true; if(!remove_rows_with_oov || !row_has_oov) it++; } } width_ = n_attributes; data.resize(it,n_attributes); length_ = it; // Making sure that the dictionaries cannot be // updated anymore, since they should contain // all the needed information about the data for(int i=0; i<dictionaries.length(); i++) dictionaries[i]->setUpdateMode(NO_UPDATE); }
string PLearn::DictionaryVMatrix::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 83 of file DictionaryVMatrix.cc.
void PLearn::DictionaryVMatrix::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declares this class' options.
Reimplemented from PLearn::VMatrix.
Definition at line 134 of file DictionaryVMatrix.cc.
References PLearn::OptionBase::buildoption, code, data, PLearn::declareOption(), PLearn::VMatrix::declareOptions(), delimiters, dictionaries, file_names, PLearn::OptionBase::learntoption, minimum_frequencies, option_fields, remove_rows_with_oov, and symbols_to_ignore.
{ declareOption(ol, "file_names", &DictionaryVMatrix::file_names, OptionBase::buildoption, "The text files from which we create the VMat"); declareOption(ol, "dictionaries", &DictionaryVMatrix::dictionaries, OptionBase::buildoption, "Vector of dictionaries\n"); declareOption(ol, "option_fields", &DictionaryVMatrix::option_fields, OptionBase::buildoption, "Vector of the fields corresponding to the options of the Dictionary, for every Dictionary\n"); declareOption(ol, "data", &DictionaryVMatrix::data, OptionBase::buildoption, "Encoded Matrix\n"); declareOption(ol, "delimiters", &DictionaryVMatrix::delimiters, OptionBase::buildoption, "Delimiters for file fields (or attributs)\n"); declareOption(ol, "code", &DictionaryVMatrix::code, OptionBase::buildoption, "Snippet of python code that processes the text in the input files\n"); declareOption(ol, "minimum_frequencies", &DictionaryVMatrix::minimum_frequencies, OptionBase::buildoption, "Minimum frequency for a token to be added in a Dictionary, for the different fields\n"); declareOption(ol, "symbols_to_ignore", &DictionaryVMatrix::symbols_to_ignore, OptionBase::buildoption, "Symbols that are ignored, i.e. considered as \"<oov>\" a priori when reading\n" "the text files. In NLP, this is usually called a stop word list.\n"); declareOption(ol, "remove_rows_with_oov", &DictionaryVMatrix::remove_rows_with_oov, OptionBase::buildoption, "Indication that a row that has a OOV symbol field should be ignored, i.e. removed\n" "of the VMatrix.\n"); declareOption(ol, "data", &DictionaryVMatrix::data, OptionBase::learntoption, "Matrix containing the concatenated and encoded text files\n"); // Now call the parent class' declareOptions inherited::declareOptions(ol); }
static const PPath& PLearn::DictionaryVMatrix::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 182 of file DictionaryVMatrix.h.
DictionaryVMatrix * PLearn::DictionaryVMatrix::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 83 of file DictionaryVMatrix.cc.
PP< Dictionary > PLearn::DictionaryVMatrix::getDictionary | ( | int | col | ) | const [virtual] |
Get Dictionary from a certain column.
Reimplemented from PLearn::VMatrix.
Definition at line 103 of file DictionaryVMatrix.cc.
References dictionaries, PLERROR, and PLearn::VMatrix::width_.
{ if(col < 0 || col >= width_) PLERROR("In DictionaryVMatrix::getDictionary() : invalid col %d, width()=%d", col, width_); return dictionaries[col]; }
Fill the vector 'v' with the content of the i-th row.
v is assumed to be the right size.
Implements PLearn::RowBufferedVMatrix.
Definition at line 85 of file DictionaryVMatrix.cc.
References data.
{ v << data(i); }
OptionList & PLearn::DictionaryVMatrix::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 83 of file DictionaryVMatrix.cc.
OptionMap & PLearn::DictionaryVMatrix::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 83 of file DictionaryVMatrix.cc.
RemoteMethodMap & PLearn::DictionaryVMatrix::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 83 of file DictionaryVMatrix.cc.
returns value associated with a string (or MISSING_VALUE if there's no association for this string)
Reimplemented from PLearn::VMatrix.
Definition at line 90 of file DictionaryVMatrix.cc.
References dictionaries, and MISSING_VALUE.
{ int ret = dictionaries[col]->getId(str); if(ret == -1) return MISSING_VALUE; else return ret; }
Returns the string associated with value val for field# col.
Or returns "" if no string is associated.
Reimplemented from PLearn::VMatrix.
Definition at line 97 of file DictionaryVMatrix.cc.
References dictionaries, PLearn::is_missing(), and PLearn::tostring().
{ if(is_missing(val))return tostring(val); return dictionaries[col]->getSymbol((int)val); }
Returns the possible values for a certain field in the VMatrix.
For example, if "col" corresponds to the target column, this function could fill "values" with the class indices of the possible target classes for the example at row "row". The default getValues(...) function gives an empty "values"
Reimplemented from PLearn::VMatrix.
Definition at line 110 of file DictionaryVMatrix.cc.
References data, dictionaries, i, PLearn::TVec< T >::length(), PLearn::VMatrix::length(), PLearn::VMatrix::length_, option_fields, PLERROR, and PLearn::VMatrix::width_.
{ if(row < 0 || row >= length_) PLERROR("In DictionaryVMatrix::getValues() : invalid row %d, length()=%d", row, length_); if(col < 0 || col >= width_) PLERROR("In DictionaryVMatrix::getValues() : invalid col %d, width()=%d", col, width_); TVec<string> options(option_fields[col].length()); for(int i=0; i<options.length(); i++) { options[i] = dictionaries[option_fields[col][i]]->getSymbol((int)data(row,option_fields[col][i])); } dictionaries[col]->getValues(options, values); }
void PLearn::DictionaryVMatrix::getValues | ( | const Vec & | input, |
int | col, | ||
Vec & | values | ||
) | const [virtual] |
Gives the possible values of a certain field (column) given the input.
Reimplemented from PLearn::VMatrix.
Definition at line 122 of file DictionaryVMatrix.cc.
References dictionaries, i, PLearn::TVec< T >::length(), PLearn::VMatrix::length(), option_fields, PLERROR, and PLearn::VMatrix::width_.
{ if(col < 0 || col >= width_) PLERROR("In DictionaryVMatrix::getValues() : invalid col %d, width()=%d", col, width_); TVec<string> options(option_fields[col].length()); for(int i=0; i<options.length(); i++) { options[i] = dictionaries[option_fields[col][i]]->getSymbol((int)input[option_fields[col][i]]); } dictionaries[col]->getValues(options, values); }
void PLearn::DictionaryVMatrix::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transform a shallow copy into a deep copy.
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 354 of file DictionaryVMatrix.cc.
References data, PLearn::deepCopyField(), dictionaries, file_names, PLearn::RowBufferedVMatrix::makeDeepCopyFromShallowCopy(), minimum_frequencies, option_fields, python, and symbols_to_ignore.
{ inherited::makeDeepCopyFromShallowCopy(copies); deepCopyField(data, copies); deepCopyField(file_names, copies); deepCopyField(dictionaries, copies); deepCopyField(option_fields, copies); deepCopyField(minimum_frequencies, copies); deepCopyField(symbols_to_ignore, copies); deepCopyField(python, copies); }
Reimplemented from PLearn::RowBufferedVMatrix.
Definition at line 182 of file DictionaryVMatrix.h.
Snippet of python code that processes the text in the input files.
Definition at line 122 of file DictionaryVMatrix.h.
Referenced by build_(), and declareOptions().
Mat PLearn::DictionaryVMatrix::data [private] |
Definition at line 89 of file DictionaryVMatrix.h.
Referenced by build_(), declareOptions(), DictionaryVMatrix(), getNewRow(), getValues(), and makeDeepCopyFromShallowCopy().
String delimiters for input file fields.
Definition at line 119 of file DictionaryVMatrix.h.
Referenced by build_(), declareOptions(), and PLearn::VMatDictionaryCommand::run().
The dictionaries, one for each attributes.
Definition at line 113 of file DictionaryVMatrix.h.
Referenced by build_(), declareOptions(), getDictionary(), getStringVal(), getValString(), getValues(), and makeDeepCopyFromShallowCopy().
The text input files which are processed with dictionaries.
Definition at line 110 of file DictionaryVMatrix.h.
Referenced by build_(), declareOptions(), makeDeepCopyFromShallowCopy(), and PLearn::VMatDictionaryCommand::run().
Minimum frequency for a token to be added in a Dictionary, for the different fields.
Definition at line 126 of file DictionaryVMatrix.h.
Referenced by build_(), declareOptions(), and makeDeepCopyFromShallowCopy().
int PLearn::DictionaryVMatrix::n_attributes [protected] |
Number of attributes in the input text file (\t separated)
Definition at line 98 of file DictionaryVMatrix.h.
Referenced by build_().
The options fields of every dictionaries.
Definition at line 116 of file DictionaryVMatrix.h.
Referenced by build_(), declareOptions(), getValues(), and makeDeepCopyFromShallowCopy().
PP<PythonCodeSnippet> PLearn::DictionaryVMatrix::python [protected] |
Python code snippet.
Definition at line 101 of file DictionaryVMatrix.h.
Referenced by build_(), and makeDeepCopyFromShallowCopy().
Indication that a row that has a OOV_TAG field should be ignored, i.e.
removed of the VMatrix
Definition at line 133 of file DictionaryVMatrix.h.
Referenced by build_(), and declareOptions().
TVec< TVec<string> > PLearn::DictionaryVMatrix::symbols_to_ignore |
Stop word list, mapped to OOV_TAG.
Definition at line 129 of file DictionaryVMatrix.h.
Referenced by build_(), declareOptions(), and makeDeepCopyFromShallowCopy().