PLearn 0.1
|
#include <Dictionary.h>
Public Member Functions | |
Dictionary () | |
Default constructor. | |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual Dictionary * | deepCopy (CopiesMap &copies) const |
virtual void | setUpdateMode (int up_mode) |
Set update dictionary mode : UPDATE/NO_UPDATE. | |
virtual int | getId (string symbol, TVec< string > options=TVec< string >(0)) |
Gives the ID of a symbol in the dictionary If the symbol is not in the dictionary, returns the index of oov token if update_mode = NO_UPDATE. | |
virtual string | getSymbol (int id, TVec< string > options=TVec< string >(0)) const |
Gives the symbol from an id of the dictionary If the id is invalid, the string returned is "" Options can be specified ... | |
virtual int | size (TVec< string > options=TVec< string >(0)) |
Get size of the dictionary (number of symbols in the dictionary) Options can be specified to restrict the number of possible values. | |
virtual void | getValues (TVec< string > options, Vec &values) |
Fills a Vec containing every possible ID values of the Dictionary Options can be specified to restrict the number of possible values. | |
virtual bool | isIn (string symbol, TVec< string > options=TVec< string >(0)) |
Indicates if a symbol is in the dictionary The OOV symbol is considered as in the dictionary if dont_insert_oov_symbol is false. | |
virtual bool | isIn (int id, TVec< string > options=TVec< string >(0)) |
Indicates if an id is in the dictionary The OOV symbol's ID is considered as in the dictionary if dont_insert_oov_symbol is false. | |
virtual void | clear () |
virtual void | build () |
Post-constructor. | |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Transforms a shallow copy into a deep copy. | |
Static Public Member Functions | |
static string | _classname_ () |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
int | update_mode |
update mode update/no update | |
bool | dont_insert_oov_symbol |
Indication that "oov" should not be considered as part of the dictionary. | |
string | oov_symbol |
Symbol for "out-of-vocabulary" token. | |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declares this class' options. | |
Protected Attributes | |
map< string, int > | string_to_int |
string to int mapping | |
map< int, string > | int_to_string |
int to string mapping | |
Private Types | |
typedef Object | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. |
A dictionary is a mapping between a string and an index or ID (int). Depending on the update mode, the dictionay can include an unknown word when asking for its ID with getId(); if update_mode == UPDATE, add the word and return its ID if update_mode == NO_UPDATE, return OOV symbol's ID (see below)
Also, when asking for the symbol associated to an ID, no update is possible. If the ID has not been assigned by the dictionary, then the returned symbol is "".
A Dictionary object is instantiated empty, and then symbols can be added in the dictionary by using getId() (with update_mode == UPDATE). By default, IDs are assigned by starting with the ID 0 and incrementing the assigned IDs as symbols are added in the dictionary.
Also, a "out-of-vocabulary" (OOV) symbol is assigned an ID which is equal to the number of symbols (other than the OOV symbol) that are in the dictionary at the current state. By default, this symbol corresponds to the string "<oov>".
The OOV symbol's ID is returned by the getId(symbol) function if "symbol" is not in the dictionary. Also by default, this OOV token is considered as part of the dictionary, but this can be changed using the dont_insert_oov_symbol option. If dont_insert_oov_token is true, then the size(), isIn() and getValues() functions do not consider the oov symbol as part of the dictionary. Note that getId() will still return the oov symbol's ID for unknown symbols, and getSymbol() will still return the OOV symbol for its ID.
Definition at line 87 of file Dictionary.h.
typedef Object PLearn::Dictionary::inherited [private] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 92 of file Dictionary.h.
PLearn::Dictionary::Dictionary | ( | ) |
Default constructor.
Definition at line 49 of file Dictionary.cc.
: update_mode(UPDATE), dont_insert_oov_symbol(false), oov_symbol("<oov>") { }
string PLearn::Dictionary::_classname_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 87 of file Dictionary.cc.
OptionList & PLearn::Dictionary::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 87 of file Dictionary.cc.
RemoteMethodMap & PLearn::Dictionary::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 87 of file Dictionary.cc.
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 87 of file Dictionary.cc.
Object * PLearn::Dictionary::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 87 of file Dictionary.cc.
StaticInitializer Dictionary::_static_initializer_ & PLearn::Dictionary::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 87 of file Dictionary.cc.
void PLearn::Dictionary::build | ( | ) | [virtual] |
Post-constructor.
The normal implementation should call simply inherited::build(), then this class's build_(). This method should be callable again at later times, after modifying some option fields to change the "architecture" of the object.
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 120 of file Dictionary.cc.
References PLearn::Object::build(), and build_().
Referenced by PLearn::WordNetSenseDictionary::build(), PLearn::VecDictionary::build(), PLearn::FileDictionary::build(), and PLearn::ConditionalDictionary::build().
{ inherited::build(); build_(); }
void PLearn::Dictionary::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 116 of file Dictionary.cc.
Referenced by build().
{ }
string PLearn::Dictionary::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 87 of file Dictionary.cc.
void PLearn::Dictionary::clear | ( | ) | [virtual] |
Definition at line 209 of file Dictionary.cc.
References int_to_string, and string_to_int.
{ string_to_int.clear(); int_to_string.clear(); }
void PLearn::Dictionary::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declares this class' options.
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 89 of file Dictionary.cc.
References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::Object::declareOptions(), dont_insert_oov_symbol, int_to_string, PLearn::OptionBase::learntoption, oov_symbol, string_to_int, and update_mode.
Referenced by PLearn::WordNetSenseDictionary::declareOptions(), PLearn::VecDictionary::declareOptions(), PLearn::FileDictionary::declareOptions(), and PLearn::ConditionalDictionary::declareOptions().
{ declareOption(ol, "update_mode", &Dictionary::update_mode, OptionBase::buildoption, "update_mode : 0(no_update)/1(update). Default is update"); declareOption(ol, "oov_symbol", &Dictionary::oov_symbol, OptionBase::buildoption, "String symbol for \"out-of-vocabulary\" token."); declareOption(ol, "string_to_int", &Dictionary::string_to_int, OptionBase::learntoption, "string to int mapping"); declareOption(ol, "int_to_string", &Dictionary::int_to_string, OptionBase::learntoption, "int to string mapping"); // For backward compatibility... declareOption(ol, "oov_not_in_possible_values", &Dictionary::dont_insert_oov_symbol, OptionBase::buildoption, "DEPRECATED, use dont_insert_oov_symbol instead: Indication that\n" "the OOV symbol should not be considered as part of the dictionary"); declareOption(ol, "dont_insert_oov_symbol", &Dictionary::dont_insert_oov_symbol, OptionBase::buildoption, "Indication that the OOV symbol should not be considered\n" "as part of the dictionary"); inherited::declareOptions(ol); }
static const PPath& PLearn::Dictionary::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 138 of file Dictionary.h.
Dictionary * PLearn::Dictionary::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 87 of file Dictionary.cc.
Gives the ID of a symbol in the dictionary If the symbol is not in the dictionary, returns the index of oov token if update_mode = NO_UPDATE.
Insert the new word otherwise and return its index When a symbol is added to the dictionary, the following fields are updated: string_to_int, int_to_string, values Options can be specified ...
Reimplemented in PLearn::WordNetSenseDictionary.
Definition at line 133 of file Dictionary.cc.
References int_to_string, oov_symbol, string_to_int, UPDATE, and update_mode.
Referenced by PLearn::FileDictionary::build_(), PLearn::ConditionalDictionary::build_(), PLearn::VecDictionary::build_(), PLearn::WordNetSenseDictionary::getId(), PLearn::WordNetSenseDictionary::getSensesFromWordNet(), PLearn::WordNetSenseDictionary::getValues(), isIn(), and PLearn::WordNetSenseDictionary::rootNode().
{ // Gives the id of a symbol in the dictionary // If the symbol is not in the dictionary, // returns index of OOV symbol if update_mode = NO_UPDATE, // insert the new word otherwise and return its index if(symbol == oov_symbol) return string_to_int.size(); int index; if(update_mode== UPDATE) { if(string_to_int.find(symbol) == string_to_int.end()){ // word not found, add it index=int(string_to_int.size()); string_to_int[symbol] = index; int_to_string[index] = symbol; } return string_to_int[symbol]; } else { // No update mode if(string_to_int.find(symbol) == string_to_int.end()){ // word not found, return OOV symbol's ID return string_to_int.size(); }else{ return string_to_int[symbol]; } } }
OptionList & PLearn::Dictionary::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 87 of file Dictionary.cc.
OptionMap & PLearn::Dictionary::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 87 of file Dictionary.cc.
RemoteMethodMap & PLearn::Dictionary::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 87 of file Dictionary.cc.
string PLearn::Dictionary::getSymbol | ( | int | id, |
TVec< string > | options = TVec<string>(0) |
||
) | const [virtual] |
Gives the symbol from an id of the dictionary If the id is invalid, the string returned is "" Options can be specified ...
Definition at line 165 of file Dictionary.cc.
References int_to_string, oov_symbol, and string_to_int.
{ if(int_to_string.find(id) == int_to_string.end()) { if(id == ((int)string_to_int.size())) return oov_symbol; else return ""; } else return int_to_string.find(id)->second; }
Fills a Vec containing every possible ID values of the Dictionary Options can be specified to restrict the number of possible values.
A Vec instead of a TVec<int> is required, for compatibility with the getValues() function of VMatrix objects.
Reimplemented in PLearn::ConditionalDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 185 of file Dictionary.cc.
References dont_insert_oov_symbol, PLearn::TVec< T >::first(), i, int_to_string, PLearn::TVec< T >::resize(), and size().
Referenced by PLearn::WordNetSenseDictionary::getValues(), and PLearn::ConditionalDictionary::getValues().
{ values.resize(size()); int i=0; for(map<int,string>::iterator it = int_to_string.begin(); it != int_to_string.end(); it++) values[i++] = it->first; if(!dont_insert_oov_symbol) values[i] = i; // OOV symbol's ID }
Indicates if a symbol is in the dictionary The OOV symbol is considered as in the dictionary if dont_insert_oov_symbol is false.
Options can be specified ...
Definition at line 195 of file Dictionary.cc.
References dont_insert_oov_symbol, getId(), NO_UPDATE, oov_symbol, string_to_int, and update_mode.
Referenced by PLearn::WordNetSenseDictionary::getSensesFromWordNet().
{ if(symbol == oov_symbol) return !dont_insert_oov_symbol; int last_update_mode = update_mode; update_mode = NO_UPDATE; int id = getId(symbol,options); update_mode = last_update_mode; return id != ((int)string_to_int.size()); }
Indicates if an id is in the dictionary The OOV symbol's ID is considered as in the dictionary if dont_insert_oov_symbol is false.
Options can be specified ...
Definition at line 204 of file Dictionary.cc.
{
return id < size() && id >= 0;
}
void PLearn::Dictionary::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 215 of file Dictionary.cc.
References PLearn::deepCopyField(), int_to_string, PLearn::Object::makeDeepCopyFromShallowCopy(), and string_to_int.
Referenced by PLearn::WordNetSenseDictionary::makeDeepCopyFromShallowCopy(), PLearn::VecDictionary::makeDeepCopyFromShallowCopy(), PLearn::FileDictionary::makeDeepCopyFromShallowCopy(), and PLearn::ConditionalDictionary::makeDeepCopyFromShallowCopy().
{ inherited::makeDeepCopyFromShallowCopy(copies); deepCopyField(string_to_int, copies); deepCopyField(int_to_string, copies); }
void PLearn::Dictionary::setUpdateMode | ( | int | up_mode | ) | [virtual] |
Set update dictionary mode : UPDATE/NO_UPDATE.
Definition at line 126 of file Dictionary.cc.
References NO_UPDATE, PLERROR, UPDATE, and update_mode.
Referenced by PLearn::FileDictionary::FileDictionary(), and PLearn::VecDictionary::VecDictionary().
{ if(up_mode != UPDATE && up_mode != NO_UPDATE) PLERROR("In Dictionary::setUpdateMode(): incompatible update mode %d", up_mode); update_mode =up_mode; }
Get size of the dictionary (number of symbols in the dictionary) Options can be specified to restrict the number of possible values.
Reimplemented in PLearn::ConditionalDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 178 of file Dictionary.cc.
References dont_insert_oov_symbol, and string_to_int.
Referenced by PLearn::VecDictionary::build_(), PLearn::FileDictionary::build_(), getValues(), PLearn::WordNetSenseDictionary::size(), and PLearn::ConditionalDictionary::size().
{ if(dont_insert_oov_symbol) return int(string_to_int.size()); else return int(string_to_int.size())+1; }
Reimplemented from PLearn::Object.
Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.
Definition at line 138 of file Dictionary.h.
Indication that "oov" should not be considered as part of the dictionary.
Definition at line 113 of file Dictionary.h.
Referenced by declareOptions(), getValues(), isIn(), and size().
map<int,string> PLearn::Dictionary::int_to_string [protected] |
int to string mapping
Definition at line 102 of file Dictionary.h.
Referenced by clear(), declareOptions(), getId(), getSymbol(), getValues(), and makeDeepCopyFromShallowCopy().
Symbol for "out-of-vocabulary" token.
Definition at line 115 of file Dictionary.h.
Referenced by declareOptions(), getId(), getSymbol(), and isIn().
map<string,int> PLearn::Dictionary::string_to_int [protected] |
string to int mapping
Definition at line 100 of file Dictionary.h.
Referenced by clear(), declareOptions(), getId(), getSymbol(), isIn(), makeDeepCopyFromShallowCopy(), and size().
update mode update/no update
Definition at line 110 of file Dictionary.h.
Referenced by PLearn::VecDictionary::build_(), PLearn::FileDictionary::build_(), PLearn::ConditionalDictionary::build_(), declareOptions(), PLearn::WordNetSenseDictionary::getId(), getId(), isIn(), and setUpdateMode().