PLearn 0.1
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Static Protected Member Functions | Protected Attributes | Private Types | Private Member Functions
PLearn::Dictionary Class Reference

#include <Dictionary.h>

Inheritance diagram for PLearn::Dictionary:
Inheritance graph
[legend]
Collaboration diagram for PLearn::Dictionary:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 Dictionary ()
 Default constructor.
virtual string classname () const
virtual OptionListgetOptionList () const
virtual OptionMapgetOptionMap () const
virtual RemoteMethodMapgetRemoteMethodMap () const
virtual DictionarydeepCopy (CopiesMap &copies) const
virtual void setUpdateMode (int up_mode)
 Set update dictionary mode : UPDATE/NO_UPDATE.
virtual int getId (string symbol, TVec< string > options=TVec< string >(0))
 Gives the ID of a symbol in the dictionary If the symbol is not in the dictionary, returns the index of oov token if update_mode = NO_UPDATE.
virtual string getSymbol (int id, TVec< string > options=TVec< string >(0)) const
 Gives the symbol from an id of the dictionary If the id is invalid, the string returned is "" Options can be specified ...
virtual int size (TVec< string > options=TVec< string >(0))
 Get size of the dictionary (number of symbols in the dictionary) Options can be specified to restrict the number of possible values.
virtual void getValues (TVec< string > options, Vec &values)
 Fills a Vec containing every possible ID values of the Dictionary Options can be specified to restrict the number of possible values.
virtual bool isIn (string symbol, TVec< string > options=TVec< string >(0))
 Indicates if a symbol is in the dictionary The OOV symbol is considered as in the dictionary if dont_insert_oov_symbol is false.
virtual bool isIn (int id, TVec< string > options=TVec< string >(0))
 Indicates if an id is in the dictionary The OOV symbol's ID is considered as in the dictionary if dont_insert_oov_symbol is false.
virtual void clear ()
virtual void build ()
 Post-constructor.
virtual void makeDeepCopyFromShallowCopy (CopiesMap &copies)
 Transforms a shallow copy into a deep copy.

Static Public Member Functions

static string _classname_ ()
static OptionList_getOptionList_ ()
static RemoteMethodMap_getRemoteMethodMap_ ()
static Object_new_instance_for_typemap_ ()
static bool _isa_ (const Object *o)
static void _static_initialize_ ()
static const PPathdeclaringFile ()

Public Attributes

int update_mode
 update mode update/no update
bool dont_insert_oov_symbol
 Indication that "oov" should not be considered as part of the dictionary.
string oov_symbol
 Symbol for "out-of-vocabulary" token.

Static Public Attributes

static StaticInitializer _static_initializer_

Static Protected Member Functions

static void declareOptions (OptionList &ol)
 Declares this class' options.

Protected Attributes

map< string, intstring_to_int
 string to int mapping
map< int, string > int_to_string
 int to string mapping

Private Types

typedef Object inherited

Private Member Functions

void build_ ()
 This does the actual building.

Detailed Description

A dictionary is a mapping between a string and an index or ID (int). Depending on the update mode, the dictionay can include an unknown word when asking for its ID with getId(); if update_mode == UPDATE, add the word and return its ID if update_mode == NO_UPDATE, return OOV symbol's ID (see below)

Also, when asking for the symbol associated to an ID, no update is possible. If the ID has not been assigned by the dictionary, then the returned symbol is "".

A Dictionary object is instantiated empty, and then symbols can be added in the dictionary by using getId() (with update_mode == UPDATE). By default, IDs are assigned by starting with the ID 0 and incrementing the assigned IDs as symbols are added in the dictionary.

Also, a "out-of-vocabulary" (OOV) symbol is assigned an ID which is equal to the number of symbols (other than the OOV symbol) that are in the dictionary at the current state. By default, this symbol corresponds to the string "<oov>".

The OOV symbol's ID is returned by the getId(symbol) function if "symbol" is not in the dictionary. Also by default, this OOV token is considered as part of the dictionary, but this can be changed using the dont_insert_oov_symbol option. If dont_insert_oov_token is true, then the size(), isIn() and getValues() functions do not consider the oov symbol as part of the dictionary. Note that getId() will still return the oov symbol's ID for unknown symbols, and getSymbol() will still return the OOV symbol for its ID.

Definition at line 87 of file Dictionary.h.


Member Typedef Documentation


Constructor & Destructor Documentation

PLearn::Dictionary::Dictionary ( )

Default constructor.

Definition at line 49 of file Dictionary.cc.


Member Function Documentation

string PLearn::Dictionary::_classname_ ( ) [static]
OptionList & PLearn::Dictionary::_getOptionList_ ( ) [static]
RemoteMethodMap & PLearn::Dictionary::_getRemoteMethodMap_ ( ) [static]
bool PLearn::Dictionary::_isa_ ( const Object o) [static]
Object * PLearn::Dictionary::_new_instance_for_typemap_ ( ) [static]
StaticInitializer Dictionary::_static_initializer_ & PLearn::Dictionary::_static_initialize_ ( ) [static]
void PLearn::Dictionary::build ( ) [virtual]

Post-constructor.

The normal implementation should call simply inherited::build(), then this class's build_(). This method should be callable again at later times, after modifying some option fields to change the "architecture" of the object.

Reimplemented from PLearn::Object.

Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.

Definition at line 120 of file Dictionary.cc.

References PLearn::Object::build(), and build_().

Referenced by PLearn::WordNetSenseDictionary::build(), PLearn::VecDictionary::build(), PLearn::FileDictionary::build(), and PLearn::ConditionalDictionary::build().

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::Dictionary::build_ ( ) [private]

This does the actual building.

Reimplemented from PLearn::Object.

Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.

Definition at line 116 of file Dictionary.cc.

Referenced by build().

                       {
}

Here is the caller graph for this function:

string PLearn::Dictionary::classname ( ) const [virtual]
void PLearn::Dictionary::clear ( ) [virtual]

Definition at line 209 of file Dictionary.cc.

References int_to_string, and string_to_int.

{
    string_to_int.clear(); 
    int_to_string.clear();
}
void PLearn::Dictionary::declareOptions ( OptionList ol) [static, protected]

Declares this class' options.

Reimplemented from PLearn::Object.

Reimplemented in PLearn::ConditionalDictionary, PLearn::FileDictionary, PLearn::VecDictionary, and PLearn::WordNetSenseDictionary.

Definition at line 89 of file Dictionary.cc.

References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::Object::declareOptions(), dont_insert_oov_symbol, int_to_string, PLearn::OptionBase::learntoption, oov_symbol, string_to_int, and update_mode.

Referenced by PLearn::WordNetSenseDictionary::declareOptions(), PLearn::VecDictionary::declareOptions(), PLearn::FileDictionary::declareOptions(), and PLearn::ConditionalDictionary::declareOptions().

{
    declareOption(ol, "update_mode", &Dictionary::update_mode, 
                  OptionBase::buildoption, 
                  "update_mode : 0(no_update)/1(update). Default is update");
    declareOption(ol, "oov_symbol", &Dictionary::oov_symbol, 
                  OptionBase::buildoption, 
                  "String symbol for \"out-of-vocabulary\" token.");
    declareOption(ol, "string_to_int", &Dictionary::string_to_int, 
                  OptionBase::learntoption, "string to int mapping");
    declareOption(ol, "int_to_string", &Dictionary::int_to_string, 
                  OptionBase::learntoption, "int to string mapping");
    // For backward compatibility...
    declareOption(ol, "oov_not_in_possible_values", 
                  &Dictionary::dont_insert_oov_symbol, 
                  OptionBase::buildoption,
                  "DEPRECATED, use dont_insert_oov_symbol instead: Indication that\n"
                  "the OOV symbol should not be considered as part of the dictionary"); 
    declareOption(ol, "dont_insert_oov_symbol", 
                  &Dictionary::dont_insert_oov_symbol, 
                  OptionBase::buildoption, 
                  "Indication that the OOV symbol should not be considered\n"
                  "as part of the dictionary"); 

    inherited::declareOptions(ol);
}

Here is the call graph for this function:

Here is the caller graph for this function:

static const PPath& PLearn::Dictionary::declaringFile ( ) [inline, static]
Dictionary * PLearn::Dictionary::deepCopy ( CopiesMap copies) const [virtual]
int PLearn::Dictionary::getId ( string  symbol,
TVec< string >  options = TVec<string>(0) 
) [virtual]

Gives the ID of a symbol in the dictionary If the symbol is not in the dictionary, returns the index of oov token if update_mode = NO_UPDATE.

Insert the new word otherwise and return its index When a symbol is added to the dictionary, the following fields are updated: string_to_int, int_to_string, values Options can be specified ...

Reimplemented in PLearn::WordNetSenseDictionary.

Definition at line 133 of file Dictionary.cc.

References int_to_string, oov_symbol, string_to_int, UPDATE, and update_mode.

Referenced by PLearn::FileDictionary::build_(), PLearn::ConditionalDictionary::build_(), PLearn::VecDictionary::build_(), PLearn::WordNetSenseDictionary::getId(), PLearn::WordNetSenseDictionary::getSensesFromWordNet(), PLearn::WordNetSenseDictionary::getValues(), isIn(), and PLearn::WordNetSenseDictionary::rootNode().

{
    // Gives the id of a symbol in the dictionary
    // If the symbol is not in the dictionary, 
    // returns index of OOV symbol if update_mode = NO_UPDATE,
    // insert the new word otherwise and return its index

    if(symbol == oov_symbol) return string_to_int.size();

    int index;
    if(update_mode== UPDATE)
    {
        if(string_to_int.find(symbol) == string_to_int.end()){
            // word not found, add it
            index=int(string_to_int.size());
            string_to_int[symbol] = index;
            int_to_string[index] = symbol;
        }
        return string_to_int[symbol];
    }
    else
    {
        // No update mode
        if(string_to_int.find(symbol) == string_to_int.end()){
            // word not found, return OOV symbol's ID
            return string_to_int.size();
        }else{
            return string_to_int[symbol];
        }
    }
}

Here is the caller graph for this function:

OptionList & PLearn::Dictionary::getOptionList ( ) const [virtual]
OptionMap & PLearn::Dictionary::getOptionMap ( ) const [virtual]
RemoteMethodMap & PLearn::Dictionary::getRemoteMethodMap ( ) const [virtual]
string PLearn::Dictionary::getSymbol ( int  id,
TVec< string >  options = TVec<string>(0) 
) const [virtual]

Gives the symbol from an id of the dictionary If the id is invalid, the string returned is "" Options can be specified ...

Definition at line 165 of file Dictionary.cc.

References int_to_string, oov_symbol, and string_to_int.

{
    if(int_to_string.find(id) == int_to_string.end())
    {
        if(id == ((int)string_to_int.size()))
            return oov_symbol;
        else
            return "";
    }
    else
        return int_to_string.find(id)->second;
}
void PLearn::Dictionary::getValues ( TVec< string >  options,
Vec values 
) [virtual]

Fills a Vec containing every possible ID values of the Dictionary Options can be specified to restrict the number of possible values.

A Vec instead of a TVec<int> is required, for compatibility with the getValues() function of VMatrix objects.

Reimplemented in PLearn::ConditionalDictionary, and PLearn::WordNetSenseDictionary.

Definition at line 185 of file Dictionary.cc.

References dont_insert_oov_symbol, PLearn::TVec< T >::first(), i, int_to_string, PLearn::TVec< T >::resize(), and size().

Referenced by PLearn::WordNetSenseDictionary::getValues(), and PLearn::ConditionalDictionary::getValues().

{ 
    values.resize(size());
    int i=0;
    for(map<int,string>::iterator it = int_to_string.begin(); it != int_to_string.end(); it++)
        values[i++] = it->first;
    if(!dont_insert_oov_symbol)
        values[i] = i; // OOV symbol's ID
}

Here is the call graph for this function:

Here is the caller graph for this function:

bool PLearn::Dictionary::isIn ( string  symbol,
TVec< string >  options = TVec<string>(0) 
) [virtual]

Indicates if a symbol is in the dictionary The OOV symbol is considered as in the dictionary if dont_insert_oov_symbol is false.

Options can be specified ...

Definition at line 195 of file Dictionary.cc.

References dont_insert_oov_symbol, getId(), NO_UPDATE, oov_symbol, string_to_int, and update_mode.

Referenced by PLearn::WordNetSenseDictionary::getSensesFromWordNet().

                                                        {
    if(symbol == oov_symbol) return !dont_insert_oov_symbol;
    int last_update_mode = update_mode;
    update_mode = NO_UPDATE;
    int id = getId(symbol,options);
    update_mode = last_update_mode;
    return id != ((int)string_to_int.size());
}

Here is the call graph for this function:

Here is the caller graph for this function:

bool PLearn::Dictionary::isIn ( int  id,
TVec< string >  options = TVec<string>(0) 
) [virtual]

Indicates if an id is in the dictionary The OOV symbol's ID is considered as in the dictionary if dont_insert_oov_symbol is false.

Options can be specified ...

Definition at line 204 of file Dictionary.cc.

{ 
    return id < size() && id >= 0;
}
void PLearn::Dictionary::makeDeepCopyFromShallowCopy ( CopiesMap copies) [virtual]
void PLearn::Dictionary::setUpdateMode ( int  up_mode) [virtual]

Set update dictionary mode : UPDATE/NO_UPDATE.

Definition at line 126 of file Dictionary.cc.

References NO_UPDATE, PLERROR, UPDATE, and update_mode.

Referenced by PLearn::FileDictionary::FileDictionary(), and PLearn::VecDictionary::VecDictionary().

{
    if(up_mode != UPDATE && up_mode != NO_UPDATE)
        PLERROR("In Dictionary::setUpdateMode(): incompatible update mode %d", up_mode);
    update_mode =up_mode;
}

Here is the caller graph for this function:

int PLearn::Dictionary::size ( TVec< string >  options = TVec<string>(0)) [virtual]

Get size of the dictionary (number of symbols in the dictionary) Options can be specified to restrict the number of possible values.

Reimplemented in PLearn::ConditionalDictionary, and PLearn::WordNetSenseDictionary.

Definition at line 178 of file Dictionary.cc.

References dont_insert_oov_symbol, and string_to_int.

Referenced by PLearn::VecDictionary::build_(), PLearn::FileDictionary::build_(), getValues(), PLearn::WordNetSenseDictionary::size(), and PLearn::ConditionalDictionary::size().

                                        {
    if(dont_insert_oov_symbol)
        return int(string_to_int.size());
    else
        return int(string_to_int.size())+1;
}

Here is the caller graph for this function:


Member Data Documentation

Indication that "oov" should not be considered as part of the dictionary.

Definition at line 113 of file Dictionary.h.

Referenced by declareOptions(), getValues(), isIn(), and size().

map<int,string> PLearn::Dictionary::int_to_string [protected]

int to string mapping

Definition at line 102 of file Dictionary.h.

Referenced by clear(), declareOptions(), getId(), getSymbol(), getValues(), and makeDeepCopyFromShallowCopy().

Symbol for "out-of-vocabulary" token.

Definition at line 115 of file Dictionary.h.

Referenced by declareOptions(), getId(), getSymbol(), and isIn().

map<string,int> PLearn::Dictionary::string_to_int [protected]

string to int mapping

Definition at line 100 of file Dictionary.h.

Referenced by clear(), declareOptions(), getId(), getSymbol(), isIn(), makeDeepCopyFromShallowCopy(), and size().


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines