PLearn 0.1
|
Base class for feature sets that maintains an explicit mapping between index and string form features", This class facilitates the conception of FeatureSet objects. More...
#include <HashMapFeatureSet.h>
Public Member Functions | |
HashMapFeatureSet () | |
Default constructor. | |
virtual HashMapFeatureSet * | deepCopy (CopiesMap &copies) const |
virtual void | build () |
Post-constructor. | |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Transforms a shallow copy into a deep copy. | |
virtual void | getFeatures (string token, TVec< int > &feats) |
Gives features of token in index form. | |
virtual string | getStringFeature (int index) |
Gives string form of a feature in index form. | |
virtual int | getIndexFeature (string str) |
Gives index form of a feature in string form. | |
virtual int | size () |
Gives the number of features in the set. | |
virtual void | addFeatures (string token) |
Adds the features for a given token in the set. | |
virtual void | addFeatures (VMat tokens, int min_freq=-1) |
Adds the features for the tokens contained in the set. | |
virtual void | clear () |
Clears all features from the feature set. | |
virtual void | getNewFeaturesString (string token, TVec< string > &feats_str)=0 |
Gives the possibly new features in string form for a token. | |
Static Public Member Functions | |
static string | _classname_ () |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
bool | report_progress |
Indication that progress should be reported in a ProgressBar when filling the feature set. | |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declares the class options. | |
Protected Attributes | |
hash_map< int, string > | index_to_string_feats |
Index feature to string feature mapping. | |
hash_map< string, int > | string_to_index_feats |
String feature to index feature mapping. | |
Private Types | |
typedef FeatureSet | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. | |
Private Attributes | |
TVec< string > | f_str |
Temporary computations vectors. | |
int | index |
Base class for feature sets that maintains an explicit mapping between index and string form features", This class facilitates the conception of FeatureSet objects.
"
Classes that inherits from this class only need to define the\n" getNewFeaturesString() function, which is called when adding new features
"
or when asking for the features of a token (in index or string format).\n" This function should return the list of features in string format
"
for a given token. It is assumed that no feature is present more than once\n" in the output of that function, for any input token.
"
This base class maintains two hash_maps between the index and string forms\n" of the features. These mappings contain all the necessary
"
information to define the different functions of a FeatureSet object.\n"
Definition at line 61 of file HashMapFeatureSet.h.
typedef FeatureSet PLearn::HashMapFeatureSet::inherited [private] |
Reimplemented from PLearn::FeatureSet.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 63 of file HashMapFeatureSet.h.
PLearn::HashMapFeatureSet::HashMapFeatureSet | ( | ) |
Default constructor.
Definition at line 61 of file HashMapFeatureSet.cc.
: report_progress(false) {}
string PLearn::HashMapFeatureSet::_classname_ | ( | ) | [static] |
Reimplemented from PLearn::FeatureSet.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 59 of file HashMapFeatureSet.cc.
OptionList & PLearn::HashMapFeatureSet::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::FeatureSet.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 59 of file HashMapFeatureSet.cc.
RemoteMethodMap & PLearn::HashMapFeatureSet::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::FeatureSet.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 59 of file HashMapFeatureSet.cc.
Reimplemented from PLearn::FeatureSet.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 59 of file HashMapFeatureSet.cc.
StaticInitializer HashMapFeatureSet::_static_initializer_ & PLearn::HashMapFeatureSet::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::FeatureSet.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 59 of file HashMapFeatureSet.cc.
void PLearn::HashMapFeatureSet::addFeatures | ( | string | token | ) | [virtual] |
Adds the features for a given token in the set.
Implements PLearn::FeatureSet.
Definition at line 142 of file HashMapFeatureSet.cc.
References f_str, getNewFeaturesString(), index, index_to_string_feats, PLearn::TVec< T >::length(), and string_to_index_feats.
{ string str; int index; getNewFeaturesString(token,f_str); for(int t=0; t<f_str.length(); t++) { str = f_str[t]; if(string_to_index_feats.find(str) == string_to_index_feats.end()){ index = int(string_to_index_feats.size()); string_to_index_feats[str] = index; index_to_string_feats[index] = str; } } }
Adds the features for the tokens contained in the set.
Features with frequency smaller than min_freq are not added.
Implements PLearn::FeatureSet.
Definition at line 160 of file HashMapFeatureSet.cc.
References PLearn::TVec< T >::copy(), f_str, getNewFeaturesString(), i, index, index_to_string_feats, j, PLearn::TVec< T >::length(), PLearn::VMat::length(), report_progress, PLearn::TVec< T >::resize(), string_to_index_feats, PLearn::tostring(), and PLearn::VMat::width().
{ string str, token; int index; map<string,int> frequencies; map<string,TVec<string> > token_features; map<string,bool> token_feat_inserted; Vec row(tokens->width()); TVec<string> f_str_temp; PP<ProgressBar> pb; int id=0; if(min_freq > 1) { if(report_progress) pb = new ProgressBar("Counting frequencies for the features of " + tostring(tokens->length()*tokens->width()) + " tokens", tokens->length()*tokens->width()); for(int i=0; i<tokens->length(); i++) { tokens->getRow(i,row); for(int j=0; j<tokens->width(); j++) { token = tokens->getValString(j,row[j]); if(token_features.find(token) == token_features.end()) { getNewFeaturesString(token,f_str); token_features[token] = f_str.copy(); } else { f_str_temp = token_features[token]; f_str.resize(f_str_temp.length()); f_str << f_str_temp; } for(int t=0; t<f_str.length(); t++) { str = f_str[t]; if(frequencies.find(str) == frequencies.end()) frequencies[str] = 1; else frequencies[str]++; } if(report_progress) pb->update(++id); } } token_features.clear(); } id=0; if(report_progress) pb = new ProgressBar("Filling FeatureSet with the features of " + tostring(tokens->length()*tokens->width()) + " tokens", tokens->length()*tokens->width()); for(int i=0; i<tokens->length(); i++) { tokens->getRow(i,row); for(int j=0; j<tokens->width(); j++) { token = tokens->getValString(j,row[j]); if(token_feat_inserted.find(token) == token_feat_inserted.end()) { getNewFeaturesString(token,f_str); for(int t=0; t<f_str.length(); t++) { str = f_str[t]; if(string_to_index_feats.find(str) == string_to_index_feats.end() && (min_freq <=1 || frequencies[str] >= min_freq)){ index = int(string_to_index_feats.size()); string_to_index_feats[str] = index; index_to_string_feats[index] = str; } } token_feat_inserted[token] = true; } if(report_progress) pb->update(++id); } } }
void PLearn::HashMapFeatureSet::build | ( | ) | [virtual] |
Post-constructor.
The normal implementation should call simply inherited::build(), then this class's build_(). This method should be callable again at later times, after modifying some option fields to change the "architecture" of the object.
Reimplemented from PLearn::FeatureSet.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 66 of file HashMapFeatureSet.cc.
References PLearn::FeatureSet::build(), and build_().
Referenced by PLearn::WordNetFeatureSet::build(), and PLearn::PythonFeatureSet::build().
{ inherited::build(); build_(); }
void PLearn::HashMapFeatureSet::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::FeatureSet.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 98 of file HashMapFeatureSet.cc.
Referenced by build().
{}
void PLearn::HashMapFeatureSet::clear | ( | ) | [virtual] |
Clears all features from the feature set.
Implements PLearn::FeatureSet.
Definition at line 242 of file HashMapFeatureSet.cc.
References index_to_string_feats, and string_to_index_feats.
{ index_to_string_feats.clear(); string_to_index_feats.clear(); }
void PLearn::HashMapFeatureSet::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declares the class options.
Reimplemented from PLearn::FeatureSet.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 81 of file HashMapFeatureSet.cc.
References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::FeatureSet::declareOptions(), index_to_string_feats, PLearn::OptionBase::learntoption, report_progress, and string_to_index_feats.
Referenced by PLearn::WordNetFeatureSet::declareOptions(), and PLearn::PythonFeatureSet::declareOptions().
{ declareOption(ol, "report_progress", &HashMapFeatureSet::report_progress, OptionBase::buildoption, "Indication that progress should be reported in a ProgressBar\n" "when filling the feature set.\n"); declareOption(ol, "index_to_string_feats", &HashMapFeatureSet::index_to_string_feats, OptionBase::learntoption, "Index feature to string feature mapping"); declareOption(ol, "string_to_index_feats", &HashMapFeatureSet::string_to_index_feats, OptionBase::learntoption, "String feature to index feature mapping"); // Now call the parent class' declareOptions inherited::declareOptions(ol); }
static const PPath& PLearn::HashMapFeatureSet::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::FeatureSet.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 81 of file HashMapFeatureSet.h.
:
//##### Protected Options ###############################################
HashMapFeatureSet * PLearn::HashMapFeatureSet::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::FeatureSet.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 59 of file HashMapFeatureSet.cc.
Gives features of token in index form.
Implements PLearn::FeatureSet.
Definition at line 101 of file HashMapFeatureSet.cc.
References f_str, getNewFeaturesString(), index, PLearn::TVec< T >::length(), PLearn::TVec< T >::push_back(), PLearn::TVec< T >::resize(), and string_to_index_feats.
{ int index; string str; getNewFeaturesString(token,f_str); feats.resize(0); for(int t=0; t<f_str.length(); t++) { str = f_str[t]; if(string_to_index_feats.find(str) != string_to_index_feats.end()) { index = string_to_index_feats[str]; feats.push_back(index); } } }
int PLearn::HashMapFeatureSet::getIndexFeature | ( | string | str | ) | [virtual] |
Gives index form of a feature in string form.
Implements PLearn::FeatureSet.
Definition at line 129 of file HashMapFeatureSet.cc.
References PLERROR, and string_to_index_feats.
{ if(string_to_index_feats.find(str) == string_to_index_feats.end()) PLERROR("In HashMapFeatureSet::getIndexFeature(): string %s is an invalid feature", str.c_str()); return string_to_index_feats[str]; }
virtual void PLearn::HashMapFeatureSet::getNewFeaturesString | ( | string | token, |
TVec< string > & | feats_str | ||
) | [pure virtual] |
Gives the possibly new features in string form for a token.
Implements PLearn::FeatureSet.
Implemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Referenced by addFeatures(), and getFeatures().
string PLearn::HashMapFeatureSet::getStringFeature | ( | int | index | ) | [virtual] |
Gives string form of a feature in index form.
Implements PLearn::FeatureSet.
Definition at line 119 of file HashMapFeatureSet.cc.
References index, index_to_string_feats, and PLERROR.
{ #ifdef BOUNDCHECK if(index_to_string_feats.find(index) == index_to_string_feats.end()) PLERROR("In HashMapFeatureSet::getStringFeature(): index %d is an invalid feature index", index); #endif return index_to_string_feats[index]; }
void PLearn::HashMapFeatureSet::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::FeatureSet.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 72 of file HashMapFeatureSet.cc.
References PLearn::deepCopyField(), index_to_string_feats, PLearn::FeatureSet::makeDeepCopyFromShallowCopy(), and string_to_index_feats.
Referenced by PLearn::WordNetFeatureSet::makeDeepCopyFromShallowCopy(), and PLearn::PythonFeatureSet::makeDeepCopyFromShallowCopy().
{ inherited::makeDeepCopyFromShallowCopy(copies); deepCopyField(index_to_string_feats, copies); deepCopyField(string_to_index_feats, copies); //PLERROR("HashMapFeatureSet::makeDeepCopyFromShallowCopy not fully (correctly) implemented yet!"); }
int PLearn::HashMapFeatureSet::size | ( | ) | [virtual] |
Gives the number of features in the set.
Implements PLearn::FeatureSet.
Definition at line 137 of file HashMapFeatureSet.cc.
References index_to_string_feats.
{ return int(index_to_string_feats.size()); }
Reimplemented from PLearn::FeatureSet.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 81 of file HashMapFeatureSet.h.
TVec<string> PLearn::HashMapFeatureSet::f_str [private] |
Temporary computations vectors.
Reimplemented in PLearn::IdentityFeatureSet, PLearn::PythonFeatureSet, and PLearn::WordNetFeatureSet.
Definition at line 138 of file HashMapFeatureSet.h.
Referenced by addFeatures(), and getFeatures().
int PLearn::HashMapFeatureSet::index [private] |
Definition at line 139 of file HashMapFeatureSet.h.
Referenced by addFeatures(), getFeatures(), and getStringFeature().
hash_map<int,string> PLearn::HashMapFeatureSet::index_to_string_feats [protected] |
Index feature to string feature mapping.
Definition at line 118 of file HashMapFeatureSet.h.
Referenced by addFeatures(), clear(), declareOptions(), getStringFeature(), makeDeepCopyFromShallowCopy(), and size().
Indication that progress should be reported in a ProgressBar when filling the feature set.
Definition at line 70 of file HashMapFeatureSet.h.
Referenced by addFeatures(), and declareOptions().
hash_map<string,int> PLearn::HashMapFeatureSet::string_to_index_feats [protected] |
String feature to index feature mapping.
Definition at line 120 of file HashMapFeatureSet.h.
Referenced by addFeatures(), clear(), declareOptions(), getFeatures(), getIndexFeature(), and makeDeepCopyFromShallowCopy().