PLearn 0.1
|
Takes a VMatrix with a word and a POS field and adds a field consisting of the lemma form of the word. More...
#include <LemmatizeVMatrix.h>
Public Member Functions | |
LemmatizeVMatrix () | |
Default constructor. | |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual LemmatizeVMatrix * | deepCopy (CopiesMap &copies) const |
virtual void | build () |
Simply calls inherited::build() then build_(). | |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Transforms a shallow copy into a deep copy. | |
virtual real | getStringVal (int col, const string &str) const |
returns value associated with a string (or MISSING_VALUE if there's no association for this string) | |
virtual string | getValString (int col, real val) const |
Returns the string associated with value val for field# col. | |
virtual void | getValues (int row, int col, Vec &values) const |
Gives the possible values for a certain field in the VMatrix. | |
virtual void | getValues (const Vec &input, int col, Vec &values) const |
Gives the possible values of a certain field (column) given the input. | |
virtual PP< Dictionary > | getDictionary (int col) const |
Get Dictionary from a certain column. | |
Static Public Member Functions | |
static string | _classname_ () |
Declares name and deepCopy methods. | |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
int | word_field |
Index (position) of word field. | |
int | pos_field |
Index (position) of POS field. | |
TMat< string > | pos_to_lemma_table |
Customized table that uses the POS tag to obtain a lemma It must have exactly two columns [POS lemma]. | |
TMat< string > | word_to_lemma_table |
Customized table that uses the word to obtain a lemma It must have exactly two columns [word lemma]. | |
TMat< string > | word_pos_to_lemma_table |
Customized table that uses the word and POS tag to obtain a lemma It must have exactly three columns [word POS lemma]. | |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Protected Member Functions | |
virtual void | getNewRow (int i, const Vec &v) const |
Fill the vector 'v' with the content of the i-th row. | |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declares the class options. | |
Protected Attributes | |
PP< Dictionary > | lemma_dict |
Lemma Dictionary, constructed automatically. | |
Vec | src_row |
Temporary variable for source row. | |
Private Types | |
typedef SourceVMatrix | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. | |
string | getLemma (int row) const |
Gives the lemma for a certain row. |
Takes a VMatrix with a word and a POS field and adds a field consisting of the lemma form of the word.
This field becomes the last input field. First, customized mapping tables (with different priorities) are used, and finally, if the previous step was unsuccesful, WordNet is used to obtain a lemma.
Definition at line 62 of file LemmatizeVMatrix.h.
typedef SourceVMatrix PLearn::LemmatizeVMatrix::inherited [private] |
Reimplemented from PLearn::SourceVMatrix.
Definition at line 64 of file LemmatizeVMatrix.h.
PLearn::LemmatizeVMatrix::LemmatizeVMatrix | ( | ) |
Default constructor.
Definition at line 62 of file LemmatizeVMatrix.cc.
References lemma_dict, and PLERROR.
: word_field(-1), pos_field(-1) { lemma_dict = new Dictionary(); if(morphinit<0) PLERROR("In WordNetSenseDictionary(): could not open WordNet exception list files"); // if(wninit()<0) // PLERROR("In WordNetSenseDictionary(): could not open WordNet database files"); }
string PLearn::LemmatizeVMatrix::_classname_ | ( | ) | [static] |
Declares name and deepCopy methods.
Reimplemented from PLearn::SourceVMatrix.
Definition at line 57 of file LemmatizeVMatrix.cc.
OptionList & PLearn::LemmatizeVMatrix::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::SourceVMatrix.
Definition at line 57 of file LemmatizeVMatrix.cc.
RemoteMethodMap & PLearn::LemmatizeVMatrix::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::SourceVMatrix.
Definition at line 57 of file LemmatizeVMatrix.cc.
Reimplemented from PLearn::SourceVMatrix.
Definition at line 57 of file LemmatizeVMatrix.cc.
Object * PLearn::LemmatizeVMatrix::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::SourceVMatrix.
Definition at line 57 of file LemmatizeVMatrix.cc.
StaticInitializer LemmatizeVMatrix::_static_initializer_ & PLearn::LemmatizeVMatrix::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::SourceVMatrix.
Definition at line 57 of file LemmatizeVMatrix.cc.
void PLearn::LemmatizeVMatrix::build | ( | ) | [virtual] |
Simply calls inherited::build() then build_().
Reimplemented from PLearn::SourceVMatrix.
Definition at line 190 of file LemmatizeVMatrix.cc.
References PLearn::SourceVMatrix::build(), and build_().
{ inherited::build(); build_(); }
void PLearn::LemmatizeVMatrix::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::SourceVMatrix.
Definition at line 199 of file LemmatizeVMatrix.cc.
References getLemma(), i, PLearn::VMatrix::inputsize_, lemma_dict, PLearn::TMat< T >::length(), PLearn::VMat::length(), PLearn::VMatrix::length_, NO_UPDATE, PLERROR, pos_field, pos_to_lemma_table, PLearn::TVec< T >::resize(), PLearn::SourceVMatrix::source, src_row, PLearn::VMatrix::targetsize_, PLearn::VMatrix::updateMtime(), PLearn::VMatrix::weightsize_, PLearn::VMat::width(), PLearn::TMat< T >::width(), PLearn::VMatrix::width_, word_field, word_pos_to_lemma_table, and word_to_lemma_table.
Referenced by build().
{ if(source) { if(word_field < 0 || word_field >= source->inputsize()) PLERROR("In LemmatizeVMatrix::build_(): word_field(%d) is not in input part of source VMatrix", word_field); if(pos_field < 0 || pos_field >= source->inputsize()) PLERROR("In LemmatizeVMatrix::build_(): pos_field(%d) is not in input part of source VMatrix", pos_field); inputsize_ = source->inputsize() + 1; targetsize_ = source->targetsize(); weightsize_ = source->weightsize(); width_ = inputsize_+targetsize_+weightsize_; length_ = source->length(); updateMtime(source); if(word_pos_to_lemma_table.length() != 0 && word_pos_to_lemma_table.width() != 3) PLERROR("In LemmatizeVMatrix::build_(): word_pos_to_lemma_table doesn't have three columns"); if(word_to_lemma_table.length() != 0 && word_to_lemma_table.width() != 2) PLERROR("In LemmatizeVMatrix::build_(): word_to_lemma_table doesn't have two columns"); if(pos_to_lemma_table.length() != 0 && pos_to_lemma_table.width() != 2) PLERROR("In LemmatizeVMatrix::build_(): pos_to_lemma_table doesn't have two columns"); string lemma; for(int i=0; i<length_;i++) { lemma = getLemma(i); lemma_dict->getId(lemma); } lemma_dict->update_mode = NO_UPDATE; src_row.resize(source->width()); } }
string PLearn::LemmatizeVMatrix::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::SourceVMatrix.
Definition at line 57 of file LemmatizeVMatrix.cc.
void PLearn::LemmatizeVMatrix::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declares the class options.
Reimplemented from PLearn::SourceVMatrix.
Definition at line 165 of file LemmatizeVMatrix.cc.
References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::SourceVMatrix::declareOptions(), PLearn::OptionBase::learntoption, lemma_dict, pos_field, pos_to_lemma_table, word_field, word_pos_to_lemma_table, and word_to_lemma_table.
{ declareOption(ol, "lemma_dict", &LemmatizeVMatrix::lemma_dict, OptionBase::learntoption, "Dictionary for the lemma field\n"); declareOption(ol, "word_field", &LemmatizeVMatrix::word_field, OptionBase::buildoption, "Index (position) of word field\n"); declareOption(ol, "pos_field", &LemmatizeVMatrix::pos_field, OptionBase::buildoption, "Index (position) of POS field\n"); declareOption(ol, "word_pos_to_lemma_table", &LemmatizeVMatrix::word_pos_to_lemma_table, OptionBase::buildoption, "Customized table that uses the word and POS tag to obtain a lemma\n" "It must have exactly three columns [word POS lemma]. It has the first priority.\n"); declareOption(ol, "word_to_lemma_table", &LemmatizeVMatrix::word_to_lemma_table, OptionBase::buildoption, "Customized table that uses the word to obtain a lemma\n" "It must have exactly two columns [word lemma]. It has the second priority.\n"); declareOption(ol, "pos_to_lemma_table", &LemmatizeVMatrix::pos_to_lemma_table, OptionBase::buildoption, "Customized table that uses the POS tag to obtain a lemma\n" "It must have exactly two columns [POS lemma] . It has the third priority.\n"); // Now call the parent class' declareOptions inherited::declareOptions(ol); }
static const PPath& PLearn::LemmatizeVMatrix::declaringFile | ( | ) | [inline, static] |
LemmatizeVMatrix * PLearn::LemmatizeVMatrix::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::SourceVMatrix.
Definition at line 57 of file LemmatizeVMatrix.cc.
PP< Dictionary > PLearn::LemmatizeVMatrix::getDictionary | ( | int | col | ) | const [virtual] |
Get Dictionary from a certain column.
Reimplemented from PLearn::SourceVMatrix.
Definition at line 263 of file LemmatizeVMatrix.cc.
References PLearn::VMatrix::inputsize_, lemma_dict, and PLearn::SourceVMatrix::source.
{ if(col == inputsize_-1) return lemma_dict; return source->getDictionary(col - (col < inputsize_-1?0:1)); }
string PLearn::LemmatizeVMatrix::getLemma | ( | int | row | ) | const [private] |
Gives the lemma for a certain row.
Definition at line 74 of file LemmatizeVMatrix.cc.
References i, PLearn::TMat< T >::length(), PLearn::lowerstring(), pos_field, pos_to_lemma_table, PLearn::SourceVMatrix::source, PLearn::stemWord(), word_field, word_pos_to_lemma_table, and word_to_lemma_table.
Referenced by build_(), and getNewRow().
{ string pos; string word; word = lowerstring(source->getString(row,word_field)); pos = source->getString(row,pos_field); // Verify if POS tag is compatible for word, using WordNet // wninit() most be called for that... /* TVec<string> ret_noun(0); TVec<string> ret_verb(0); TVec<string> ret_adj(0); TVec<string> ret_adv(0); TVec<string> stems; if(strstr(pos.c_str(),"NN")) { ret_noun.append(extractSenses(word,NOUN,"sense_key")); stems = stemsOfWord(word,NOUN); for(int i=0; i<stems.length(); i++) if(word != stems[i]) ret_noun.append(extractSenses(stems[i],NOUN,"sense_key")); } if(strstr(pos.c_str(),"VB")) { ret_verb.append(extractSenses(word,VERB,"sense_key")); stems = stemsOfWord(word,VERB); for(int i=0; i<stems.length(); i++) if(word != stems[i]) ret_verb.append(extractSenses(stems[i],VERB,"sense_key")); } if(strstr(pos.c_str(),"JJ")) { ret_adj.append(extractSenses(word,ADJ,"sense_key")); stems = stemsOfWord(word,ADJ); for(int i=0; i<stems.length(); i++) if(word != stems[i]) ret_adj.append(extractSenses(stems[i],ADJ,"sense_key")); } if(strstr(pos.c_str(),"RB")) { ret_adv.append(extractSenses(word,ADV,"sense_key")); stems = stemsOfWord(word,ADV); for(int i=0; i<stems.length(); i++) if(word != stems[i]) ret_adv.append(extractSenses(stems[i],ADV,"sense_key")); } if(!(ret_noun.length() == 0 && ret_verb.length() == 0 && ret_adj.length() == 0 && ret_adv.length() == 0)) { if(strstr(pos.c_str(),"NN") && ret_noun.length() == 0) PLWARNING("In LemmatizeVMatrix::getLemma(): word %s cannot be a noun", word.c_str()); if(strstr(pos.c_str(),"VB") && ret_verb.length() == 0) PLWARNING("In LemmatizeVMatrix::getLemma(): word %s cannot be a verb", word.c_str()); if(strstr(pos.c_str(),"JJ") && ret_adj.length() == 0) PLWARNING("In LemmatizeVMatrix::getLemma(): word %s cannot be an adjective", word.c_str()); if(strstr(pos.c_str(),"RB") && ret_adv.length() == 0) PLWARNING("In LemmatizeVMatrix::getLemma(): word %s cannot be an adverb", word.c_str()); } */ // Use word_pos_to_lemma_table for(int i=0; i<word_pos_to_lemma_table.length(); i++) if(word_pos_to_lemma_table(i,0) == word && word_pos_to_lemma_table(i,1) == pos) return word_pos_to_lemma_table(i,2); // Use word_to_lemma_table for(int i=0; i<word_to_lemma_table.length(); i++) if(word_to_lemma_table(i,0) == word) return word_to_lemma_table(i,1); // Use pos_to_lemma_table for(int i=0; i<pos_to_lemma_table.length(); i++) if(pos_to_lemma_table(i,0) == pos) return pos_to_lemma_table(i,1); if(strstr(pos.c_str(),"NN")) return stemWord(word,NOUN); else if(strstr(pos.c_str(),"VB")) return stemWord(word,VERB); else if(strstr(pos.c_str(),"JJ")) return stemWord(word,ADJ); else if(strstr(pos.c_str(),"RB")) return stemWord(word,ADV); return word; }
Fill the vector 'v' with the content of the i-th row.
v is assumed to be the right size.
Reimplemented from PLearn::SourceVMatrix.
Definition at line 234 of file LemmatizeVMatrix.cc.
References getLemma(), PLearn::VMatrix::inputsize_, lemma_dict, PLearn::SourceVMatrix::source, src_row, PLearn::TVec< T >::subVec(), PLearn::VMatrix::targetsize_, and PLearn::VMatrix::weightsize_.
{ source->getRow(i,src_row); v.subVec(0,inputsize_-1) << src_row.subVec(0,inputsize_-1); v[inputsize_-1] = lemma_dict->getId(getLemma(i)); v.subVec(inputsize_,targetsize_+weightsize_) << src_row.subVec(inputsize_-1,targetsize_+weightsize_); }
OptionList & PLearn::LemmatizeVMatrix::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::SourceVMatrix.
Definition at line 57 of file LemmatizeVMatrix.cc.
OptionMap & PLearn::LemmatizeVMatrix::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::SourceVMatrix.
Definition at line 57 of file LemmatizeVMatrix.cc.
RemoteMethodMap & PLearn::LemmatizeVMatrix::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::SourceVMatrix.
Definition at line 57 of file LemmatizeVMatrix.cc.
returns value associated with a string (or MISSING_VALUE if there's no association for this string)
Reimplemented from PLearn::VMatrix.
Definition at line 243 of file LemmatizeVMatrix.cc.
References PLearn::VMatrix::inputsize_, lemma_dict, MISSING_VALUE, and PLearn::SourceVMatrix::source.
{ int ret; if(col == inputsize_-1) { ret = lemma_dict->getId(str); if(ret == -1) return MISSING_VALUE; else return ret; } return source->getStringVal(col - (col < inputsize_-1?0:1),str); }
Returns the string associated with value val for field# col.
Or returns "" if no string is associated.
Reimplemented from PLearn::VMatrix.
Definition at line 255 of file LemmatizeVMatrix.cc.
References PLearn::VMatrix::inputsize_, PLearn::is_missing(), lemma_dict, PLearn::SourceVMatrix::source, and PLearn::tostring().
{ if(is_missing(val))return tostring(val); if(col == inputsize_-1) return lemma_dict->getSymbol((int)val); return source->getValString(col - (col < inputsize_-1?0:1),val); }
Gives the possible values for a certain field in the VMatrix.
Reimplemented from PLearn::SourceVMatrix.
Definition at line 271 of file LemmatizeVMatrix.cc.
References PLearn::VMatrix::inputsize_, lemma_dict, PLearn::VMatrix::length_, PLERROR, and PLearn::SourceVMatrix::source.
{ if(row < 0 || row >= length_) PLERROR("In LemmatizeVMatrix::getValues() : invalid row %d, length()=%d", row, length_); if(col == inputsize_-1) { TVec<string> options(0); lemma_dict->getValues(options,values); } else source->getValues(row,col - (col < inputsize_-1?0:1),values); }
void PLearn::LemmatizeVMatrix::getValues | ( | const Vec & | input, |
int | col, | ||
Vec & | values | ||
) | const [virtual] |
Gives the possible values of a certain field (column) given the input.
Reimplemented from PLearn::SourceVMatrix.
Definition at line 283 of file LemmatizeVMatrix.cc.
References PLearn::VMatrix::inputsize_, lemma_dict, and PLearn::SourceVMatrix::source.
{ if(col == inputsize_-1) { TVec<string> options(0); lemma_dict->getValues(options,values); } else source->getValues(input,col - (col < inputsize_-1?0:1),values); }
void PLearn::LemmatizeVMatrix::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::SourceVMatrix.
Definition at line 299 of file LemmatizeVMatrix.cc.
References PLearn::deepCopyField(), lemma_dict, PLearn::SourceVMatrix::makeDeepCopyFromShallowCopy(), pos_to_lemma_table, src_row, word_pos_to_lemma_table, and word_to_lemma_table.
{ inherited::makeDeepCopyFromShallowCopy(copies); deepCopyField(pos_to_lemma_table, copies); deepCopyField(word_to_lemma_table, copies); deepCopyField(word_pos_to_lemma_table, copies); deepCopyField(lemma_dict, copies); deepCopyField(src_row, copies); //PLERROR("LemmatizeVMatrix::makeDeepCopyFromShallowCopy not fully (correctly) implemented yet!"); }
Reimplemented from PLearn::SourceVMatrix.
Definition at line 85 of file LemmatizeVMatrix.h.
PP<Dictionary> PLearn::LemmatizeVMatrix::lemma_dict [protected] |
Lemma Dictionary, constructed automatically.
Definition at line 109 of file LemmatizeVMatrix.h.
Referenced by build_(), declareOptions(), getDictionary(), getNewRow(), getStringVal(), getValString(), getValues(), LemmatizeVMatrix(), and makeDeepCopyFromShallowCopy().
Index (position) of POS field.
Definition at line 70 of file LemmatizeVMatrix.h.
Referenced by build_(), declareOptions(), and getLemma().
Customized table that uses the POS tag to obtain a lemma It must have exactly two columns [POS lemma].
It has the third priority.
Definition at line 73 of file LemmatizeVMatrix.h.
Referenced by build_(), declareOptions(), getLemma(), and makeDeepCopyFromShallowCopy().
Vec PLearn::LemmatizeVMatrix::src_row [protected] |
Temporary variable for source row.
Definition at line 111 of file LemmatizeVMatrix.h.
Referenced by build_(), getNewRow(), and makeDeepCopyFromShallowCopy().
Index (position) of word field.
Definition at line 68 of file LemmatizeVMatrix.h.
Referenced by build_(), declareOptions(), and getLemma().
Customized table that uses the word and POS tag to obtain a lemma It must have exactly three columns [word POS lemma].
It has the first priority.
Definition at line 79 of file LemmatizeVMatrix.h.
Referenced by build_(), declareOptions(), getLemma(), and makeDeepCopyFromShallowCopy().
Customized table that uses the word to obtain a lemma It must have exactly two columns [word lemma].
It has the second priority.
Definition at line 76 of file LemmatizeVMatrix.h.
Referenced by build_(), declareOptions(), getLemma(), and makeDeepCopyFromShallowCopy().