PLearn 0.1
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Protected Member Functions | Static Protected Member Functions | Protected Attributes | Private Types | Private Member Functions
PLearn::LemmatizeVMatrix Class Reference

Takes a VMatrix with a word and a POS field and adds a field consisting of the lemma form of the word. More...

#include <LemmatizeVMatrix.h>

Inheritance diagram for PLearn::LemmatizeVMatrix:
Inheritance graph
[legend]
Collaboration diagram for PLearn::LemmatizeVMatrix:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 LemmatizeVMatrix ()
 Default constructor.
virtual string classname () const
virtual OptionListgetOptionList () const
virtual OptionMapgetOptionMap () const
virtual RemoteMethodMapgetRemoteMethodMap () const
virtual LemmatizeVMatrixdeepCopy (CopiesMap &copies) const
virtual void build ()
 Simply calls inherited::build() then build_().
virtual void makeDeepCopyFromShallowCopy (CopiesMap &copies)
 Transforms a shallow copy into a deep copy.
virtual real getStringVal (int col, const string &str) const
 returns value associated with a string (or MISSING_VALUE if there's no association for this string)
virtual string getValString (int col, real val) const
 Returns the string associated with value val for field# col.
virtual void getValues (int row, int col, Vec &values) const
 Gives the possible values for a certain field in the VMatrix.
virtual void getValues (const Vec &input, int col, Vec &values) const
 Gives the possible values of a certain field (column) given the input.
virtual PP< DictionarygetDictionary (int col) const
 Get Dictionary from a certain column.

Static Public Member Functions

static string _classname_ ()
 Declares name and deepCopy methods.
static OptionList_getOptionList_ ()
static RemoteMethodMap_getRemoteMethodMap_ ()
static Object_new_instance_for_typemap_ ()
static bool _isa_ (const Object *o)
static void _static_initialize_ ()
static const PPathdeclaringFile ()

Public Attributes

int word_field
 Index (position) of word field.
int pos_field
 Index (position) of POS field.
TMat< string > pos_to_lemma_table
 Customized table that uses the POS tag to obtain a lemma It must have exactly two columns [POS lemma].
TMat< string > word_to_lemma_table
 Customized table that uses the word to obtain a lemma It must have exactly two columns [word lemma].
TMat< string > word_pos_to_lemma_table
 Customized table that uses the word and POS tag to obtain a lemma It must have exactly three columns [word POS lemma].

Static Public Attributes

static StaticInitializer _static_initializer_

Protected Member Functions

virtual void getNewRow (int i, const Vec &v) const
 Fill the vector 'v' with the content of the i-th row.

Static Protected Member Functions

static void declareOptions (OptionList &ol)
 Declares the class options.

Protected Attributes

PP< Dictionarylemma_dict
 Lemma Dictionary, constructed automatically.
Vec src_row
 Temporary variable for source row.

Private Types

typedef SourceVMatrix inherited

Private Member Functions

void build_ ()
 This does the actual building.
string getLemma (int row) const
 Gives the lemma for a certain row.

Detailed Description

Takes a VMatrix with a word and a POS field and adds a field consisting of the lemma form of the word.

This field becomes the last input field. First, customized mapping tables (with different priorities) are used, and finally, if the previous step was unsuccesful, WordNet is used to obtain a lemma.

Definition at line 62 of file LemmatizeVMatrix.h.


Member Typedef Documentation

Reimplemented from PLearn::SourceVMatrix.

Definition at line 64 of file LemmatizeVMatrix.h.


Constructor & Destructor Documentation

PLearn::LemmatizeVMatrix::LemmatizeVMatrix ( )

Default constructor.

Definition at line 62 of file LemmatizeVMatrix.cc.

References lemma_dict, and PLERROR.

    : word_field(-1), pos_field(-1)
{
    lemma_dict = new Dictionary();

    if(morphinit<0)
        PLERROR("In WordNetSenseDictionary(): could not open WordNet exception list files");
//    if(wninit()<0)
//        PLERROR("In WordNetSenseDictionary(): could not open WordNet database files");
}

Member Function Documentation

string PLearn::LemmatizeVMatrix::_classname_ ( ) [static]

Declares name and deepCopy methods.

Reimplemented from PLearn::SourceVMatrix.

Definition at line 57 of file LemmatizeVMatrix.cc.

OptionList & PLearn::LemmatizeVMatrix::_getOptionList_ ( ) [static]

Reimplemented from PLearn::SourceVMatrix.

Definition at line 57 of file LemmatizeVMatrix.cc.

RemoteMethodMap & PLearn::LemmatizeVMatrix::_getRemoteMethodMap_ ( ) [static]

Reimplemented from PLearn::SourceVMatrix.

Definition at line 57 of file LemmatizeVMatrix.cc.

bool PLearn::LemmatizeVMatrix::_isa_ ( const Object o) [static]

Reimplemented from PLearn::SourceVMatrix.

Definition at line 57 of file LemmatizeVMatrix.cc.

Object * PLearn::LemmatizeVMatrix::_new_instance_for_typemap_ ( ) [static]

Reimplemented from PLearn::SourceVMatrix.

Definition at line 57 of file LemmatizeVMatrix.cc.

StaticInitializer LemmatizeVMatrix::_static_initializer_ & PLearn::LemmatizeVMatrix::_static_initialize_ ( ) [static]

Reimplemented from PLearn::SourceVMatrix.

Definition at line 57 of file LemmatizeVMatrix.cc.

void PLearn::LemmatizeVMatrix::build ( ) [virtual]

Simply calls inherited::build() then build_().

Reimplemented from PLearn::SourceVMatrix.

Definition at line 190 of file LemmatizeVMatrix.cc.

References PLearn::SourceVMatrix::build(), and build_().

Here is the call graph for this function:

void PLearn::LemmatizeVMatrix::build_ ( ) [private]

This does the actual building.

Reimplemented from PLearn::SourceVMatrix.

Definition at line 199 of file LemmatizeVMatrix.cc.

References getLemma(), i, PLearn::VMatrix::inputsize_, lemma_dict, PLearn::TMat< T >::length(), PLearn::VMat::length(), PLearn::VMatrix::length_, NO_UPDATE, PLERROR, pos_field, pos_to_lemma_table, PLearn::TVec< T >::resize(), PLearn::SourceVMatrix::source, src_row, PLearn::VMatrix::targetsize_, PLearn::VMatrix::updateMtime(), PLearn::VMatrix::weightsize_, PLearn::VMat::width(), PLearn::TMat< T >::width(), PLearn::VMatrix::width_, word_field, word_pos_to_lemma_table, and word_to_lemma_table.

Referenced by build().

{
    if(source)
    {
        if(word_field < 0 || word_field >= source->inputsize())
            PLERROR("In LemmatizeVMatrix::build_(): word_field(%d) is not in input part of source VMatrix", word_field);
        if(pos_field < 0 || pos_field >= source->inputsize())
            PLERROR("In LemmatizeVMatrix::build_(): pos_field(%d) is not in input part of source VMatrix", pos_field);
        inputsize_ = source->inputsize() + 1;
        targetsize_ = source->targetsize();
        weightsize_ = source->weightsize();
        width_ = inputsize_+targetsize_+weightsize_;
        length_ = source->length();
        updateMtime(source);
        if(word_pos_to_lemma_table.length() != 0 && word_pos_to_lemma_table.width() != 3)
           PLERROR("In LemmatizeVMatrix::build_(): word_pos_to_lemma_table doesn't have three columns");
        if(word_to_lemma_table.length() != 0 && word_to_lemma_table.width() != 2)
           PLERROR("In LemmatizeVMatrix::build_(): word_to_lemma_table doesn't have two columns");
        if(pos_to_lemma_table.length() != 0 && pos_to_lemma_table.width() != 2)
           PLERROR("In LemmatizeVMatrix::build_(): pos_to_lemma_table doesn't have two columns");

        string lemma;
        for(int i=0; i<length_;i++)
        {
            lemma = getLemma(i);
            lemma_dict->getId(lemma);
        }
        lemma_dict->update_mode = NO_UPDATE;
        src_row.resize(source->width());
    }
}

Here is the call graph for this function:

Here is the caller graph for this function:

string PLearn::LemmatizeVMatrix::classname ( ) const [virtual]

Reimplemented from PLearn::SourceVMatrix.

Definition at line 57 of file LemmatizeVMatrix.cc.

void PLearn::LemmatizeVMatrix::declareOptions ( OptionList ol) [static, protected]

Declares the class options.

Reimplemented from PLearn::SourceVMatrix.

Definition at line 165 of file LemmatizeVMatrix.cc.

References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::SourceVMatrix::declareOptions(), PLearn::OptionBase::learntoption, lemma_dict, pos_field, pos_to_lemma_table, word_field, word_pos_to_lemma_table, and word_to_lemma_table.

{
    declareOption(ol, "lemma_dict", &LemmatizeVMatrix::lemma_dict, OptionBase::learntoption,
                  "Dictionary for the lemma field\n");
    declareOption(ol, "word_field", &LemmatizeVMatrix::word_field, OptionBase::buildoption,
                  "Index (position) of word field\n");
    declareOption(ol, "pos_field", &LemmatizeVMatrix::pos_field, OptionBase::buildoption,
                  "Index (position) of POS field\n");
    declareOption(ol, "word_pos_to_lemma_table", &LemmatizeVMatrix::word_pos_to_lemma_table, OptionBase::buildoption,
                  "Customized table that uses the word and POS tag to obtain a lemma\n"
                  "It must have exactly three columns [word POS lemma]. It has the first priority.\n");
    declareOption(ol, "word_to_lemma_table", &LemmatizeVMatrix::word_to_lemma_table, OptionBase::buildoption,
                  "Customized table that uses the word to obtain a lemma\n"
                  "It must have exactly two columns [word lemma]. It has the second priority.\n");
    declareOption(ol, "pos_to_lemma_table", &LemmatizeVMatrix::pos_to_lemma_table, OptionBase::buildoption,
                  "Customized table that uses the POS tag to obtain a lemma\n"
                  "It must have exactly two columns [POS lemma] . It has the third priority.\n");

    // Now call the parent class' declareOptions
    inherited::declareOptions(ol);
}

Here is the call graph for this function:

static const PPath& PLearn::LemmatizeVMatrix::declaringFile ( ) [inline, static]

Reimplemented from PLearn::SourceVMatrix.

Definition at line 85 of file LemmatizeVMatrix.h.

:
LemmatizeVMatrix * PLearn::LemmatizeVMatrix::deepCopy ( CopiesMap copies) const [virtual]

Reimplemented from PLearn::SourceVMatrix.

Definition at line 57 of file LemmatizeVMatrix.cc.

PP< Dictionary > PLearn::LemmatizeVMatrix::getDictionary ( int  col) const [virtual]

Get Dictionary from a certain column.

Reimplemented from PLearn::SourceVMatrix.

Definition at line 263 of file LemmatizeVMatrix.cc.

References PLearn::VMatrix::inputsize_, lemma_dict, and PLearn::SourceVMatrix::source.

{
    if(col == inputsize_-1)
        return lemma_dict;
    return source->getDictionary(col - (col < inputsize_-1?0:1));
}
string PLearn::LemmatizeVMatrix::getLemma ( int  row) const [private]

Gives the lemma for a certain row.

Definition at line 74 of file LemmatizeVMatrix.cc.

References i, PLearn::TMat< T >::length(), PLearn::lowerstring(), pos_field, pos_to_lemma_table, PLearn::SourceVMatrix::source, PLearn::stemWord(), word_field, word_pos_to_lemma_table, and word_to_lemma_table.

Referenced by build_(), and getNewRow().

{
    string pos;
    string word;

    word = lowerstring(source->getString(row,word_field));
    pos = source->getString(row,pos_field);

    // Verify if POS tag is compatible for word, using WordNet
    // wninit() most be called for that...
   /*
    TVec<string> ret_noun(0);
    TVec<string> ret_verb(0);
    TVec<string> ret_adj(0);
    TVec<string> ret_adv(0);
    TVec<string> stems;
    if(strstr(pos.c_str(),"NN"))
    {
        ret_noun.append(extractSenses(word,NOUN,"sense_key"));
        stems = stemsOfWord(word,NOUN);
        for(int i=0; i<stems.length(); i++)
            if(word != stems[i])
                ret_noun.append(extractSenses(stems[i],NOUN,"sense_key"));
    }

    if(strstr(pos.c_str(),"VB"))
    {
        ret_verb.append(extractSenses(word,VERB,"sense_key"));
        stems = stemsOfWord(word,VERB);
        for(int i=0; i<stems.length(); i++)
            if(word != stems[i])
                ret_verb.append(extractSenses(stems[i],VERB,"sense_key"));
    }

    if(strstr(pos.c_str(),"JJ"))
    {
        ret_adj.append(extractSenses(word,ADJ,"sense_key"));
        stems = stemsOfWord(word,ADJ);
        for(int i=0; i<stems.length(); i++)
            if(word != stems[i])
                ret_adj.append(extractSenses(stems[i],ADJ,"sense_key"));
    }

    if(strstr(pos.c_str(),"RB"))
    {
        ret_adv.append(extractSenses(word,ADV,"sense_key"));
        stems = stemsOfWord(word,ADV);
        for(int i=0; i<stems.length(); i++)
            if(word != stems[i])
                ret_adv.append(extractSenses(stems[i],ADV,"sense_key"));
    }

    if(!(ret_noun.length() == 0 && ret_verb.length() == 0 && ret_adj.length() == 0 && ret_adv.length() == 0))
    {
        if(strstr(pos.c_str(),"NN") && ret_noun.length() == 0)
            PLWARNING("In LemmatizeVMatrix::getLemma(): word %s cannot be a noun", word.c_str());
        if(strstr(pos.c_str(),"VB") && ret_verb.length() == 0)
            PLWARNING("In LemmatizeVMatrix::getLemma(): word %s cannot be a verb", word.c_str());
        if(strstr(pos.c_str(),"JJ") && ret_adj.length() == 0)
            PLWARNING("In LemmatizeVMatrix::getLemma(): word %s cannot be an adjective", word.c_str());
        if(strstr(pos.c_str(),"RB") && ret_adv.length() == 0)
            PLWARNING("In LemmatizeVMatrix::getLemma(): word %s cannot be an adverb", word.c_str());
    }
    */

    // Use word_pos_to_lemma_table
    for(int i=0; i<word_pos_to_lemma_table.length(); i++)
        if(word_pos_to_lemma_table(i,0) == word && word_pos_to_lemma_table(i,1) == pos)
            return word_pos_to_lemma_table(i,2);

    // Use word_to_lemma_table
    for(int i=0; i<word_to_lemma_table.length(); i++)
        if(word_to_lemma_table(i,0) == word)
            return word_to_lemma_table(i,1);

    // Use pos_to_lemma_table
    for(int i=0; i<pos_to_lemma_table.length(); i++)
        if(pos_to_lemma_table(i,0) == pos)
            return pos_to_lemma_table(i,1);

    if(strstr(pos.c_str(),"NN")) return stemWord(word,NOUN);
    else if(strstr(pos.c_str(),"VB")) return stemWord(word,VERB);
    else if(strstr(pos.c_str(),"JJ")) return stemWord(word,ADJ);
    else if(strstr(pos.c_str(),"RB")) return stemWord(word,ADV);

    return word;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::LemmatizeVMatrix::getNewRow ( int  i,
const Vec v 
) const [protected, virtual]

Fill the vector 'v' with the content of the i-th row.

v is assumed to be the right size.

Reimplemented from PLearn::SourceVMatrix.

Definition at line 234 of file LemmatizeVMatrix.cc.

References getLemma(), PLearn::VMatrix::inputsize_, lemma_dict, PLearn::SourceVMatrix::source, src_row, PLearn::TVec< T >::subVec(), PLearn::VMatrix::targetsize_, and PLearn::VMatrix::weightsize_.

{
    source->getRow(i,src_row);
    v.subVec(0,inputsize_-1) << src_row.subVec(0,inputsize_-1);
    v[inputsize_-1] = lemma_dict->getId(getLemma(i));
    v.subVec(inputsize_,targetsize_+weightsize_) << src_row.subVec(inputsize_-1,targetsize_+weightsize_);
}

Here is the call graph for this function:

OptionList & PLearn::LemmatizeVMatrix::getOptionList ( ) const [virtual]

Reimplemented from PLearn::SourceVMatrix.

Definition at line 57 of file LemmatizeVMatrix.cc.

OptionMap & PLearn::LemmatizeVMatrix::getOptionMap ( ) const [virtual]

Reimplemented from PLearn::SourceVMatrix.

Definition at line 57 of file LemmatizeVMatrix.cc.

RemoteMethodMap & PLearn::LemmatizeVMatrix::getRemoteMethodMap ( ) const [virtual]

Reimplemented from PLearn::SourceVMatrix.

Definition at line 57 of file LemmatizeVMatrix.cc.

real PLearn::LemmatizeVMatrix::getStringVal ( int  col,
const string &  str 
) const [virtual]

returns value associated with a string (or MISSING_VALUE if there's no association for this string)

Reimplemented from PLearn::VMatrix.

Definition at line 243 of file LemmatizeVMatrix.cc.

References PLearn::VMatrix::inputsize_, lemma_dict, MISSING_VALUE, and PLearn::SourceVMatrix::source.

{
    int ret;
    if(col == inputsize_-1)
    {
        ret = lemma_dict->getId(str);
        if(ret == -1) return MISSING_VALUE;
        else return ret;
    }
    return source->getStringVal(col - (col < inputsize_-1?0:1),str);
}
string PLearn::LemmatizeVMatrix::getValString ( int  col,
real  val 
) const [virtual]

Returns the string associated with value val for field# col.

Or returns "" if no string is associated.

Reimplemented from PLearn::VMatrix.

Definition at line 255 of file LemmatizeVMatrix.cc.

References PLearn::VMatrix::inputsize_, PLearn::is_missing(), lemma_dict, PLearn::SourceVMatrix::source, and PLearn::tostring().

{
    if(is_missing(val))return tostring(val);
    if(col == inputsize_-1)
        return lemma_dict->getSymbol((int)val);
    return source->getValString(col - (col < inputsize_-1?0:1),val);
}

Here is the call graph for this function:

void PLearn::LemmatizeVMatrix::getValues ( int  row,
int  col,
Vec values 
) const [virtual]

Gives the possible values for a certain field in the VMatrix.

Reimplemented from PLearn::SourceVMatrix.

Definition at line 271 of file LemmatizeVMatrix.cc.

References PLearn::VMatrix::inputsize_, lemma_dict, PLearn::VMatrix::length_, PLERROR, and PLearn::SourceVMatrix::source.

{
    if(row < 0 || row >= length_) PLERROR("In LemmatizeVMatrix::getValues() : invalid row %d, length()=%d", row, length_);
    if(col == inputsize_-1)
    {
        TVec<string> options(0);
        lemma_dict->getValues(options,values);
    }
    else
        source->getValues(row,col - (col < inputsize_-1?0:1),values);
}
void PLearn::LemmatizeVMatrix::getValues ( const Vec input,
int  col,
Vec values 
) const [virtual]

Gives the possible values of a certain field (column) given the input.

Reimplemented from PLearn::SourceVMatrix.

Definition at line 283 of file LemmatizeVMatrix.cc.

References PLearn::VMatrix::inputsize_, lemma_dict, and PLearn::SourceVMatrix::source.

{
    if(col == inputsize_-1)
    {
        TVec<string> options(0);
        lemma_dict->getValues(options,values);
    }
    else
        source->getValues(input,col - (col < inputsize_-1?0:1),values);
}
void PLearn::LemmatizeVMatrix::makeDeepCopyFromShallowCopy ( CopiesMap copies) [virtual]

Transforms a shallow copy into a deep copy.

Reimplemented from PLearn::SourceVMatrix.

Definition at line 299 of file LemmatizeVMatrix.cc.

References PLearn::deepCopyField(), lemma_dict, PLearn::SourceVMatrix::makeDeepCopyFromShallowCopy(), pos_to_lemma_table, src_row, word_pos_to_lemma_table, and word_to_lemma_table.

{
    inherited::makeDeepCopyFromShallowCopy(copies);
    deepCopyField(pos_to_lemma_table, copies);
    deepCopyField(word_to_lemma_table, copies);
    deepCopyField(word_pos_to_lemma_table, copies);
    deepCopyField(lemma_dict, copies);
    deepCopyField(src_row, copies);

    //PLERROR("LemmatizeVMatrix::makeDeepCopyFromShallowCopy not fully (correctly) implemented yet!");
}

Here is the call graph for this function:


Member Data Documentation

Reimplemented from PLearn::SourceVMatrix.

Definition at line 85 of file LemmatizeVMatrix.h.

Index (position) of POS field.

Definition at line 70 of file LemmatizeVMatrix.h.

Referenced by build_(), declareOptions(), and getLemma().

Customized table that uses the POS tag to obtain a lemma It must have exactly two columns [POS lemma].

It has the third priority.

Definition at line 73 of file LemmatizeVMatrix.h.

Referenced by build_(), declareOptions(), getLemma(), and makeDeepCopyFromShallowCopy().

Temporary variable for source row.

Definition at line 111 of file LemmatizeVMatrix.h.

Referenced by build_(), getNewRow(), and makeDeepCopyFromShallowCopy().

Index (position) of word field.

Definition at line 68 of file LemmatizeVMatrix.h.

Referenced by build_(), declareOptions(), and getLemma().

Customized table that uses the word and POS tag to obtain a lemma It must have exactly three columns [word POS lemma].

It has the first priority.

Definition at line 79 of file LemmatizeVMatrix.h.

Referenced by build_(), declareOptions(), getLemma(), and makeDeepCopyFromShallowCopy().

Customized table that uses the word to obtain a lemma It must have exactly two columns [word lemma].

It has the second priority.

Definition at line 76 of file LemmatizeVMatrix.h.

Referenced by build_(), declareOptions(), getLemma(), and makeDeepCopyFromShallowCopy().


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines