PLearn 0.1
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Protected Member Functions | Static Protected Member Functions | Protected Attributes | Private Types | Private Member Functions
PLearn::PythonProcessedLearner Class Reference

Allows preprocessing operations to be carried out by a Python code snippet. More...

#include <PythonProcessedLearner.h>

Inheritance diagram for PLearn::PythonProcessedLearner:
Inheritance graph
[legend]
Collaboration diagram for PLearn::PythonProcessedLearner:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 PythonProcessedLearner ()
 Default constructor.
virtual void setTrainingSet (VMat training_set, bool call_forget=true)
 Calls the Python getOutputNames() function and remember the result.
virtual int outputsize () const
 Returns the size of this learner's output, (which typically may depend on its inputsize(), targetsize() and set options).
virtual void forget ()
 Currently a no-op.
virtual void train ()
 Currently a no-op.
virtual void computeOutput (const Vec &input, Vec &output) const
 Computes the output from the input: actually calls Python.
virtual void computeCostsFromOutputs (const Vec &input, const Vec &output, const Vec &target, Vec &costs) const
 Computes the costs from already computed output: currently a no-op.
virtual TVec< std::string > getTestCostNames () const
 Empty vector: no test costs are defined.
virtual TVec< std::string > getTrainCostNames () const
 Empty vector: no train costs are defined.
virtual TVec< string > getOutputNames () const
 Returns the output names cached from the last setTrainingSet()
virtual string classname () const
virtual OptionListgetOptionList () const
virtual OptionMapgetOptionMap () const
virtual RemoteMethodMapgetRemoteMethodMap () const
virtual PythonProcessedLearnerdeepCopy (CopiesMap &copies) const
virtual void build ()
 Finish building the object; just call inherited::build followed by build_()
virtual void makeDeepCopyFromShallowCopy (CopiesMap &copies)
 Transforms a shallow copy into a deep copy.

Static Public Member Functions

static string _classname_ ()
static OptionList_getOptionList_ ()
static RemoteMethodMap_getRemoteMethodMap_ ()
static Object_new_instance_for_typemap_ ()
static bool _isa_ (const Object *o)
static void _static_initialize_ ()
static const PPathdeclaringFile ()

Public Attributes

string m_code
 The Python code snippet.
map< string, string > m_params
 General-purpose parameters that are injected into the Python code snippet and accessible via the getParam/setParam functions.

Static Public Attributes

static StaticInitializer _static_initializer_

Protected Member Functions

PythonObjectWrapper getParams (const TVec< PythonObjectWrapper > &args) const
 Injected into the Python code to return m_params.
PythonObjectWrapper getParam (const TVec< PythonObjectWrapper > &args) const
 Injected into the Python code to return the m_param value associated to the given key (sole argument).
PythonObjectWrapper setParam (const TVec< PythonObjectWrapper > &args)
 Injected into the Python code to set the m_param value associated to the given key.
void compileAndInject ()
 If not already done, compile the Python snippet and inject the required stuff into the Python environment.
void setOutputNamesFromParams ()
 Sets the outputnames from the fieldnames/*size stored in params.

Static Protected Member Functions

static void declareOptions (OptionList &ol)
 Declares the class options.

Protected Attributes

TVec< string > m_outputnames
 Cached version of the output names.
PP< PythonCodeSnippetpython
 Actual Python environment.

Private Types

typedef PLearner inherited

Private Member Functions

void build_ ()
 This does the actual building.

Detailed Description

Allows preprocessing operations to be carried out by a Python code snippet.

This PLearner allows embedding a PythonCodeSnippet, and define Python operations to be carried out during the computeOutput. The current implementation does not attempt to provide a full implementability of the PLearner protocol in Python -- only computeOutput is supported for now, and the intended use is to specify some fixed preprocessing inside a ChainedLearner, such as what would otherwise be performed by a VPLPreprocessedLearner.

The Python code snippet (see the option 'code' below) must define the following functions:

def getOutputNames(train_fieldnames, inputsize, targetsize, weightsize, extrasize): """Return the names of the outputs computed by the learner, namely the implementation of the PLearner::getOutputNames() function. This is called every time setTrainingSet() is called on the PLearner."""

def computeOutput(input): """Return the result of the computation. The size of the output vector must be the same as the number of elements returned by the getOutputNames() function."""

The Python code snippet has access to the following (injected) interface:

NOTE ON RELOADED MODELS. If a PythonProcessedLearner is saved to disk and later reloaded, it is likely that a setTrainingSet will not be called on the model (which would establish the outputsize through a call to getOutputNames). If this happens, the PythonProcessedLearner looks in its 'params' to locate the following entries:

If it can find them all, the outputsize() function automatically invokes getOutputNames to establish the proper dimensions of the learner. Note that your Python code should normally manually update these options within the getOutputNames() function to ensure that they are saved with the learner after training.

Definition at line 101 of file PythonProcessedLearner.h.


Member Typedef Documentation

Reimplemented from PLearn::PLearner.

Definition at line 103 of file PythonProcessedLearner.h.


Constructor & Destructor Documentation

PLearn::PythonProcessedLearner::PythonProcessedLearner ( )

Default constructor.

Definition at line 99 of file PythonProcessedLearner.cc.

    : m_code(""),
      python()
{ }

Member Function Documentation

string PLearn::PythonProcessedLearner::_classname_ ( ) [static]

Reimplemented from PLearn::PLearner.

Definition at line 97 of file PythonProcessedLearner.cc.

OptionList & PLearn::PythonProcessedLearner::_getOptionList_ ( ) [static]

Reimplemented from PLearn::PLearner.

Definition at line 97 of file PythonProcessedLearner.cc.

RemoteMethodMap & PLearn::PythonProcessedLearner::_getRemoteMethodMap_ ( ) [static]

Reimplemented from PLearn::PLearner.

Definition at line 97 of file PythonProcessedLearner.cc.

bool PLearn::PythonProcessedLearner::_isa_ ( const Object o) [static]

Reimplemented from PLearn::PLearner.

Definition at line 97 of file PythonProcessedLearner.cc.

Object * PLearn::PythonProcessedLearner::_new_instance_for_typemap_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 97 of file PythonProcessedLearner.cc.

StaticInitializer PythonProcessedLearner::_static_initializer_ & PLearn::PythonProcessedLearner::_static_initialize_ ( ) [static]

Reimplemented from PLearn::PLearner.

Definition at line 97 of file PythonProcessedLearner.cc.

void PLearn::PythonProcessedLearner::build ( ) [virtual]

Finish building the object; just call inherited::build followed by build_()

Reimplemented from PLearn::PLearner.

Definition at line 126 of file PythonProcessedLearner.cc.

References PLearn::PLearner::build(), and build_().

Here is the call graph for this function:

void PLearn::PythonProcessedLearner::build_ ( ) [private]

This does the actual building.

Reimplemented from PLearn::PLearner.

Definition at line 132 of file PythonProcessedLearner.cc.

References compileAndInject(), PLASSERT, PLERROR, and python.

Referenced by build().

{
    // First step, compile the Python code
    compileAndInject();
    PLASSERT( python );

    // Ensure that the required functions are defined
    const char* FUNC_ERR = "%s: the Python code snippet must define the function '%s'";
    if (! python->isInvokable("getOutputNames"))
        PLERROR(FUNC_ERR, __FUNCTION__, "getOutputNames");
    if (! python->isInvokable("computeOutput"))
        PLERROR(FUNC_ERR, __FUNCTION__, "computeOutput");
}

Here is the call graph for this function:

Here is the caller graph for this function:

string PLearn::PythonProcessedLearner::classname ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 97 of file PythonProcessedLearner.cc.

void PLearn::PythonProcessedLearner::compileAndInject ( ) [protected]

If not already done, compile the Python snippet and inject the required stuff into the Python environment.

Definition at line 277 of file PythonProcessedLearner.cc.

References getParam(), getParams(), m_code, PLASSERT, python, and setParam().

Referenced by build_(), and makeDeepCopyFromShallowCopy().

{
    if (! python) {
        python = new PythonCodeSnippet(m_code);
        PLASSERT( python );
        python->build();
        python->inject("getParams",    this, &PythonProcessedLearner::getParams);
        python->inject("getParam",     this, &PythonProcessedLearner::getParam);
        python->inject("setParam",     this, &PythonProcessedLearner::setParam);
    }
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::PythonProcessedLearner::computeCostsFromOutputs ( const Vec input,
const Vec output,
const Vec target,
Vec costs 
) const [virtual]

Computes the costs from already computed output: currently a no-op.

Implements PLearn::PLearner.

Definition at line 208 of file PythonProcessedLearner.cc.

{
    // No-op in current version
}
void PLearn::PythonProcessedLearner::computeOutput ( const Vec input,
Vec output 
) const [virtual]

Computes the output from the input: actually calls Python.

Reimplemented from PLearn::PLearner.

Definition at line 199 of file PythonProcessedLearner.cc.

References PLearn::Profiler::end(), PLASSERT, python, and PLearn::Profiler::start().

{
    PLASSERT( python );
    Profiler::start("PythonProcessedLearner");
    Vec processed = python->invoke("computeOutput", input).as<Vec>();
    output << processed;
    Profiler::end("PythonProcessedLearner");
}

Here is the call graph for this function:

void PLearn::PythonProcessedLearner::declareOptions ( OptionList ol) [static, protected]

Declares the class options.

Reimplemented from PLearn::PLearner.

Definition at line 104 of file PythonProcessedLearner.cc.

References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::PLearner::declareOptions(), m_code, and m_params.

{
    declareOption(
        ol, "code", &PythonProcessedLearner::m_code, OptionBase::buildoption,
        "The Python code snippet.  The functions described in the class\n"
        "documentation must be provided.  Note that, after an initial build(),\n"
        "changing this string calling build() again DOES NOT result in the\n"
        "recompilation of the code.\n");

    declareOption(
        ol, "params", &PythonProcessedLearner::m_params, OptionBase::buildoption,
        "General-purpose parameters that are injected into the Python code\n"
        "snippet and accessible via the getParam/setParam functions.  Can be\n"
        "used for passing processing arguments to the Python code.\n");
    
    // Now call the parent class' declareOptions
    inherited::declareOptions(ol);
}

Here is the call graph for this function:

static const PPath& PLearn::PythonProcessedLearner::declaringFile ( ) [inline, static]

Reimplemented from PLearn::PLearner.

Definition at line 165 of file PythonProcessedLearner.h.

:
PythonProcessedLearner * PLearn::PythonProcessedLearner::deepCopy ( CopiesMap copies) const [virtual]

Reimplemented from PLearn::PLearner.

Definition at line 97 of file PythonProcessedLearner.cc.

void PLearn::PythonProcessedLearner::forget ( ) [virtual]

Currently a no-op.

Reimplemented from PLearn::PLearner.

Definition at line 188 of file PythonProcessedLearner.cc.

{
    // no-op in current version
}
OptionList & PLearn::PythonProcessedLearner::getOptionList ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 97 of file PythonProcessedLearner.cc.

OptionMap & PLearn::PythonProcessedLearner::getOptionMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 97 of file PythonProcessedLearner.cc.

TVec< string > PLearn::PythonProcessedLearner::getOutputNames ( ) const [virtual]

Returns the output names cached from the last setTrainingSet()

Reimplemented from PLearn::PLearner.

Definition at line 224 of file PythonProcessedLearner.cc.

References m_outputnames.

{
    return m_outputnames;
}
PythonObjectWrapper PLearn::PythonProcessedLearner::getParam ( const TVec< PythonObjectWrapper > &  args) const [protected]

Injected into the Python code to return the m_param value associated to the given key (sole argument).

Returns None if not found.

Definition at line 246 of file PythonProcessedLearner.cc.

References m_params, PLERROR, and PLearn::TVec< T >::size().

Referenced by compileAndInject().

{
    if (args.size() != 1)
        PLERROR("PythonProcessedLearner::getParam: expected 1 argument; got %d",
                args.size());
    string key = args[0].as<string>();
    map<string,string>::const_iterator found = m_params.find(key);
    if (found != m_params.end())
        return PythonObjectWrapper(found->second);
    else
        return PythonObjectWrapper();        // None
}

Here is the call graph for this function:

Here is the caller graph for this function:

PythonObjectWrapper PLearn::PythonProcessedLearner::getParams ( const TVec< PythonObjectWrapper > &  args) const [protected]

Injected into the Python code to return m_params.

Definition at line 233 of file PythonProcessedLearner.cc.

References m_params, PLERROR, and PLearn::TVec< T >::size().

Referenced by compileAndInject().

{
    if (args.size() != 0)
        PLERROR("PythonProcessedLearner::getParams: expected 0 argument; got %d",
                args.size());

    return PythonObjectWrapper(m_params);
}

Here is the call graph for this function:

Here is the caller graph for this function:

RemoteMethodMap & PLearn::PythonProcessedLearner::getRemoteMethodMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 97 of file PythonProcessedLearner.cc.

TVec< string > PLearn::PythonProcessedLearner::getTestCostNames ( ) const [virtual]

Empty vector: no test costs are defined.

Implements PLearn::PLearner.

Definition at line 214 of file PythonProcessedLearner.cc.

{
    return TVec<string>();
}
TVec< string > PLearn::PythonProcessedLearner::getTrainCostNames ( ) const [virtual]

Empty vector: no train costs are defined.

Implements PLearn::PLearner.

Definition at line 219 of file PythonProcessedLearner.cc.

{
    return TVec<string>();
}
void PLearn::PythonProcessedLearner::makeDeepCopyFromShallowCopy ( CopiesMap copies) [virtual]

Transforms a shallow copy into a deep copy.

Reimplemented from PLearn::PLearner.

Definition at line 147 of file PythonProcessedLearner.cc.

References compileAndInject(), PLearn::deepCopyField(), m_outputnames, PLearn::PLearner::makeDeepCopyFromShallowCopy(), and python.

{
    inherited::makeDeepCopyFromShallowCopy(copies);

    deepCopyField(m_outputnames, copies);

    // Recompile the Python code in a fresh environment
    python = 0;
    compileAndInject();
}

Here is the call graph for this function:

int PLearn::PythonProcessedLearner::outputsize ( ) const [virtual]

Returns the size of this learner's output, (which typically may depend on its inputsize(), targetsize() and set options).

Implements PLearn::PLearner.

Definition at line 177 of file PythonProcessedLearner.cc.

References m_outputnames, PLASSERT, python, and PLearn::TVec< T >::size().

{
    PLASSERT( python );

    // Reloaded model for which we have not yet established the outputsize
    if (m_outputnames.size() == 0)
        const_cast<PythonProcessedLearner*>(this)->setOutputNamesFromParams();
    
    return m_outputnames.size();
}

Here is the call graph for this function:

void PLearn::PythonProcessedLearner::setOutputNamesFromParams ( ) [protected]

Sets the outputnames from the fieldnames/*size stored in params.

Definition at line 292 of file PythonProcessedLearner.cc.

References PLearn::PLearner::inputsize(), PLearn::lexical_cast(), m_outputnames, m_params, PLASSERT, PLERROR, python, PLearn::search_replace(), PLearn::PLearner::targetsize(), and PLearn::PLearner::weightsize().

{
    PLASSERT( python );
    map<string,string>::iterator NOT_FOUND = m_params.end();
    if (m_params.find("fieldnames") == NOT_FOUND ||
        m_params.find("inputsize")  == NOT_FOUND ||
        m_params.find("targetsize") == NOT_FOUND ||
        m_params.find("weightsize") == NOT_FOUND ||
        m_params.find("extrasize")  == NOT_FOUND )
        PLERROR("%s: Cannot find one of {'fieldnames','inputsize','targetsize',\n"
                "'weightsize','extrasize'} in params in order to set outputnames\n"
                "from params", __FUNCTION__);

    string fieldnames = m_params["fieldnames"];
    search_replace(fieldnames, "'", "\"");
    TVec<string> fields = lexical_cast< TVec<string> >(fieldnames);
    int inputsize       = lexical_cast<int>(m_params["inputsize" ]);
    int targetsize      = lexical_cast<int>(m_params["targetsize"]);
    int weightsize      = lexical_cast<int>(m_params["weightsize"]);
    int extrasize       = lexical_cast<int>(m_params["extrasize" ]);

    m_outputnames = python->invoke("getOutputNames", fields, inputsize, targetsize,
                                   weightsize, extrasize).as< TVec<string> >();
}

Here is the call graph for this function:

PythonObjectWrapper PLearn::PythonProcessedLearner::setParam ( const TVec< PythonObjectWrapper > &  args) [protected]

Injected into the Python code to set the m_param value associated to the given key.

Always return None.

Definition at line 263 of file PythonProcessedLearner.cc.

References m_params, PLERROR, and PLearn::TVec< T >::size().

Referenced by compileAndInject().

{
    if (args.size() != 2)
        PLERROR("PythonProcessedLearner::setParam: expected 2 arguments; got %d",
                args.size());
    string key   = args[0].as<string>();
    string value = args[1].as<string>();
    m_params[key] = value;
    return PythonObjectWrapper();
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::PythonProcessedLearner::setTrainingSet ( VMat  training_set,
bool  call_forget = true 
) [virtual]

Calls the Python getOutputNames() function and remember the result.

Reimplemented from PLearn::PLearner.

Definition at line 159 of file PythonProcessedLearner.cc.

References PLearn::PLearner::getTrainingSet(), PLearn::PLearner::inputsize(), m_outputnames, PLASSERT, python, PLearn::PLearner::setTrainingSet(), PLearn::PLearner::targetsize(), and PLearn::PLearner::weightsize().

{
    inherited::setTrainingSet(training_set, call_forget);
    VMat trset = getTrainingSet();
    PLASSERT( trset  );
    PLASSERT( python );
    
    TVec<string> fields = trset->fieldNames();
    int inputsize  = trset->inputsize();
    int targetsize = trset->targetsize();
    int weightsize = trset->weightsize();
    int extrasize  = trset->extrasize();

    m_outputnames = python->invoke("getOutputNames", fields, inputsize, targetsize,
                                   weightsize, extrasize).as< TVec<string> >();
}

Here is the call graph for this function:

void PLearn::PythonProcessedLearner::train ( ) [virtual]

Currently a no-op.

Implements PLearn::PLearner.

Definition at line 193 of file PythonProcessedLearner.cc.

{
    // No-op in current version
}

Member Data Documentation

Reimplemented from PLearn::PLearner.

Definition at line 165 of file PythonProcessedLearner.h.

The Python code snippet.

The functions described in the class documentation must be provided. Note that, after an initial build(), changing this string calling build() again DOES NOT result in the recompilation of the code.

Definition at line 114 of file PythonProcessedLearner.h.

Referenced by compileAndInject(), and declareOptions().

Cached version of the output names.

Definition at line 197 of file PythonProcessedLearner.h.

Referenced by getOutputNames(), makeDeepCopyFromShallowCopy(), outputsize(), setOutputNamesFromParams(), and setTrainingSet().

General-purpose parameters that are injected into the Python code snippet and accessible via the getParam/setParam functions.

Can be used for passing processing arguments to the Python code.

Definition at line 121 of file PythonProcessedLearner.h.

Referenced by declareOptions(), getParam(), getParams(), setOutputNamesFromParams(), and setParam().


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines