PLearn 0.1
|
Allows preprocessing operations to be carried out by a Python code snippet. More...
#include <PythonProcessedLearner.h>
Public Member Functions | |
PythonProcessedLearner () | |
Default constructor. | |
virtual void | setTrainingSet (VMat training_set, bool call_forget=true) |
Calls the Python getOutputNames() function and remember the result. | |
virtual int | outputsize () const |
Returns the size of this learner's output, (which typically may depend on its inputsize(), targetsize() and set options). | |
virtual void | forget () |
Currently a no-op. | |
virtual void | train () |
Currently a no-op. | |
virtual void | computeOutput (const Vec &input, Vec &output) const |
Computes the output from the input: actually calls Python. | |
virtual void | computeCostsFromOutputs (const Vec &input, const Vec &output, const Vec &target, Vec &costs) const |
Computes the costs from already computed output: currently a no-op. | |
virtual TVec< std::string > | getTestCostNames () const |
Empty vector: no test costs are defined. | |
virtual TVec< std::string > | getTrainCostNames () const |
Empty vector: no train costs are defined. | |
virtual TVec< string > | getOutputNames () const |
Returns the output names cached from the last setTrainingSet() | |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual PythonProcessedLearner * | deepCopy (CopiesMap &copies) const |
virtual void | build () |
Finish building the object; just call inherited::build followed by build_() | |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Transforms a shallow copy into a deep copy. | |
Static Public Member Functions | |
static string | _classname_ () |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
string | m_code |
The Python code snippet. | |
map< string, string > | m_params |
General-purpose parameters that are injected into the Python code snippet and accessible via the getParam/setParam functions. | |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Protected Member Functions | |
PythonObjectWrapper | getParams (const TVec< PythonObjectWrapper > &args) const |
Injected into the Python code to return m_params. | |
PythonObjectWrapper | getParam (const TVec< PythonObjectWrapper > &args) const |
Injected into the Python code to return the m_param value associated to the given key (sole argument). | |
PythonObjectWrapper | setParam (const TVec< PythonObjectWrapper > &args) |
Injected into the Python code to set the m_param value associated to the given key. | |
void | compileAndInject () |
If not already done, compile the Python snippet and inject the required stuff into the Python environment. | |
void | setOutputNamesFromParams () |
Sets the outputnames from the fieldnames/*size stored in params. | |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declares the class options. | |
Protected Attributes | |
TVec< string > | m_outputnames |
Cached version of the output names. | |
PP< PythonCodeSnippet > | python |
Actual Python environment. | |
Private Types | |
typedef PLearner | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. |
Allows preprocessing operations to be carried out by a Python code snippet.
This PLearner allows embedding a PythonCodeSnippet, and define Python operations to be carried out during the computeOutput. The current implementation does not attempt to provide a full implementability of the PLearner protocol in Python -- only computeOutput is supported for now, and the intended use is to specify some fixed preprocessing inside a ChainedLearner, such as what would otherwise be performed by a VPLPreprocessedLearner.
The Python code snippet (see the option 'code' below) must define the following functions:
def getOutputNames(train_fieldnames, inputsize, targetsize, weightsize, extrasize): """Return the names of the outputs computed by the learner, namely the implementation of the PLearner::getOutputNames() function. This is called every time setTrainingSet() is called on the PLearner."""
def computeOutput(input): """Return the result of the computation. The size of the output vector must be the same as the number of elements returned by the getOutputNames() function."""
The Python code snippet has access to the following (injected) interface:
NOTE ON RELOADED MODELS. If a PythonProcessedLearner is saved to disk and later reloaded, it is likely that a setTrainingSet will not be called on the model (which would establish the outputsize through a call to getOutputNames). If this happens, the PythonProcessedLearner looks in its 'params' to locate the following entries:
If it can find them all, the outputsize() function automatically invokes getOutputNames to establish the proper dimensions of the learner. Note that your Python code should normally manually update these options within the getOutputNames() function to ensure that they are saved with the learner after training.
Definition at line 101 of file PythonProcessedLearner.h.
typedef PLearner PLearn::PythonProcessedLearner::inherited [private] |
Reimplemented from PLearn::PLearner.
Definition at line 103 of file PythonProcessedLearner.h.
PLearn::PythonProcessedLearner::PythonProcessedLearner | ( | ) |
Default constructor.
Definition at line 99 of file PythonProcessedLearner.cc.
string PLearn::PythonProcessedLearner::_classname_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 97 of file PythonProcessedLearner.cc.
OptionList & PLearn::PythonProcessedLearner::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 97 of file PythonProcessedLearner.cc.
RemoteMethodMap & PLearn::PythonProcessedLearner::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 97 of file PythonProcessedLearner.cc.
Reimplemented from PLearn::PLearner.
Definition at line 97 of file PythonProcessedLearner.cc.
Object * PLearn::PythonProcessedLearner::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Definition at line 97 of file PythonProcessedLearner.cc.
StaticInitializer PythonProcessedLearner::_static_initializer_ & PLearn::PythonProcessedLearner::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::PLearner.
Definition at line 97 of file PythonProcessedLearner.cc.
void PLearn::PythonProcessedLearner::build | ( | ) | [virtual] |
Finish building the object; just call inherited::build followed by build_()
Reimplemented from PLearn::PLearner.
Definition at line 126 of file PythonProcessedLearner.cc.
References PLearn::PLearner::build(), and build_().
{ inherited::build(); build_(); }
void PLearn::PythonProcessedLearner::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::PLearner.
Definition at line 132 of file PythonProcessedLearner.cc.
References compileAndInject(), PLASSERT, PLERROR, and python.
Referenced by build().
{ // First step, compile the Python code compileAndInject(); PLASSERT( python ); // Ensure that the required functions are defined const char* FUNC_ERR = "%s: the Python code snippet must define the function '%s'"; if (! python->isInvokable("getOutputNames")) PLERROR(FUNC_ERR, __FUNCTION__, "getOutputNames"); if (! python->isInvokable("computeOutput")) PLERROR(FUNC_ERR, __FUNCTION__, "computeOutput"); }
string PLearn::PythonProcessedLearner::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 97 of file PythonProcessedLearner.cc.
void PLearn::PythonProcessedLearner::compileAndInject | ( | ) | [protected] |
If not already done, compile the Python snippet and inject the required stuff into the Python environment.
Definition at line 277 of file PythonProcessedLearner.cc.
References getParam(), getParams(), m_code, PLASSERT, python, and setParam().
Referenced by build_(), and makeDeepCopyFromShallowCopy().
{ if (! python) { python = new PythonCodeSnippet(m_code); PLASSERT( python ); python->build(); python->inject("getParams", this, &PythonProcessedLearner::getParams); python->inject("getParam", this, &PythonProcessedLearner::getParam); python->inject("setParam", this, &PythonProcessedLearner::setParam); } }
void PLearn::PythonProcessedLearner::computeCostsFromOutputs | ( | const Vec & | input, |
const Vec & | output, | ||
const Vec & | target, | ||
Vec & | costs | ||
) | const [virtual] |
Computes the costs from already computed output: currently a no-op.
Implements PLearn::PLearner.
Definition at line 208 of file PythonProcessedLearner.cc.
{
// No-op in current version
}
void PLearn::PythonProcessedLearner::computeOutput | ( | const Vec & | input, |
Vec & | output | ||
) | const [virtual] |
Computes the output from the input: actually calls Python.
Reimplemented from PLearn::PLearner.
Definition at line 199 of file PythonProcessedLearner.cc.
References PLearn::Profiler::end(), PLASSERT, python, and PLearn::Profiler::start().
{ PLASSERT( python ); Profiler::start("PythonProcessedLearner"); Vec processed = python->invoke("computeOutput", input).as<Vec>(); output << processed; Profiler::end("PythonProcessedLearner"); }
void PLearn::PythonProcessedLearner::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declares the class options.
Reimplemented from PLearn::PLearner.
Definition at line 104 of file PythonProcessedLearner.cc.
References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::PLearner::declareOptions(), m_code, and m_params.
{ declareOption( ol, "code", &PythonProcessedLearner::m_code, OptionBase::buildoption, "The Python code snippet. The functions described in the class\n" "documentation must be provided. Note that, after an initial build(),\n" "changing this string calling build() again DOES NOT result in the\n" "recompilation of the code.\n"); declareOption( ol, "params", &PythonProcessedLearner::m_params, OptionBase::buildoption, "General-purpose parameters that are injected into the Python code\n" "snippet and accessible via the getParam/setParam functions. Can be\n" "used for passing processing arguments to the Python code.\n"); // Now call the parent class' declareOptions inherited::declareOptions(ol); }
static const PPath& PLearn::PythonProcessedLearner::declaringFile | ( | ) | [inline, static] |
PythonProcessedLearner * PLearn::PythonProcessedLearner::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::PLearner.
Definition at line 97 of file PythonProcessedLearner.cc.
void PLearn::PythonProcessedLearner::forget | ( | ) | [virtual] |
Currently a no-op.
Reimplemented from PLearn::PLearner.
Definition at line 188 of file PythonProcessedLearner.cc.
{
// no-op in current version
}
OptionList & PLearn::PythonProcessedLearner::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 97 of file PythonProcessedLearner.cc.
OptionMap & PLearn::PythonProcessedLearner::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 97 of file PythonProcessedLearner.cc.
TVec< string > PLearn::PythonProcessedLearner::getOutputNames | ( | ) | const [virtual] |
Returns the output names cached from the last setTrainingSet()
Reimplemented from PLearn::PLearner.
Definition at line 224 of file PythonProcessedLearner.cc.
References m_outputnames.
{ return m_outputnames; }
PythonObjectWrapper PLearn::PythonProcessedLearner::getParam | ( | const TVec< PythonObjectWrapper > & | args | ) | const [protected] |
Injected into the Python code to return the m_param value associated to the given key (sole argument).
Returns None if not found.
Definition at line 246 of file PythonProcessedLearner.cc.
References m_params, PLERROR, and PLearn::TVec< T >::size().
Referenced by compileAndInject().
{ if (args.size() != 1) PLERROR("PythonProcessedLearner::getParam: expected 1 argument; got %d", args.size()); string key = args[0].as<string>(); map<string,string>::const_iterator found = m_params.find(key); if (found != m_params.end()) return PythonObjectWrapper(found->second); else return PythonObjectWrapper(); // None }
PythonObjectWrapper PLearn::PythonProcessedLearner::getParams | ( | const TVec< PythonObjectWrapper > & | args | ) | const [protected] |
Injected into the Python code to return m_params.
Definition at line 233 of file PythonProcessedLearner.cc.
References m_params, PLERROR, and PLearn::TVec< T >::size().
Referenced by compileAndInject().
{ if (args.size() != 0) PLERROR("PythonProcessedLearner::getParams: expected 0 argument; got %d", args.size()); return PythonObjectWrapper(m_params); }
RemoteMethodMap & PLearn::PythonProcessedLearner::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Definition at line 97 of file PythonProcessedLearner.cc.
TVec< string > PLearn::PythonProcessedLearner::getTestCostNames | ( | ) | const [virtual] |
Empty vector: no test costs are defined.
Implements PLearn::PLearner.
Definition at line 214 of file PythonProcessedLearner.cc.
{
return TVec<string>();
}
TVec< string > PLearn::PythonProcessedLearner::getTrainCostNames | ( | ) | const [virtual] |
Empty vector: no train costs are defined.
Implements PLearn::PLearner.
Definition at line 219 of file PythonProcessedLearner.cc.
{
return TVec<string>();
}
void PLearn::PythonProcessedLearner::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::PLearner.
Definition at line 147 of file PythonProcessedLearner.cc.
References compileAndInject(), PLearn::deepCopyField(), m_outputnames, PLearn::PLearner::makeDeepCopyFromShallowCopy(), and python.
{ inherited::makeDeepCopyFromShallowCopy(copies); deepCopyField(m_outputnames, copies); // Recompile the Python code in a fresh environment python = 0; compileAndInject(); }
int PLearn::PythonProcessedLearner::outputsize | ( | ) | const [virtual] |
Returns the size of this learner's output, (which typically may depend on its inputsize(), targetsize() and set options).
Implements PLearn::PLearner.
Definition at line 177 of file PythonProcessedLearner.cc.
References m_outputnames, PLASSERT, python, and PLearn::TVec< T >::size().
{ PLASSERT( python ); // Reloaded model for which we have not yet established the outputsize if (m_outputnames.size() == 0) const_cast<PythonProcessedLearner*>(this)->setOutputNamesFromParams(); return m_outputnames.size(); }
void PLearn::PythonProcessedLearner::setOutputNamesFromParams | ( | ) | [protected] |
Sets the outputnames from the fieldnames/*size stored in params.
Definition at line 292 of file PythonProcessedLearner.cc.
References PLearn::PLearner::inputsize(), PLearn::lexical_cast(), m_outputnames, m_params, PLASSERT, PLERROR, python, PLearn::search_replace(), PLearn::PLearner::targetsize(), and PLearn::PLearner::weightsize().
{ PLASSERT( python ); map<string,string>::iterator NOT_FOUND = m_params.end(); if (m_params.find("fieldnames") == NOT_FOUND || m_params.find("inputsize") == NOT_FOUND || m_params.find("targetsize") == NOT_FOUND || m_params.find("weightsize") == NOT_FOUND || m_params.find("extrasize") == NOT_FOUND ) PLERROR("%s: Cannot find one of {'fieldnames','inputsize','targetsize',\n" "'weightsize','extrasize'} in params in order to set outputnames\n" "from params", __FUNCTION__); string fieldnames = m_params["fieldnames"]; search_replace(fieldnames, "'", "\""); TVec<string> fields = lexical_cast< TVec<string> >(fieldnames); int inputsize = lexical_cast<int>(m_params["inputsize" ]); int targetsize = lexical_cast<int>(m_params["targetsize"]); int weightsize = lexical_cast<int>(m_params["weightsize"]); int extrasize = lexical_cast<int>(m_params["extrasize" ]); m_outputnames = python->invoke("getOutputNames", fields, inputsize, targetsize, weightsize, extrasize).as< TVec<string> >(); }
PythonObjectWrapper PLearn::PythonProcessedLearner::setParam | ( | const TVec< PythonObjectWrapper > & | args | ) | [protected] |
Injected into the Python code to set the m_param value associated to the given key.
Always return None.
Definition at line 263 of file PythonProcessedLearner.cc.
References m_params, PLERROR, and PLearn::TVec< T >::size().
Referenced by compileAndInject().
{ if (args.size() != 2) PLERROR("PythonProcessedLearner::setParam: expected 2 arguments; got %d", args.size()); string key = args[0].as<string>(); string value = args[1].as<string>(); m_params[key] = value; return PythonObjectWrapper(); }
void PLearn::PythonProcessedLearner::setTrainingSet | ( | VMat | training_set, |
bool | call_forget = true |
||
) | [virtual] |
Calls the Python getOutputNames() function and remember the result.
Reimplemented from PLearn::PLearner.
Definition at line 159 of file PythonProcessedLearner.cc.
References PLearn::PLearner::getTrainingSet(), PLearn::PLearner::inputsize(), m_outputnames, PLASSERT, python, PLearn::PLearner::setTrainingSet(), PLearn::PLearner::targetsize(), and PLearn::PLearner::weightsize().
{ inherited::setTrainingSet(training_set, call_forget); VMat trset = getTrainingSet(); PLASSERT( trset ); PLASSERT( python ); TVec<string> fields = trset->fieldNames(); int inputsize = trset->inputsize(); int targetsize = trset->targetsize(); int weightsize = trset->weightsize(); int extrasize = trset->extrasize(); m_outputnames = python->invoke("getOutputNames", fields, inputsize, targetsize, weightsize, extrasize).as< TVec<string> >(); }
void PLearn::PythonProcessedLearner::train | ( | ) | [virtual] |
Currently a no-op.
Implements PLearn::PLearner.
Definition at line 193 of file PythonProcessedLearner.cc.
{
// No-op in current version
}
Reimplemented from PLearn::PLearner.
Definition at line 165 of file PythonProcessedLearner.h.
The Python code snippet.
The functions described in the class documentation must be provided. Note that, after an initial build(), changing this string calling build() again DOES NOT result in the recompilation of the code.
Definition at line 114 of file PythonProcessedLearner.h.
Referenced by compileAndInject(), and declareOptions().
TVec<string> PLearn::PythonProcessedLearner::m_outputnames [protected] |
Cached version of the output names.
Definition at line 197 of file PythonProcessedLearner.h.
Referenced by getOutputNames(), makeDeepCopyFromShallowCopy(), outputsize(), setOutputNamesFromParams(), and setTrainingSet().
map<string,string> PLearn::PythonProcessedLearner::m_params |
General-purpose parameters that are injected into the Python code snippet and accessible via the getParam/setParam functions.
Can be used for passing processing arguments to the Python code.
Definition at line 121 of file PythonProcessedLearner.h.
Referenced by declareOptions(), getParam(), getParams(), setOutputNamesFromParams(), and setParam().
PP<PythonCodeSnippet> PLearn::PythonProcessedLearner::python [protected] |
Actual Python environment.
Definition at line 200 of file PythonProcessedLearner.h.
Referenced by build_(), compileAndInject(), computeOutput(), makeDeepCopyFromShallowCopy(), outputsize(), setOutputNamesFromParams(), and setTrainingSet().