|
PLearn 0.1
|
Trains a Neural Network Language Model (NNLM). More...
#include <NnlmOnlineLearner.h>


Public Member Functions | |
| NnlmOnlineLearner () | |
| Default constructor. | |
| void | buildLayers () |
| builds the layers, ie modules and output_modules | |
| void | buildCandidates () |
| Specific to the gaussian model. | |
| void | reevaluateGaussianParameters () const |
| Reevaluates "fresh" gaussian mus and sigmas - make sure you want to do this. | |
| void | myGetExample (const VMat &example_set, int &sample, Vec &input, Vec &target, real &weight) const |
| Interfaces with the ProcessSymbolicSequenceVMatrix's getRow() | |
| virtual int | outputsize () const |
| Returns the size of this learner's output, (which typically may depend on its inputsize(), targetsize() and set options). | |
| virtual void | forget () |
| (Re-)initializes the PLearner in its fresh state (that state may depend on the 'seed' option) and sets 'stage' back to 0 (this is the stage of a fresh learner!). | |
| virtual void | train () |
| The role of the train method is to bring the learner up to stage==nstages, updating the train_stats collector with training costs measured on-line in the process. | |
| void | test (VMat testset, PP< VecStatsCollector > test_stats, VMat testoutputs, VMat testcosts) const |
| Performs test on testset, updating test cost statistics, and optionally filling testoutputs and testcosts. | |
| virtual void | computeOutput (const Vec &input, Vec &output) const |
| Computes the output from the input. | |
| virtual void | computeCostsFromOutputs (const Vec &input, const Vec &output, const Vec &target, Vec &costs) const |
| Computes the costs from already computed output. | |
| virtual void | computeTrainCostsFromOutputs (const Vec &input, const Vec &output, const Vec &target, Vec &costs) const |
| virtual TVec< std::string > | getTestCostNames () const |
| Returns the names of the costs computed by computeCostsFromOutpus (and thus the test method). | |
| virtual TVec< std::string > | getTrainCostNames () const |
| Returns the names of the objective costs that the train method computes and for which it updates the VecStatsCollector train_stats. | |
| virtual string | classname () const |
| virtual OptionList & | getOptionList () const |
| virtual OptionMap & | getOptionMap () const |
| virtual RemoteMethodMap & | getRemoteMethodMap () const |
| virtual NnlmOnlineLearner * | deepCopy (CopiesMap &copies) const |
| virtual void | build () |
| Finish building the object; just call inherited::build followed by build_() | |
| virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
| Transforms a shallow copy into a deep copy. | |
Static Public Member Functions | |
| static string | _classname_ () |
| static OptionList & | _getOptionList_ () |
| static RemoteMethodMap & | _getRemoteMethodMap_ () |
| static Object * | _new_instance_for_typemap_ () |
| static bool | _isa_ (const Object *o) |
| static void | _static_initialize_ () |
| static const PPath & | declaringFile () |
Public Attributes | |
| string | str_input_model |
| Defines which model is used. | |
| string | str_output_model |
| int | word_representation_size |
| Size of the real ditributed word representations. | |
| int | semantic_layer_size |
| Size of the semantic layer. | |
| real | wrl_slr |
| Neural part parameters. | |
| real | wrl_dc |
| real | wrl_wd_l1 |
| real | wrl_wd_l2 |
| real | sl_slr |
| real | sl_dc |
| real | sl_wd_l1 |
| real | sl_wd_l2 |
| string | str_gaussian_model_train_cost |
| Define behavior. | |
| string | str_gaussian_model_learning |
| real | gaussian_model_sigma2_min |
| real | gaussian_model_dl_slr |
| real | gaussian_model_dl_dc |
| int | shared_candidates_size |
| Number of candidates to use from different sources in the gaussian model when we use the approx_discriminant cost. | |
| int | ngram_candidates_size |
| int | self_candidates_size |
| VMat | ngram_train_set |
| Used in determining the C sets of candidate words for normalization in the evaluated discriminant cost. | |
| real | sm_slr |
| real | sm_dc |
| real | sm_wd_l1 |
| real | sm_wd_l2 |
| TVec< PP< OnlineLearningModule > > | modules |
| Layers of the learner Separated between the fixed part which computes up to the "semantic layer" and the variable part (gaussian or softmax) | |
| TVec< PP< OnlineLearningModule > > | output_modules |
| int | vocabulary_size |
| NNLM related - determined from train_set. | |
| int | context_size |
| PP< NGramDistribution > | theNGram |
| Used in determining the C sets of candidate words for normalization in the evaluated discriminant cost. | |
| TVec< int > | shared_candidates |
| Holds candidates. | |
| TVec< TVec< int > > | candidates |
Static Public Attributes | |
| static StaticInitializer | _static_initializer_ |
Static Protected Member Functions | |
| static void | declareOptions (OptionList &ol) |
| Declares the class options. | |
Protected Attributes | |
| TVec< Vec > | values |
| stores the input and output values of the functions | |
| TVec< Vec > | gradients |
| stores the gradients | |
| TVec< Vec > | output_values |
| for the second variable part of the model (starts from 'r', the semantic layer, on) | |
| TVec< Vec > | output_gradients |
Private Types | |
| enum | { MODEL_TYPE_GAUSSIAN = 0, MODEL_TYPE_SOFTMAX = 1 } |
| enum | { GAUSSIAN_COST_DISCR = 0, GAUSSIAN_COST_APPROX_DISCR = 1, GAUSSIAN_COST_NON_DISCR = 2 } |
| enum | { GAUSSIAN_LEARNING_DISCR = 0, GAUSSIAN_LEARNING_EMPIRICAL = 1 } |
| typedef PLearner | inherited |
Private Member Functions | |
| void | build_ () |
| This does the actual building. | |
Private Attributes | |
| int | nmodules |
| Used for loops. | |
| int | output_nmodules |
| int | model_type |
| Holds model type. | |
| int | gaussian_model_cost |
| int | gaussian_model_learning |
Trains a Neural Network Language Model (NNLM).
This learner is based upon the online module architecture.
Definition at line 61 of file NnlmOnlineLearner.h.
typedef PLearner PLearn::NnlmOnlineLearner::inherited [private] |
Reimplemented from PLearn::PLearner.
Definition at line 63 of file NnlmOnlineLearner.h.
anonymous enum [private] |
Definition at line 274 of file NnlmOnlineLearner.h.
anonymous enum [private] |
Definition at line 275 of file NnlmOnlineLearner.h.
anonymous enum [private] |
Definition at line 276 of file NnlmOnlineLearner.h.
| PLearn::NnlmOnlineLearner::NnlmOnlineLearner | ( | ) |
Default constructor.
Definition at line 84 of file NnlmOnlineLearner.cc.
References PLearn::PLearner::random_gen.
: PLearner(), str_input_model( "wrl" ), str_output_model( "gaussian" ), word_representation_size( 30 ), semantic_layer_size( 100 ), wrl_slr( 0.001 ), wrl_dc( 0.0 ), wrl_wd_l1( 0.0 ), wrl_wd_l2( 0.0 ), sl_slr( 0.001 ), sl_dc( 0.0 ), sl_wd_l1( 0.0 ), sl_wd_l2( 0.0 ), str_gaussian_model_train_cost( "approx_discriminant" ), str_gaussian_model_learning( "non_discriminant" ), gaussian_model_sigma2_min(0.000001), gaussian_model_dl_slr(0.001), shared_candidates_size( 0 ), ngram_candidates_size( 50 ), self_candidates_size( 0 ), sm_slr( 0.001 ), sm_dc( 0.0 ), sm_wd_l1( 0.0 ), sm_wd_l2( 0.0 ), vocabulary_size( -1 ), context_size( -1 ), nmodules( -1 ), output_nmodules( -1 ), model_type( -1 ), gaussian_model_cost( -1 ), gaussian_model_learning( -1 ) { // ### You may (or not) want to call build_() to finish building the object // ### (doing so assumes the parent classes' build_() have been called too // ### in the parent classes' constructors, something that you must ensure) random_gen = new PRandom(); }
| string PLearn::NnlmOnlineLearner::_classname_ | ( | ) | [static] |
| OptionList & PLearn::NnlmOnlineLearner::_getOptionList_ | ( | ) | [static] |
| RemoteMethodMap & PLearn::NnlmOnlineLearner::_getRemoteMethodMap_ | ( | ) | [static] |
| Object * PLearn::NnlmOnlineLearner::_new_instance_for_typemap_ | ( | ) | [static] |
| StaticInitializer NnlmOnlineLearner::_static_initializer_ & PLearn::NnlmOnlineLearner::_static_initialize_ | ( | ) | [static] |
| void PLearn::NnlmOnlineLearner::build | ( | ) | [virtual] |
Finish building the object; just call inherited::build followed by build_()
Reimplemented from PLearn::PLearner.
Definition at line 278 of file NnlmOnlineLearner.cc.
References PLearn::PLearner::build(), and build_().
{
inherited::build();
build_();
}

| void PLearn::NnlmOnlineLearner::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::PLearner.
Definition at line 288 of file NnlmOnlineLearner.cc.
References buildLayers(), context_size, PLearn::endl(), GAUSSIAN_COST_APPROX_DISCR, GAUSSIAN_COST_DISCR, GAUSSIAN_COST_NON_DISCR, GAUSSIAN_LEARNING_DISCR, GAUSSIAN_LEARNING_EMPIRICAL, gaussian_model_cost, gaussian_model_learning, PLearn::PLearner::inputsize(), PLearn::lowerstring(), model_type, MODEL_TYPE_GAUSSIAN, MODEL_TYPE_SOFTMAX, ngram_train_set, PLERROR, str_gaussian_model_learning, str_gaussian_model_train_cost, str_output_model, PLearn::PLearner::train_set, PLearn::PLearner::verbosity, and vocabulary_size.
Referenced by build().
{
cout << "NnlmOnlineLearner::build_()" << endl;
if( !train_set ) {
return;
}
// *** Sanity Checks ***
// *** Sanity Checks ***
/*int word_representation_size
int semantic_layer_size
real wrl_slr;
real wrl_dc;
real wrl_wd_l1;
real wrl_wd_l2;
real sl_slr;
real sl_dc;
real sl_wd_l1;
real sl_wd_l2;
real gaussian_model_sigma2_min
int shared_candidates_size;
int ngram_candidates_size;
int self_candidates_size;
real sm_slr;
real sm_dc;
real sm_wd_l1;
real sm_wd_l2;*/
// *** Determine Model ***
// *** Determine Model ***
// * Model type *
string mt = lowerstring( str_output_model );
if( mt == "gaussian" || mt == "" ) {
model_type = MODEL_TYPE_GAUSSIAN;
} else if( mt == "softmax" ) {
model_type = MODEL_TYPE_SOFTMAX;
} else {
PLERROR( "'%s' model type is unknown.\n", mt.c_str() );
}
if( model_type == MODEL_TYPE_GAUSSIAN ) {
// * Gaussian model cost *
string gmc = lowerstring( str_gaussian_model_train_cost );
if( gmc == "approx_discriminant" || gmc == "" ) {
gaussian_model_cost = GAUSSIAN_COST_APPROX_DISCR;
} else if( gmc == "non_discriminant" ) {
gaussian_model_cost = GAUSSIAN_COST_NON_DISCR;
} else if( gmc == "discriminant" ) {
gaussian_model_cost = GAUSSIAN_COST_DISCR;
} else {
PLERROR( "'%s' gaussian model train cost is unknown.\n", gmc.c_str() );
}
// * Gaussian model learning *
string gml = lowerstring( str_gaussian_model_learning );
if( gml == "non_discriminant" || gml == "" ) {
gaussian_model_learning = GAUSSIAN_LEARNING_EMPIRICAL;
} else if( gml == "discriminant" ) {
gaussian_model_learning = GAUSSIAN_LEARNING_DISCR;
} else {
PLERROR( "'%s' gaussian model learning is unknown.\n", gml.c_str() );
}
}
// *** Vocabulary size ***
// *** Vocabulary size ***
// the train set's dictionary_size +1 for the 'OOV' tag (tag 0) +1 for the 'missing' tag (tag 'dict_size+1')
vocabulary_size = (train_set->getDictionary(0))->size()+2;
if( verbosity > 0 ) {
cout << "\tvocabulary_size = " << vocabulary_size << endl;
}
// Ensure MINIMAL dictionary coherence, ie size, with ngram set
if( model_type == MODEL_TYPE_GAUSSIAN ) {
if( vocabulary_size != (ngram_train_set->getDictionary(0))->size()+2 ) {
PLERROR("train_set and ngram_train_set have dictionaries of different sizes.\n");
}
}
// *** Context size ***
// *** Context size ***
// The ProcessSymbolicSequenceVMatrix has only input. Last input is used as target.
context_size = inputsize()-1;
if( verbosity > 0 ) {
cout << "\tcontext_size = " << context_size << endl;
}
// *** Build modules and output_module ***
// *** Build modules and output_module ***
buildLayers();
cout << "NnlmOnlineLearner::build_() - DONE!" << endl;
}


| void PLearn::NnlmOnlineLearner::buildCandidates | ( | ) |
Specific to the gaussian model.
Definition at line 646 of file NnlmOnlineLearner.cc.
References candidates, PLearn::endl(), PLearn::find(), i, model_type, MODEL_TYPE_GAUSSIAN, ngram_candidates_size, ngram_train_set, PLWARNING, PLearn::TVec< T >::push_back(), PLearn::TVec< T >::resize(), shared_candidates, shared_candidates_size, theNGram, PLearn::PLearner::train_set, vocabulary_size, and PLearn::wordAndFreqGT().
Referenced by buildLayers().
{
if( model_type != MODEL_TYPE_GAUSSIAN ) {
PLWARNING("NnlmOnlineLearner::buildCandidates() - model is not of gaussian type. Ignoring call.\n");
return;
}
// *** Train ngram ***
// *** Train ngram ***
cout << "NnlmOnlineLearner::buildCandidates()" << endl;
cout << "\ttraining ngram..." << endl;
theNGram = new NGramDistribution();
theNGram->n = ngram_train_set->inputsize();
theNGram->smoothing = "no_smoothing";
theNGram->nan_replace = true;
theNGram->setTrainingSet( ngram_train_set );
//theNGram->build(); Done in setTrainingSet
theNGram->train();
// *** Effective building ***
// *** Effective building ***
cout << "\tbuilding candidates..." << endl;
shared_candidates.resize( shared_candidates_size );
candidates.resize( vocabulary_size );
std::vector< wordAndFreq > tmp;
// temporary list containing the shared candidates
list<int> l_tmp_shared_candidates;
list<int>::iterator itr_tmp_shared_candidates;
// * Determine most frequent words and so the shared_candidates
TVec<int> unigram( 1 );
TVec<int> unifreq( 1 );
// wt means "word tag"
// Note -> wt=vocabulary_size-1 corresponds to the (-1) tag in the NGramDistribution
// we skip this tag, the 'missing' tag
// NOTE Is this appropriate treatment?
// I don't see how the missing values could occur anywhere except at the beginning so yes.
for(int wt=0; wt<vocabulary_size-1; wt++) {
unigram[0] = wt;
unifreq = (theNGram->tree)->freq(unigram);
tmp.push_back( wordAndFreq(wt, unifreq[0]) );
}
std::sort(tmp.begin(), tmp.end(), wordAndFreqGT);
//cout << "These are the shared candidates:" << endl;
// HACK we don't check if itr has hit the end... unlikely vocabulary_size is smaller
// than shared_candidates_size
std::vector< wordAndFreq >::iterator itr_vec;
itr_vec=tmp.begin();
for(int i=0; i< shared_candidates_size; i++) {
cout << (train_set->getDictionary(0))->getSymbol( itr_vec->wordtag ) << "\t";
shared_candidates[i] = itr_vec->wordtag;
l_tmp_shared_candidates.push_back(itr_vec->wordtag);
itr_vec++;
}
tmp.clear();
cout << endl;
// * Add best candidates according to a bigram
// wt means "word tag"
// Note -> wt=vocabulary_size-1 corresponds to the (-1) tag in the NGramDistribution
// we skip this tag, the 'missing' tag
// NOTE Is this appropriate treatment?
map<int, int> frequenciesCopy;
map<int,int>::iterator itr;
int n_candidates;
for(int wt=-1; wt<vocabulary_size-1; wt++) {
// - fill list of candidates, then sort
PP<SymbolNode> node = ((theNGram->tree)->getRoot())->child(wt);
if(node) {
frequenciesCopy = node->getFrequencies();
itr = frequenciesCopy.begin();
while( itr != frequenciesCopy.end() ) {
// -1 is the NGram's missing tag, our vocabulary_size-1 tag
// Actually, we should not see it as a follower to anything except itself...
if( itr->first != -1) {
tmp.push_back( wordAndFreq( itr->first, itr->second ) );
} else {
tmp.push_back( wordAndFreq( vocabulary_size-1, itr->second ) );
}
itr++;
}
std::sort(tmp.begin(), tmp.end(), wordAndFreqGT);
// - resize candidates entry
if( ngram_candidates_size < (int) tmp.size() ) {
n_candidates = ngram_candidates_size;
} else {
n_candidates = tmp.size();
}
if(wt!=-1) {
candidates[wt].resize( n_candidates );
} else {
candidates[ vocabulary_size-1 ].resize( n_candidates );
}
// - fill candidates entry
itr_vec=tmp.begin();
for(int i=0; i< n_candidates; i++) {
//cout << (train_set->getDictionary(0))->getSymbol( itr_vec->wordtag ) << "\t";
// ONLY ADD IF NOT IN THE SHARED CANDIDATES
// Search the list.
itr_tmp_shared_candidates = find( l_tmp_shared_candidates.begin(), l_tmp_shared_candidates.end(), itr_vec->wordtag);
// if not found -> add it
if (itr_tmp_shared_candidates == l_tmp_shared_candidates.end())
{
if( itr_vec->wordtag > vocabulary_size -1 )
cout << "NnlmOnlineLearner::buildCandidates() - problem " << itr_vec->wordtag <<endl;
if(wt!=-1) {
candidates[wt][i] = itr_vec->wordtag;
} else {
candidates[ vocabulary_size-1 ][i] = itr_vec->wordtag;
}
// compensate for not adding this word
} else {
i--;
n_candidates--;
}
itr_vec++;
}
// compensate for not adding words
if(wt!=-1) {
candidates[wt].resize( n_candidates );
} else {
candidates[ vocabulary_size-1 ].resize( n_candidates );
}
tmp.clear();
}
}
l_tmp_shared_candidates.clear();
}


| void PLearn::NnlmOnlineLearner::buildLayers | ( | ) |
builds the layers, ie modules and output_modules
Definition at line 400 of file NnlmOnlineLearner.cc.
References buildCandidates(), candidates, context_size, GAUSSIAN_COST_APPROX_DISCR, GAUSSIAN_COST_DISCR, GAUSSIAN_COST_NON_DISCR, gaussian_model_cost, gaussian_model_learning, gaussian_model_sigma2_min, gradients, i, PLearn::PLearner::inputsize(), PLearn::lowerstring(), model_type, MODEL_TYPE_GAUSSIAN, modules, nmodules, output_gradients, output_modules, output_nmodules, output_values, PLERROR, PLWARNING, PLearn::PLearner::random_gen, PLearn::TVec< T >::resize(), semantic_layer_size, shared_candidates, sl_dc, sl_slr, sl_wd_l1, sl_wd_l2, sm_dc, sm_slr, sm_wd_l1, sm_wd_l2, PLearn::sqrt(), str_input_model, values, vocabulary_size, word_representation_size, wrl_dc, wrl_slr, wrl_wd_l1, and wrl_wd_l2.
Referenced by build_().
{
// *** Do we have to build the layers, or did we load them? ***
if( nmodules <= 0 ) {
//------------------------------------------
// 1) Fixed part - up to the semantic layer
//------------------------------------------
nmodules = 3;
modules.resize( nmodules );
// *** First layer ***
string ilm = lowerstring( str_input_model );
if( ilm == "wrl" || ilm == "" ) {
// *** Word representation layer ***
// *** Word representation layer ***
PP< NnlmWordRepresentationLayer > p_wrl = new NnlmWordRepresentationLayer();
p_wrl->input_size = context_size;
p_wrl->output_size = context_size * word_representation_size;
p_wrl->start_learning_rate = wrl_slr;
p_wrl->decrease_constant = wrl_dc;
//TODO
//p_wrl->L1_penalty_factor = wrl_wd_l1;
//p_wrl->L2_penalty_factor = wrl_wd_l2;
p_wrl->vocabulary_size = vocabulary_size;
p_wrl->word_representation_size = word_representation_size;
p_wrl->context_size = context_size;
p_wrl->random_gen = random_gen;
modules[0] = p_wrl;
} else if( ilm == "gnnl" ) {
PP< GradNNetLayerModule > p_nnl = new GradNNetLayerModule();
p_nnl->input_size = inputsize();
p_nnl->output_size = inputsize() * word_representation_size;
p_nnl->start_learning_rate = wrl_slr;
p_nnl->decrease_constant = wrl_dc;
p_nnl->L1_penalty_factor = wrl_wd_l1;
p_nnl->L2_penalty_factor = wrl_wd_l2;
p_nnl->init_weights_random_scale=sqrt(p_nnl->input_size);
p_nnl->random_gen = random_gen;
modules[0] = p_nnl;
} else {
PLERROR( "'%s' input layer model is unknown.\n", ilm.c_str() );
}
// *** GradNNetLayer ***
// *** GradNNetLayer ***
PP< GradNNetLayerModule > p_nnl = new GradNNetLayerModule();
p_nnl->input_size = context_size * word_representation_size;
p_nnl->output_size = semantic_layer_size;
p_nnl->start_learning_rate = sl_slr;
p_nnl->decrease_constant = sl_dc;
p_nnl->L1_penalty_factor = sl_wd_l1;
p_nnl->L2_penalty_factor = sl_wd_l2;
p_nnl->init_weights_random_scale=3.0*sqrt(p_nnl->input_size);
p_nnl->random_gen = random_gen;
modules[1] = p_nnl;
// *** Tanh layer ***
// *** Tanh layer ***
PP< TanhModule > p_thm = new TanhModule();
p_thm->input_size = semantic_layer_size;
p_thm->output_size = semantic_layer_size;
modules[2] = p_thm;
//------------------------------------------
// 2) Variable part - over semantic layer
//------------------------------------------
if( model_type == MODEL_TYPE_GAUSSIAN ) {
output_nmodules = 1;
output_modules.resize( output_nmodules );
// *** NnlmOutputLayer ***
PP< NnlmOutputLayer > p_nol = new NnlmOutputLayer();
p_nol->input_size = semantic_layer_size;
p_nol->output_size = 1;
// the missing tag does NOT get an output (never is the target)
p_nol->target_cardinality = vocabulary_size-1;
p_nol->sigma2min = gaussian_model_sigma2_min;
p_nol->context_cardinality = vocabulary_size;
p_nol->dl_start_learning_rate = 0.0001;
//TODO Set cost and learning
//int gaussian_model_cost;
//int gaussian_model_learning;
output_modules[0] = p_nol;
output_modules[0]->build();
} else {
output_nmodules = 2;
output_modules.resize( output_nmodules );
// *** GradNNetLayer ***
// *** GradNNetLayer ***
PP< GradNNetLayerModule > p_sm_nnl = new GradNNetLayerModule();
p_sm_nnl->input_size = semantic_layer_size;
// the missing tag does NOT get an output (never is the target)
p_sm_nnl->output_size = vocabulary_size-1;
p_sm_nnl->start_learning_rate = sm_slr;
p_sm_nnl->decrease_constant = sm_dc;
p_sm_nnl->L1_penalty_factor = sm_wd_l1;
p_sm_nnl->L2_penalty_factor = sm_wd_l2;
p_sm_nnl->init_weights_random_scale=3.0*sqrt(p_sm_nnl->input_size);
p_sm_nnl->random_gen = random_gen;
output_modules[0] = p_sm_nnl;
output_modules[0]->build();
// *** Softmax ***
output_modules[1] = new NLLErrModule();
// the missing tag does NOT get an output (never is the target)
output_modules[1]->input_size = vocabulary_size-1;
output_modules[1]->output_size = 1;
output_modules[1]->build();
//
output_values.resize( 1 );
output_gradients.resize( 1 );
// TODO should improve this
// +1 so we can add the target in the last spot
output_values[0].resize( vocabulary_size );
output_gradients[0].resize( vocabulary_size-1 );
}
}
// *** Check on layer size compatibilities, resize values and gradients, and build ***
// *** Check on layer size compatibilities, resize values and gradients, and build ***
// TODO Right now we simply check up to the semantic layer. And we don't check compatibility
// with context_size and word_representation_size and semantic_layer_size.
// variables
values.resize( nmodules+1 );
gradients.resize( nmodules+1 );
// first values will be "input" values
int size = context_size;
values[0].resize( size );
gradients[0].resize( size );
for( int i=0 ; i<nmodules ; i++ )
{
PP<OnlineLearningModule> p_module = modules[i];
if( p_module->input_size != size )
{
PLWARNING( "NnlmOnlineLearner::buildLayers(): module '%d'\n"
"has an input size of '%d', but previous layer's output"
" size\n"
"is '%d'. Resizing module '%d'.\n",
i, p_module->input_size, size, i);
p_module->input_size = size;
}
p_module->estimate_simpler_diag_hessian = true;
p_module->build();
size = p_module->output_size;
values[i+1].resize( size );
gradients[i+1].resize( size );
}
// *** Gaussian Model ***
// *** Gaussian Model ***
if( model_type == MODEL_TYPE_GAUSSIAN ) {
// * Build candidates
if( gaussian_model_cost == GAUSSIAN_COST_APPROX_DISCR ) {
buildCandidates();
}
// * Set
PP<NnlmOutputLayer> p_nol;
if( !(p_nol = dynamic_cast<NnlmOutputLayer*>( (OnlineLearningModule*) output_modules[0] ) ) )
{
PLERROR("NnlmOnlineLearner::build_() - MODEL_TYPE_GAUSSIAN but output_modules[0] is not an NnlmOutputLayer");
}
// TODO clean this
// point to the same place
p_nol->shared_candidates = shared_candidates;
p_nol->candidates = candidates;
// TODO Set learning method - discriminant or non-discriminant
p_nol->setLearning(gaussian_model_learning);
// Set Cost
if( gaussian_model_cost == GAUSSIAN_COST_APPROX_DISCR ) {
p_nol->setCost(GAUSSIAN_COST_APPROX_DISCR);
} else if( gaussian_model_cost == GAUSSIAN_COST_NON_DISCR ) {
p_nol->setCost(GAUSSIAN_COST_NON_DISCR);
} else { //GAUSSIAN_COST_DISCR
p_nol->setCost(GAUSSIAN_COST_DISCR);
}
//evaluateGaussianCounts();
//reevaluateGaussianParameters();
// *
// Not here, because forget will be called after and it resets mus and sigmas
// Initialize mus and sigmas using 1 pass
//reevaluateGaussianParameters();
// ### Should only be evaluated once
//p_nol->sumI << p_nol->test_sumI;
//p_nol->s_sumI = p_nol->test_s_sumI;
}
}


| string PLearn::NnlmOnlineLearner::classname | ( | ) | const [virtual] |
| void PLearn::NnlmOnlineLearner::computeCostsFromOutputs | ( | const Vec & | input, |
| const Vec & | output, | ||
| const Vec & | target, | ||
| Vec & | costs | ||
| ) | const [virtual] |
Computes the costs from already computed output.
Computes the test costs, ie the full disriminant cost (NLL).
See about how to include/print the perplexity.
Implements PLearn::PLearner.
Definition at line 1377 of file NnlmOnlineLearner.cc.
References GAUSSIAN_COST_APPROX_DISCR, GAUSSIAN_COST_DISCR, GAUSSIAN_COST_NON_DISCR, gaussian_model_cost, model_type, MODEL_TYPE_GAUSSIAN, output_modules, output_values, PLERROR, PLearn::TVec< T >::subVec(), and vocabulary_size.
{
if( model_type == MODEL_TYPE_GAUSSIAN ) {
PP<NnlmOutputLayer> p_nol;
if( !(p_nol = dynamic_cast<NnlmOutputLayer*>( (OnlineLearningModule*) output_modules[0] ) ) )
{
PLERROR("NnlmOnlineLearner::computeCostsFromOutputs - MODEL_TYPE_GAUSSIAN but output_modules[0] is not an NnlmOutputLayer");
}
p_nol->setCost(GAUSSIAN_COST_DISCR);
p_nol->setTarget( (int)target[0] );
p_nol->fprop( output, costs);
// Re-Set Cost
if( gaussian_model_cost == GAUSSIAN_COST_APPROX_DISCR ) {
p_nol->setCost(GAUSSIAN_COST_APPROX_DISCR);
} else { //GAUSSIAN_COST_NON_DISCR
p_nol->setCost(GAUSSIAN_COST_NON_DISCR);
}
} else {
Vec example_cost(1);
Vec bob(vocabulary_size-1);
output_modules[0]->fprop( output, bob );
output_values[0].subVec( 0, vocabulary_size-1 ) << bob;
// output_values[0][vocabulary_size-1] contains the target index from myGetExample
output_modules[1]->fprop( output_values[0], example_cost);
costs[0] = example_cost[0];
}
}

Computes the output from the input.
Reimplemented from PLearn::PLearner.
Definition at line 1301 of file NnlmOnlineLearner.cc.
References i, modules, nmodules, outputsize(), PLearn::TVec< T >::resize(), and values.
Referenced by reevaluateGaussianParameters(), and train().
{
//cout << "************************************" << endl;
// fprop
values[0] << input;
for( int i=0 ; i<nmodules ; i++ ) {
modules[i]->fprop( values[i], values[i+1] );
//cout << "-= " << i << " =-" << endl;
//cout << values[i] << endl;
}
//cout << "-= " << nmodules << " =-" << endl;
//cout <<values[ nmodules ] << endl;
//
output.resize( outputsize() );
output << values[ nmodules ];
}


| void PLearn::NnlmOnlineLearner::computeTrainCostsFromOutputs | ( | const Vec & | input, |
| const Vec & | output, | ||
| const Vec & | target, | ||
| Vec & | costs | ||
| ) | const [virtual] |
Definition at line 1327 of file NnlmOnlineLearner.cc.
References PLearn::PLearner::inputsize(), model_type, MODEL_TYPE_GAUSSIAN, output_modules, output_values, PLERROR, PLearn::TVec< T >::subVec(), and vocabulary_size.
Referenced by train().
{
if( model_type == MODEL_TYPE_GAUSSIAN ) {
PP<NnlmOutputLayer> p_nol;
if( !(p_nol = dynamic_cast<NnlmOutputLayer*>( (OnlineLearningModule*) output_modules[0] ) ) )
{
PLERROR("NnlmOnlineLearner::computeTrainCostsAndGradientsFromOutputs - MODEL_TYPE_GAUSSIAN but output_modules[0] is not an NnlmOutputLayer");
}
p_nol->setTarget( (int) target[0] );
p_nol->setContext( (int) input[ (int) (inputsize()-2) ] );
p_nol->fprop( output, costs );
} else {
Vec example_cost(1);
// don't give the target to the gradnnetlayermodule
Vec bob(vocabulary_size-1);
/*
Vec out_tgt = output.copy();
out_tgt.append( target );
for( int i=0 ; i<ncosts ; i++ )
{
Vec cost(1);
cost_modules[i]->fprop( out_tgt, cost );
costs[i] = cost[0];
}
*/
//output_modules[0]->fprop( output, output_values[0].subVec( 0, vocabulary_size-1 ) );
// output_values[0][vocabulary_size-1] contains the target index myGetExample
output_modules[0]->fprop( output, bob );
output_values[0].subVec( 0, vocabulary_size-1 ) << bob;
output_modules[1]->fprop( output_values[0], example_cost);
costs[0] = example_cost[0];
}
}


| void PLearn::NnlmOnlineLearner::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declares the class options.
Reimplemented from PLearn::PLearner.
Definition at line 127 of file NnlmOnlineLearner.cc.
References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::PLearner::declareOptions(), gaussian_model_dl_dc, gaussian_model_dl_slr, gaussian_model_sigma2_min, modules, ngram_candidates_size, ngram_train_set, output_modules, self_candidates_size, semantic_layer_size, shared_candidates_size, sl_dc, sl_slr, sl_wd_l1, sl_wd_l2, sm_dc, sm_slr, sm_wd_l1, sm_wd_l2, str_gaussian_model_learning, str_gaussian_model_train_cost, str_input_model, str_output_model, word_representation_size, wrl_dc, wrl_slr, wrl_wd_l1, and wrl_wd_l2.
{
// *** Build Options ***
// * Model type *
declareOption(ol, "str_input_model",
&NnlmOnlineLearner::str_input_model,
OptionBase::buildoption,
"Specifies what's used as input layer: wrl (default - word representation layer) or gnnl (gradnnetlayer).");
declareOption(ol, "str_output_model",
&NnlmOnlineLearner::str_output_model,
OptionBase::buildoption,
"Specifies what's used on top of the semantic layer: 'softmax' or 'gaussian'(default).");
// * Model size *
declareOption(ol, "word_representation_size",
&NnlmOnlineLearner::word_representation_size,
OptionBase::buildoption,
"Size of the real distributed word representation.");
declareOption(ol, "semantic_layer_size",
&NnlmOnlineLearner::semantic_layer_size,
OptionBase::buildoption,
"Size of the semantic layer.");
// * Same part parameters
declareOption(ol, "wrl_slr",
&NnlmOnlineLearner::wrl_slr,
OptionBase::buildoption,
"Word representation layer start learning rate.");
declareOption(ol, "wrl_dc",
&NnlmOnlineLearner::wrl_dc,
OptionBase::buildoption,
"Word representation layer decrease constant.");
declareOption(ol, "wrl_wd_l1",
&NnlmOnlineLearner::wrl_wd_l1,
OptionBase::buildoption,
"Word representation layer L1 penalty factor.");
declareOption(ol, "wrl_wd_l2",
&NnlmOnlineLearner::wrl_wd_l2,
OptionBase::buildoption,
"Word representation layer L2 penalty factor.");
declareOption(ol, "sl_slr",
&NnlmOnlineLearner::sl_slr,
OptionBase::buildoption,
"Semantic layer start learning rate.");
declareOption(ol, "sl_dc",
&NnlmOnlineLearner::sl_dc,
OptionBase::buildoption,
"Semantic layer decrease constant.");
declareOption(ol, "sl_wd_l1",
&NnlmOnlineLearner::sl_wd_l1,
OptionBase::buildoption,
"Semantic layer L1 penalty factor.");
declareOption(ol, "sl_wd_l2",
&NnlmOnlineLearner::sl_wd_l2,
OptionBase::buildoption,
"Semantic layer L2 penalty factor.");
// * Gaussian model specific
// - model behavior
// TODO how about combining the two costs: maybe jumpstart with one
declareOption(ol, "str_gaussian_model_train_cost",
&NnlmOnlineLearner::str_gaussian_model_train_cost,
OptionBase::buildoption,
"In case of a gaussian output module, specifies the cost used for training (i a word, r a semantic layer representation) : 'discriminant' (computes p(i|r) exactly, with full computation of normalizer), 'approx_discriminant' (default - uses some candidate words for normalization) or 'non_discriminant' (uses p(r|i)).");
declareOption(ol, "str_gaussian_model_learning",
&NnlmOnlineLearner::str_gaussian_model_learning,
OptionBase::buildoption,
"In case of a gaussian output module, specifies the learning technique: 'discriminant' or 'non_discriminant' (default - evaluates empirical mu and sigma).");
declareOption(ol, "gaussian_model_sigma2_min",
&NnlmOnlineLearner::gaussian_model_sigma2_min,
OptionBase::buildoption,
"In case of a gaussian output module, specifies the minimal sigma^2.");
declareOption(ol, "gaussian_model_dl_slr",
&NnlmOnlineLearner::gaussian_model_dl_slr,
OptionBase::buildoption,
"In case of a gaussian output module with discriminant learning, this specifies the starting learning rate.");
declareOption(ol, "gaussian_model_dl_dc",
&NnlmOnlineLearner::gaussian_model_dl_dc,
OptionBase::buildoption,
"In case of a gaussian output module with discriminant learning, this specifies the decrease constant.");
// - Candidate set sizes
declareOption(ol, "shared_candidates_size",
&NnlmOnlineLearner::shared_candidates_size,
OptionBase::buildoption,
"Number of candidates drawn from frequent words in aproximate discriminant cost evaluation.");
declareOption(ol, "ngram_candidates_size",
&NnlmOnlineLearner::ngram_candidates_size,
OptionBase::buildoption,
"Number of candidates drawn from the context (using bigram) in aproximated discriminant cost evaluation.");
declareOption(ol, "self_candidates_size",
&NnlmOnlineLearner::self_candidates_size,
OptionBase::buildoption,
"Number of candidates drawn from the nnlm in aproximated discriminant cost evaluation (evaluated periodically). NOT IMPLEMENTED!!");
// - Ngram (for evaluating ngram candidates) train set
declareOption(ol, "ngram_train_set",
&NnlmOnlineLearner::ngram_train_set,
OptionBase::buildoption,
"Train set used for training the bigram used in the evaluation of the set of candidate words used for normalization in the evaluated discriminant cost (ProcessSymbolicSequenceVMatrix) (ONLY BIGRAMS).");
// * Softmax specific
declareOption(ol, "sm_slr",
&NnlmOnlineLearner::sm_slr,
OptionBase::buildoption,
"Softmax layer start learning rate.");
declareOption(ol, "sm_dc",
&NnlmOnlineLearner::sm_dc,
OptionBase::buildoption,
"Softmax layer decrease constant.");
declareOption(ol, "sm_wd_l1",
&NnlmOnlineLearner::sm_wd_l1,
OptionBase::buildoption,
"Softmax layer L1 penalty factor.");
declareOption(ol, "sm_wd_l2",
&NnlmOnlineLearner::sm_wd_l2,
OptionBase::buildoption,
"Softmax layer L2 penalty factor.");
// *** Learnt Options ***
declareOption(ol, "modules", &NnlmOnlineLearner::modules,
OptionBase::buildoption,
"Layers of the learner");
declareOption(ol, "output_modules", &NnlmOnlineLearner::output_modules,
OptionBase::buildoption,
"Output layers");
// TODO Are there missing things here?
// Now call the parent class' declareOptions
inherited::declareOptions(ol);
}

| static const PPath& PLearn::NnlmOnlineLearner::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::PLearner.
Definition at line 221 of file NnlmOnlineLearner.h.
:
//##### Protected Options ###############################################
| NnlmOnlineLearner * PLearn::NnlmOnlineLearner::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
| void PLearn::NnlmOnlineLearner::forget | ( | ) | [virtual] |
(Re-)initializes the PLearner in its fresh state (that state may depend on the 'seed' option) and sets 'stage' back to 0 (this is the stage of a fresh learner!).
Reimplemented from PLearn::PLearner.
Definition at line 939 of file NnlmOnlineLearner.cc.
References PLearn::TVec< T >::clear(), PLearn::PLearner::forget(), gradients, i, model_type, MODEL_TYPE_SOFTMAX, modules, nmodules, output_gradients, output_modules, output_nmodules, output_values, PLearn::PLearner::stage, and values.
{
inherited::forget();
// reset inputs
values[0].clear();
gradients[0].clear();
// reset modules and outputs
for( int i=0 ; i<nmodules ; i++ )
{
modules[i]->forget();
values[i+1].clear();
gradients[i+1].clear();
}
if( model_type == MODEL_TYPE_SOFTMAX ) {
output_values[0].clear();
output_gradients[0].clear();
}
for( int i=0 ; i<output_nmodules; i++ )
{
output_modules[i]->forget();
}
stage = 0;
}

| OptionList & PLearn::NnlmOnlineLearner::getOptionList | ( | ) | const [virtual] |
| OptionMap & PLearn::NnlmOnlineLearner::getOptionMap | ( | ) | const [virtual] |
| RemoteMethodMap & PLearn::NnlmOnlineLearner::getRemoteMethodMap | ( | ) | const [virtual] |
| TVec< string > PLearn::NnlmOnlineLearner::getTestCostNames | ( | ) | const [virtual] |
Returns the names of the costs computed by computeCostsFromOutpus (and thus the test method).
Implements PLearn::PLearner.
Definition at line 1418 of file NnlmOnlineLearner.cc.
References PLearn::TVec< T >::resize().
Referenced by test().
{
// Return the names of the costs computed by computeCostsFromOutputs
// (these may or may not be exactly the same as what's returned by
// getTrainCostNames).
TVec<string> ret;
ret.resize(1);
ret[0] = "NLL";
return ret;
}


| TVec< string > PLearn::NnlmOnlineLearner::getTrainCostNames | ( | ) | const [virtual] |
Returns the names of the objective costs that the train method computes and for which it updates the VecStatsCollector train_stats.
Implements PLearn::PLearner.
Definition at line 1433 of file NnlmOnlineLearner.cc.
References model_type, MODEL_TYPE_GAUSSIAN, and PLearn::TVec< T >::resize().
Referenced by train().
{
// Return the names of the objective costs that the train method computes
// and for which it updates the VecStatsCollector train_stats
// (these may or may not be exactly the same as what's returned by
// getTestCostNames).
TVec<string> ret;
if( model_type == MODEL_TYPE_GAUSSIAN ) {
ret.resize(2);
ret[0] = "non_discriminant";
ret[1] = "approx_discriminant";
} else {
ret.resize(1);
ret[0] = "NLL";
}
return ret;
}


| void PLearn::NnlmOnlineLearner::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::PLearner.
Definition at line 899 of file NnlmOnlineLearner.cc.
References PLearn::deepCopyField(), gradients, PLearn::PLearner::makeDeepCopyFromShallowCopy(), modules, output_gradients, output_modules, output_values, and values.
{
inherited::makeDeepCopyFromShallowCopy(copies);
deepCopyField(modules, copies);
deepCopyField(values, copies);
deepCopyField(gradients, copies);
deepCopyField(output_modules, copies);
deepCopyField(output_values, copies);
deepCopyField(output_gradients, copies);
// ### How about these?
//ngram_train_set
//theNGram
//shared_candidates
//candidates
}

| void PLearn::NnlmOnlineLearner::myGetExample | ( | const VMat & | example_set, |
| int & | sample, | ||
| Vec & | input, | ||
| Vec & | target, | ||
| real & | weight | ||
| ) | const |
Interfaces with the ProcessSymbolicSequenceVMatrix's getRow()
Definition at line 975 of file NnlmOnlineLearner.cc.
References i, PLearn::PLearner::inputsize(), PLearn::is_missing(), model_type, MODEL_TYPE_SOFTMAX, output_values, PLearn::TVec< T >::resize(), PLearn::TVec< T >::subVec(), vocabulary_size, and PLearn::PLearner::weightsize().
Referenced by reevaluateGaussianParameters(), test(), and train().
{
static Vec row;
// the actual inputsize is (inputsize()-1) and targetsize() is 1
row.resize( inputsize() + weightsize() );
example_set->getRow( sample, row);
input << row.subVec( 0, inputsize()-1 );
target << row.subVec( inputsize()-1, 1 );
weight = 1.0;
if( weightsize() ) {
weight = row[ inputsize() ];
}
// *** SHOULD BE DONE IN PRETREATMENT!!! -> but we have a ProcessSymbolicSequenceVMatrix...
// * Replace nan in input by '(train_set->getDictionary(0))->size()+1',
// the missing value tag
for( int i=0 ; i < inputsize()-1 ; i++ ) {
if( is_missing(input[i]) ) {
input[i] = vocabulary_size - 1;
}
}
// * Replace a 'nan' in the target by OOV
// this nan should not be missing data (seeing the train_set is a
// ProcessSymbolicSequenceVMatrix)
// but the word "nan", ie "Mrs Nan said she would blabla"
// *** Problem however - current vocabulary is full for train_set,
// ie we train OOV on nan-word instances.
// DO a pretreatment to replace Nan by *Nan* or something like it
if( is_missing(target[0]) ) {
target[0] = 0;
}
// *** SHOULD BE DONE IN PRETREATMENT!!!
// set target for the nllerrmodule
if( model_type == MODEL_TYPE_SOFTMAX ) {
output_values[0][vocabulary_size-1]=target[0];
}
}


| int PLearn::NnlmOnlineLearner::outputsize | ( | ) | const [virtual] |
Returns the size of this learner's output, (which typically may depend on its inputsize(), targetsize() and set options).
This is the output size of the layer before the cost layer.
Implements PLearn::PLearner.
Definition at line 924 of file NnlmOnlineLearner.cc.
References PLearn::TVec< T >::length(), nmodules, and values.
Referenced by computeOutput(), reevaluateGaussianParameters(), test(), and train().
{
if( nmodules < 0 || values.length() <= nmodules )
return -1;
else
return values[ nmodules ].length();
}


| void PLearn::NnlmOnlineLearner::reevaluateGaussianParameters | ( | ) | const |
Reevaluates "fresh" gaussian mus and sigmas - make sure you want to do this.
Definition at line 851 of file NnlmOnlineLearner.cc.
References computeOutput(), PLearn::endl(), PLearn::PLearner::inputsize(), PLearn::VMat::length(), model_type, MODEL_TYPE_GAUSSIAN, myGetExample(), output_modules, outputsize(), PLERROR, PLWARNING, PLearn::sample(), and PLearn::PLearner::train_set.
Referenced by test(), and train().
{
cout << "Evaluating gaussian parameters..." << endl;
if( model_type != MODEL_TYPE_GAUSSIAN ) {
PLWARNING( "NnlmOnlineLearner::reevaluateGaussianParameters(): not a gaussian model. Ignoring call.\n");
return;
}
Vec input( inputsize()-1 );
Vec target( 1 );
real weight;
Vec output( outputsize() ); // the output of the semantic layer
int nsamples = train_set->length();
PP<NnlmOutputLayer> p_nol;
if( !(p_nol = dynamic_cast<NnlmOutputLayer*>( (OnlineLearningModule*) output_modules[0] ) ) )
{
PLERROR("NnlmOnlineLearner::reevaluateGaussianParameters() - output_modules[0] is not an NnlmOutputLayer");
}
p_nol->resetAllClassVars();
// * Compute stats
for( int sample=0 ; sample < nsamples ; sample++ )
{
myGetExample(train_set, sample, input, target, weight );
// * fprop
computeOutput(input, output);
//p_nol->setTarget( (int) target[0]);
//p_nol->setContext( (int) input[ (inputsize()-2) ] );
p_nol->updateClassVars((int) target[0], output);
}
// * Apply values
p_nol->applyAllClassVars();
}


| void PLearn::NnlmOnlineLearner::test | ( | VMat | testset, |
| PP< VecStatsCollector > | test_stats, | ||
| VMat | testoutputs, | ||
| VMat | testcosts | ||
| ) | const [virtual] |
Performs test on testset, updating test cost statistics, and optionally filling testoutputs and testcosts.
The default version repeatedly calls computeOutputAndCosts or computeCostsOnly. Note that neither test_stats->forget() nor test_stats->finalize() is called, so that you should call them yourself (respectively before and after calling this method) if you don't plan to accumulate statistics.
Reimplemented from PLearn::PLearner.
Definition at line 1190 of file NnlmOnlineLearner.cc.
References candidates, PLearn::PLearner::computeOutputAndCosts(), PLearn::endl(), PLearn::entropy(), PLearn::TVec< T >::fill(), getTestCostNames(), i, PLearn::PLearner::inputsize(), PLearn::VMat::length(), model_type, MODEL_TYPE_GAUSSIAN, myGetExample(), output_modules, outputsize(), PLERROR, reevaluateGaussianParameters(), PLearn::PLearner::report_progress, PLearn::safeexp(), PLearn::sample(), PLearn::TVec< T >::size(), PLearn::PLearner::stage, and vocabulary_size.
{
Vec input( inputsize()-1 );
Vec output( outputsize() );
Vec target( 1 );
real weight;
Vec test_costs( getTestCostNames().length() );
real entropy = 0.0;
real perplexity = 0.0;
int nsamples = testset->length();
// * Empty test set: we give -1 cost arbitrarily.
if (nsamples == 0) {
test_costs.fill(-1);
test_stats->update(test_costs);
}
if( stage == 0 ) {
// Initialize mus and sigmas using 1 pass
reevaluateGaussianParameters();
}
// * TODO Should we do this?
//reevaluateGaussianParameters();
if( model_type == MODEL_TYPE_GAUSSIAN ) {
PP<NnlmOutputLayer> p_nol;
if( !(p_nol = dynamic_cast<NnlmOutputLayer*>( (OnlineLearningModule*) output_modules[0] ) ) )
{
PLERROR("NnlmOnlineLearner::train() - MODEL_TYPE_GAUSSIAN but output_modules[0] is not an NnlmOutputLayer");
}
cout << "global_mu " << p_nol->global_mu << endl;
cout << "global_sigma2 " << p_nol->global_sigma2 << endl;
}
PP<ProgressBar> pb;
if(report_progress)
pb = new ProgressBar("Testing learner",nsamples);
for( int sample=0 ; sample < nsamples ; sample++ )
{
myGetExample(testset, sample, input, target, weight );
// Always call computeOutputAndCosts, since this is better
// behaved with stateful learners
computeOutputAndCosts(input, target, output, test_costs);
if(testoutputs)
testoutputs->putOrAppendRow(sample,output);
if(testcosts)
testcosts->putOrAppendRow(sample, test_costs);
if(test_stats)
test_stats->update(test_costs,weight);
if(report_progress)
pb->update(sample);
entropy += test_costs[0];
// Do some outputing
// Do some outputing
if( sample < 50 ) {
cout << "---> ";
for( int i=0; i<inputsize()-1; i++) {
if( (int)input[i] == vocabulary_size - 1) {
cout << "\\missing\\ ";
} else {
cout << (testset->getDictionary(0))->getSymbol( (int)input[i] ) << " ";
}
}
cout << "\t\t " << (testset->getDictionary(0))->getSymbol( (int)target[0] ) << " p(t|r) " << safeexp( - test_costs[0] ) << endl;
if( model_type == MODEL_TYPE_GAUSSIAN ) {
PP<NnlmOutputLayer> p_nol;
if( !(p_nol = dynamic_cast<NnlmOutputLayer*>( (OnlineLearningModule*) output_modules[0] ) ) )
{
PLERROR("NnlmOnlineLearner::test - MODEL_TYPE_GAUSSIAN but output_modules[0] is not an NnlmOutputLayer");
}
Vec candidates, probabilities;
p_nol->getBestCandidates(output, candidates, probabilities);
for(int i=0; i<candidates.size(); i++) {
cout << "\t" << (testset->getDictionary(0))->getSymbol( (int)candidates[i] ) << " " << probabilities[i] << endl;
}
}
}
// Do some outputing - END
// Do some outputing - END
}
entropy /= nsamples;
perplexity = safeexp(entropy);
cout << "entropy: " << entropy << " perplexity " << perplexity << endl;
}

| void PLearn::NnlmOnlineLearner::train | ( | ) | [virtual] |
The role of the train method is to bring the learner up to stage==nstages, updating the train_stats collector with training costs measured on-line in the process.
Implements PLearn::PLearner.
Definition at line 1024 of file NnlmOnlineLearner.cc.
References PLearn::OnlineLearningModule::bpropUpdate(), computeOutput(), computeTrainCostsFromOutputs(), PLearn::endl(), GAUSSIAN_COST_APPROX_DISCR, gaussian_model_cost, getTrainCostNames(), gradients, i, PLearn::PLearner::initTrain(), PLearn::PLearner::inputsize(), PLearn::VMat::length(), model_type, MODEL_TYPE_GAUSSIAN, modules, myGetExample(), nmodules, PLearn::PLearner::nstages, output_gradients, output_modules, output_values, outputsize(), PLERROR, reevaluateGaussianParameters(), PLearn::PLearner::report_progress, PLearn::sample(), semantic_layer_size, PLearn::PLearner::stage, PLearn::TVec< T >::subVec(), PLearn::PLearner::train_set, PLearn::PLearner::train_stats, values, and vocabulary_size.
{
if (!initTrain())
return;
Vec input( inputsize()-1 );
Vec target( 1 );
real weight;
Vec output( outputsize() ); // the output of the semantic layer
Vec train_costs( getTrainCostNames().length() );
Vec out_gradient(1,1); // the gradient wrt the cost is '1'
Vec gradient( semantic_layer_size );
int nsamples = train_set->length();
// Initialize mus and sigmas using 1 pass
reevaluateGaussianParameters();
if(stage==0) {
PP<NnlmOutputLayer> p_nol;
if( !(p_nol = dynamic_cast<NnlmOutputLayer*>( (OnlineLearningModule*) output_modules[0] ) ) )
{
PLERROR("NnlmOnlineLearner::train() - MODEL_TYPE_GAUSSIAN but output_modules[0] is not an NnlmOutputLayer");
}
p_nol->computeEmpiricalLearningRateParameters();
}
if( model_type == MODEL_TYPE_GAUSSIAN ) {
PP<NnlmOutputLayer> p_nol;
if( !(p_nol = dynamic_cast<NnlmOutputLayer*>( (OnlineLearningModule*) output_modules[0] ) ) )
{
PLERROR("NnlmOnlineLearner::train() - MODEL_TYPE_GAUSSIAN but output_modules[0] is not an NnlmOutputLayer");
}
p_nol->is_learning = true;
}
//---------------
/* PP<GradNNetLayerModule> p_gnn;
if( !(p_gnn = dynamic_cast<GradNNetLayerModule*>( (OnlineLearningModule*) modules[1] ) ) )
{
PLERROR("NnlmOnlineLearner::train - modules[1] is not a GradNNetLayerModule");
}
p_gnn->printVariance();*/
//---------------
PP<ProgressBar> pb;
if(report_progress) {
pb = new ProgressBar("Training", nsamples);
}
// *** For stages ***
for( ; stage < nstages ; stage++ )
{
if(report_progress) {
cout << "*** Stage " << stage << " ***" << endl;
//cout << "uniform_mixture_coeff " << output_modules[0]->umc << " " << 1 - output_modules[0]->umc<< endl;
}
if( model_type == MODEL_TYPE_GAUSSIAN ) {
PP<NnlmOutputLayer> p_nol;
if( !(p_nol = dynamic_cast<NnlmOutputLayer*>( (OnlineLearningModule*) output_modules[0] ) ) )
{
PLERROR("NnlmOnlineLearner::train() - MODEL_TYPE_GAUSSIAN but output_modules[0] is not an NnlmOutputLayer");
}
cout << "global_mu " << p_nol->global_mu << endl;
cout << "global_sigma2 " << p_nol->global_sigma2 << endl;
}
// * clear stats of previous epoch *
train_stats->forget();
// * for examples *
for( int sample=0 ; sample < nsamples ; sample++ )
{
if(report_progress)
pb->update(sample);
// - Get example -
myGetExample(train_set, sample, input, target, weight );
// - Fixed part fprop -
computeOutput(input, output);
// - Variable part fprop - cost and gradient for this part -
// (we don't want to duplicate some computations in gaussian model gradient evaluation)
// In gaussian case, gradients[nmodules] is computed here.
computeTrainCostsFromOutputs(input, output, target, train_costs );
// - bpropUpdate -
// Variable part
if( model_type == MODEL_TYPE_GAUSSIAN ) {
PP<NnlmOutputLayer> p_nol;
if( !(p_nol = dynamic_cast<NnlmOutputLayer*>( (OnlineLearningModule*) output_modules[0] ) ) )
{
PLERROR("NnlmOnlineLearner::train() - MODEL_TYPE_GAUSSIAN but output_modules[0] is not an NnlmOutputLayer");
}
if( gaussian_model_cost == GAUSSIAN_COST_APPROX_DISCR ) {
output_modules[0]->bpropUpdate( output, train_costs.subVec(1,1), out_gradient );
gradients[nmodules] << p_nol->ad_gradient;
} else { //if( gaussian_model_cost == GAUSSIAN_COST_NON_DISCR )
output_modules[0]->bpropUpdate( output, train_costs.subVec(0,1), out_gradient );
gradients[nmodules] << p_nol->nd_gradient;
}
} else {
output_modules[1]->bpropUpdate( output_values[0], train_costs, output_gradients[0], out_gradient );
output_modules[0]->bpropUpdate( output, output_values[0].subVec( 0, vocabulary_size-1 ), gradients[nmodules], output_gradients[0] );
}
// Fixed (common to both models) part
for( int i=nmodules-1 ; i>0 ; i-- ) {
modules[i]->bpropUpdate( values[i], values[i+1], gradients[i], gradients[i+1] );
}
modules[0]->bpropUpdate( values[0], values[1], gradients[1] );
// - Update stats -
train_stats->update( train_costs );
}// * for examples - END
train_stats->finalize(); // finalize statistics for this epoch
// Initialize mus and sigmas using 1 pass
reevaluateGaussianParameters();
}// *** For stages - END
if( model_type == MODEL_TYPE_GAUSSIAN ) {
PP<NnlmOutputLayer> p_nol;
if( !(p_nol = dynamic_cast<NnlmOutputLayer*>( (OnlineLearningModule*) output_modules[0] ) ) )
{
PLERROR("NnlmOnlineLearner::train() - MODEL_TYPE_GAUSSIAN but output_modules[0] is not an NnlmOutputLayer");
}
p_nol->is_learning = false;
}
}

Reimplemented from PLearn::PLearner.
Definition at line 221 of file NnlmOnlineLearner.h.
Definition at line 144 of file NnlmOnlineLearner.h.
Referenced by buildCandidates(), buildLayers(), and test().
Definition at line 132 of file NnlmOnlineLearner.h.
Referenced by build_(), and buildLayers().
--- Gaussian output model specific stuff ------------------------------
Definition at line 271 of file NnlmOnlineLearner.h.
Referenced by build_(), buildLayers(), computeCostsFromOutputs(), and train().
Definition at line 98 of file NnlmOnlineLearner.h.
Referenced by declareOptions().
Definition at line 97 of file NnlmOnlineLearner.h.
Referenced by declareOptions().
Definition at line 272 of file NnlmOnlineLearner.h.
Referenced by build_(), and buildLayers().
Definition at line 96 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
TVec<Vec> PLearn::NnlmOnlineLearner::gradients [protected] |
stores the gradients
Definition at line 240 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), forget(), makeDeepCopyFromShallowCopy(), and train().
int PLearn::NnlmOnlineLearner::model_type [private] |
Holds model type.
Definition at line 268 of file NnlmOnlineLearner.h.
Referenced by build_(), buildCandidates(), buildLayers(), computeCostsFromOutputs(), computeTrainCostsFromOutputs(), forget(), getTrainCostNames(), myGetExample(), reevaluateGaussianParameters(), test(), and train().
Layers of the learner Separated between the fixed part which computes up to the "semantic layer" and the variable part (gaussian or softmax)
Definition at line 125 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), computeOutput(), declareOptions(), forget(), makeDeepCopyFromShallowCopy(), and train().
Definition at line 104 of file NnlmOnlineLearner.h.
Referenced by buildCandidates(), and declareOptions().
Used in determining the C sets of candidate words for normalization in the evaluated discriminant cost.
Definition at line 110 of file NnlmOnlineLearner.h.
Referenced by build_(), buildCandidates(), and declareOptions().
int PLearn::NnlmOnlineLearner::nmodules [private] |
Used for loops.
Definition at line 264 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), computeOutput(), forget(), outputsize(), and train().
TVec<Vec> PLearn::NnlmOnlineLearner::output_gradients [protected] |
Definition at line 245 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), forget(), makeDeepCopyFromShallowCopy(), and train().
Definition at line 126 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), computeCostsFromOutputs(), computeTrainCostsFromOutputs(), declareOptions(), forget(), makeDeepCopyFromShallowCopy(), reevaluateGaussianParameters(), test(), and train().
Definition at line 265 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and forget().
TVec<Vec> PLearn::NnlmOnlineLearner::output_values [protected] |
for the second variable part of the model (starts from 'r', the semantic layer, on)
Definition at line 244 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), computeCostsFromOutputs(), computeTrainCostsFromOutputs(), forget(), makeDeepCopyFromShallowCopy(), myGetExample(), and train().
Definition at line 105 of file NnlmOnlineLearner.h.
Referenced by declareOptions().
Size of the semantic layer.
Definition at line 79 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), declareOptions(), and train().
Holds candidates.
Definition at line 143 of file NnlmOnlineLearner.h.
Referenced by buildCandidates(), and buildLayers().
Number of candidates to use from different sources in the gaussian model when we use the approx_discriminant cost.
Definition at line 103 of file NnlmOnlineLearner.h.
Referenced by buildCandidates(), and declareOptions().
Definition at line 87 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
Definition at line 86 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
Definition at line 88 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
Definition at line 89 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
Definition at line 115 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
--- Softmax output model specific stuff -------------------------------
Definition at line 114 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
Definition at line 116 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
Definition at line 117 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
Definition at line 95 of file NnlmOnlineLearner.h.
Referenced by build_(), and declareOptions().
Define behavior.
--- Gaussian output model specific stuff ------------------------------
Definition at line 94 of file NnlmOnlineLearner.h.
Referenced by build_(), and declareOptions().
Defines which model is used.
Definition at line 69 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
Definition at line 70 of file NnlmOnlineLearner.h.
Referenced by build_(), and declareOptions().
Used in determining the C sets of candidate words for normalization in the evaluated discriminant cost.
--- Gaussian output model specific stuff ------------------------------
Definition at line 139 of file NnlmOnlineLearner.h.
Referenced by buildCandidates().
TVec<Vec> PLearn::NnlmOnlineLearner::values [protected] |
stores the input and output values of the functions
Definition at line 238 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), computeOutput(), forget(), makeDeepCopyFromShallowCopy(), outputsize(), and train().
NNLM related - determined from train_set.
Definition at line 131 of file NnlmOnlineLearner.h.
Referenced by build_(), buildCandidates(), buildLayers(), computeCostsFromOutputs(), computeTrainCostsFromOutputs(), myGetExample(), test(), and train().
Size of the real ditributed word representations.
--- Fixed (same in both output models) part ----------------------------------
Definition at line 76 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
Definition at line 83 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
Neural part parameters.
Definition at line 82 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
Definition at line 84 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
Definition at line 85 of file NnlmOnlineLearner.h.
Referenced by buildLayers(), and declareOptions().
1.7.4