| 
    PLearn 0.1 
   | 
 
Does the same thing as Hinton's deep belief nets. More...
#include <GaussianDBNRegression.h>


Public Member Functions | |
| GaussianDBNRegression () | |
| Default constructor.   | |
| virtual real | density (const Vec &y) const | 
| Return probability density p(y | x)   | |
| virtual real | log_density (const Vec &y) const | 
| Return log of probability density log(p(y | x)).   | |
| virtual real | survival_fn (const Vec &y) const | 
| Return survival function: P(Y>y | x).   | |
| virtual real | cdf (const Vec &y) const | 
| Return cdf: P(Y<y | x).   | |
| virtual void | expectation (Vec &mu) const | 
| Return E[Y | x].   | |
| virtual void | variance (Mat &cov) const | 
| Return Var[Y | x].   | |
| virtual void | generate (Vec &y) const | 
| Return a pseudo-random sample generated from the conditional distribution, of density p(y | x).   | |
| virtual bool | setPredictorPredictedSizes (int the_predictor_size, int the_predicted_size, bool call_parent=true) | 
| Generates a pseudo-random sample x from the reversed conditional distribution, of density p(x | y) (and NOT p(y | x)).   | |
| virtual void | setPredictor (const Vec &predictor, bool call_parent=true) const | 
| Set the value for the predictor part of a conditional probability.   | |
| virtual void | forget () | 
| (Re-)initializes the PDistribution in its fresh state (that state may depend on the 'seed' option).   | |
| virtual void | train () | 
| The role of the train method is to bring the learner up to stage == nstages, updating the train_stats collector with training costs measured on-line in the process.   | |
| virtual void | computeCostsFromOutputs (const Vec &input, const Vec &output, const Vec &target, Vec &costs) const | 
| Compute a cost, depending on the type of the first output : if it is the density or the log-density: NLL if it is the expectation: NLL and class error.   | |
| virtual TVec< string > | getTestCostNames () const | 
| Return [ "NLL" ] (the only cost computed by a PDistribution).   | |
| virtual string | classname () const | 
| virtual OptionList & | getOptionList () const | 
| virtual OptionMap & | getOptionMap () const | 
| virtual RemoteMethodMap & | getRemoteMethodMap () const | 
| virtual GaussianDBNRegression * | deepCopy (CopiesMap &copies) const | 
| virtual void | build () | 
| Simply calls inherited::build() then build_().   | |
| virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) | 
| Transforms a shallow copy into a deep copy.   | |
Static Public Member Functions | |
| static string | _classname_ () | 
| static OptionList & | _getOptionList_ () | 
| static RemoteMethodMap & | _getRemoteMethodMap_ () | 
| static Object * | _new_instance_for_typemap_ () | 
| static bool | _isa_ (const Object *o) | 
| static void | _static_initialize_ () | 
| static const PPath & | declaringFile () | 
Public Attributes | |
| real | learning_rate | 
| ### declare public option fields (such as build options) here Start your comments with Doxygen-compatible comments such as //!   | |
| real | weight_decay | 
| The weight decay.   | |
| string | initialization_method | 
| The method used to initialize the weights:   | |
| int | n_layers | 
| Number of layers, including input layer and last layer, but not target layer.   | |
| TVec< PP< RBMLayer > > | layers | 
| Layers that learn representations of the input, layers[0] is input layer, layers[n_layers-1] is last layer.   | |
| PP< RBMLayer > | last_layer | 
| last_layer is layer[n_layers-1]   | |
| PP< RBMLayer > | target_layer | 
| Target (or label) layer.   | |
| TVec< PP< RBMLLParameters > > | params | 
| RBMParameters linking the unsupervised layers.   | |
| PP< RBMQLParameters > | input_params | 
| Parameters linking input layer[0] and layer[1].   | |
| PP< RBMLQParameters > | target_params | 
| Parameters linking target_layer and last_layer.   | |
| TVec< int > | training_schedule | 
| Number of examples to use during each of the different greedy steps of the training phase.   | |
| string | fine_tuning_method | 
| Method for fine-tuning the whole network after greedy learning.   | |
| bool | use_sample_rather_than_expectation_in_positive_phase_statistics | 
Static Public Attributes | |
| static StaticInitializer | _static_initializer_ | 
Protected Member Functions | |
| virtual void | greedyStep (const Vec &predictor, int params_index) | 
| virtual void | fineTuneByGradientDescent (const Vec &input) | 
| void | fineTuneByGradientDescentLastLayer (const Vec &input) | 
Static Protected Member Functions | |
| static void | declareOptions (OptionList &ol) | 
| Declares the class options.   | |
Protected Attributes | |
| TVec< Vec > | activation_gradients | 
| gradients of cost wrt the activations (output of params)   | |
| TVec< Vec > | expectation_gradients | 
| gradients of cost wrt the expectations (output of layers)   | |
| Vec | output_gradient | 
| gradient wrt output activations   | |
Private Types | |
| typedef PDistribution | inherited | 
Private Member Functions | |
| void | build_ () | 
| This does the actual building.   | |
| void | build_layers () | 
| Build the layers.   | |
| void | build_params () | 
| Build the parameters if needed.   | |
Does the same thing as Hinton's deep belief nets.
Definition at line 60 of file GaussianDBNRegression.h.
typedef PDistribution PLearn::GaussianDBNRegression::inherited [private] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 62 of file GaussianDBNRegression.h.
| PLearn::GaussianDBNRegression::GaussianDBNRegression | ( | ) | 
Default constructor.
Definition at line 64 of file GaussianDBNRegression.cc.
References PLearn::PLearner::random_gen.
                                             :
    learning_rate(0.),
    weight_decay(0.),
    use_sample_rather_than_expectation_in_positive_phase_statistics(false)
{
    random_gen = new PRandom();
}
| string PLearn::GaussianDBNRegression::_classname_ | ( | ) |  [static] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
| OptionList & PLearn::GaussianDBNRegression::_getOptionList_ | ( | ) |  [static] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
| RemoteMethodMap & PLearn::GaussianDBNRegression::_getRemoteMethodMap_ | ( | ) |  [static] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
| Object * PLearn::GaussianDBNRegression::_new_instance_for_typemap_ | ( | ) |  [static] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
| StaticInitializer GaussianDBNRegression::_static_initializer_ & PLearn::GaussianDBNRegression::_static_initialize_ | ( | ) |  [static] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
| void PLearn::GaussianDBNRegression::build | ( | ) |  [virtual] | 
        
Simply calls inherited::build() then build_().
Reimplemented from PLearn::PDistribution.
Definition at line 156 of file GaussianDBNRegression.cc.
References PLearn::PDistribution::build(), and build_().
{
    // ### Nothing to add here, simply calls build_().
    inherited::build();
    build_();
}

| void PLearn::GaussianDBNRegression::build_ | ( | ) |  [private] | 
        
This does the actual building.
Reimplemented from PLearn::PDistribution.
Definition at line 166 of file GaussianDBNRegression.cc.
References build_layers(), build_params(), PLearn::endl(), fine_tuning_method, initialization_method, layers, PLearn::TVec< T >::length(), PLearn::lowerstring(), n_layers, PLERROR, and training_schedule.
Referenced by build().
{
    MODULE_LOG << "build_() called" << endl;
    n_layers = layers.length();
    if( n_layers <= 1 )
        return;
    // check value of initialization_method
    string im = lowerstring( initialization_method );
    if( im == "" || im == "uniform_sqrt" )
        initialization_method = "uniform_sqrt";
    else if( im == "uniform_linear" )
        initialization_method = im;
    else if( im == "zero" )
        initialization_method = im;
    else
        PLERROR( "RBMParameters::build_ - initialization_method\n"
                 "\"%s\" unknown.\n", initialization_method.c_str() );
    MODULE_LOG << "  initialization_method = \"" << initialization_method
        << "\"" << endl;
    // check value of fine_tuning_method
    string ftm = lowerstring( fine_tuning_method );
    if( ftm == "" | ftm == "none" )
        fine_tuning_method = "";
    else if( ftm == "cd" | ftm == "contrastive_divergence" )
        fine_tuning_method = "CD";
    else if( ftm == "egd" | ftm == "error_gradient_descent" )
        fine_tuning_method = "EGD";
    else if( ftm == "ws" | ftm == "wake_sleep" )
        fine_tuning_method = "WS";
    else
        PLERROR( "GaussianDBNRegression::build_ - fine_tuning_method \"%s\"\n"
                 "is unknown.\n", fine_tuning_method.c_str() );
    MODULE_LOG << "  fine_tuning_method = \"" << fine_tuning_method << "\""
        <<  endl;
    //TODO: build structure to store gradients during gradient descent
    if( training_schedule.length() != n_layers )
        training_schedule = TVec<int>( n_layers, 1000000 );
    MODULE_LOG << "  training_schedule = " << training_schedule << endl;
    MODULE_LOG << endl;
    build_layers();
    build_params();
}


| void PLearn::GaussianDBNRegression::build_layers | ( | ) |  [private] | 
        
Build the layers.
Definition at line 213 of file GaussianDBNRegression.cc.
References PLearn::endl(), i, PLearn::PLearner::inputsize(), PLearn::PLearner::inputsize_, last_layer, layers, n_layers, PLearn::PDistribution::n_predicted, PLearn::PDistribution::n_predictor, PLASSERT, PLearn::PLearner::random_gen, setPredictorPredictedSizes(), and target_layer.
Referenced by build_().
{
    MODULE_LOG << "build_layers() called" << endl;
    if( inputsize_ >= 0 )
    {
        PLASSERT( layers[0]->size + target_layer->size == inputsize() );
        setPredictorPredictedSizes( layers[0]->size,
                                    target_layer->size, false );
        MODULE_LOG << "  n_predictor = " << n_predictor << endl;
        MODULE_LOG << "  n_predicted = " << n_predicted << endl;
    }
    for( int i=0 ; i<n_layers ; i++ )
        layers[i]->random_gen = random_gen;
    target_layer->random_gen = random_gen;
    
    last_layer = layers[n_layers-1];
}


| void PLearn::GaussianDBNRegression::build_params | ( | ) |  [private] | 
        
Build the parameters if needed.
Definition at line 233 of file GaussianDBNRegression.cc.
References activation_gradients, PLearn::endl(), expectation_gradients, i, initialization_method, input_params, last_layer, layers, learning_rate, PLearn::TVec< T >::length(), n_layers, PLearn::PDistribution::n_predicted, output_gradient, params, PLERROR, PLearn::PLearner::random_gen, PLearn::TVec< T >::resize(), target_layer, and target_params.
Referenced by build_().
{
    MODULE_LOG << "build_params() called" << endl;
    if( params.length() == 0 )
    {
        input_params = new RBMQLParameters() ; 
        params.resize( n_layers-1 );
        for( int i=1 ; i<n_layers-1 ; i++ )
            params[i] = new RBMLLParameters();
        // params[0] is not being using, it is not being created
    }
    else if( params.length() != n_layers-1 )
        PLERROR( "GaussianDBNRegression::build_params - params.length() should\n"
                 "be equal to layers.length()-1 (%d != %d).\n",
                 params.length(), n_layers-1 );
    activation_gradients.resize( n_layers+1 );
    expectation_gradients.resize( n_layers+1 );
    output_gradient.resize( n_predicted );
    input_params->down_units_types = layers[0]->units_types;
    input_params->up_units_types = layers[1]->units_types;
    input_params->learning_rate = learning_rate;
    input_params->initialization_method = initialization_method;
    input_params->random_gen = random_gen;
    input_params->build();
    activation_gradients[0].resize( input_params->down_layer_size );
    expectation_gradients[0].resize( input_params->down_layer_size );
    for( int i=1 ; i<n_layers-1 ; i++ )
    {
        //TODO: call changeOptions instead
        
        params[i]->down_units_types = layers[i]->units_types;
        params[i]->up_units_types = layers[i+1]->units_types;
        params[i]->learning_rate = learning_rate;
        params[i]->initialization_method = initialization_method;
        params[i]->random_gen = random_gen;
        params[i]->build();
        
        activation_gradients[i].resize( params[i]->down_layer_size );
        expectation_gradients[i].resize( params[i]->down_layer_size );
        
    }
    if( target_layer && !target_params )
        target_params = new RBMLQParameters();
    //TODO: call changeOptions instead
    target_params->down_units_types = last_layer->units_types;
    target_params->up_units_types = target_layer->units_types;
    target_params->learning_rate = learning_rate;
    target_params->initialization_method = initialization_method;
    target_params->random_gen = random_gen;
    target_params->build();
}


Return cdf: P(Y<y | x).
Reimplemented from PLearn::PDistribution.
Definition at line 331 of file GaussianDBNRegression.cc.
References PLERROR.
{
    PLERROR("cdf not implemented for GaussianDBNRegression"); return 0;
}
| string PLearn::GaussianDBNRegression::classname | ( | ) |  const [virtual] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
Referenced by train().

| void PLearn::GaussianDBNRegression::computeCostsFromOutputs | ( | const Vec & | input, | 
| const Vec & | output, | ||
| const Vec & | target, | ||
| Vec & | costs | ||
| ) |  const [virtual] | 
        
Compute a cost, depending on the type of the first output : if it is the density or the log-density: NLL if it is the expectation: NLL and class error.
Reimplemented from PLearn::PDistribution.
Definition at line 837 of file GaussianDBNRegression.cc.
References c, PLearn::PDistribution::computeCostsFromOutputs(), i, PLearn::TVec< T >::length(), PLearn::PDistribution::outputs_def, PLearn::PDistribution::predicted_part, PLearn::TVec< T >::resize(), PLearn::PDistribution::splitCond(), and PLearn::square().
{
    char c = outputs_def[0];
    if( c == 'l' || c == 'd' )
        inherited::computeCostsFromOutputs(input, output, target, costs);
    else if( c == 'e' )
    {
        costs.resize( 1 );
        costs[0] = .0 ; 
        splitCond(input);
        
        int output_size = output.length(); 
        for(int i=0 ; i<output_size ; ++i) { 
            costs[0] += square(output[i] - predicted_part[i]) ;
        }
        costs[0] /= output_size ; 
    }
}

| void PLearn::GaussianDBNRegression::declareOptions | ( | OptionList & | ol | ) |  [static, protected] | 
        
Declares the class options.
Reimplemented from PLearn::PDistribution.
Definition at line 75 of file GaussianDBNRegression.cc.
References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::PDistribution::declareOptions(), fine_tuning_method, initialization_method, input_params, layers, learning_rate, PLearn::OptionBase::learntoption, n_layers, params, target_layer, target_params, training_schedule, use_sample_rather_than_expectation_in_positive_phase_statistics, and weight_decay.
{
    declareOption(ol, "learning_rate", &GaussianDBNRegression::learning_rate,
                  OptionBase::buildoption,
                  "Learning rate");
    declareOption(ol, "weight_decay", &GaussianDBNRegression::weight_decay,
                  OptionBase::buildoption,
                  "Weight decay");
    declareOption(ol, "initialization_method",
                  &GaussianDBNRegression::initialization_method,
                  OptionBase::buildoption,
                  "The method used to initialize the weights:\n"
                  "  - \"uniform_linear\" = a uniform law in [-1/d, 1/d]\n"
                  "  - \"uniform_sqrt\"   = a uniform law in [-1/sqrt(d),"
                  " 1/sqrt(d)]\n"
                  "  - \"zero\"           = all weights are set to 0,\n"
                  "where d = max( up_layer_size, down_layer_size ).\n");
    declareOption(ol, "training_schedule",
                  &GaussianDBNRegression::training_schedule,
                  OptionBase::buildoption,
                  "Number of examples to use during each of the different"
                  " greedy\n"
                  "steps of the training phase.\n");
    declareOption(ol, "fine_tuning_method",
                  &GaussianDBNRegression::fine_tuning_method,
                  OptionBase::buildoption,
                  "Method for fine-tuning the whole network after greedy"
                  " learning.\n"
                  "One of:\n"
                  "  - \"none\"\n"
                  "  - \"CD\" or \"contrastive_divergence\"\n"
                  "  - \"EGD\" or \"error_gradient_descent\"\n"
                  "  - \"WS\" or \"wake_sleep\".\n");
    declareOption(ol, "layers", &GaussianDBNRegression::layers,
                  OptionBase::buildoption,
                  "Layers that learn representations of the input,"
                  " unsupervisedly.\n"
                  "layers[0] is input layer.\n");
    declareOption(ol, "target_layer", &GaussianDBNRegression::target_layer,
                  OptionBase::buildoption,
                  "Target (or label) layer");
    declareOption(ol, "params", &GaussianDBNRegression::params,
                  OptionBase::buildoption,
                  "RBMParameters linking the unsupervised layers.\n"
                  "params[i] links layers[i] and layers[i+1], except for"
                  "params[n_layers-1],\n"
                  "that links layers[n_layers-1] and last_layer.\n");
    declareOption(ol, "target_params", &GaussianDBNRegression::target_params,
                  OptionBase::buildoption,
                  "Parameters linking target_layer and last_layer");
    
    declareOption(ol, "input_params", &GaussianDBNRegression::input_params,
                  OptionBase::buildoption,
                  "Parameters linking layer[0] and layer[1]");
    declareOption(ol, "use_sample_rather_than_expectation_in_positive_phase_statistics",
                  &GaussianDBNRegression::use_sample_rather_than_expectation_in_positive_phase_statistics,
                  OptionBase::buildoption,
                  "In positive phase statistics use output->sample * input\n"
                  "rather than output->expectation * input.\n");
    declareOption(ol, "n_layers", &GaussianDBNRegression::n_layers,
                  OptionBase::learntoption,
                  "Number of unsupervised layers, including input layer");
    // Now call the parent class' declareOptions().
    inherited::declareOptions(ol);
}

| static const PPath& PLearn::GaussianDBNRegression::declaringFile | ( | ) |  [inline, static] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 232 of file GaussianDBNRegression.h.
:
    //#####  Protected Options  ###############################################
| GaussianDBNRegression * PLearn::GaussianDBNRegression::deepCopy | ( | CopiesMap & | copies | ) |  const [virtual] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
Return probability density p(y | x)
Reimplemented from PLearn::PDistribution.
Definition at line 367 of file GaussianDBNRegression.cc.
References PLearn::argmax(), expectation(), i, PLearn::is_equal(), PLearn::PDistribution::n_predicted, PLASSERT, PLearn::TVec< T >::size(), and PLearn::PDistribution::store_expect.
Referenced by log_density().
{
    PLASSERT( y.size() == n_predicted );
    // TODO: 'y'[0] devrait plutot etre l'entier "index" lui-meme!
    int index = argmax( y );
    // If y != onehot( index ), then density is 0
    if( !is_equal( y[index], 1. ) )
        return 0;
    for( int i=0 ; i<n_predicted ; i++ )
        if( !is_equal( y[i], 0 ) && i != index )
            return 0;
    expectation( store_expect );
    return store_expect[index];
}


| void PLearn::GaussianDBNRegression::expectation | ( | Vec & | mu | ) |  const [virtual] | 
        
Return E[Y | x].
Reimplemented from PLearn::PDistribution.
Definition at line 339 of file GaussianDBNRegression.cc.
References i, input_params, last_layer, layers, n_layers, params, PLearn::PDistribution::predicted_size, PLearn::PDistribution::predictor_part, PLearn::TVec< T >::resize(), target_layer, and target_params.
Referenced by density(), fineTuneByGradientDescent(), fineTuneByGradientDescentLastLayer(), greedyStep(), and train().
{
    mu.resize( predicted_size );
    // Propagate input (predictor_part) until penultimate layer
    layers[0]->expectation << predictor_part;
    input_params->setAsDownInput(layers[0]->expectation) ; 
    layers[1]->getAllActivations( (RBMQLParameters*) input_params );
    layers[1]->computeExpectation();
    
    for( int i=1 ; i<n_layers-1 ; i++ )
    {
        params[i]->setAsDownInput( layers[i]->expectation );
        layers[i+1]->getAllActivations( (RBMLLParameters*) params[i] );
        layers[i+1]->computeExpectation();
    }
        
    target_params->setAsDownInput( last_layer->expectation );
    target_layer->getAllActivations( (RBMLQParameters*) target_params );
    target_layer->computeExpectation();
    mu << target_layer->expectation;
}


| void PLearn::GaussianDBNRegression::fineTuneByGradientDescent | ( | const Vec & | input | ) |  [protected, virtual] | 
        
Definition at line 785 of file GaussianDBNRegression.cc.
References activation_gradients, expectation(), expectation_gradients, i, input_params, layers, n_layers, output_gradient, params, PLearn::PDistribution::predicted_part, PLearn::TVec< T >::resize(), PLearn::TVec< T >::size(), PLearn::PDistribution::splitCond(), target_layer, and target_params.
{
    // split input in predictor_part and predicted_part
    splitCond(input);
    // compute predicted_part expectation, conditioned on predictor_part
    // (forward pass)
    expectation( output_gradient );
    int target_size = predicted_part.size() ; 
    expectation_gradients[n_layers].resize(target_size) ; 
    
    for(int i=0 ; i < target_size ; ++i) { 
        expectation_gradients[n_layers][i] = 2 * (output_gradient[i] - predicted_part[i]) ;
    }
    target_layer->bpropUpdate( target_layer->activations,
                               target_layer->expectation,
                               activation_gradients[n_layers] ,
                               expectation_gradients[n_layers]) ; 
    
    target_params->bpropUpdate( layers[n_layers-1]->expectation,
                               target_layer->activations,
                               expectation_gradients[n_layers-1],
                               activation_gradients[n_layers] );
    for( int i=n_layers-1 ; i>1 ; i-- )
    {
        layers[i]->bpropUpdate( layers[i]->activations,
                                layers[i]->expectation,
                                activation_gradients[i],
                                expectation_gradients[i] );
        params[i-1]->bpropUpdate( layers[i-1]->expectation,
                                  layers[i]->activations,
                                  expectation_gradients[i-1],
                                  activation_gradients[i] );
    }
    
        layers[1]->bpropUpdate( layers[1]->activations,
                                layers[1]->expectation,
                                activation_gradients[1],
                                expectation_gradients[1] );
        
        input_params->bpropUpdate( layers[0]->expectation,
                                  layers[1]->activations,
                                  expectation_gradients[0],
                                  activation_gradients[1] );
                                  
}

| void PLearn::GaussianDBNRegression::fineTuneByGradientDescentLastLayer | ( | const Vec & | input | ) |  [protected] | 
        
Definition at line 756 of file GaussianDBNRegression.cc.
References activation_gradients, expectation(), expectation_gradients, i, layers, n_layers, output_gradient, PLearn::PDistribution::predicted_part, PLearn::TVec< T >::resize(), PLearn::TVec< T >::size(), PLearn::PDistribution::splitCond(), target_layer, and target_params.
{
    // split input in predictor_part and predicted_part
    splitCond(input);
    // compute predicted_part expectation, conditioned on predictor_part
    // (forward pass)
    expectation( output_gradient );
    int target_size = predicted_part.size() ; 
    expectation_gradients[n_layers].resize(target_size) ; 
    
    for(int i=0 ; i < target_size ; ++i) { 
        expectation_gradients[n_layers][i] = 2 * (output_gradient[i] - predicted_part[i]) ;
    }
    target_layer->bpropUpdate( target_layer->activations,
                               target_layer->expectation,
                               activation_gradients[n_layers] ,
                               expectation_gradients[n_layers]) ; 
    
    target_params->bpropUpdate( layers[n_layers-1]->expectation,
                               target_layer->activations,
                               expectation_gradients[n_layers-1],
                               activation_gradients[n_layers] );
    
}

| void PLearn::GaussianDBNRegression::forget | ( | ) |  [virtual] | 
        
(Re-)initializes the PDistribution in its fresh state (that state may depend on the 'seed' option).
And sets 'stage' back to 0 (this is the stage of a fresh learner!). ### You may remove this method if your distribution does not ### implement it.
A typical forget() method should do the following:
Reimplemented from PLearn::PDistribution.
Definition at line 297 of file GaussianDBNRegression.cc.
References PLearn::endl(), i, input_params, layers, n_layers, params, PLearn::PDistribution::resetGenerator(), PLearn::PLearner::seed_, PLearn::PLearner::stage, target_layer, and target_params.
{
    MODULE_LOG << "forget() called" << endl;
    resetGenerator(seed_);
    input_params->forget() ; 
    for( int i=1 ; i<n_layers-1 ; i++ )
        params[i]->forget();
    for( int i=0 ; i<n_layers ; i++ )
        layers[i]->reset();
    target_params->forget();
    target_layer->reset();
    stage = 0;
}

| void PLearn::GaussianDBNRegression::generate | ( | Vec & | y | ) |  const [virtual] | 
        
Return a pseudo-random sample generated from the conditional distribution, of density p(y | x).
Reimplemented from PLearn::PDistribution.
Definition at line 323 of file GaussianDBNRegression.cc.
References PLERROR.
{
    PLERROR("generate not implemented for GaussianDBNRegression");
}
| OptionList & PLearn::GaussianDBNRegression::getOptionList | ( | ) |  const [virtual] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
| OptionMap & PLearn::GaussianDBNRegression::getOptionMap | ( | ) |  const [virtual] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
| RemoteMethodMap & PLearn::GaussianDBNRegression::getRemoteMethodMap | ( | ) |  const [virtual] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 59 of file GaussianDBNRegression.cc.
| TVec< string > PLearn::GaussianDBNRegression::getTestCostNames | ( | ) |  const [virtual] | 
        
Return [ "NLL" ] (the only cost computed by a PDistribution).
Reimplemented from PLearn::PDistribution.
Definition at line 861 of file GaussianDBNRegression.cc.
References PLearn::TVec< T >::append(), c, and PLearn::PDistribution::outputs_def.
{
    char c = outputs_def[0];
    TVec<string> result;
    if( c == 'l' || c == 'd' )
        result.append( "NLL" );
    else if( c == 'e' )
    {
        result.append( "MSE" );
    }
    return result;
}

| void PLearn::GaussianDBNRegression::greedyStep | ( | const Vec & | predictor, | 
| int | params_index | ||
| ) |  [protected, virtual] | 
        
Definition at line 677 of file GaussianDBNRegression.cc.
References PLearn::RBMQLParameters::accumulateNegStats(), PLearn::RBMQLParameters::accumulatePosStats(), expectation(), i, input_params, layers, params, PLearn::sample(), PLearn::RBMParameters::setAsDownInput(), PLearn::RBMParameters::setAsUpInput(), PLearn::RBMQLParameters::update(), and use_sample_rather_than_expectation_in_positive_phase_statistics.
Referenced by train().
{
    // deterministic propagation until we reach index
    layers[0]->expectation << predictor;
    input_params->setAsDownInput( layers[0]->expectation );
    layers[1]->getAllActivations( (RBMQLParameters*) input_params );
    layers[1]->computeExpectation();
        
    for( int i=1 ; i<index ; i++ )
    {
        params[i]->setAsDownInput( layers[i]->expectation );
        layers[i+1]->getAllActivations( (RBMLLParameters*) params[i] );
        layers[i+1]->computeExpectation();
    }
    // positive phase
    if (index == 0) {
        input_params->setAsDownInput( layers[index]->expectation );
        layers[index+1]->getAllActivations((RBMQLParameters*) input_params);
        layers[index+1]->computeExpectation();
        layers[index+1]->generateSample();
        if (use_sample_rather_than_expectation_in_positive_phase_statistics)
            input_params->accumulatePosStats(layers[index]->expectation,
                    layers[index+1]->sample );
        else
            input_params->accumulatePosStats(layers[index]->expectation,
                    layers[index+1]->expectation );
        // down propagation
        input_params->setAsUpInput( layers[index+1]->sample );
        layers[index]->getAllActivations( (RBMQLParameters*) input_params );
        // negative phase
        layers[index]->generateSample();
        input_params->setAsDownInput( layers[index]->sample );
        layers[index+1]->getAllActivations((RBMQLParameters*) input_params);
        layers[index+1]->computeExpectation();
        input_params->accumulateNegStats( layers[index]->sample,
                layers[index+1]->expectation );
        // update
        input_params->update();
    }
    else {
        params[index]->setAsDownInput( layers[index]->expectation );
        layers[index+1]->getAllActivations((RBMLLParameters*) params[index]);
        layers[index+1]->computeExpectation();
        layers[index+1]->generateSample();
        if (use_sample_rather_than_expectation_in_positive_phase_statistics)
            params[index]->accumulatePosStats(layers[index]->expectation,
                    layers[index+1]->sample );
        else
            params[index]->accumulatePosStats(layers[index]->expectation,
                    layers[index+1]->expectation );
        // down propagation
        params[index]->setAsUpInput( layers[index+1]->sample );
        layers[index]->getAllActivations( (RBMLLParameters*) params[index] );
        // negative phase
        layers[index]->generateSample();
        params[index]->setAsDownInput( layers[index]->sample );
        layers[index+1]->getAllActivations((RBMLLParameters*) params[index]);
        layers[index+1]->computeExpectation();
        params[index]->accumulateNegStats( layers[index]->sample,
                layers[index+1]->expectation );
        // update
        params[index]->update();
    }
    
}


Return log of probability density log(p(y | x)).
Reimplemented from PLearn::PDistribution.
Definition at line 389 of file GaussianDBNRegression.cc.
References density(), and pl_log.

| void PLearn::GaussianDBNRegression::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) |  [virtual] | 
        
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::PDistribution.
Definition at line 413 of file GaussianDBNRegression.cc.
References PLearn::deepCopyField(), input_params, last_layer, layers, PLearn::PDistribution::makeDeepCopyFromShallowCopy(), params, target_layer, target_params, and training_schedule.
{
    inherited::makeDeepCopyFromShallowCopy(copies);
    deepCopyField(layers, copies);
    deepCopyField(last_layer, copies);
    deepCopyField(target_layer, copies);
    deepCopyField(params, copies);
    deepCopyField(input_params, copies);
    deepCopyField(target_params, copies);
    deepCopyField(training_schedule, copies);
}

| void PLearn::GaussianDBNRegression::setPredictor | ( | const Vec & | predictor, | 
| bool | call_parent = true  | 
        ||
| ) |  const [virtual] | 
        
Set the value for the predictor part of a conditional probability.
Reimplemented from PLearn::PDistribution.
Definition at line 429 of file GaussianDBNRegression.cc.
References PLearn::PDistribution::setPredictor().
{
    if (call_parent)
        inherited::setPredictor(predictor, true);
    // ### Add here any specific code required by your subclass.
}

| bool PLearn::GaussianDBNRegression::setPredictorPredictedSizes | ( | int | the_predictor_size, | 
| int | the_predicted_size, | ||
| bool | call_parent = true  | 
        ||
| ) |  [virtual] | 
        
Generates a pseudo-random sample x from the reversed conditional distribution, of density p(x | y) (and NOT p(y | x)).
i.e., generates a "predictor" part given a "predicted" part, regardless of any previously set predictor. Set the 'predictor' and 'predicted' sizes for this distribution.
Reimplemented from PLearn::PDistribution.
Definition at line 440 of file GaussianDBNRegression.cc.
References layers, PLearn::PDistribution::n_predicted, PLearn::PDistribution::n_predictor, PLERROR, PLearn::PDistribution::setPredictorPredictedSizes(), PLearn::TVec< T >::size(), and target_layer.
Referenced by build_layers().
{
    bool sizes_have_changed = false;
    if (call_parent)
        sizes_have_changed = inherited::setPredictorPredictedSizes(
            the_predictor_size, the_predicted_size, true);
    // ### Add here any specific code required by your subclass.
    if( the_predictor_size >= 0 && the_predictor_size != layers[0]->size ||
        the_predicted_size >= 0 && the_predicted_size != target_layer->size )
        PLERROR( "GaussianDBNRegression::setPredictorPredictedSizes - \n"
                 "n_predictor should be equal to layer[0]->size (%d)\n"
                 "n_predicted should be equal to target_layer->size (%d).\n",
                 layers[0]->size, target_layer->size );
    n_predictor = layers[0]->size;
    n_predicted = target_layer->size;
    // Returned value.
    return sizes_have_changed;
}


Return survival function: P(Y>y | x).
Reimplemented from PLearn::PDistribution.
Definition at line 397 of file GaussianDBNRegression.cc.
References PLERROR.
{
    PLERROR("survival_fn not implemented for GaussianDBNRegression"); return 0;
}
| void PLearn::GaussianDBNRegression::train | ( | ) |  [virtual] | 
        
The role of the train method is to bring the learner up to stage == nstages, updating the train_stats collector with training costs measured on-line in the process.
Reimplemented from PLearn::PDistribution.
Definition at line 468 of file GaussianDBNRegression.cc.
References classname(), PLearn::endl(), expectation(), PLearn::VMat::getExample(), greedyStep(), i, PLearn::PLearner::initTrain(), PLearn::PLearner::inputsize(), j, last_layer, PLearn::VMat::length(), PLearn::linearRegression(), PLearn::min(), n_layers, PLearn::PDistribution::n_predicted, PLearn::PDistribution::n_predictor, PLearn::PLearner::nstages, PLearn::PLearner::report_progress, PLearn::sample(), PLearn::PDistribution::splitCond(), PLearn::PLearner::stage, PLearn::TVec< T >::subVec(), target_params, PLearn::PLearner::targetsize(), PLearn::tostring(), PLearn::PLearner::train_set, PLearn::PLearner::train_stats, training_schedule, and PLearn::ProgressBar::update().
{
    MODULE_LOG << "train() called" << endl;
    // The role of the train method is to bring the learner up to
    // stage==nstages, updating train_stats with training costs measured
    // on-line in the process.
    /* TYPICAL CODE:
    static Vec input;  // static so we don't reallocate memory each time...
    static Vec target; // (but be careful that static means shared!)
    input.resize(inputsize());    // the train_set's inputsize()
    target.resize(targetsize());  // the train_set's targetsize()
    real weight;
    // This generic PLearner method does a number of standard stuff useful for
    // (almost) any learner, and return 'false' if no training should take
    // place. See PLearner.h for more details.
    if (!initTrain())
        return;
    while(stage<nstages)
    {
        // clear statistics of previous epoch
        train_stats->forget();
        //... train for 1 stage, and update train_stats,
        // using train_set->getExample(input, target, weight)
        // and train_stats->update(train_costs)
        ++stage;
        train_stats->finalize(); // finalize statistics for this epoch
    }
    */
    Vec input( inputsize() );
    Vec target( targetsize() ); // unused
    real weight; // unused
    if( !initTrain() )
    {
        MODULE_LOG << "train() aborted" << endl;
        return;
    }
    int nsamples = train_set->length();
    MODULE_LOG << "  nsamples = " << nsamples << endl;
    // Let's define stage and nstages:
    //   - 0: fresh state, nothing is done
    //   - 1..n_layers-2: params[stage-1] is trained
    //   - n_layers-1: joint_params is trained (including params[n_layers-2])
    //   - n_layers: after the fine tuning
    MODULE_LOG << "initial stage = " << stage << endl;
    MODULE_LOG << "objective: nstages = " << nstages << endl;
        
    // clear stats of previous epoch
    train_stats->forget();
    for(int layer=0 ; layer<n_layers-1 ; ++layer) { 
            
        MODULE_LOG << "Training parameters between layers " << layer
            << " and " << layer+1 << endl;
        
        // this progress bar shows the number of loops through the whole
        // training set
        ProgressBar* pb = 0;
              
        int end_stage = min( training_schedule[layer], nstages );              
        if( report_progress && stage < end_stage )
        {
            pb = new ProgressBar( "Training layer "+tostring(layer)+
                    "of" + classname(), end_stage - stage );
        }
        for( ; stage < end_stage ; stage++ )
        {
                // sample is the index in the training set
                int sample = stage % train_set->length();
                train_set->getExample(sample, input, target, weight);
                greedyStep( input.subVec(0, n_predictor), layer );
                if( pb )
                {
                    if( layer == 0 )
                        pb->update(stage + 1);
                    else
                        pb->update(stage - training_schedule[layer-1] + 1);
                }
                Mat inputs(train_set.length() , n_predictor) ; 
                Mat outputs(train_set.length() , n_predicted);
                Mat theta(1 + n_predictor , n_predicted) ; 
                Vec output_value(n_predicted) ; 
                for(int i=0 ; i<train_set.length() ; ++i) { 
                    train_set->getExample(i, input, target, weight);
                    // split input in predictor_part and predicted_part
                    splitCond(input);
                    // compute predicted_part expectation, conditioned on predictor_part
                    // (forward pass)
                    expectation( output_value );
                    for(int j=0 ; j<n_predictor ; ++j) { 
                        inputs[i][j] = last_layer->expectation[j] ; 
//                        cout << last_layer->expectation[j] << " " ; 
                    }
                    for(int j=0 ; j<n_predicted ; ++j) { 
                        outputs[i][j] = input[j+n_predictor] ; 
                    }
                }
//                pout << "inputs " << endl << inputs << endl  ; 
                
//                pout << "outputs " << endl << outputs << endl  ; 
                
                linearRegression(inputs,outputs,0.0,theta); 
                // init the a_i term
                target_params->up_units_params[1].fill(1) ; 
//                pout << "Theta" << theta << endl ; 
                // set the bias (b_i)
                for(int i=0 ; i<n_predicted ; ++i) { 
                    target_params->up_units_params[0][i] = - 2.0 * theta[i][0] ; 
                }
                for(int i=0 ; i<n_predicted ; ++i) { 
                    for(int j=0 ; j<n_predictor ; ++j) { 
                        target_params->weights[i][j] = -2.0 * theta[j][i+1] ; 
                    }
                }
                
        }
            
    }
/*            
    MODULE_LOG << "Fine-tuning all parameters, using method "
    << fine_tuning_method << endl;
            if( fine_tuning_method == "" ) // do nothing
                sample += n_samples_to_see;
            else if( fine_tuning_method == "EGD" )
            {
                if( report_progress )
                    pb = new ProgressBar( "Training all " + classname()
                                          + " parameters by fine tuning",
                                          n_samples_to_see );
*/
                                          
/*
pout << "==================" << endl
    << "Before update:" << endl
    << "up:      " << joint_params->up_units_params << endl
    << "weights: " << endl << joint_params->weights << endl
    << "down:    " << joint_params->down_units_params << endl
    << endl;
// */
                // linear regression for last weights
                
                
/*
                int begin_sample = sample;
                int end_sample = begin_sample + n_samples_to_see;
                for( ; sample < end_sample ; sample++ )
                {
                    // sample is the index in the training set
                    int i = sample % train_set->length();
                    train_set->getExample(i, input, target, weight);
                    fineTuneByGradientDescentLastLayer( input );
                    if( pb )
                        pb->update( sample - begin_sample + 1 );
                }
                sample = begin_sample ; 
                for( ; sample < 100 ; sample++ )
                {
                    // sample is the index in the training set
                    int i = sample % train_set->length();
                    train_set->getExample(i, input, target, weight);
                    fineTuneByGradientDescent( input );
                    if( pb )
                        pb->update( sample - begin_sample + 1 );
                }
*/                
                
/*
pout << "-------" << endl
    << "After update:" << endl
    << "up:      " << joint_params->up_units_params << endl
    << "weights: " << endl << joint_params->weights << endl
    << "down:    " << joint_params->down_units_params << endl
    << endl;
// */
    train_stats->finalize(); // finalize statistics for this epoch
    MODULE_LOG << endl;
}

| void PLearn::GaussianDBNRegression::variance | ( | Mat & | cov | ) |  const [virtual] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 405 of file GaussianDBNRegression.cc.
References PLERROR.
{
    PLERROR("variance not implemented for GaussianDBNRegression");
}
Reimplemented from PLearn::PDistribution.
Definition at line 232 of file GaussianDBNRegression.h.
TVec< Vec > PLearn::GaussianDBNRegression::activation_gradients [mutable, protected] | 
        
gradients of cost wrt the activations (output of params)
Definition at line 248 of file GaussianDBNRegression.h.
Referenced by build_params(), fineTuneByGradientDescent(), and fineTuneByGradientDescentLastLayer().
TVec< Vec > PLearn::GaussianDBNRegression::expectation_gradients [mutable, protected] | 
        
gradients of cost wrt the expectations (output of layers)
Definition at line 251 of file GaussianDBNRegression.h.
Referenced by build_params(), fineTuneByGradientDescent(), and fineTuneByGradientDescentLastLayer().
Method for fine-tuning the whole network after greedy learning.
One of:
Definition at line 117 of file GaussianDBNRegression.h.
Referenced by build_(), and declareOptions().
The method used to initialize the weights:
Definition at line 81 of file GaussianDBNRegression.h.
Referenced by build_(), build_params(), and declareOptions().
Parameters linking input layer[0] and layer[1].
Definition at line 102 of file GaussianDBNRegression.h.
Referenced by build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), forget(), greedyStep(), and makeDeepCopyFromShallowCopy().
last_layer is layer[n_layers-1]
Definition at line 92 of file GaussianDBNRegression.h.
Referenced by build_layers(), build_params(), expectation(), makeDeepCopyFromShallowCopy(), and train().
Layers that learn representations of the input, layers[0] is input layer, layers[n_layers-1] is last layer.
Definition at line 89 of file GaussianDBNRegression.h.
Referenced by build_(), build_layers(), build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), fineTuneByGradientDescentLastLayer(), forget(), greedyStep(), makeDeepCopyFromShallowCopy(), and setPredictorPredictedSizes().
### declare public option fields (such as build options) here Start your comments with Doxygen-compatible comments such as //!
The learning rate
Definition at line 71 of file GaussianDBNRegression.h.
Referenced by build_params(), and declareOptions().
Number of layers, including input layer and last layer, but not target layer.
Definition at line 85 of file GaussianDBNRegression.h.
Referenced by build_(), build_layers(), build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), fineTuneByGradientDescentLastLayer(), forget(), and train().
Vec PLearn::GaussianDBNRegression::output_gradient [mutable, protected] | 
        
gradient wrt output activations
Definition at line 254 of file GaussianDBNRegression.h.
Referenced by build_params(), fineTuneByGradientDescent(), and fineTuneByGradientDescentLastLayer().
RBMParameters linking the unsupervised layers.
params[i] links layers[i] and layers[i+1], i>0
Definition at line 99 of file GaussianDBNRegression.h.
Referenced by build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), forget(), greedyStep(), and makeDeepCopyFromShallowCopy().
Target (or label) layer.
Definition at line 95 of file GaussianDBNRegression.h.
Referenced by build_layers(), build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), fineTuneByGradientDescentLastLayer(), forget(), makeDeepCopyFromShallowCopy(), and setPredictorPredictedSizes().
Parameters linking target_layer and last_layer.
Definition at line 105 of file GaussianDBNRegression.h.
Referenced by build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), fineTuneByGradientDescentLastLayer(), forget(), makeDeepCopyFromShallowCopy(), and train().
Number of examples to use during each of the different greedy steps of the training phase.
Definition at line 109 of file GaussianDBNRegression.h.
Referenced by build_(), declareOptions(), makeDeepCopyFromShallowCopy(), and train().
Definition at line 119 of file GaussianDBNRegression.h.
Referenced by declareOptions(), and greedyStep().
The weight decay.
Definition at line 74 of file GaussianDBNRegression.h.
Referenced by declareOptions().
 1.7.4