| 
    PLearn 0.1 
   | 
 
Does the same thing as Hinton's deep belief nets. More...
#include <GaussianDBNClassification.h>


Public Member Functions | |
| GaussianDBNClassification () | |
| Default constructor.   | |
| virtual real | density (const Vec &y) const | 
| Return probability density p(y | x)   | |
| virtual real | log_density (const Vec &y) const | 
| Return log of probability density log(p(y | x)).   | |
| virtual real | survival_fn (const Vec &y) const | 
| Return survival function: P(Y>y | x).   | |
| virtual real | cdf (const Vec &y) const | 
| Return cdf: P(Y<y | x).   | |
| virtual void | expectation (Vec &mu) const | 
| Return E[Y | x].   | |
| virtual void | variance (Mat &cov) const | 
| Return Var[Y | x].   | |
| virtual void | generate (Vec &y) const | 
| Return a pseudo-random sample generated from the conditional distribution, of density p(y | x).   | |
| virtual bool | setPredictorPredictedSizes (int the_predictor_size, int the_predicted_size, bool call_parent=true) | 
| Generates a pseudo-random sample x from the reversed conditional distribution, of density p(x | y) (and NOT p(y | x)).   | |
| virtual void | setPredictor (const Vec &predictor, bool call_parent=true) const | 
| Set the value for the predictor part of a conditional probability.   | |
| virtual void | forget () | 
| (Re-)initializes the PDistribution in its fresh state (that state may depend on the 'seed' option).   | |
| virtual void | train () | 
| The role of the train method is to bring the learner up to stage == nstages, updating the train_stats collector with training costs measured on-line in the process.   | |
| virtual void | computeCostsFromOutputs (const Vec &input, const Vec &output, const Vec &target, Vec &costs) const | 
| Compute a cost, depending on the type of the first output : if it is the density or the log-density: NLL if it is the expectation: NLL and class error.   | |
| virtual TVec< string > | getTestCostNames () const | 
| Return [ "NLL" ] (the only cost computed by a PDistribution).   | |
| virtual string | classname () const | 
| virtual OptionList & | getOptionList () const | 
| virtual OptionMap & | getOptionMap () const | 
| virtual RemoteMethodMap & | getRemoteMethodMap () const | 
| virtual GaussianDBNClassification * | deepCopy (CopiesMap &copies) const | 
| virtual void | build () | 
| Simply calls inherited::build() then build_().   | |
| virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) | 
| Transforms a shallow copy into a deep copy.   | |
Static Public Member Functions | |
| static string | _classname_ () | 
| static OptionList & | _getOptionList_ () | 
| static RemoteMethodMap & | _getRemoteMethodMap_ () | 
| static Object * | _new_instance_for_typemap_ () | 
| static bool | _isa_ (const Object *o) | 
| static void | _static_initialize_ () | 
| static const PPath & | declaringFile () | 
Public Attributes | |
| real | learning_rate | 
| ### declare public option fields (such as build options) here Start your comments with Doxygen-compatible comments such as //!   | |
| real | weight_decay | 
| The weight decay.   | |
| string | initialization_method | 
| The method used to initialize the weights:   | |
| int | n_layers | 
| Number of layers, including input layer and last layer, but not target layer.   | |
| TVec< PP< RBMLayer > > | layers | 
| Layers that learn representations of the input, layers[0] is input layer, layers[n_layers-1] is last layer.   | |
| PP< RBMLayer > | last_layer | 
| Last layer, learning joint representations of input and target.   | |
| PP< RBMMultinomialLayer > | target_layer | 
| Target (or label) layer.   | |
| PP< RBMMixedLayer > | joint_layer | 
| Concatenation of target_layer and layers[n_layers-2].   | |
| TVec< PP< RBMLLParameters > > | params | 
| RBMParameters linking the unsupervised layers.   | |
| PP< RBMQLParameters > | input_params | 
| PP< RBMLLParameters > | target_params | 
| Parameters linking target_layer and last_layer.   | |
| PP< RBMJointLLParameters > | joint_params | 
| Parameters linking joint_layer and last_layer.   | |
| TVec< int > | training_schedule | 
| Number of examples to use during each of the different greedy steps of the training phase.   | |
| string | fine_tuning_method | 
| Method for fine-tuning the whole network after greedy learning.   | |
| bool | use_sample_rather_than_expectation_in_positive_phase_statistics | 
Static Public Attributes | |
| static StaticInitializer | _static_initializer_ | 
Protected Member Functions | |
| virtual void | greedyStep (const Vec &predictor, int params_index) | 
| virtual void | jointGreedyStep (const Vec &input) | 
| virtual void | fineTuneByGradientDescent (const Vec &input) | 
Static Protected Member Functions | |
| static void | declareOptions (OptionList &ol) | 
| Declares the class options.   | |
Protected Attributes | |
| TVec< Vec > | activation_gradients | 
| gradients of cost wrt the activations (output of params)   | |
| TVec< Vec > | expectation_gradients | 
| gradients of cost wrt the expectations (output of layers)   | |
| Vec | output_gradient | 
| gradient wrt output activations   | |
Private Types | |
| typedef PDistribution | inherited | 
Private Member Functions | |
| void | build_ () | 
| This does the actual building.   | |
| void | build_layers () | 
| Build the layers.   | |
| void | build_params () | 
| Build the parameters if needed.   | |
Does the same thing as Hinton's deep belief nets.
Definition at line 67 of file GaussianDBNClassification.h.
typedef PDistribution PLearn::GaussianDBNClassification::inherited [private] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 69 of file GaussianDBNClassification.h.
| PLearn::GaussianDBNClassification::GaussianDBNClassification | ( | ) | 
Default constructor.
Definition at line 63 of file GaussianDBNClassification.cc.
References PLearn::PLearner::random_gen.
                                                     :
    learning_rate(0.),
    weight_decay(0.),
    use_sample_rather_than_expectation_in_positive_phase_statistics(false)
{
    random_gen = new PRandom();
}
| string PLearn::GaussianDBNClassification::_classname_ | ( | ) |  [static] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 58 of file GaussianDBNClassification.cc.
| OptionList & PLearn::GaussianDBNClassification::_getOptionList_ | ( | ) |  [static] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 58 of file GaussianDBNClassification.cc.
| RemoteMethodMap & PLearn::GaussianDBNClassification::_getRemoteMethodMap_ | ( | ) |  [static] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 58 of file GaussianDBNClassification.cc.
Reimplemented from PLearn::PDistribution.
Definition at line 58 of file GaussianDBNClassification.cc.
| Object * PLearn::GaussianDBNClassification::_new_instance_for_typemap_ | ( | ) |  [static] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 58 of file GaussianDBNClassification.cc.
| StaticInitializer GaussianDBNClassification::_static_initializer_ & PLearn::GaussianDBNClassification::_static_initialize_ | ( | ) |  [static] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 58 of file GaussianDBNClassification.cc.
| void PLearn::GaussianDBNClassification::build | ( | ) |  [virtual] | 
        
Simply calls inherited::build() then build_().
Reimplemented from PLearn::PDistribution.
Definition at line 168 of file GaussianDBNClassification.cc.
References PLearn::PDistribution::build(), and build_().
{
    // ### Nothing to add here, simply calls build_().
    inherited::build();
    build_();
}

| void PLearn::GaussianDBNClassification::build_ | ( | ) |  [private] | 
        
This does the actual building.
Reimplemented from PLearn::PDistribution.
Definition at line 178 of file GaussianDBNClassification.cc.
References build_layers(), build_params(), PLearn::endl(), fine_tuning_method, initialization_method, layers, PLearn::TVec< T >::length(), PLearn::lowerstring(), n_layers, PLERROR, and training_schedule.
Referenced by build().
{
    MODULE_LOG << "build_() called" << endl;
    n_layers = layers.length();
    if( n_layers <= 1 )
        return;
    // check value of initialization_method
    string im = lowerstring( initialization_method );
    if( im == "" || im == "uniform_sqrt" )
        initialization_method = "uniform_sqrt";
    else if( im == "uniform_linear" )
        initialization_method = im;
    else if( im == "zero" )
        initialization_method = im;
    else
        PLERROR( "RBMParameters::build_ - initialization_method\n"
                 "\"%s\" unknown.\n", initialization_method.c_str() );
    MODULE_LOG << "  initialization_method = \"" << initialization_method
        << "\"" << endl;
    // check value of fine_tuning_method
    string ftm = lowerstring( fine_tuning_method );
    if( ftm == "" | ftm == "none" )
        fine_tuning_method = "";
    else if( ftm == "cd" | ftm == "contrastive_divergence" )
        fine_tuning_method = "CD";
    else if( ftm == "egd" | ftm == "error_gradient_descent" )
        fine_tuning_method = "EGD";
    else if( ftm == "ws" | ftm == "wake_sleep" )
        fine_tuning_method = "WS";
    else
        PLERROR( "GaussianDBNClassification::build_ - fine_tuning_method \"%s\"\n"
                 "is unknown.\n", fine_tuning_method.c_str() );
    MODULE_LOG << "  fine_tuning_method = \"" << fine_tuning_method << "\""
        <<  endl;
    //TODO: build structure to store gradients during gradient descent
    if( training_schedule.length() != n_layers )
        training_schedule = TVec<int>( n_layers, 1000000 );
    MODULE_LOG << "  training_schedule = " << training_schedule << endl;
    MODULE_LOG << endl;
    build_layers();
    build_params();
}


| void PLearn::GaussianDBNClassification::build_layers | ( | ) |  [private] | 
        
Build the layers.
Definition at line 225 of file GaussianDBNClassification.cc.
References PLearn::endl(), i, PLearn::PLearner::inputsize(), PLearn::PLearner::inputsize_, joint_layer, last_layer, layers, n_layers, PLearn::PDistribution::n_predicted, PLearn::PDistribution::n_predictor, PLASSERT, PLearn::PLearner::random_gen, setPredictorPredictedSizes(), and target_layer.
Referenced by build_().
{
    MODULE_LOG << "build_layers() called" << endl;
    if( inputsize_ >= 0 )
    {
        PLASSERT( layers[0]->size + target_layer->size == inputsize() );
        setPredictorPredictedSizes( layers[0]->size,
                                    target_layer->size, false );
        MODULE_LOG << "  n_predictor = " << n_predictor << endl;
        MODULE_LOG << "  n_predicted = " << n_predicted << endl;
    }
    for( int i=0 ; i<n_layers ; i++ )
        layers[i]->random_gen = random_gen;
    target_layer->random_gen = random_gen;
    last_layer = layers[n_layers-1];
    // concatenate target_layer and layers[n_layers-2] into joint_layer
    TVec< PP<RBMLayer> > the_sub_layers( 2 );
    the_sub_layers[0] = target_layer;
    the_sub_layers[1] = layers[n_layers-2];
    joint_layer = new RBMMixedLayer( the_sub_layers );
    joint_layer->random_gen = random_gen;
}


| void PLearn::GaussianDBNClassification::build_params | ( | ) |  [private] | 
        
Build the parameters if needed.
Definition at line 251 of file GaussianDBNClassification.cc.
References activation_gradients, PLearn::endl(), expectation_gradients, i, initialization_method, input_params, joint_params, last_layer, layers, learning_rate, PLearn::TVec< T >::length(), n_layers, PLearn::PDistribution::n_predicted, output_gradient, params, PLERROR, PLearn::PLearner::random_gen, PLearn::TVec< T >::resize(), target_layer, and target_params.
Referenced by build_().
{
    MODULE_LOG << "build_params() called" << endl;
    if( params.length() == 0 )
    {
        input_params = new RBMQLParameters() ; 
        params.resize( n_layers-1 );
        for( int i=1 ; i<n_layers-1 ; i++ )
            params[i] = new RBMLLParameters();
        // params[0] is not being using, it is not being created
    }
    else if( params.length() != n_layers-1 )
        PLERROR( "GaussianDBNClassification::build_params - params.length() should\n"
                 "be equal to layers.length()-1 (%d != %d).\n",
                 params.length(), n_layers-1 );
    activation_gradients.resize( n_layers-1 );
    expectation_gradients.resize( n_layers-1 );
    output_gradient.resize( n_predicted );
    input_params->down_units_types = layers[0]->units_types;
    input_params->up_units_types = layers[1]->units_types;
    input_params->learning_rate = learning_rate;
    input_params->initialization_method = initialization_method;
    input_params->random_gen = random_gen;
    input_params->build();
    activation_gradients[0].resize( input_params->down_layer_size );
    expectation_gradients[0].resize( input_params->down_layer_size );
    for( int i=1 ; i<n_layers-1 ; i++ )
    {
        //TODO: call changeOptions instead
        
        params[i]->down_units_types = layers[i]->units_types;
        params[i]->up_units_types = layers[i+1]->units_types;
        params[i]->learning_rate = learning_rate;
        params[i]->initialization_method = initialization_method;
        params[i]->random_gen = random_gen;
        params[i]->build();
        activation_gradients[i].resize( params[i]->down_layer_size );
        expectation_gradients[i].resize( params[i]->down_layer_size );
    }
    if( target_layer && !target_params )
        target_params = new RBMLLParameters();
    //TODO: call changeOptions instead
    target_params->down_units_types = target_layer->units_types;
    target_params->up_units_types = last_layer->units_types;
    target_params->learning_rate = learning_rate;
    target_params->initialization_method = initialization_method;
    target_params->random_gen = random_gen;
    target_params->build();
    // build joint_params from params[n_layers-1] and target_params
    joint_params = new RBMJointLLParameters( target_params,
                                             params[n_layers-2] );
    joint_params->learning_rate = learning_rate;
    joint_params->random_gen = random_gen;
}


Return cdf: P(Y<y | x).
Reimplemented from PLearn::PDistribution.
Definition at line 352 of file GaussianDBNClassification.cc.
References PLERROR.
{
    PLERROR("cdf not implemented for GaussianDBNClassification"); return 0;
}
| string PLearn::GaussianDBNClassification::classname | ( | ) |  const [virtual] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 58 of file GaussianDBNClassification.cc.
Referenced by train().

| void PLearn::GaussianDBNClassification::computeCostsFromOutputs | ( | const Vec & | input, | 
| const Vec & | output, | ||
| const Vec & | target, | ||
| Vec & | costs | ||
| ) |  const [virtual] | 
        
Compute a cost, depending on the type of the first output : if it is the density or the log-density: NLL if it is the expectation: NLL and class error.
Reimplemented from PLearn::PDistribution.
Definition at line 827 of file GaussianDBNClassification.cc.
References PLearn::argmax(), c, PLearn::PDistribution::computeCostsFromOutputs(), i, PLearn::is_equal(), PLearn::PDistribution::n_predicted, PLearn::PDistribution::outputs_def, pl_log, PLASSERT, PLearn::PDistribution::predicted_part, PLearn::TVec< T >::resize(), and PLearn::PDistribution::splitCond().
{
    char c = outputs_def[0];
    if( c == 'l' || c == 'd' )
        inherited::computeCostsFromOutputs(input, output, target, costs);
    else if( c == 'e' )
    {
        costs.resize( 2 );
        splitCond(input);
        // actual_index is the actual 'target'
        int actual_index = argmax(predicted_part);
#ifdef BOUNDCHECK
        for( int i=0 ; i<n_predicted ; i++ )
            PLASSERT( is_equal( predicted_part[i], 0. ) ||
                    i == actual_index && is_equal( predicted_part[i], 1. ) );
#endif
        costs[0] = -pl_log( output[actual_index] );
        // predicted_index is the most probable predicted class
        int predicted_index = argmax(output);
        if( predicted_index == actual_index )
            costs[1] = 0;
        else
            costs[1] = 1;
    }
}

| void PLearn::GaussianDBNClassification::declareOptions | ( | OptionList & | ol | ) |  [static, protected] | 
        
Declares the class options.
Reimplemented from PLearn::PDistribution.
Definition at line 74 of file GaussianDBNClassification.cc.
References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::PDistribution::declareOptions(), fine_tuning_method, initialization_method, input_params, joint_layer, joint_params, last_layer, layers, learning_rate, PLearn::OptionBase::learntoption, n_layers, params, target_layer, target_params, training_schedule, use_sample_rather_than_expectation_in_positive_phase_statistics, and weight_decay.
{
    declareOption(ol, "learning_rate", &GaussianDBNClassification::learning_rate,
                  OptionBase::buildoption,
                  "Learning rate");
    declareOption(ol, "weight_decay", &GaussianDBNClassification::weight_decay,
                  OptionBase::buildoption,
                  "Weight decay");
    declareOption(ol, "initialization_method",
                  &GaussianDBNClassification::initialization_method,
                  OptionBase::buildoption,
                  "The method used to initialize the weights:\n"
                  "  - \"uniform_linear\" = a uniform law in [-1/d, 1/d]\n"
                  "  - \"uniform_sqrt\"   = a uniform law in [-1/sqrt(d),"
                  " 1/sqrt(d)]\n"
                  "  - \"zero\"           = all weights are set to 0,\n"
                  "where d = max( up_layer_size, down_layer_size ).\n");
    declareOption(ol, "training_schedule",
                  &GaussianDBNClassification::training_schedule,
                  OptionBase::buildoption,
                  "Number of examples to use during each of the different"
                  " greedy\n"
                  "steps of the training phase.\n");
    declareOption(ol, "fine_tuning_method",
                  &GaussianDBNClassification::fine_tuning_method,
                  OptionBase::buildoption,
                  "Method for fine-tuning the whole network after greedy"
                  " learning.\n"
                  "One of:\n"
                  "  - \"none\"\n"
                  "  - \"CD\" or \"contrastive_divergence\"\n"
                  "  - \"EGD\" or \"error_gradient_descent\"\n"
                  "  - \"WS\" or \"wake_sleep\".\n");
    declareOption(ol, "layers", &GaussianDBNClassification::layers,
                  OptionBase::buildoption,
                  "Layers that learn representations of the input,"
                  " unsupervisedly.\n"
                  "layers[0] is input layer.\n");
    declareOption(ol, "target_layer", &GaussianDBNClassification::target_layer,
                  OptionBase::buildoption,
                  "Target (or label) layer");
    declareOption(ol, "params", &GaussianDBNClassification::params,
                  OptionBase::buildoption,
                  "RBMParameters linking the unsupervised layers.\n"
                  "params[i] links layers[i] and layers[i+1], except for"
                  "params[n_layers-1],\n"
                  "that links layers[n_layers-1] and last_layer.\n");
    declareOption(ol, "target_params", &GaussianDBNClassification::target_params,
                  OptionBase::buildoption,
                  "Parameters linking target_layer and last_layer");
    
    declareOption(ol, "input_params", &GaussianDBNClassification::input_params,
                  OptionBase::buildoption,
                  "Parameters linking layer[0] and layer[1]");
    declareOption(ol, "use_sample_rather_than_expectation_in_positive_phase_statistics",
                  &GaussianDBNClassification::use_sample_rather_than_expectation_in_positive_phase_statistics,
                  OptionBase::buildoption,
                  "In positive phase statistics use output->sample * input\n"
                  "rather than output->expectation * input.\n");
    declareOption(ol, "n_layers", &GaussianDBNClassification::n_layers,
                  OptionBase::learntoption,
                  "Number of unsupervised layers, including input layer");
    declareOption(ol, "last_layer", &GaussianDBNClassification::last_layer,
                  OptionBase::learntoption,
                  "Last layer, learning joint representations of input and"
                  " target");
    declareOption(ol, "joint_layer", &GaussianDBNClassification::joint_layer,
                  OptionBase::learntoption,
                  "Concatenation of target_layer and layers[n_layers-1]");
    declareOption(ol, "joint_params", &GaussianDBNClassification::joint_params,
                  OptionBase::learntoption,
                  "Parameters linking joint_layer and last_layer");
    // Now call the parent class' declareOptions().
    inherited::declareOptions(ol);
}

| static const PPath& PLearn::GaussianDBNClassification::declaringFile | ( | ) |  [inline, static] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 245 of file GaussianDBNClassification.h.
:
    //#####  Protected Options  ###############################################
| GaussianDBNClassification * PLearn::GaussianDBNClassification::deepCopy | ( | CopiesMap & | copies | ) |  const [virtual] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 58 of file GaussianDBNClassification.cc.
Return probability density p(y | x)
Reimplemented from PLearn::PDistribution.
Definition at line 391 of file GaussianDBNClassification.cc.
References PLearn::argmax(), expectation(), i, PLearn::is_equal(), PLearn::PDistribution::n_predicted, PLASSERT, PLearn::TVec< T >::size(), and PLearn::PDistribution::store_expect.
Referenced by log_density().
{
    PLASSERT( y.size() == n_predicted );
    // TODO: 'y'[0] devrait plutot etre l'entier "index" lui-meme!
    int index = argmax( y );
    // If y != onehot( index ), then density is 0
    if( !is_equal( y[index], 1. ) )
        return 0;
    for( int i=0 ; i<n_predicted ; i++ )
        if( !is_equal( y[i], 0 ) && i != index )
            return 0;
    expectation( store_expect );
    return store_expect[index];
}


| void PLearn::GaussianDBNClassification::expectation | ( | Vec & | mu | ) |  const [virtual] | 
        
Return E[Y | x].
Reimplemented from PLearn::PDistribution.
Definition at line 360 of file GaussianDBNClassification.cc.
References i, input_params, joint_params, layers, n_layers, params, PLearn::PDistribution::predicted_size, PLearn::PDistribution::predictor_part, PLearn::TVec< T >::resize(), and target_layer.
Referenced by density(), fineTuneByGradientDescent(), greedyStep(), and jointGreedyStep().
{
    mu.resize( predicted_size );
    // Propagate input (predictor_part) until penultimate layer
    layers[0]->expectation << predictor_part;
    input_params->setAsDownInput(layers[0]->expectation) ; 
    layers[1]->getAllActivations( (RBMQLParameters*) input_params );
    layers[1]->computeExpectation();
    
    for( int i=1 ; i<n_layers-2 ; i++ )
    {
        params[i]->setAsDownInput( layers[i]->expectation );
        layers[i+1]->getAllActivations( (RBMLLParameters*) params[i] );
        layers[i+1]->computeExpectation();
    }
    // Set layers[n_layers-2]->expectation (penultimate) as conditionning input
    // of joint_params
    joint_params->setAsCondInput( layers[n_layers-2]->expectation );
    // Get all activations on target_layer from target_params
    target_layer->getAllActivations( (RBMLLParameters*) joint_params );
    target_layer->computeExpectation();
    mu << target_layer->expectation;
}


| void PLearn::GaussianDBNClassification::fineTuneByGradientDescent | ( | const Vec & | input | ) |  [protected, virtual] | 
        
Definition at line 797 of file GaussianDBNClassification.cc.
References activation_gradients, PLearn::argmax(), expectation(), expectation_gradients, i, joint_params, layers, n_layers, output_gradient, params, PLearn::PDistribution::predicted_part, PLearn::PDistribution::splitCond(), and target_layer.
Referenced by train().
{
    // split input in predictor_part and predicted_part
    splitCond(input);
    // compute predicted_part expectation, conditioned on predictor_part
    // (forward pass)
    expectation( output_gradient );
    int actual_index = argmax(predicted_part);
    output_gradient[actual_index] -= 1.;
    joint_params->bpropUpdate( layers[n_layers-2]->expectation,
                               target_layer->expectation,
                               expectation_gradients[n_layers-2],
                               output_gradient );
    for( int i=n_layers-2 ; i>0 ; i-- )
    {
        layers[i]->bpropUpdate( layers[i]->activations,
                                layers[i]->expectation,
                                activation_gradients[i],
                                expectation_gradients[i] );
        params[i-1]->bpropUpdate( layers[i-1]->expectation,
                                  layers[i]->activations,
                                  expectation_gradients[i-1],
                                  activation_gradients[i] );
    }
}


| void PLearn::GaussianDBNClassification::forget | ( | ) |  [virtual] | 
        
(Re-)initializes the PDistribution in its fresh state (that state may depend on the 'seed' option).
And sets 'stage' back to 0 (this is the stage of a fresh learner!). ### You may remove this method if your distribution does not ### implement it.
A typical forget() method should do the following:
Reimplemented from PLearn::PDistribution.
Definition at line 318 of file GaussianDBNClassification.cc.
References PLearn::endl(), i, input_params, layers, n_layers, params, PLearn::PDistribution::resetGenerator(), PLearn::PLearner::seed_, PLearn::PLearner::stage, target_layer, and target_params.
{
    MODULE_LOG << "forget() called" << endl;
    resetGenerator(seed_);
    input_params->forget() ; 
    for( int i=1 ; i<n_layers-1 ; i++ )
        params[i]->forget();
    for( int i=0 ; i<n_layers ; i++ )
        layers[i]->reset();
    target_params->forget();
    target_layer->reset();
    stage = 0;
}

| void PLearn::GaussianDBNClassification::generate | ( | Vec & | y | ) |  const [virtual] | 
        
Return a pseudo-random sample generated from the conditional distribution, of density p(y | x).
Reimplemented from PLearn::PDistribution.
Definition at line 344 of file GaussianDBNClassification.cc.
References PLERROR.
{
    PLERROR("generate not implemented for GaussianDBNClassification");
}
| OptionList & PLearn::GaussianDBNClassification::getOptionList | ( | ) |  const [virtual] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 58 of file GaussianDBNClassification.cc.
| OptionMap & PLearn::GaussianDBNClassification::getOptionMap | ( | ) |  const [virtual] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 58 of file GaussianDBNClassification.cc.
| RemoteMethodMap & PLearn::GaussianDBNClassification::getRemoteMethodMap | ( | ) |  const [virtual] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 58 of file GaussianDBNClassification.cc.
| TVec< string > PLearn::GaussianDBNClassification::getTestCostNames | ( | ) |  const [virtual] | 
        
Return [ "NLL" ] (the only cost computed by a PDistribution).
Reimplemented from PLearn::PDistribution.
Definition at line 858 of file GaussianDBNClassification.cc.
References PLearn::TVec< T >::append(), c, and PLearn::PDistribution::outputs_def.
{
    char c = outputs_def[0];
    TVec<string> result;
    if( c == 'l' || c == 'd' )
        result.append( "NLL" );
    else if( c == 'e' )
    {
        result.append( "NLL" );
        result.append( "class_error" );
    }
    return result;
}

| void PLearn::GaussianDBNClassification::greedyStep | ( | const Vec & | predictor, | 
| int | params_index | ||
| ) |  [protected, virtual] | 
        
Definition at line 672 of file GaussianDBNClassification.cc.
References PLearn::RBMQLParameters::accumulateNegStats(), PLearn::RBMQLParameters::accumulatePosStats(), expectation(), i, input_params, layers, params, PLearn::sample(), PLearn::RBMParameters::setAsDownInput(), PLearn::RBMParameters::setAsUpInput(), PLearn::RBMQLParameters::update(), and use_sample_rather_than_expectation_in_positive_phase_statistics.
Referenced by train().
{
    // deterministic propagation until we reach index
    layers[0]->expectation << predictor;
    input_params->setAsDownInput( layers[0]->expectation );
    layers[1]->getAllActivations( (RBMQLParameters*) input_params );
    layers[1]->computeExpectation();
        
    for( int i=1 ; i<index ; i++ )
    {
        params[i]->setAsDownInput( layers[i]->expectation );
        layers[i+1]->getAllActivations( (RBMLLParameters*) params[i] );
        layers[i+1]->computeExpectation();
    }
    // positive phase
    if (index == 0) {
        input_params->setAsDownInput( layers[index]->expectation );
        layers[index+1]->getAllActivations((RBMQLParameters*) input_params);
        layers[index+1]->computeExpectation();
        layers[index+1]->generateSample();
        if (use_sample_rather_than_expectation_in_positive_phase_statistics)
            input_params->accumulatePosStats(layers[index]->expectation,
                    layers[index+1]->sample );
        else
            input_params->accumulatePosStats(layers[index]->expectation,
                    layers[index+1]->expectation );
        // down propagation
        input_params->setAsUpInput( layers[index+1]->sample );
        layers[index]->getAllActivations( (RBMQLParameters*) input_params );
        // negative phase
        layers[index]->generateSample();
        input_params->setAsDownInput( layers[index]->sample );
        layers[index+1]->getAllActivations((RBMQLParameters*) input_params);
        layers[index+1]->computeExpectation();
        input_params->accumulateNegStats( layers[index]->sample,
                layers[index+1]->expectation );
        // update
        input_params->update();
    }
    else {
        params[index]->setAsDownInput( layers[index]->expectation );
        layers[index+1]->getAllActivations((RBMLLParameters*) params[index]);
        layers[index+1]->computeExpectation();
        layers[index+1]->generateSample();
        if (use_sample_rather_than_expectation_in_positive_phase_statistics)
            params[index]->accumulatePosStats(layers[index]->expectation,
                    layers[index+1]->sample );
        else
            params[index]->accumulatePosStats(layers[index]->expectation,
                    layers[index+1]->expectation );
        // down propagation
        params[index]->setAsUpInput( layers[index+1]->sample );
        layers[index]->getAllActivations( (RBMLLParameters*) params[index] );
        // negative phase
        layers[index]->generateSample();
        params[index]->setAsDownInput( layers[index]->sample );
        layers[index+1]->getAllActivations((RBMLLParameters*) params[index]);
        layers[index+1]->computeExpectation();
        params[index]->accumulateNegStats( layers[index]->sample,
                layers[index+1]->expectation );
        // update
        params[index]->update();
    }
    
}


| void PLearn::GaussianDBNClassification::jointGreedyStep | ( | const Vec & | input | ) |  [protected, virtual] | 
        
Definition at line 749 of file GaussianDBNClassification.cc.
References PLearn::RBMLLParameters::accumulateNegStats(), PLearn::RBMLLParameters::accumulatePosStats(), expectation(), i, input_params, joint_layer, joint_params, last_layer, layers, n_layers, PLearn::PDistribution::n_predicted, PLearn::PDistribution::n_predictor, params, PLearn::RBMParameters::setAsDownInput(), PLearn::RBMParameters::setAsUpInput(), PLearn::TVec< T >::subVec(), target_layer, PLearn::RBMLLParameters::update(), and use_sample_rather_than_expectation_in_positive_phase_statistics.
Referenced by train().
{
    // deterministic propagation until we reach n_layers-2, setting the input
    // of the "input" part of joint_layer
    layers[0]->expectation << input.subVec( 0, n_predictor );
    input_params->setAsDownInput( layers[0]->expectation );
    layers[1]->getAllActivations( (RBMQLParameters*) input_params );
    layers[1]->computeExpectation();
    
    
    for( int i=1 ; i<n_layers-2 ; i++ )
    {
        params[i]->setAsDownInput( layers[i]->expectation );
        layers[i+1]->getAllActivations( (RBMLLParameters*) params[i] );
        layers[i+1]->computeExpectation();
    }
    // now fill the "target" part of joint_layer
    target_layer->expectation << input.subVec( n_predictor, n_predicted );
    // positive phase
    joint_params->setAsDownInput( joint_layer->expectation );
    last_layer->getAllActivations( (RBMLLParameters*) joint_params );
    last_layer->computeExpectation();
    last_layer->generateSample();
    if (use_sample_rather_than_expectation_in_positive_phase_statistics)
        joint_params->accumulatePosStats( joint_layer->expectation,
                                          last_layer->sample );
    else
        joint_params->accumulatePosStats( joint_layer->expectation,
                                          last_layer->expectation );
    // down propagation
    joint_params->setAsUpInput( last_layer->sample );
    joint_layer->getAllActivations( (RBMLLParameters*) joint_params );
    // negative phase
    joint_layer->generateSample();
    joint_params->setAsDownInput( joint_layer->sample );
    last_layer->getAllActivations( (RBMLLParameters*) joint_params );
    last_layer->computeExpectation();
    joint_params->accumulateNegStats( joint_layer->sample,
                                      last_layer->expectation );
    // update
    joint_params->update();
}


Return log of probability density log(p(y | x)).
Reimplemented from PLearn::PDistribution.
Definition at line 413 of file GaussianDBNClassification.cc.
References density(), and pl_log.

| void PLearn::GaussianDBNClassification::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) |  [virtual] | 
        
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::PDistribution.
Definition at line 437 of file GaussianDBNClassification.cc.
References PLearn::deepCopyField(), input_params, joint_layer, joint_params, last_layer, layers, PLearn::PDistribution::makeDeepCopyFromShallowCopy(), params, target_layer, target_params, and training_schedule.
{
    inherited::makeDeepCopyFromShallowCopy(copies);
    deepCopyField(layers, copies);
    deepCopyField(last_layer, copies);
    deepCopyField(target_layer, copies);
    deepCopyField(joint_layer, copies);
    deepCopyField(params, copies);
    deepCopyField(joint_params, copies);
    deepCopyField(input_params, copies);
    deepCopyField(target_params, copies);
    deepCopyField(training_schedule, copies);
}

| void PLearn::GaussianDBNClassification::setPredictor | ( | const Vec & | predictor, | 
| bool | call_parent = true  | 
        ||
| ) |  const [virtual] | 
        
Set the value for the predictor part of a conditional probability.
Reimplemented from PLearn::PDistribution.
Definition at line 455 of file GaussianDBNClassification.cc.
References PLearn::PDistribution::setPredictor().
{
    if (call_parent)
        inherited::setPredictor(predictor, true);
    // ### Add here any specific code required by your subclass.
}

| bool PLearn::GaussianDBNClassification::setPredictorPredictedSizes | ( | int | the_predictor_size, | 
| int | the_predicted_size, | ||
| bool | call_parent = true  | 
        ||
| ) |  [virtual] | 
        
Generates a pseudo-random sample x from the reversed conditional distribution, of density p(x | y) (and NOT p(y | x)).
i.e., generates a "predictor" part given a "predicted" part, regardless of any previously set predictor. Set the 'predictor' and 'predicted' sizes for this distribution.
Reimplemented from PLearn::PDistribution.
Definition at line 466 of file GaussianDBNClassification.cc.
References layers, PLearn::PDistribution::n_predicted, PLearn::PDistribution::n_predictor, PLERROR, PLearn::PDistribution::setPredictorPredictedSizes(), PLearn::TVec< T >::size(), and target_layer.
Referenced by build_layers().
{
    bool sizes_have_changed = false;
    if (call_parent)
        sizes_have_changed = inherited::setPredictorPredictedSizes(
            the_predictor_size, the_predicted_size, true);
    // ### Add here any specific code required by your subclass.
    if( the_predictor_size >= 0 && the_predictor_size != layers[0]->size ||
        the_predicted_size >= 0 && the_predicted_size != target_layer->size )
        PLERROR( "GaussianDBNClassification::setPredictorPredictedSizes - \n"
                 "n_predictor should be equal to layer[0]->size (%d)\n"
                 "n_predicted should be equal to target_layer->size (%d).\n",
                 layers[0]->size, target_layer->size );
    n_predictor = layers[0]->size;
    n_predicted = target_layer->size;
    // Returned value.
    return sizes_have_changed;
}


Return survival function: P(Y>y | x).
Reimplemented from PLearn::PDistribution.
Definition at line 421 of file GaussianDBNClassification.cc.
References PLERROR.
{
    PLERROR("survival_fn not implemented for GaussianDBNClassification"); return 0;
}
| void PLearn::GaussianDBNClassification::train | ( | ) |  [virtual] | 
        
The role of the train method is to bring the learner up to stage == nstages, updating the train_stats collector with training costs measured on-line in the process.
Reimplemented from PLearn::PDistribution.
Definition at line 494 of file GaussianDBNClassification.cc.
References classname(), PLearn::endl(), fine_tuning_method, fineTuneByGradientDescent(), PLearn::VMat::getExample(), greedyStep(), i, PLearn::PLearner::initTrain(), PLearn::PLearner::inputsize(), jointGreedyStep(), PLearn::VMat::length(), n_layers, PLearn::PDistribution::n_predictor, PLearn::PLearner::nstages, PLERROR, PLearn::PLearner::report_progress, PLearn::sample(), PLearn::PLearner::stage, PLearn::TVec< T >::subVec(), PLearn::PLearner::targetsize(), PLearn::tostring(), PLearn::PLearner::train_set, PLearn::PLearner::train_stats, training_schedule, and PLearn::ProgressBar::update().
{
    MODULE_LOG << "train() called" << endl;
    // The role of the train method is to bring the learner up to
    // stage==nstages, updating train_stats with training costs measured
    // on-line in the process.
    /* TYPICAL CODE:
    static Vec input;  // static so we don't reallocate memory each time...
    static Vec target; // (but be careful that static means shared!)
    input.resize(inputsize());    // the train_set's inputsize()
    target.resize(targetsize());  // the train_set's targetsize()
    real weight;
    // This generic PLearner method does a number of standard stuff useful for
    // (almost) any learner, and return 'false' if no training should take
    // place. See PLearner.h for more details.
    if (!initTrain())
        return;
    while(stage<nstages)
    {
        // clear statistics of previous epoch
        train_stats->forget();
        //... train for 1 stage, and update train_stats,
        // using train_set->getExample(input, target, weight)
        // and train_stats->update(train_costs)
        ++stage;
        train_stats->finalize(); // finalize statistics for this epoch
    }
    */
    Vec input( inputsize() );
    Vec target( targetsize() ); // unused
    real weight; // unused
    if( !initTrain() )
    {
        MODULE_LOG << "train() aborted" << endl;
        return;
    }
    int nsamples = train_set->length();
    int sample = 0;
    MODULE_LOG << "  nsamples = " << nsamples << endl;
    // Let's define stage and nstages:
    //   - 0: fresh state, nothing is done
    //   - 1..n_layers-2: params[stage-1] is trained
    //   - n_layers-1: joint_params is trained (including params[n_layers-2])
    //   - n_layers: after the fine tuning
    MODULE_LOG << "initial stage = " << stage << endl;
    MODULE_LOG << "objective: nstages = " << nstages << endl;
    for( ; stage < nstages ; stage++ )
    {
        // clear stats of previous epoch
        train_stats->forget();
        // loops over the training set, until training_schedule[stage] examples
        // have been seen.
        // TODO: modify the training set used?
        int layer = stage;
        int n_samples_to_see = training_schedule[stage];
        // this progress bar shows the number of loops through the whole
        // training set
        ProgressBar* pb = 0;
        if( stage < n_layers-2 )
        {
            MODULE_LOG << "Training parameters between layers " << stage
                << " and " << stage+1 << endl;
            if( report_progress )
                pb = new ProgressBar( "Training " + classname()
                                      + " parameters between layers "
                                      + tostring(stage) + " and "
                                      + tostring(stage+1),
                                      n_samples_to_see );
            int begin_sample = sample;
            int end_sample = begin_sample + n_samples_to_see;
            for( ; sample < end_sample ; sample++ )
            {
                // sample is the index in the training set
                int i = sample % train_set->length();
                train_set->getExample(i, input, target, weight);
                greedyStep( input.subVec(0, n_predictor), layer );
                if( pb )
                    pb->update( sample - begin_sample + 1 );
            }
        }
        else if( stage == n_layers-2 )
        {
            MODULE_LOG << "Training joint parameters, between target,"
                << " penultimate (" << n_layers-2 << ")," << endl
                << "and last (" << n_layers-1 << ") layers." << endl;
            if( report_progress )
                pb = new ProgressBar( "Training " + classname()
                                      + " parameters between target, "
                                      + tostring(stage) + " and "
                                      + tostring(stage+1) + " layers",
                                      n_samples_to_see );
            int begin_sample = sample;
            int end_sample = begin_sample + n_samples_to_see;
            for( ; sample < end_sample ; sample++ )
            {
                // sample is the index in the training set
                int i = sample % train_set->length();
                train_set->getExample(i, input, target, weight);
                jointGreedyStep( input );
                if( pb )
                    pb->update( sample - begin_sample + 1 );
            }
        }
        else if( stage == n_layers-1 )
        {
            MODULE_LOG << "Fine-tuning all parameters, using method "
                << fine_tuning_method << endl;
            if( fine_tuning_method == "" ) // do nothing
                sample += n_samples_to_see;
            else if( fine_tuning_method == "EGD" )
            {
                if( report_progress )
                    pb = new ProgressBar( "Training all " + classname()
                                          + " parameters by fine tuning",
                                          n_samples_to_see );
/*
pout << "==================" << endl
    << "Before update:" << endl
    << "up:      " << joint_params->up_units_params << endl
    << "weights: " << endl << joint_params->weights << endl
    << "down:    " << joint_params->down_units_params << endl
    << endl;
// */
                int begin_sample = sample;
                int end_sample = begin_sample + n_samples_to_see;
                for( ; sample < end_sample ; sample++ )
                {
                    // sample is the index in the training set
                    int i = sample % train_set->length();
                    train_set->getExample(i, input, target, weight);
                    fineTuneByGradientDescent( input );
                    if( pb )
                        pb->update( sample - begin_sample + 1 );
                }
/*
pout << "-------" << endl
    << "After update:" << endl
    << "up:      " << joint_params->up_units_params << endl
    << "weights: " << endl << joint_params->weights << endl
    << "down:    " << joint_params->down_units_params << endl
    << endl;
// */
            }
            else
                PLERROR( "Fine-tuning methods other than \"EGD\" are not"
                         " implemented yet." );
        }
        train_stats->finalize(); // finalize statistics for this epoch
    }
    MODULE_LOG << endl;
}

| void PLearn::GaussianDBNClassification::variance | ( | Mat & | cov | ) |  const [virtual] | 
        
Reimplemented from PLearn::PDistribution.
Definition at line 429 of file GaussianDBNClassification.cc.
References PLERROR.
{
    PLERROR("variance not implemented for GaussianDBNClassification");
}
Reimplemented from PLearn::PDistribution.
Definition at line 245 of file GaussianDBNClassification.h.
TVec< Vec > PLearn::GaussianDBNClassification::activation_gradients [mutable, protected] | 
        
gradients of cost wrt the activations (output of params)
Definition at line 261 of file GaussianDBNClassification.h.
Referenced by build_params(), and fineTuneByGradientDescent().
TVec< Vec > PLearn::GaussianDBNClassification::expectation_gradients [mutable, protected] | 
        
gradients of cost wrt the expectations (output of layers)
Definition at line 264 of file GaussianDBNClassification.h.
Referenced by build_params(), and fineTuneByGradientDescent().
Method for fine-tuning the whole network after greedy learning.
One of:
Definition at line 130 of file GaussianDBNClassification.h.
Referenced by build_(), declareOptions(), and train().
The method used to initialize the weights:
Definition at line 88 of file GaussianDBNClassification.h.
Referenced by build_(), build_params(), and declareOptions().
Definition at line 111 of file GaussianDBNClassification.h.
Referenced by build_params(), declareOptions(), expectation(), forget(), greedyStep(), jointGreedyStep(), and makeDeepCopyFromShallowCopy().
Concatenation of target_layer and layers[n_layers-2].
Definition at line 105 of file GaussianDBNClassification.h.
Referenced by build_layers(), declareOptions(), jointGreedyStep(), and makeDeepCopyFromShallowCopy().
Parameters linking joint_layer and last_layer.
Contains params[n_layers-2] and target_params.
Definition at line 118 of file GaussianDBNClassification.h.
Referenced by build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), jointGreedyStep(), and makeDeepCopyFromShallowCopy().
Last layer, learning joint representations of input and target.
Definition at line 99 of file GaussianDBNClassification.h.
Referenced by build_layers(), build_params(), declareOptions(), jointGreedyStep(), and makeDeepCopyFromShallowCopy().
Layers that learn representations of the input, layers[0] is input layer, layers[n_layers-1] is last layer.
Definition at line 96 of file GaussianDBNClassification.h.
Referenced by build_(), build_layers(), build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), forget(), greedyStep(), jointGreedyStep(), makeDeepCopyFromShallowCopy(), and setPredictorPredictedSizes().
### declare public option fields (such as build options) here Start your comments with Doxygen-compatible comments such as //!
The learning rate
Definition at line 78 of file GaussianDBNClassification.h.
Referenced by build_params(), and declareOptions().
Number of layers, including input layer and last layer, but not target layer.
Definition at line 92 of file GaussianDBNClassification.h.
Referenced by build_(), build_layers(), build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), forget(), jointGreedyStep(), and train().
Vec PLearn::GaussianDBNClassification::output_gradient [mutable, protected] | 
        
gradient wrt output activations
Definition at line 267 of file GaussianDBNClassification.h.
Referenced by build_params(), and fineTuneByGradientDescent().
RBMParameters linking the unsupervised layers.
params[i] links layers[i] and layers[i+1], i>0
Definition at line 109 of file GaussianDBNClassification.h.
Referenced by build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), forget(), greedyStep(), jointGreedyStep(), and makeDeepCopyFromShallowCopy().
Target (or label) layer.
Definition at line 102 of file GaussianDBNClassification.h.
Referenced by build_layers(), build_params(), declareOptions(), expectation(), fineTuneByGradientDescent(), forget(), jointGreedyStep(), makeDeepCopyFromShallowCopy(), and setPredictorPredictedSizes().
Parameters linking target_layer and last_layer.
Definition at line 114 of file GaussianDBNClassification.h.
Referenced by build_params(), declareOptions(), forget(), and makeDeepCopyFromShallowCopy().
Number of examples to use during each of the different greedy steps of the training phase.
Definition at line 122 of file GaussianDBNClassification.h.
Referenced by build_(), declareOptions(), makeDeepCopyFromShallowCopy(), and train().
| bool PLearn::GaussianDBNClassification::use_sample_rather_than_expectation_in_positive_phase_statistics | 
Definition at line 132 of file GaussianDBNClassification.h.
Referenced by declareOptions(), greedyStep(), and jointGreedyStep().
The weight decay.
Definition at line 81 of file GaussianDBNClassification.h.
Referenced by declareOptions().
 1.7.4