PLearn 0.1
|
Stores and learns the parameters between two linear layers of an RBM. More...
#include <RBMLLParameters.h>
Public Member Functions | |
RBMLLParameters (real the_learning_rate=0) | |
Default constructor. | |
RBMLLParameters (string down_types, string up_types, real the_learning_rate=0) | |
Constructor from two string prototymes. | |
virtual void | accumulatePosStats (const Vec &down_values, const Vec &up_values) |
Accumulates positive phase statistics to *_pos_stats. | |
virtual void | accumulateNegStats (const Vec &down_values, const Vec &up_values) |
Accumulates negative phase statistics to *_neg_stats. | |
virtual void | update () |
Updates parameters according to contrastive divergence gradient. | |
virtual void | update (const Vec &pos_down_values, const Vec &pos_up_values, const Vec &neg_down_values, const Vec &neg_up_values) |
Updates parameters according to contrastive divergence gradient, not using the statistics but the explicit values passed. | |
virtual void | clearStats () |
Clear all information accumulated during stats. | |
virtual void | computeUnitActivations (int start, int length, const Vec &activations) const |
Computes the vectors of activation of "length" units, starting from "start", and concatenates them into "activations". | |
virtual void | bpropUpdate (const Vec &input, const Vec &output, Vec &input_gradient, const Vec &output_gradient) |
Adapt based on the output gradient: this method should only be called just after a corresponding fprop; it should be called with the same arguments as fprop for the first two arguments (and output should not have been modified since then). | |
virtual void | forget () |
reset the parameters to the state they would be BEFORE starting training. | |
virtual int | nParameters (bool share_up_params, bool share_down_params) const |
optionally perform some processing after training, or after a series of fprop/bpropUpdate calls to prepare the model for truly out-of-sample operation. | |
virtual Vec | makeParametersPointHere (const Vec &global_parameters, bool share_up_params, bool share_down_params) |
Make the parameters data be sub-vectors of the given global_parameters. | |
virtual string | classname () const |
virtual OptionList & | getOptionList () const |
virtual OptionMap & | getOptionMap () const |
virtual RemoteMethodMap & | getRemoteMethodMap () const |
virtual RBMLLParameters * | deepCopy (CopiesMap &copies) const |
virtual void | build () |
Post-constructor. | |
virtual void | makeDeepCopyFromShallowCopy (CopiesMap &copies) |
Transforms a shallow copy into a deep copy. | |
Static Public Member Functions | |
static string | _classname_ () |
static OptionList & | _getOptionList_ () |
static RemoteMethodMap & | _getRemoteMethodMap_ () |
static Object * | _new_instance_for_typemap_ () |
static bool | _isa_ (const Object *o) |
static void | _static_initialize_ () |
static const PPath & | declaringFile () |
Public Attributes | |
real | momentum |
Momentum factor. | |
Mat | weights |
Matrix containing unit-to-unit weights (output_size × input_size) | |
Vec | up_units_bias |
Element i contains the bias of up unit i. | |
Vec | down_units_bias |
Element i contains the bias of down unit i. | |
Mat | weights_pos_stats |
Accumulates positive contribution to the weights' gradient. | |
Mat | weights_neg_stats |
Accumulates negative contribution to the weights' gradient. | |
Vec | up_units_bias_pos_stats |
Accumulates positive contribution to the gradient of up_units_bias. | |
Vec | up_units_bias_neg_stats |
Accumulates negative contribution to the gradient of up_units_bias. | |
Vec | down_units_bias_pos_stats |
Accumulates positive contribution to the gradient of down_units_bias. | |
Vec | down_units_bias_neg_stats |
Accumulates negative contribution to the gradient of down_units_bias. | |
Mat | weights_inc |
Used if momentum != 0. | |
Vec | down_units_bias_inc |
Vec | up_units_bias_inc |
Static Public Attributes | |
static StaticInitializer | _static_initializer_ |
Static Protected Member Functions | |
static void | declareOptions (OptionList &ol) |
Declares the class options. | |
Private Types | |
typedef RBMParameters | inherited |
Private Member Functions | |
void | build_ () |
This does the actual building. |
Stores and learns the parameters between two linear layers of an RBM.
Definition at line 55 of file RBMLLParameters.h.
typedef RBMParameters PLearn::RBMLLParameters::inherited [private] |
Reimplemented from PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 57 of file RBMLLParameters.h.
PLearn::RBMLLParameters::RBMLLParameters | ( | real | the_learning_rate = 0 | ) |
Default constructor.
Definition at line 52 of file RBMLLParameters.cc.
PLearn::RBMLLParameters::RBMLLParameters | ( | string | down_types, |
string | up_types, | ||
real | the_learning_rate = 0 |
||
) |
string PLearn::RBMLLParameters::_classname_ | ( | ) | [static] |
Reimplemented from PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 50 of file RBMLLParameters.cc.
OptionList & PLearn::RBMLLParameters::_getOptionList_ | ( | ) | [static] |
Reimplemented from PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 50 of file RBMLLParameters.cc.
RemoteMethodMap & PLearn::RBMLLParameters::_getRemoteMethodMap_ | ( | ) | [static] |
Reimplemented from PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 50 of file RBMLLParameters.cc.
Reimplemented from PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 50 of file RBMLLParameters.cc.
Object * PLearn::RBMLLParameters::_new_instance_for_typemap_ | ( | ) | [static] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 50 of file RBMLLParameters.cc.
StaticInitializer RBMLLParameters::_static_initializer_ & PLearn::RBMLLParameters::_static_initialize_ | ( | ) | [static] |
Reimplemented from PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 50 of file RBMLLParameters.cc.
void PLearn::RBMLLParameters::accumulateNegStats | ( | const Vec & | down_values, |
const Vec & | up_values | ||
) | [virtual] |
Accumulates negative phase statistics to *_neg_stats.
Implements PLearn::RBMParameters.
Definition at line 186 of file RBMLLParameters.cc.
References down_units_bias_neg_stats, PLearn::externalProductAcc(), PLearn::RBMParameters::neg_count, up_units_bias_neg_stats, and weights_neg_stats.
Referenced by PLearn::GaussianDBNClassification::jointGreedyStep(), and PLearn::UnfrozenDeepBeliefNet::train().
{ // weights_neg_stats += up_values * down_values' externalProductAcc( weights_neg_stats, up_values, down_values ); down_units_bias_neg_stats += down_values; up_units_bias_neg_stats += up_values; neg_count++; }
void PLearn::RBMLLParameters::accumulatePosStats | ( | const Vec & | down_values, |
const Vec & | up_values | ||
) | [virtual] |
Accumulates positive phase statistics to *_pos_stats.
Implements PLearn::RBMParameters.
Definition at line 174 of file RBMLLParameters.cc.
References down_units_bias_pos_stats, PLearn::externalProductAcc(), PLearn::RBMParameters::pos_count, up_units_bias_pos_stats, and weights_pos_stats.
Referenced by PLearn::GaussianDBNClassification::jointGreedyStep(), and PLearn::UnfrozenDeepBeliefNet::train().
{ // weights_pos_stats += up_values * down_values' externalProductAcc( weights_pos_stats, up_values, down_values ); down_units_bias_pos_stats += down_values; up_units_bias_pos_stats += up_values; pos_count++; }
void PLearn::RBMLLParameters::bpropUpdate | ( | const Vec & | input, |
const Vec & | output, | ||
Vec & | input_gradient, | ||
const Vec & | output_gradient | ||
) | [virtual] |
Adapt based on the output gradient: this method should only be called just after a corresponding fprop; it should be called with the same arguments as fprop for the first two arguments (and output should not have been modified since then).
this version allows to obtain the input gradient as well
Since sub-classes are supposed to learn ONLINE, the object is 'ready-to-be-used' just after any bpropUpdate. N.B. A DEFAULT IMPLEMENTATION IS PROVIDED IN THE SUPER-CLASS, WHICH JUST CALLS bpropUpdate(input, output, input_gradient, output_gradient) AND IGNORES INPUT GRADIENT. this version allows to obtain the input gradient as well N.B. THE DEFAULT IMPLEMENTATION IN SUPER-CLASS JUST RAISES A PLERROR.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 467 of file RBMLLParameters.cc.
References PLearn::RBMParameters::down_layer_size, PLearn::externalProductScaleAcc(), PLearn::RBMParameters::learning_rate, PLearn::multiplyAcc(), PLASSERT, PLearn::TVec< T >::resize(), PLearn::TVec< T >::size(), PLearn::transposeProduct(), PLearn::RBMParameters::up_layer_size, up_units_bias, and weights.
Referenced by PLearn::PartSupervisedDBN::jointGreedyStep(), and PLearn::GaussPartSupervisedDBN::jointGreedyStep().
{ PLASSERT( input.size() == down_layer_size ); PLASSERT( output.size() == up_layer_size ); PLASSERT( output_gradient.size() == up_layer_size ); input_gradient.resize( down_layer_size ); // input_gradient = weights' * output_gradient transposeProduct( input_gradient, weights, output_gradient ); // weights -= learning_rate * output_gradient * input' externalProductScaleAcc( weights, output_gradient, input, -learning_rate ); // (up) bias -= learning_rate * output_gradient multiplyAcc( up_units_bias, output_gradient, -learning_rate ); }
void PLearn::RBMLLParameters::build | ( | ) | [virtual] |
Post-constructor.
The normal implementation should call simply inherited::build(), then this class's build_(). This method should be callable again at later times, after modifying some option fields to change the "architecture" of the object.
Reimplemented from PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 149 of file RBMLLParameters.cc.
References PLearn::RBMParameters::build(), and build_().
Referenced by PLearn::RBMJointLLParameters::build(), and RBMLLParameters().
{ inherited::build(); build_(); }
void PLearn::RBMLLParameters::build_ | ( | ) | [private] |
This does the actual building.
Reimplemented from PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 92 of file RBMLLParameters.cc.
References clearStats(), PLearn::RBMParameters::down_layer_size, down_units_bias, down_units_bias_inc, down_units_bias_neg_stats, down_units_bias_pos_stats, PLearn::RBMParameters::down_units_types, forget(), i, PLearn::TMat< T >::length(), momentum, PLearn::OnlineLearningModule::output_size, PLERROR, PLearn::TVec< T >::resize(), PLearn::TMat< T >::resize(), PLearn::RBMParameters::up_layer_size, up_units_bias, up_units_bias_inc, up_units_bias_neg_stats, up_units_bias_pos_stats, PLearn::RBMParameters::up_units_types, weights, weights_inc, weights_neg_stats, weights_pos_stats, and PLearn::TMat< T >::width().
Referenced by build().
{ if( up_layer_size == 0 || down_layer_size == 0 ) return; output_size = 0; bool needs_forget = false; // do we need to reinitialize the parameters? if( weights.length() != up_layer_size || weights.width() != down_layer_size ) { weights.resize( up_layer_size, down_layer_size ); needs_forget = true; } weights_pos_stats.resize( up_layer_size, down_layer_size ); weights_neg_stats.resize( up_layer_size, down_layer_size ); down_units_bias.resize( down_layer_size ); down_units_bias_pos_stats.resize( down_layer_size ); down_units_bias_neg_stats.resize( down_layer_size ); for( int i=0 ; i<down_layer_size ; i++ ) { char dut_i = down_units_types[i]; if( dut_i != 'l' ) // not linear activation unit PLERROR( "RBMLLParameters::build_() - value '%c' for" " down_units_types[%d]\n" "should be 'l'.\n", dut_i, i ); } up_units_bias.resize( up_layer_size ); up_units_bias_pos_stats.resize( up_layer_size ); up_units_bias_neg_stats.resize( up_layer_size ); for( int i=0 ; i<up_layer_size ; i++ ) { char uut_i = up_units_types[i]; if( uut_i != 'l' ) // not linear activation unit PLERROR( "RBMLLParameters::build_() - value '%c' for" " up_units_types[%d]\n" "should be 'l'.\n", uut_i, i ); } if( momentum != 0. ) { weights_inc.resize( up_layer_size, down_layer_size ); down_units_bias_inc.resize( down_layer_size ); up_units_bias_inc.resize( up_layer_size ); } if( needs_forget ) forget(); clearStats(); }
string PLearn::RBMLLParameters::classname | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 50 of file RBMLLParameters.cc.
void PLearn::RBMLLParameters::clearStats | ( | ) | [virtual] |
Clear all information accumulated during stats.
Implements PLearn::RBMParameters.
Definition at line 430 of file RBMLLParameters.cc.
References PLearn::TMat< T >::clear(), PLearn::TVec< T >::clear(), down_units_bias_neg_stats, down_units_bias_pos_stats, PLearn::RBMParameters::neg_count, PLearn::RBMParameters::pos_count, up_units_bias_neg_stats, up_units_bias_pos_stats, weights_neg_stats, and weights_pos_stats.
Referenced by build_(), forget(), PLearn::RBMJointLLParameters::forget(), and update().
{ weights_pos_stats.clear(); weights_neg_stats.clear(); down_units_bias_pos_stats.clear(); down_units_bias_neg_stats.clear(); up_units_bias_pos_stats.clear(); up_units_bias_neg_stats.clear(); pos_count = 0; neg_count = 0; }
void PLearn::RBMLLParameters::computeUnitActivations | ( | int | start, |
int | length, | ||
const Vec & | activations | ||
) | const [virtual] |
Computes the vectors of activation of "length" units, starting from "start", and concatenates them into "activations".
"start" indexes an up unit if "going_up", else a down unit.
Implements PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 446 of file RBMLLParameters.cc.
References PLearn::TVec< T >::length(), PLASSERT, PLearn::product(), PLearn::TVec< T >::subVec(), and PLearn::transposeProduct().
{ PLASSERT( activations.length() == length ); if( going_up ) { PLASSERT( start+length <= up_layer_size ); // activations[i-start] = sum_j weights(i,j) input_vec[j] + b[i] product( activations, weights.subMatRows(start, length), input_vec ); activations += up_units_bias.subVec(start, length); } else { PLASSERT( start+length <= down_layer_size ); // activations[i-start] = sum_j weights(j,i) input_vec[j] + b[i] transposeProduct( activations, weights.subMatColumns(start, length), input_vec ); activations += down_units_bias.subVec(start, length); } }
void PLearn::RBMLLParameters::declareOptions | ( | OptionList & | ol | ) | [static, protected] |
Declares the class options.
Reimplemented from PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 67 of file RBMLLParameters.cc.
References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::RBMParameters::declareOptions(), down_units_bias, PLearn::OptionBase::learntoption, momentum, up_units_bias, and weights.
Referenced by PLearn::RBMJointLLParameters::declareOptions().
{ declareOption(ol, "momentum", &RBMLLParameters::momentum, OptionBase::buildoption, "Momentum factor (should be between 0 and 1)"); declareOption(ol, "weights", &RBMLLParameters::weights, OptionBase::learntoption, "Matrix containing unit-to-unit weights (output_size ×" " input_size)"); declareOption(ol, "up_units_bias", &RBMLLParameters::up_units_bias, OptionBase::learntoption, "Element i contains the bias of up unit i"); declareOption(ol, "down_units_bias", &RBMLLParameters::down_units_bias, OptionBase::learntoption, "Element i contains the bias of down unit i"); // Now call the parent class' declareOptions inherited::declareOptions(ol); }
static const PPath& PLearn::RBMLLParameters::declaringFile | ( | ) | [inline, static] |
Reimplemented from PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 180 of file RBMLLParameters.h.
:
RBMLLParameters * PLearn::RBMLLParameters::deepCopy | ( | CopiesMap & | copies | ) | const [virtual] |
Reimplemented from PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 50 of file RBMLLParameters.cc.
void PLearn::RBMLLParameters::forget | ( | ) | [virtual] |
reset the parameters to the state they would be BEFORE starting training.
Note that this method is necessarily called from build().
Implements PLearn::OnlineLearningModule.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 489 of file RBMLLParameters.cc.
References PLearn::TVec< T >::clear(), PLearn::TMat< T >::clear(), clearStats(), d, PLearn::RBMParameters::down_layer_size, down_units_bias, PLearn::RBMParameters::initialization_method, PLearn::max(), PLearn::OnlineLearningModule::random_gen, PLearn::sqrt(), PLearn::RBMParameters::up_layer_size, up_units_bias, and weights.
Referenced by build_().
{ if( initialization_method == "zero" ) weights.clear(); else { if( !random_gen ) random_gen = new PRandom(); real d = 1. / max( down_layer_size, up_layer_size ); if( initialization_method == "uniform_sqrt" ) d = sqrt( d ); random_gen->fill_random_uniform( weights, -d, d ); } down_units_bias.clear(); up_units_bias.clear(); clearStats(); }
OptionList & PLearn::RBMLLParameters::getOptionList | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 50 of file RBMLLParameters.cc.
OptionMap & PLearn::RBMLLParameters::getOptionMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 50 of file RBMLLParameters.cc.
RemoteMethodMap & PLearn::RBMLLParameters::getRemoteMethodMap | ( | ) | const [virtual] |
Reimplemented from PLearn::Object.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 50 of file RBMLLParameters.cc.
void PLearn::RBMLLParameters::makeDeepCopyFromShallowCopy | ( | CopiesMap & | copies | ) | [virtual] |
Transforms a shallow copy into a deep copy.
Reimplemented from PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 156 of file RBMLLParameters.cc.
References PLearn::deepCopyField(), down_units_bias, down_units_bias_inc, down_units_bias_neg_stats, down_units_bias_pos_stats, PLearn::RBMParameters::makeDeepCopyFromShallowCopy(), up_units_bias, up_units_bias_inc, up_units_bias_neg_stats, up_units_bias_pos_stats, weights, weights_inc, weights_neg_stats, and weights_pos_stats.
Referenced by PLearn::RBMJointLLParameters::makeDeepCopyFromShallowCopy().
{ inherited::makeDeepCopyFromShallowCopy(copies); deepCopyField(weights, copies); deepCopyField(up_units_bias, copies); deepCopyField(down_units_bias, copies); deepCopyField(weights_pos_stats, copies); deepCopyField(weights_neg_stats, copies); deepCopyField(up_units_bias_pos_stats, copies); deepCopyField(up_units_bias_neg_stats, copies); deepCopyField(down_units_bias_pos_stats, copies); deepCopyField(down_units_bias_neg_stats, copies); deepCopyField(weights_inc, copies); deepCopyField(up_units_bias_inc, copies); deepCopyField(down_units_bias_inc, copies); }
Vec PLearn::RBMLLParameters::makeParametersPointHere | ( | const Vec & | global_parameters, |
bool | share_up_params, | ||
bool | share_down_params | ||
) | [virtual] |
Make the parameters data be sub-vectors of the given global_parameters.
The argument should have size >= nParameters. The result is a Vec that starts just after this object's parameters end, i.e. result = global_parameters.subVec(nParameters(),global_parameters.size()-nParameters()); This allows to easily chain calls of this method on multiple RBMParameters.
Implements PLearn::RBMParameters.
Definition at line 534 of file RBMLLParameters.cc.
References PLearn::TVec< T >::data(), down_units_bias, m, PLearn::TMat< T >::makeSharedValue(), PLearn::TVec< T >::makeSharedValue(), n, PLERROR, PLearn::TVec< T >::size(), PLearn::TMat< T >::size(), PLearn::TVec< T >::subVec(), up_units_bias, and weights.
{ int n1=weights.size(); int n2=up_units_bias.size(); int n3=down_units_bias.size(); int n = n1+(share_up_params?n2:0)+(share_down_params?n3:0); // should be = nParameters() int m = global_parameters.size(); if (m<n) PLERROR("RBMLLParameters::makeParametersPointHere: argument has length %d, should be longer than nParameters()=%d",m,n); real* p = global_parameters.data(); weights.makeSharedValue(p,n1); p+=n1; if (share_up_params) { up_units_bias.makeSharedValue(p,n2); p+=n2; } if (share_down_params) down_units_bias.makeSharedValue(p,n3); return global_parameters.subVec(n,m-n); }
int PLearn::RBMLLParameters::nParameters | ( | bool | share_up_params, |
bool | share_down_params | ||
) | const [virtual] |
optionally perform some processing after training, or after a series of fprop/bpropUpdate calls to prepare the model for truly out-of-sample operation.
return the number of parameters
THE DEFAULT IMPLEMENTATION PROVIDED IN THE SUPER-CLASS DOES NOT DO ANYTHING. return the number of parameters
Implements PLearn::RBMParameters.
Definition at line 523 of file RBMLLParameters.cc.
References down_units_bias, PLearn::TVec< T >::size(), PLearn::TMat< T >::size(), up_units_bias, and weights.
{ return weights.size() + (share_up_params?up_units_bias.size():0) + (share_down_params?down_units_bias.size():0); }
void PLearn::RBMLLParameters::update | ( | ) | [virtual] |
Updates parameters according to contrastive divergence gradient.
Implements PLearn::RBMParameters.
Definition at line 198 of file RBMLLParameters.cc.
References clearStats(), PLearn::TVec< T >::data(), PLearn::TMat< T >::data(), down_units_bias, down_units_bias_inc, down_units_bias_neg_stats, down_units_bias_pos_stats, i, j, PLearn::RBMParameters::learning_rate, PLearn::TVec< T >::length(), PLearn::TMat< T >::length(), PLearn::TMat< T >::mod(), momentum, PLearn::RBMParameters::neg_count, PLearn::RBMParameters::pos_count, PLearn::TVec< T >::resize(), PLearn::TMat< T >::resize(), up_units_bias, up_units_bias_inc, up_units_bias_neg_stats, up_units_bias_pos_stats, w, weights, weights_inc, weights_neg_stats, weights_pos_stats, and PLearn::TMat< T >::width().
Referenced by PLearn::GaussianDBNClassification::jointGreedyStep(), and PLearn::UnfrozenDeepBeliefNet::train().
{ // updates parameters //weights -= learning_rate * (weights_pos_stats/pos_count // - weights_neg_stats/neg_count) real pos_factor = -learning_rate / pos_count; real neg_factor = learning_rate / neg_count; int l = weights.length(); int w = weights.width(); real* w_i = weights.data(); real* wps_i = weights_pos_stats.data(); real* wns_i = weights_neg_stats.data(); int w_mod = weights.mod(); int wps_mod = weights_pos_stats.mod(); int wns_mod = weights_neg_stats.mod(); if( momentum == 0. ) { // no need to use weights_inc for( int i=0 ; i<l ; i++, w_i+=w_mod, wps_i+=wps_mod, wns_i+=wns_mod ) for( int j=0 ; j<w ; j++ ) w_i[j] += pos_factor * wps_i[j] + neg_factor * wns_i[j]; } else { // ensure that weights_inc has the right size weights_inc.resize( l, w ); // The update rule becomes: // weights_inc = momentum * weights_inc // - learning_rate * (weights_pos_stats/pos_count // - weights_neg_stats/neg_count); // weights += weights_inc; real* winc_i = weights_inc.data(); int winc_mod = weights_inc.mod(); for( int i=0 ; i<l ; i++, w_i += w_mod, wps_i += wps_mod, wns_i += wns_mod, winc_i += winc_mod ) for( int j=0 ; j<w ; j++ ) { winc_i[j] = momentum * winc_i[j] + pos_factor * wps_i[j] + neg_factor * wns_i[j]; w_i[j] += winc_i[j]; } } // down_units_bias -= learning_rate * (down_units_bias_pos_stats/pos_count // -down_units_bias_neg_stats/neg_count) l = down_units_bias.length(); real* dub = down_units_bias.data(); real* dubps = down_units_bias_pos_stats.data(); real* dubns = down_units_bias_neg_stats.data(); if( momentum == 0. ) { // no need to use down_units_bias_inc for( int i=0 ; i<l ; i++ ) dub[i] += pos_factor * dubps[i] + neg_factor * dubns[i]; } else { // ensure that down_units_bias_inc has the right size down_units_bias_inc.resize( l ); // The update rule becomes: // down_units_bias_inc = // momentum * down_units_bias_inc // - learning_rate * (down_units_bias_pos_stats/pos_count // -down_units_bias_neg_stats/neg_count); // down_units_bias += down_units_bias_inc; real* dubinc = down_units_bias_inc.data(); for( int i=0 ; i<l ; i++ ) { dubinc[i] = momentum * dubinc[i] + pos_factor * dubps[i] + neg_factor * dubns[i]; dub[i] += dubinc[i]; } } // up_units_bias -= learning_rate * (up_units_bias_pos_stats/pos_count // -up_units_bias_neg_stats/neg_count) l = up_units_bias.length(); real* uub = up_units_bias.data(); real* uubps = up_units_bias_pos_stats.data(); real* uubns = up_units_bias_neg_stats.data(); if( momentum == 0. ) { // no need to use up_units_bias_inc for( int i=0 ; i<l ; i++ ) uub[i] += pos_factor * uubps[i] + neg_factor * uubns[i]; } else { // ensure that up_units_bias_inc has the right size up_units_bias_inc.resize( l ); // The update rule becomes: // up_units_bias_inc = // momentum * up_units_bias_inc // - learning_rate * (up_units_bias_pos_stats/pos_count // -up_units_bias_neg_stats/neg_count); // up_units_bias += up_units_bias_inc; real* uubinc = up_units_bias_inc.data(); for( int i=0 ; i<l ; i++ ) { uubinc[i] = momentum * uubinc[i] + pos_factor * uubps[i] + neg_factor * uubns[i]; uub[i] += uubinc[i]; } } clearStats(); }
void PLearn::RBMLLParameters::update | ( | const Vec & | pos_down_values, |
const Vec & | pos_up_values, | ||
const Vec & | neg_down_values, | ||
const Vec & | neg_up_values | ||
) | [virtual] |
Updates parameters according to contrastive divergence gradient, not using the statistics but the explicit values passed.
Reimplemented from PLearn::RBMParameters.
Definition at line 315 of file RBMLLParameters.cc.
References PLearn::TVec< T >::data(), PLearn::TMat< T >::data(), down_units_bias, down_units_bias_inc, i, j, PLearn::RBMParameters::learning_rate, PLearn::TVec< T >::length(), PLearn::TMat< T >::length(), PLearn::TMat< T >::mod(), momentum, PLASSERT, PLearn::TVec< T >::resize(), PLearn::TMat< T >::resize(), up_units_bias, up_units_bias_inc, w, weights, weights_inc, and PLearn::TMat< T >::width().
{ // weights -= learning_rate * ( h_0 v_0' - h_1 v_1' ); // or: // weights[i][j] += learning_rate * (h_1[i] v_1[j] - h_0[i] v_0[j]); int l = weights.length(); int w = weights.width(); PLASSERT( pos_up_values.length() == l ); PLASSERT( neg_up_values.length() == l ); PLASSERT( pos_down_values.length() == w ); PLASSERT( neg_down_values.length() == w ); real* w_i = weights.data(); real* puv_i = pos_up_values.data(); real* nuv_i = neg_up_values.data(); real* pdv = pos_down_values.data(); real* ndv = neg_down_values.data(); int w_mod = weights.mod(); if( momentum == 0. ) { for( int i=0 ; i<l ; i++, w_i += w_mod, puv_i++, nuv_i++ ) for( int j=0 ; j<w ; j++ ) w_i[j] += learning_rate * (*nuv_i * ndv[j] - *puv_i * pdv[j]); } else { // ensure that weights_inc has the right size weights_inc.resize( l, w ); // The update rule becomes: // weights_inc = momentum * weights_inc // - learning_rate * ( h_0 v_0' - h_1 v_1' ); // weights += weights_inc; real* winc_i = weights_inc.data(); int winc_mod = weights_inc.mod(); for( int i=0 ; i<l ; i++, w_i += w_mod, winc_i += winc_mod, puv_i++, nuv_i++ ) for( int j=0 ; j<w ; j++ ) { winc_i[j] = momentum * winc_i[j] + learning_rate * (*nuv_i * ndv[j] - *puv_i * pdv[j]); w_i[j] += winc_i[j]; } } // down_units_bias -= learning_rate * ( v_0 - v_1 ) real* dub = down_units_bias.data(); // pdv and ndv didn't change since last time // real* pdv = pos_down_values.data(); // real* ndv = neg_down_values.data(); if( momentum == 0. ) { // no need to use down_units_bias_inc for( int j=0 ; j<w ; j++ ) dub[j] += learning_rate * ( ndv[j] - pdv[j] ); } else { // ensure that down_units_bias_inc has the right size down_units_bias_inc.resize( w ); // The update rule becomes: // down_units_bias_inc = momentum * down_units_bias_inc // - learning_rate * ( v_0 - v_1 ) // down_units_bias += down_units_bias_inc; real* dubinc = down_units_bias_inc.data(); for( int j=0 ; j<w ; j++ ) { dubinc[j] = momentum * dubinc[j] + learning_rate * ( ndv[j] - pdv[j] ); dub[j] += dubinc[j]; } } // up_units_bias -= learning_rate * ( h_0 - h_1 ) real* uub = up_units_bias.data(); real* puv = pos_up_values.data(); real* nuv = neg_up_values.data(); if( momentum == 0. ) { // no need to use up_units_bias_inc for( int i=0 ; i<l ; i++ ) uub[i] += learning_rate * (nuv[i] - puv[i] ); } else { // ensure that up_units_bias_inc has the right size up_units_bias_inc.resize( l ); // The update rule becomes: // up_units_bias_inc = // momentum * up_units_bias_inc // - learning_rate * (up_units_bias_pos_stats/pos_count // -up_units_bias_neg_stats/neg_count); // up_units_bias += up_units_bias_inc; real* uubinc = up_units_bias_inc.data(); for( int i=0 ; i<l ; i++ ) { uubinc[i] = momentum * uubinc[i] + learning_rate * ( nuv[i] - puv[i] ); uub[i] += uubinc[i]; } } }
Reimplemented from PLearn::RBMParameters.
Reimplemented in PLearn::RBMJointLLParameters.
Definition at line 180 of file RBMLLParameters.h.
Element i contains the bias of down unit i.
Definition at line 74 of file RBMLLParameters.h.
Referenced by PLearn::RBMJointLLParameters::bpropUpdate(), PLearn::RBMJointLLParameters::build_(), build_(), declareOptions(), forget(), makeDeepCopyFromShallowCopy(), makeParametersPointHere(), nParameters(), and update().
Definition at line 95 of file RBMLLParameters.h.
Referenced by build_(), makeDeepCopyFromShallowCopy(), and update().
Accumulates negative contribution to the gradient of down_units_bias.
Definition at line 91 of file RBMLLParameters.h.
Referenced by accumulateNegStats(), PLearn::RBMJointLLParameters::build_(), build_(), clearStats(), makeDeepCopyFromShallowCopy(), and update().
Accumulates positive contribution to the gradient of down_units_bias.
Definition at line 89 of file RBMLLParameters.h.
Referenced by accumulatePosStats(), PLearn::RBMJointLLParameters::build_(), build_(), clearStats(), makeDeepCopyFromShallowCopy(), and update().
Momentum factor.
Definition at line 63 of file RBMLLParameters.h.
Referenced by build_(), declareOptions(), and update().
Element i contains the bias of up unit i.
Definition at line 71 of file RBMLLParameters.h.
Referenced by bpropUpdate(), PLearn::RBMJointLLParameters::bpropUpdate(), PLearn::RBMJointLLParameters::build_(), build_(), declareOptions(), forget(), makeDeepCopyFromShallowCopy(), makeParametersPointHere(), nParameters(), and update().
Definition at line 96 of file RBMLLParameters.h.
Referenced by build_(), makeDeepCopyFromShallowCopy(), and update().
Accumulates negative contribution to the gradient of up_units_bias.
Definition at line 87 of file RBMLLParameters.h.
Referenced by accumulateNegStats(), PLearn::RBMJointLLParameters::build_(), build_(), clearStats(), makeDeepCopyFromShallowCopy(), and update().
Accumulates positive contribution to the gradient of up_units_bias.
Definition at line 85 of file RBMLLParameters.h.
Referenced by accumulatePosStats(), PLearn::RBMJointLLParameters::build_(), build_(), clearStats(), makeDeepCopyFromShallowCopy(), and update().
Matrix containing unit-to-unit weights (output_size × input_size)
Definition at line 68 of file RBMLLParameters.h.
Referenced by bpropUpdate(), PLearn::RBMJointLLParameters::bpropUpdate(), build_(), PLearn::RBMJointLLParameters::build_(), PLearn::RBMJointLLParameters::build_units_types(), declareOptions(), forget(), makeDeepCopyFromShallowCopy(), makeParametersPointHere(), nParameters(), and update().
Used if momentum != 0.
Definition at line 94 of file RBMLLParameters.h.
Referenced by build_(), makeDeepCopyFromShallowCopy(), and update().
Accumulates negative contribution to the weights' gradient.
Definition at line 82 of file RBMLLParameters.h.
Referenced by accumulateNegStats(), PLearn::RBMJointLLParameters::build_(), build_(), clearStats(), makeDeepCopyFromShallowCopy(), and update().
Accumulates positive contribution to the weights' gradient.
Definition at line 79 of file RBMLLParameters.h.
Referenced by accumulatePosStats(), PLearn::RBMJointLLParameters::build_(), build_(), clearStats(), makeDeepCopyFromShallowCopy(), and update().