PLearn 0.1
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Static Protected Member Functions | Protected Attributes | Private Types | Private Member Functions
PLearn::RBMWoodsLayer Class Reference

RBM layer with tree-structured groups of units. More...

#include <RBMWoodsLayer.h>

Inheritance diagram for PLearn::RBMWoodsLayer:
Inheritance graph
[legend]
Collaboration diagram for PLearn::RBMWoodsLayer:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 RBMWoodsLayer (real the_learning_rate=0.)
 Default constructor.
virtual void generateSample ()
 generate a sample, and update the sample field
virtual void generateSamples ()
 Inherited.
virtual void computeProbabilisticClustering (Vec &prob_clusters)
virtual void computeExpectation ()
 Compute marginal expectations of all units.
virtual void computeExpectations ()
 Compute marginal mini-batch expectations of all units.
virtual void fprop (const Vec &input, Vec &output) const
 forward propagation
virtual void fprop (const Mat &inputs, Mat &outputs)
 Batch forward propagation.
virtual void fprop (const Vec &input, const Vec &rbm_bias, Vec &output) const
 forward propagation with provided bias
virtual void bpropUpdate (const Vec &input, const Vec &output, Vec &input_gradient, const Vec &output_gradient, bool accumulate=false)
 back-propagates the output gradient to the input
virtual void bpropUpdate (const Vec &input, const Vec &rbm_bias, const Vec &output, Vec &input_gradient, Vec &rbm_bias_gradient, const Vec &output_gradient)
 back-propagates the output gradient to the input and the bias
virtual void bpropUpdate (const Mat &inputs, const Mat &outputs, Mat &input_gradients, const Mat &output_gradients, bool accumulate=false)
 Back-propagate the output gradient to the input, and update parameters.
virtual real fpropNLL (const Vec &target)
 Computes the negative log-likelihood of target given the internal activations of the layer.
virtual void fpropNLL (const Mat &targets, const Mat &costs_column)
virtual void bpropNLL (const Vec &target, real nll, Vec &bias_gradient)
 Computes the gradient of the negative log-likelihood of target with respect to the layer's bias, given the internal activations.
virtual void bpropNLL (const Mat &targets, const Mat &costs_column, Mat &bias_gradients)
virtual real energy (const Vec &unit_values) const
 compute -bias' unit_values
virtual real freeEnergyContribution (const Vec &unit_activations) const
 Computes $ -log(\sum_{possible values of h} exp(h' unit_activations))$ This quantity is used for computing the free energy of a sample x in the OTHER layer of an RBM, from which unit_activations was computed.
virtual void freeEnergyContributionGradient (const Vec &unit_activations, Vec &unit_activations_gradient, real output_gradient=1, bool accumulate=false) const
 Computes gradient of the result of freeEnergyContribution $ -log(\sum_{possible values of h} exp(h' unit_activations))$ with respect to unit_activations.
virtual int getConfigurationCount ()
 Returns a number of different configurations the layer can be in.
virtual void getConfiguration (int conf_index, Vec &output)
 Computes the conf_index configuration of the layer.
virtual string classname () const
virtual OptionListgetOptionList () const
virtual OptionMapgetOptionMap () const
virtual RemoteMethodMapgetRemoteMethodMap () const
virtual RBMWoodsLayerdeepCopy (CopiesMap &copies) const
virtual void build ()
 Post-constructor.
virtual void makeDeepCopyFromShallowCopy (CopiesMap &copies)
 Transforms a shallow copy into a deep copy.

Static Public Member Functions

static string _classname_ ()
static OptionList_getOptionList_ ()
static RemoteMethodMap_getRemoteMethodMap_ ()
static Object_new_instance_for_typemap_ ()
static bool _isa_ (const Object *o)
static void _static_initialize_ ()
static const PPathdeclaringFile ()

Public Attributes

int n_trees
 Number of trees in the woods.
int tree_depth
 Depth of the trees in the woods (1 gives the ordinary RBMBinomialLayer)
bool use_signed_samples
 Indication that samples should be in {-1,1}, not {0,1}, at nodes where a left/right decision is made.

Static Public Attributes

static StaticInitializer _static_initializer_

Static Protected Member Functions

static void declareOptions (OptionList &ol)
 Declares the class options.

Protected Attributes

Vec off_expectation
Mat off_expectations
Vec local_node_expectation
Mat local_node_expectations
Vec on_free_energy
Mat on_free_energies
Vec off_free_energy
Mat off_free_energies
Vec local_node_expectation_gradient
Vec on_tree_gradient
Vec off_tree_gradient
Vec on_free_energy_gradient
Vec off_free_energy_gradient
Vec tree_free_energies
Vec tree_energies
Vec unit_activations_pos_gradient
Vec unit_activations_neg_gradient
TVec< boolunit_activations_pos_gradient_init
TVec< boolunit_activations_neg_gradient_init

Private Types

typedef RBMLayer inherited

Private Member Functions

void build_ ()
 This does the actual building.

Detailed Description

RBM layer with tree-structured groups of units.

Definition at line 52 of file RBMWoodsLayer.h.


Member Typedef Documentation

Reimplemented from PLearn::RBMLayer.

Definition at line 54 of file RBMWoodsLayer.h.


Constructor & Destructor Documentation

PLearn::RBMWoodsLayer::RBMWoodsLayer ( real  the_learning_rate = 0.)

Default constructor.

Definition at line 53 of file RBMWoodsLayer.cc.

                                                     :
    inherited( the_learning_rate ),
    n_trees( 10 ),
    tree_depth( 3 ),
    use_signed_samples( false )
{
}

Member Function Documentation

string PLearn::RBMWoodsLayer::_classname_ ( ) [static]

Reimplemented from PLearn::RBMLayer.

Definition at line 51 of file RBMWoodsLayer.cc.

OptionList & PLearn::RBMWoodsLayer::_getOptionList_ ( ) [static]

Reimplemented from PLearn::RBMLayer.

Definition at line 51 of file RBMWoodsLayer.cc.

RemoteMethodMap & PLearn::RBMWoodsLayer::_getRemoteMethodMap_ ( ) [static]

Reimplemented from PLearn::RBMLayer.

Definition at line 51 of file RBMWoodsLayer.cc.

bool PLearn::RBMWoodsLayer::_isa_ ( const Object o) [static]

Reimplemented from PLearn::RBMLayer.

Definition at line 51 of file RBMWoodsLayer.cc.

Object * PLearn::RBMWoodsLayer::_new_instance_for_typemap_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 51 of file RBMWoodsLayer.cc.

StaticInitializer RBMWoodsLayer::_static_initializer_ & PLearn::RBMWoodsLayer::_static_initialize_ ( ) [static]

Reimplemented from PLearn::RBMLayer.

Definition at line 51 of file RBMWoodsLayer.cc.

void PLearn::RBMWoodsLayer::bpropNLL ( const Vec target,
real  nll,
Vec bias_gradient 
) [virtual]

Computes the gradient of the negative log-likelihood of target with respect to the layer's bias, given the internal activations.

Reimplemented from PLearn::RBMLayer.

Definition at line 1153 of file RBMWoodsLayer.cc.

References computeExpectation(), PLearn::RBMLayer::expectation, PLearn::OnlineLearningModule::input_size, PLASSERT, PLERROR, PLearn::TVec< T >::resize(), PLearn::TVec< T >::size(), PLearn::RBMLayer::size, and PLearn::substract().

{
    PLERROR( "RBMWoodsLayer::bpropNLL(): not implemeted yet" );
    computeExpectation();

    PLASSERT( target.size() == input_size );
    bias_gradient.resize( size );

    // bias_gradient = expectation - target
    substract(expectation, target, bias_gradient);
}

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::bpropNLL ( const Mat targets,
const Mat costs_column,
Mat bias_gradients 
) [virtual]

Reimplemented from PLearn::RBMLayer.

Definition at line 1165 of file RBMWoodsLayer.cc.

References PLearn::RBMLayer::batch_size, computeExpectations(), PLearn::RBMLayer::expectations, PLearn::OnlineLearningModule::input_size, PLearn::TMat< T >::length(), PLASSERT, PLERROR, PLearn::TMat< T >::resize(), PLearn::RBMLayer::size, PLearn::substract(), and PLearn::TMat< T >::width().

{
    PLERROR( "RBMWoodsLayer::bpropNLL(): not implemeted yet" );
    computeExpectations();

    PLASSERT( targets.width() == input_size );
    PLASSERT( targets.length() == batch_size );
    PLASSERT( costs_column.width() == 1 );
    PLASSERT( costs_column.length() == batch_size );
    bias_gradients.resize( batch_size, size );

    // bias_gradients = expectations - targets
    substract(expectations, targets, bias_gradients);

}

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::bpropUpdate ( const Mat inputs,
const Mat outputs,
Mat input_gradients,
const Mat output_gradients,
bool  accumulate = false 
) [virtual]

Back-propagate the output gradient to the input, and update parameters.

Implements PLearn::RBMLayer.

Definition at line 980 of file RBMWoodsLayer.cc.

References PLearn::RBMLayer::applyBiasDecay(), PLearn::RBMLayer::bias, PLearn::RBMLayer::bias_inc, PLearn::TMat< T >::clear(), i, j, PLearn::RBMLayer::learning_rate, PLearn::TMat< T >::length(), PLearn::RBMLayer::momentum, PLASSERT, PLASSERT_MSG, PLERROR, PLearn::TMat< T >::resize(), PLearn::TVec< T >::resize(), PLearn::RBMLayer::size, and PLearn::TMat< T >::width().

{
    PLASSERT( inputs.width() == size );
    PLASSERT( outputs.width() == size );
    PLASSERT( output_gradients.width() == size );

    int mbatch_size = inputs.length();
    PLASSERT( outputs.length() == mbatch_size );
    PLASSERT( output_gradients.length() == mbatch_size );

    if( accumulate )
    {
        PLASSERT_MSG( input_gradients.width() == size &&
                input_gradients.length() == mbatch_size,
                "Cannot resize input_gradients and accumulate into it" );
    }
    else
    {
        input_gradients.resize(mbatch_size, size);
        input_gradients.clear();
    }

    PLERROR( "RBMWoodsLayer::bpropUpdate(): not implemeted yet" );

    if( momentum != 0. )
        bias_inc.resize( size );

    // TODO Can we do this more efficiently? (using BLAS)

    // We use the average gradient over the mini-batch.
    real avg_lr = learning_rate / inputs.length();

    for (int j = 0; j < mbatch_size; j++)
    {
        for( int i=0 ; i<size ; i++ )
        {
            real output_i = outputs(j, i);
            real in_grad_i = output_i * (1-output_i) * output_gradients(j, i);
            input_gradients(j, i) += in_grad_i;

            if( momentum == 0. )
            {
                // update the bias: bias -= learning_rate * input_gradient
                bias[i] -= avg_lr * in_grad_i;
            }
            else
            {
                PLERROR("In RBMWoodsLayer:bpropUpdate - Not implemented for "
                        "momentum with mini-batches");
                // The update rule becomes:
                // bias_inc = momentum * bias_inc - learning_rate * input_gradient
                // bias += bias_inc
                bias_inc[i] = momentum * bias_inc[i] - learning_rate * in_grad_i;
                bias[i] += bias_inc[i];
            }
        }
    }

    applyBiasDecay();
}

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::bpropUpdate ( const Vec input,
const Vec rbm_bias,
const Vec output,
Vec input_gradient,
Vec rbm_bias_gradient,
const Vec output_gradient 
) [virtual]

back-propagates the output gradient to the input and the bias

TODO: add "accumulate" here.

Reimplemented from PLearn::RBMLayer.

Definition at line 1046 of file RBMWoodsLayer.cc.

References i, PLASSERT, PLERROR, PLearn::TVec< T >::resize(), PLearn::TVec< T >::size(), and PLearn::RBMLayer::size.

{
    PLASSERT( input.size() == size );
    PLASSERT( rbm_bias.size() == size );
    PLASSERT( output.size() == size );
    PLASSERT( output_gradient.size() == size );
    input_gradient.resize( size );
    rbm_bias_gradient.resize( size );

    PLERROR( "RBMWoodsLayer::bpropUpdate(): not implemeted yet" );

    for( int i=0 ; i<size ; i++ )
    {
        real output_i = output[i];
        input_gradient[i] = output_i * (1-output_i) * output_gradient[i];
    }

    rbm_bias_gradient << input_gradient;
}

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::bpropUpdate ( const Vec input,
const Vec output,
Vec input_gradient,
const Vec output_gradient,
bool  accumulate = false 
) [virtual]

back-propagates the output gradient to the input

Implements PLearn::RBMLayer.

Definition at line 717 of file RBMWoodsLayer.cc.

References PLearn::RBMLayer::applyBiasDecay(), PLearn::RBMLayer::bias, PLearn::RBMLayer::bias_inc, PLearn::TVec< T >::clear(), i, PLearn::RBMLayer::learning_rate, local_node_expectation, local_node_expectation_gradient, PLearn::RBMLayer::momentum, n, n_trees, off_expectation, off_free_energy_gradient, off_tree_gradient, on_free_energy_gradient, on_tree_gradient, PLASSERT, PLASSERT_MSG, PLearn::TVec< T >::resize(), PLearn::RBMLayer::size, PLearn::TVec< T >::size(), tree_depth, and use_signed_samples.

{
    PLASSERT( input.size() == size );
    PLASSERT( output.size() == size );
    PLASSERT( output_gradient.size() == size );

    if( accumulate )
    {
        PLASSERT_MSG( input_gradient.size() == size,
                      "Cannot resize input_gradient AND accumulate into it" );
    }
    else
    {
        input_gradient.resize( size );
        input_gradient.clear();
    }

    // Compute gradient on marginal expectations
    int n_nodes_per_tree = size / n_trees;
    int node, depth, sub_tree_size, grand_parent;
    int offset = 0;
    bool left_of_grand_parent;
    real grand_parent_prob;
    real node_exp, parent_exp, out_grad, off_grad;
    local_node_expectation_gradient.clear();
    on_tree_gradient.clear();
    off_tree_gradient.clear();

    for( int t=0; t<n_trees; t++ )
    {
        // Set other nodes, level-wise
        depth = tree_depth-1;
        sub_tree_size = 0;
        while( depth > 1 )
        {
            // Left child
            left_of_grand_parent = true;
            for( int n=sub_tree_size; n<n_nodes_per_tree; n += 4*sub_tree_size + 4 )
            {
                out_grad = output_gradient[ n + offset ] +
                    on_tree_gradient[ n + offset ] ;
                off_grad = off_tree_gradient[ n + offset ] ;
                node_exp = local_node_expectation[ n + offset ];
                parent_exp = local_node_expectation[ n + offset + sub_tree_size + 1 ];

                if( left_of_grand_parent )
                {
                    grand_parent = n + offset + 3*sub_tree_size + 3;
                    if( use_signed_samples )
                        grand_parent_prob = output[ grand_parent ] + off_expectation[grand_parent];
                    else
                        grand_parent_prob = output[ grand_parent ];
                    // Gradient for rest of the tree
                    on_tree_gradient[ grand_parent ] +=
                        ( out_grad * node_exp
                          + off_grad * (1 - node_exp) )
                        * parent_exp;
                    left_of_grand_parent = false;
                }
                else
                {
                    grand_parent = n + offset - sub_tree_size - 1;
                    grand_parent_prob = off_expectation[ grand_parent ];
                    // Gradient for rest of the tree
                    off_tree_gradient[ grand_parent ] +=
                        ( out_grad * node_exp
                          + off_grad * (1 - node_exp) )
                        * parent_exp;
                    left_of_grand_parent = true;
                }

                // Gradient w/r current node
                local_node_expectation_gradient[ n + offset ] +=
                    ( out_grad - off_grad ) * parent_exp * grand_parent_prob;
                    //* node_exp * ( 1 - node_exp );

                // Gradient w/r parent node
                local_node_expectation_gradient[ n + offset + sub_tree_size + 1 ] +=
                    ( out_grad * node_exp + off_grad * (1 - node_exp) )  * grand_parent_prob;
                    //* parent_exp * (1-parent_exp) ;

            }

            // Right child
            left_of_grand_parent = true;
            for( int n=3*sub_tree_size+2; n<n_nodes_per_tree; n += 4*sub_tree_size + 4 )
            {
                out_grad = output_gradient[ n + offset ] +
                    on_tree_gradient[ n + offset ] ;
                off_grad = off_tree_gradient[ n + offset ] ;
                node_exp = local_node_expectation[ n + offset ];
                parent_exp = local_node_expectation[ n + offset - sub_tree_size - 1 ];

                if( left_of_grand_parent )
                {
                    grand_parent = n + offset + sub_tree_size + 1;
                    if( use_signed_samples )
                        grand_parent_prob = output[ grand_parent ] + off_expectation[ grand_parent ];
                    else
                        grand_parent_prob = output[ grand_parent ];
                    // Gradient for rest of the tree
                    on_tree_gradient[ grand_parent ] +=
                        ( out_grad * node_exp
                          + off_grad * (1 - node_exp) )
                        * ( 1 - parent_exp );
                    left_of_grand_parent = false;
                }
                else
                {
                    grand_parent = n + offset - 3*sub_tree_size - 3;
                    grand_parent_prob = off_expectation[ grand_parent ];
                    // Gradient for rest of the tree
                    off_tree_gradient[ grand_parent ] +=
                        ( out_grad * node_exp
                          + off_grad * (1 - node_exp) )
                        * ( 1 - parent_exp );
                    left_of_grand_parent = true;
                }

                // Gradient w/r current node
                local_node_expectation_gradient[ n + offset ] +=
                    ( out_grad - off_grad ) * ( 1 - parent_exp ) * grand_parent_prob;
                    //* node_exp * ( 1 - node_exp );

                // Gradient w/r parent node
                local_node_expectation_gradient[ n + offset - sub_tree_size - 1 ] -=
                    ( out_grad * node_exp + off_grad * (1 - node_exp) )  * grand_parent_prob;
                    //* parent_exp * (1-parent_exp) ;
            }
            sub_tree_size = 2 * ( sub_tree_size + 1 ) - 1;
            depth--;
        }

        depth = 1;

        node = sub_tree_size;
        out_grad = output_gradient[ node + offset ] +
            on_tree_gradient[ node + offset ] ;
        off_grad = off_tree_gradient[ node + offset ] ;
        node_exp = local_node_expectation[ node + offset ];
        parent_exp = local_node_expectation[ node + offset + sub_tree_size + 1 ];

        // Gradient w/r current node
        local_node_expectation_gradient[ node + offset ] +=
            ( out_grad - off_grad ) * parent_exp;
            //* node_exp * ( 1 - node_exp );

        // Gradient w/r parent node
        local_node_expectation_gradient[ node + offset + sub_tree_size + 1 ] +=
            ( out_grad * node_exp  + off_grad * (1 - node_exp) );
            //* parent_exp * (1-parent_exp) ;

        node = 3*sub_tree_size+2;
        out_grad = output_gradient[ node + offset ] +
            on_tree_gradient[ node + offset ] ;
        off_grad = off_tree_gradient[ node + offset ] ;
        node_exp = local_node_expectation[ node + offset ];
        parent_exp = local_node_expectation[ node + offset - sub_tree_size - 1 ];

        // Gradient w/r current node
        local_node_expectation_gradient[ node + offset ] +=
            ( out_grad - off_grad ) * ( 1 - parent_exp ) ;
            //* node_exp * ( 1 - node_exp );

        // Gradient w/r parent node
        local_node_expectation_gradient[ node + offset - sub_tree_size - 1 ] -=
            ( out_grad * node_exp + off_grad * (1 - node_exp) ) ;
            //* parent_exp * (1-parent_exp) ;

        node = n_nodes_per_tree / 2;
        sub_tree_size = 2 * ( sub_tree_size + 1 ) - 1;

        out_grad = output_gradient[ node + offset ] +
            on_tree_gradient[ node + offset ] ;
        off_grad = off_tree_gradient[ node + offset ] ;
        node_exp = local_node_expectation[ node + offset ];
        local_node_expectation_gradient[ node + offset ] +=
            ( out_grad - off_grad );// * node_exp * ( 1 - node_exp );

        offset += n_nodes_per_tree;
    }

    for( int i=0 ; i<size ; i++ )
    {
        node_exp = local_node_expectation[i];
        out_grad = local_node_expectation_gradient[i];
        on_free_energy_gradient[i] = out_grad * node_exp * ( 1 - node_exp );
        off_free_energy_gradient[i] = -out_grad * node_exp * ( 1 - node_exp );
    }

    offset = 0;
    for( int t=0; t<n_trees; t++ )
    {
        depth = 0;
        sub_tree_size = n_nodes_per_tree / 2;

        while( depth < tree_depth-1 )
        {
            for( int n=sub_tree_size; n<n_nodes_per_tree; n += 2*sub_tree_size + 2 )
            {
                out_grad = on_free_energy_gradient[ n + offset ];
                node_exp = local_node_expectation[n + offset - (sub_tree_size/2+1)];
                input_gradient[n+offset] += out_grad;
                on_free_energy_gradient[n + offset - (sub_tree_size/2+1)] += out_grad * node_exp;
                off_free_energy_gradient[n + offset - (sub_tree_size/2+1)] += out_grad * (1 - node_exp);

                out_grad = off_free_energy_gradient[ n + offset ];
                node_exp = local_node_expectation[n + offset + (sub_tree_size/2+1)];
                if( use_signed_samples )
                    input_gradient[n+offset] -= out_grad;
                on_free_energy_gradient[n + offset + (sub_tree_size/2+1)] += out_grad * node_exp;
                off_free_energy_gradient[n + offset + (sub_tree_size/2+1)] +=
                    out_grad * (1 - node_exp);
            }
            sub_tree_size /= 2;
            depth++;
        }

        depth = tree_depth-1;
        sub_tree_size = 0;

        for( int n=sub_tree_size; n<n_nodes_per_tree; n += 2*sub_tree_size + 2 )
        {
            input_gradient[n+offset] += on_free_energy_gradient[ n + offset ];
            if( use_signed_samples )
                input_gradient[n+offset] -= off_free_energy_gradient[ n + offset ];
        }

        offset += n_nodes_per_tree;
    }

    if( momentum != 0. )
        bias_inc.resize( size );

    for( int i=0 ; i<size ; i++ )
    {
        real in_grad_i = input_gradient[i];

        if( momentum == 0. )
        {
            // update the bias: bias -= learning_rate * input_gradient
            bias[i] -= learning_rate * in_grad_i;
        }
        else
        {
            // The update rule becomes:
            // bias_inc = momentum * bias_inc - learning_rate * input_gradient
            // bias += bias_inc
            bias_inc[i] = momentum * bias_inc[i] - learning_rate * in_grad_i;
            bias[i] += bias_inc[i];
        }
    }

    applyBiasDecay();
}

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::build ( ) [virtual]

Post-constructor.

The normal implementation should call simply inherited::build(), then this class's build_(). This method should be callable again at later times, after modifying some option fields to change the "architecture" of the object.

Reimplemented from PLearn::RBMLayer.

Definition at line 1226 of file RBMWoodsLayer.cc.

References PLearn::RBMLayer::build(), and build_().

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::build_ ( ) [private]

This does the actual building.

Reimplemented from PLearn::RBMLayer.

Definition at line 1202 of file RBMWoodsLayer.cc.

References PLearn::RBMLayer::build(), PLearn::ipow(), local_node_expectation, local_node_expectation_gradient, n_trees, off_expectation, off_free_energy, off_free_energy_gradient, off_tree_gradient, on_free_energy, on_free_energy_gradient, on_tree_gradient, PLASSERT, PLERROR, PLearn::TVec< T >::resize(), PLearn::RBMLayer::size, and tree_depth.

Referenced by build().

{
    PLASSERT( n_trees > 0 );
    PLASSERT( tree_depth > 0 );

    if ( tree_depth < 2 )
        PLERROR("RBMWoodsLayer::build_(): tree_depth < 2 not supported, use "
                "RBMBinomialLayer instead.");

    size = n_trees * ( ipow( 2, tree_depth ) - 1 );
    local_node_expectation.resize( size );
    on_free_energy.resize( size );
    off_free_energy.resize( size );
    off_expectation.resize( size );
    local_node_expectation_gradient.resize( size );
    on_tree_gradient.resize( size );
    off_tree_gradient.resize( size );
    on_free_energy_gradient.resize( size );
    off_free_energy_gradient.resize( size );

    // Must call parent's build, since size was just set
    inherited::build();
}

Here is the call graph for this function:

Here is the caller graph for this function:

string PLearn::RBMWoodsLayer::classname ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 51 of file RBMWoodsLayer.cc.

void PLearn::RBMWoodsLayer::computeExpectation ( ) [virtual]

Compute marginal expectations of all units.

Implements PLearn::RBMLayer.

Definition at line 179 of file RBMWoodsLayer.cc.

References PLearn::RBMLayer::activation, PLearn::RBMLayer::expectation, PLearn::RBMLayer::expectation_is_up_to_date, i, PLearn::TVec< T >::length(), local_node_expectation, PLearn::logadd(), n, n_trees, off_expectation, off_free_energy, on_free_energy, PLearn::safeexp(), PLearn::RBMLayer::size, tree_depth, and use_signed_samples.

Referenced by bpropNLL(), and computeProbabilisticClustering().

{
    if( expectation_is_up_to_date )
        return;

    int n_nodes_per_tree = size / n_trees;
    int node, depth, sub_tree_size, grand_parent;
    int offset = 0;
    bool left_of_grand_parent;
    real grand_parent_prob;

    // Get local expectations at every node

    // HUGO: Note that local_node_expectation is really
    // used as a probability, even for signed samples.
    // Sorry for the misleading choice of variable name...

    // Divide and conquer computation of local (conditional) free energies
    for( int t=0; t<n_trees; t++ )
    {
        depth = tree_depth-1;
        sub_tree_size = 0;

        // Initialize last level
        for( int n=sub_tree_size; n<n_nodes_per_tree; n += 2*sub_tree_size + 2 )
        {
            //on_free_energy[ n + offset ] = safeexp(activation[n+offset]);
            //off_free_energy[ n + offset ] = 1;
            // Now working in log-domain
            on_free_energy[ n + offset ] = activation[n+offset];
            if( use_signed_samples )
                off_free_energy[ n + offset ] = -activation[n+offset];
            else
                off_free_energy[ n + offset ] = 0;
        }

        depth = tree_depth-2;
        sub_tree_size = 1;

        while( depth >= 0 )
        {
            for( int n=sub_tree_size; n<n_nodes_per_tree; n += 2*sub_tree_size + 2 )
            {
                //on_free_energy[ n + offset ] = safeexp(activation[n+offset]) *
                //    ( on_free_energy[n + offset - sub_tree_size] + off_free_energy[n + offset - sub_tree_size] ) ;
                //off_free_energy[ n + offset ] =
                //    ( on_free_energy[n + offset + sub_tree_size] + off_free_energy[n + offset + sub_tree_size] ) ;
                // Now working in log-domain
                on_free_energy[ n + offset ] = activation[n+offset] +
                    logadd( on_free_energy[n + offset - (sub_tree_size/2+1)],
                            off_free_energy[n + offset - (sub_tree_size/2+1)] ) ;
                if( use_signed_samples )
                    off_free_energy[ n + offset ] = -activation[n+offset] +
                        logadd( on_free_energy[n + offset + (sub_tree_size/2+1)],
                                off_free_energy[n + offset + (sub_tree_size/2+1)] ) ;
                else
                    off_free_energy[ n + offset ] =
                        logadd( on_free_energy[n + offset + (sub_tree_size/2+1)],
                                off_free_energy[n + offset + (sub_tree_size/2+1)] ) ;

            }
            sub_tree_size = 2 * ( sub_tree_size + 1 ) - 1;
            depth--;
        }
        offset += n_nodes_per_tree;
    }

    for( int i=0 ; i<size ; i++ )
        //local_node_expectation[i] = on_free_energy[i] / ( on_free_energy[i] + off_free_energy[i] );
        // Now working in log-domain
        local_node_expectation[i] = safeexp(on_free_energy[i]
                                            - logadd(on_free_energy[i], off_free_energy[i]));

    // Compute marginal expectations
    offset = 0;
    for( int t=0; t<n_trees; t++ )
    {
        // Initialize root
        node = n_nodes_per_tree / 2;
        expectation[ node + offset ] = local_node_expectation[ node + offset ];
        off_expectation[ node + offset ] = (1 - local_node_expectation[ node + offset ]);
        sub_tree_size = node;

        // First level nodes
        depth = 1;
        sub_tree_size /= 2;

        // Left child
        node = sub_tree_size;
        expectation[ node + offset ] = local_node_expectation[ node + offset ]
            * local_node_expectation[ node + offset + sub_tree_size + 1 ];
        off_expectation[ node + offset ] = (1 - local_node_expectation[ node + offset ])
            * local_node_expectation[ node + offset + sub_tree_size + 1 ];

        // Right child
        node = 3*sub_tree_size+2;
        expectation[ node + offset ] = local_node_expectation[ node + offset ]
            * (1 - local_node_expectation[ node + offset - sub_tree_size - 1 ]);
        off_expectation[ node + offset ] = (1 - local_node_expectation[ node + offset ])
            * (1 - local_node_expectation[ node + offset - sub_tree_size - 1 ]);

        // Set other nodes, level-wise
        depth = 2;
        sub_tree_size /= 2;
        while( depth < tree_depth )
        {
            // Left child
            left_of_grand_parent = true;
            for( int n=sub_tree_size; n<n_nodes_per_tree; n += 4*sub_tree_size + 4 )
            {
                if( left_of_grand_parent )
                {
                    grand_parent = n + offset + 3*sub_tree_size + 3;
                    grand_parent_prob = expectation[ grand_parent ];
                    left_of_grand_parent = false;
                }
                else
                {
                    grand_parent = n + offset - sub_tree_size - 1;
                    grand_parent_prob = off_expectation[ grand_parent ];
                    left_of_grand_parent = true;
                }

                expectation[ n + offset ] = local_node_expectation[ n + offset ]
                    * local_node_expectation[ n + offset + sub_tree_size + 1 ]
                    * grand_parent_prob;
                off_expectation[ n + offset ] = (1 - local_node_expectation[ n + offset ])
                    * local_node_expectation[ n + offset + sub_tree_size + 1 ]
                    * grand_parent_prob;

            }

            // Right child
            left_of_grand_parent = true;
            for( int n=3*sub_tree_size+2; n<n_nodes_per_tree; n += 4*sub_tree_size + 4 )
            {
                if( left_of_grand_parent )
                {
                    grand_parent = n + offset + sub_tree_size + 1;
                    grand_parent_prob = expectation[ grand_parent ];
                    left_of_grand_parent = false;
                }
                else
                {
                    grand_parent = n + offset - 3*sub_tree_size - 3;
                    grand_parent_prob = off_expectation[ grand_parent ];
                    left_of_grand_parent = true;
                }

                expectation[ n + offset ] = local_node_expectation[ n + offset ]
                    * (1 - local_node_expectation[ n + offset - sub_tree_size - 1 ])
                    * grand_parent_prob;
                off_expectation[ n + offset ] = (1 - local_node_expectation[ n + offset ])
                    * (1 - local_node_expectation[ n + offset - sub_tree_size - 1 ])
                    * grand_parent_prob;
            }
            sub_tree_size /= 2;
            depth++;
        }
        offset += n_nodes_per_tree;
    }

    if( use_signed_samples )
        for( int i=0; i<expectation.length(); i++ )
            expectation[i] = expectation[i] - off_expectation[i];

    expectation_is_up_to_date = true;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::RBMWoodsLayer::computeExpectations ( ) [virtual]

Compute marginal mini-batch expectations of all units.

Implements PLearn::RBMLayer.

Definition at line 351 of file RBMWoodsLayer.cc.

References PLearn::RBMLayer::activations, b, PLearn::RBMLayer::batch_size, PLearn::RBMLayer::expectation, PLearn::RBMLayer::expectations, PLearn::RBMLayer::expectations_are_up_to_date, i, PLearn::TMat< T >::length(), PLearn::TVec< T >::length(), local_node_expectations, PLearn::logadd(), n, n_trees, off_expectations, off_free_energies, on_free_energies, PLASSERT, PLearn::TMat< T >::resize(), PLearn::safeexp(), PLearn::RBMLayer::size, tree_depth, use_signed_samples, and PLearn::TMat< T >::width().

Referenced by bpropNLL().

{
    if( expectations_are_up_to_date )
        return;

    PLASSERT( expectations.width() == size
              && expectations.length() == batch_size );
    off_expectations.resize(batch_size,size);
    local_node_expectations.resize(batch_size,size);
    on_free_energies.resize(batch_size,size);
    off_free_energies.resize(batch_size,size);

    int n_nodes_per_tree = size / n_trees;
    int node, depth, sub_tree_size, grand_parent;
    int offset = 0;
    bool left_of_grand_parent;
    real grand_parent_prob;
    for( int b=0; b<batch_size; b++ )
    {
        offset=0;
        // Get local expectations at every node
        
        // HUGO: Note that local_node_expectations is really
        // used as a probability, even for signed samples.
        // Sorry for the misleading choice of variable name...
        
        // Divide and conquer computation of local (conditional) free energies
        for( int t=0; t<n_trees; t++ )
        {
            depth = tree_depth-1;
            sub_tree_size = 0;

            // Initialize last level
            for( int n=sub_tree_size; n<n_nodes_per_tree; n += 2*sub_tree_size + 2 )
            {
                //on_free_energies(b, n + offset ) = safeexp(activations(b,n+offset));
                //off_free_energies(b, n + offset ) = 1;
                // Now working in log-domain
                on_free_energies(b, n + offset ) = activations(b,n+offset);
                if( use_signed_samples )
                    off_free_energies(b, n + offset ) = -activations(b,n+offset);
                else
                    off_free_energies(b, n + offset ) = 0;
            }

            depth = tree_depth-2;
            sub_tree_size = 1;

            while( depth >= 0 )
            {
                for( int n=sub_tree_size; n<n_nodes_per_tree; n += 2*sub_tree_size + 2 )
                {
                    //on_free_energies(b, n + offset ) = safeexp(activations(b,n+offset)) *
                    //    ( on_free_energies(b,n + offset - sub_tree_size) + off_free_energies(b,n + offset - sub_tree_size) ) ;
                    //off_free_energies(b, n + offset ) =
                    //    ( on_free_energies(b,n + offset + sub_tree_size) + off_free_energies(b,n + offset + sub_tree_size) ) ;
                    // Now working in log-domain
                    on_free_energies(b, n + offset ) = activations(b,n+offset) +
                        logadd( on_free_energies(b,n + offset - (sub_tree_size/2+1)),
                                off_free_energies(b,n + offset - (sub_tree_size/2+1)) ) ;
                    if( use_signed_samples )
                        off_free_energies(b, n + offset ) = -activations(b,n+offset) +
                            logadd( on_free_energies(b,n + offset + (sub_tree_size/2+1)),
                                    off_free_energies(b,n + offset + (sub_tree_size/2+1)) ) ;
                    else
                        off_free_energies(b, n + offset ) =
                            logadd( on_free_energies(b,n + offset + (sub_tree_size/2+1)),
                                    off_free_energies(b,n + offset + (sub_tree_size/2+1)) ) ;

                }
                sub_tree_size = 2 * ( sub_tree_size + 1 ) - 1;
                depth--;
            }
            offset += n_nodes_per_tree;
        }

        for( int i=0 ; i<size ; i++ )
            //local_node_expectations(b,i) = on_free_energies(b,i) / ( on_free_energies(b,i) + off_free_energies(b,i) );
            // Now working in log-domain
            local_node_expectations(b,i) = safeexp(on_free_energies(b,i)
                                                - logadd(on_free_energies(b,i), off_free_energies(b,i)));

        // Compute marginal expectations
        offset = 0;
        for( int t=0; t<n_trees; t++ )
        {
            // Initialize root
            node = n_nodes_per_tree / 2;
            expectations(b, node + offset ) = local_node_expectations(b, node + offset );
            off_expectations(b, node + offset ) = (1 - local_node_expectations(b, node + offset ));
            sub_tree_size = node;

            // First level nodes
            depth = 1;
            sub_tree_size /= 2;

            // Left child
            node = sub_tree_size;
            expectations(b, node + offset ) = local_node_expectations(b, node + offset )
                * local_node_expectations(b, node + offset + sub_tree_size + 1 );
            off_expectations(b, node + offset ) = (1 - local_node_expectations(b, node + offset ))
                * local_node_expectations(b, node + offset + sub_tree_size + 1 );

            // Right child
            node = 3*sub_tree_size+2;
            expectations(b, node + offset ) = local_node_expectations(b, node + offset )
                * (1 - local_node_expectations(b, node + offset - sub_tree_size - 1 ));
            off_expectations(b, node + offset ) = (1 - local_node_expectations(b, node + offset ))
                * (1 - local_node_expectations(b, node + offset - sub_tree_size - 1 ));

            // Set other nodes, level-wise
            depth = 2;
            sub_tree_size /= 2;
            while( depth < tree_depth )
            {
                // Left child
                left_of_grand_parent = true;
                for( int n=sub_tree_size; n<n_nodes_per_tree; n += 4*sub_tree_size + 4 )
                {
                    if( left_of_grand_parent )
                    {
                        grand_parent = n + offset + 3*sub_tree_size + 3;
                        grand_parent_prob = expectations(b, grand_parent );
                        left_of_grand_parent = false;
                    }
                    else
                    {
                        grand_parent = n + offset - sub_tree_size - 1;
                        grand_parent_prob = off_expectations(b, grand_parent );
                        left_of_grand_parent = true;
                    }

                    expectations(b, n + offset ) = local_node_expectations(b, n + offset )
                        * local_node_expectations(b, n + offset + sub_tree_size + 1 )
                        * grand_parent_prob;
                    off_expectations(b, n + offset ) = (1 - local_node_expectations(b, n + offset ))
                        * local_node_expectations(b, n + offset + sub_tree_size + 1 )
                        * grand_parent_prob;

                }

                // Right child
                left_of_grand_parent = true;
                for( int n=3*sub_tree_size+2; n<n_nodes_per_tree; n += 4*sub_tree_size + 4 )
                {
                    if( left_of_grand_parent )
                    {
                        grand_parent = n + offset + sub_tree_size + 1;
                        grand_parent_prob = expectations(b, grand_parent );
                        left_of_grand_parent = false;
                    }
                    else
                    {
                        grand_parent = n + offset - 3*sub_tree_size - 3;
                        grand_parent_prob = off_expectations(b, grand_parent );
                        left_of_grand_parent = true;
                    }

                    expectations(b, n + offset ) = local_node_expectations(b, n + offset )
                        * (1 - local_node_expectations(b, n + offset - sub_tree_size - 1 ))
                        * grand_parent_prob;
                    off_expectations(b, n + offset ) = (1 - local_node_expectations(b, n + offset ))
                        * (1 - local_node_expectations(b, n + offset - sub_tree_size - 1 ))
                        * grand_parent_prob;
                }
                sub_tree_size /= 2;
                depth++;
            }
            offset += n_nodes_per_tree;
        }
    }
    
    if( use_signed_samples )
        for( int b=0; b<batch_size; b++ )
            for( int i=0; i<expectation.length(); i++ )
                expectations(b,i) = expectations(b,i) - off_expectations(b,i);

    expectations_are_up_to_date = true;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::RBMWoodsLayer::computeProbabilisticClustering ( Vec prob_clusters) [virtual]

Definition at line 160 of file RBMWoodsLayer.cc.

References computeExpectation(), PLearn::RBMLayer::expectation, i, n_trees, off_expectation, PLearn::TVec< T >::resize(), and PLearn::RBMLayer::size.

{
    computeExpectation();
    int offset = 0;
    int n_nodes_per_tree = size / n_trees;
    prob_clusters.resize(n_trees*(n_nodes_per_tree+1));
    for( int t=0; t<n_trees; t++ )
    {
        for( int i=0; i<n_nodes_per_tree; i = i+2)
            prob_clusters[i+offset+t] = expectation[i+offset];
        for( int i=0; i<n_nodes_per_tree; i = i+2)
            prob_clusters[i+1+offset+t] = off_expectation[i+offset];
        offset += n_nodes_per_tree;
    }
}

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::declareOptions ( OptionList ol) [static, protected]

Declares the class options.

Reimplemented from PLearn::RBMLayer.

Definition at line 1182 of file RBMWoodsLayer.cc.

References PLearn::OptionBase::buildoption, PLearn::declareOption(), PLearn::RBMLayer::declareOptions(), n_trees, tree_depth, and use_signed_samples.

{
    declareOption(ol, "n_trees", &RBMWoodsLayer::n_trees,
                  OptionBase::buildoption,
                  "Number of trees in the woods.");

    declareOption(ol, "tree_depth", &RBMWoodsLayer::tree_depth,
                  OptionBase::buildoption,
                  "Depth of the trees in the woods (1 gives the ordinary "
                  "RBMBinomialLayer).");

    declareOption(ol, "use_signed_samples", &RBMWoodsLayer::use_signed_samples,
                  OptionBase::buildoption,
                  "Indication that samples should be in {-1,1}, not {0,1}, at nodes where a\n"
                  "left/right decision is made. Other nodes are set to 0.\n");

    // Now call the parent class' declareOptions
    inherited::declareOptions(ol);
}

Here is the call graph for this function:

static const PPath& PLearn::RBMWoodsLayer::declaringFile ( ) [inline, static]

Reimplemented from PLearn::RBMLayer.

Definition at line 151 of file RBMWoodsLayer.h.

:
    //#####  Not Options  #####################################################
RBMWoodsLayer * PLearn::RBMWoodsLayer::deepCopy ( CopiesMap copies) const [virtual]

Reimplemented from PLearn::RBMLayer.

Definition at line 51 of file RBMWoodsLayer.cc.

real PLearn::RBMWoodsLayer::energy ( const Vec unit_values) const [virtual]

compute -bias' unit_values

Reimplemented from PLearn::RBMLayer.

Definition at line 1252 of file RBMWoodsLayer.cc.

References PLearn::RBMLayer::bias, PLearn::dot(), and PLERROR.

{
    PLERROR( "RBMWoodsLayer::energy(): not implemeted yet" );
    return -dot(unit_values, bias);
}

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::fprop ( const Mat inputs,
Mat outputs 
) [virtual]

Batch forward propagation.

Reimplemented from PLearn::RBMLayer.

Definition at line 695 of file RBMWoodsLayer.cc.

References PLearn::TMat< T >::length(), PLASSERT, PLERROR, PLearn::TMat< T >::resize(), PLearn::RBMLayer::size, and PLearn::TMat< T >::width().

{
    int mbatch_size = inputs.length();
    PLASSERT( inputs.width() == size );
    outputs.resize( mbatch_size, size );

    PLERROR( "RBMWoodsLayer::fprop(): not implemented yet" );
}

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::fprop ( const Vec input,
const Vec rbm_bias,
Vec output 
) const [virtual]

forward propagation with provided bias

Reimplemented from PLearn::RBMLayer.

Definition at line 704 of file RBMWoodsLayer.cc.

References PLearn::OnlineLearningModule::input_size, PLearn::OnlineLearningModule::output_size, PLASSERT, PLERROR, PLearn::TVec< T >::resize(), and PLearn::TVec< T >::size().

{
    PLASSERT( input.size() == input_size );
    PLASSERT( rbm_bias.size() == input_size );
    output.resize( output_size );

    PLERROR( "RBMWoodsLayer::fprop(): not implemented yet" );
}

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::fprop ( const Vec input,
Vec output 
) const [virtual]

forward propagation

Reimplemented from PLearn::RBMLayer.

Definition at line 534 of file RBMWoodsLayer.cc.

References PLearn::RBMLayer::bias, i, PLearn::OnlineLearningModule::input_size, PLearn::TVec< T >::length(), local_node_expectation, PLearn::logadd(), n, n_trees, off_expectation, off_free_energy, on_free_energy, PLearn::OnlineLearningModule::output_size, PLASSERT, PLearn::TVec< T >::resize(), PLearn::safeexp(), PLearn::RBMLayer::size, PLearn::TVec< T >::size(), tree_depth, and use_signed_samples.

{
    PLASSERT( input.size() == input_size );
    output.resize( output_size );

    int n_nodes_per_tree = size / n_trees;
    int node, depth, sub_tree_size, grand_parent;
    int offset = 0;
    bool left_of_grand_parent;
    real grand_parent_prob;

    // Get local expectations at every node

    // Divide and conquer computation of local (conditional) free energies
    for( int t=0; t<n_trees; t++ )
    {
        depth = tree_depth-1;
        sub_tree_size = 0;

        // Initialize last level
        for( int n=sub_tree_size; n<n_nodes_per_tree; n += 2*sub_tree_size + 2 )
        {
            //on_free_energy[ n + offset ] = safeexp(input[n+offset] + bias[n+offset]);
            //off_free_energy[ n + offset ] = 1;
            // Now working in log-domain
            on_free_energy[ n + offset ] = input[n+offset] + bias[n+offset];
            if( use_signed_samples )
                off_free_energy[ n + offset ] = -(input[n+offset] + bias[n+offset]);
            else
                off_free_energy[ n + offset ] = 0;
        }

        depth = tree_depth-2;
        sub_tree_size = 1;

        while( depth >= 0 )
        {
            for( int n=sub_tree_size; n<n_nodes_per_tree; n += 2*sub_tree_size + 2 )
            {
                //on_free_energy[ n + offset ] = safeexp(input[n+offset] + bias[n+offset]) *
                //    ( on_free_energy[n + offset - sub_tree_size] + off_free_energy[n + offset - sub_tree_size] ) ;
                //off_free_energy[ n + offset ] =
                //    ( on_free_energy[n + offset + sub_tree_size] + off_free_energy[n + offset + sub_tree_size] ) ;
                // Now working in the log-domain
                on_free_energy[ n + offset ] = input[n+offset] + bias[n+offset] +
                    logadd( on_free_energy[n + offset - (sub_tree_size/2+1)],
                            off_free_energy[n + offset - (sub_tree_size/2+1)] ) ;
                if( use_signed_samples )
                    off_free_energy[ n + offset ] = -(input[n+offset] + bias[n+offset]) +
                        logadd( on_free_energy[n + offset + (sub_tree_size/2+1)],
                                off_free_energy[n + offset + (sub_tree_size/2+1)] ) ;
                else
                    off_free_energy[ n + offset ] =
                        logadd( on_free_energy[n + offset + (sub_tree_size/2+1)],
                                off_free_energy[n + offset + (sub_tree_size/2+1)] ) ;
            }
            sub_tree_size = 2 * ( sub_tree_size + 1 ) - 1;
            depth--;
        }
        offset += n_nodes_per_tree;
    }

    for( int i=0 ; i<size ; i++ )
        //local_node_expectation[i] = on_free_energy[i] / ( on_free_energy[i] + off_free_energy[i] );
        // Now working in log-domain
        local_node_expectation[i] = safeexp(on_free_energy[i]
                                            - logadd(on_free_energy[i], off_free_energy[i]));

    // Compute marginal expectations
    offset = 0;
    for( int t=0; t<n_trees; t++ )
    {
        // Initialize root
        node = n_nodes_per_tree / 2;
        output[ node + offset ] = local_node_expectation[ node + offset ];
        off_expectation[ node + offset ] = (1 - local_node_expectation[ node + offset ]);
        sub_tree_size = node;

        // First level nodes
        depth = 1;
        sub_tree_size /= 2;

        // Left child
        node = sub_tree_size;
        output[ node + offset ] = local_node_expectation[ node + offset ]
            * local_node_expectation[ node + offset + sub_tree_size + 1 ];
        off_expectation[ node + offset ] = (1 - local_node_expectation[ node + offset ])
            * local_node_expectation[ node + offset + sub_tree_size + 1 ];

        // Right child
        node = 3*sub_tree_size+2;
        output[ node + offset ] = local_node_expectation[ node + offset ]
            * (1 - local_node_expectation[ node + offset - sub_tree_size - 1 ]);
        off_expectation[ node + offset ] = (1 - local_node_expectation[ node + offset ])
            * (1 - local_node_expectation[ node + offset - sub_tree_size - 1 ]);

        // Set other nodes, level-wise
        depth = 2;
        sub_tree_size /= 2;
        while( depth < tree_depth )
        {
            // Left child
            left_of_grand_parent = true;
            for( int n=sub_tree_size; n<n_nodes_per_tree; n += 4*sub_tree_size + 4 )
            {
                if( left_of_grand_parent )
                {
                    grand_parent = n + offset + 3*sub_tree_size + 3;
                    grand_parent_prob = output[ grand_parent ];
                    left_of_grand_parent = false;
                }
                else
                {
                    grand_parent = n + offset - sub_tree_size - 1;
                    grand_parent_prob = off_expectation[ grand_parent ];
                    left_of_grand_parent = true;
                }

                output[ n + offset ] = local_node_expectation[ n + offset ]
                    * local_node_expectation[ n + offset + sub_tree_size + 1 ]
                    * grand_parent_prob;
                off_expectation[ n + offset ] = (1 - local_node_expectation[ n + offset ])
                    * local_node_expectation[ n + offset + sub_tree_size + 1 ]
                    * grand_parent_prob;
            }

            // Right child
            left_of_grand_parent = true;
            for( int n=3*sub_tree_size+2; n<n_nodes_per_tree; n += 4*sub_tree_size + 4 )
            {
                if( left_of_grand_parent )
                {
                    grand_parent = n + offset + sub_tree_size + 1;
                    grand_parent_prob = output[ grand_parent ];
                    left_of_grand_parent = false;
                }
                else
                {
                    grand_parent = n + offset - 3*sub_tree_size - 3;
                    grand_parent_prob = off_expectation[ grand_parent ];
                    left_of_grand_parent = true;
                }

                output[ n + offset ] = local_node_expectation[ n + offset ]
                    * (1 - local_node_expectation[ n + offset - sub_tree_size - 1 ])
                    * grand_parent_prob;
                off_expectation[ n + offset ] = (1 - local_node_expectation[ n + offset ])
                    * (1 - local_node_expectation[ n + offset - sub_tree_size - 1 ])
                    * grand_parent_prob;
            }
            sub_tree_size /= 2;
            depth++;
        }
        offset += n_nodes_per_tree;
    }

    if( use_signed_samples )
        for( int i=0; i<output.length(); i++ )
            output[i] = output[i] - off_expectation[i];
}

Here is the call graph for this function:

real PLearn::RBMWoodsLayer::fpropNLL ( const Vec target) [virtual]

Computes the negative log-likelihood of target given the internal activations of the layer.

Reimplemented from PLearn::RBMLayer.

Definition at line 1069 of file RBMWoodsLayer.cc.

References PLearn::RBMLayer::activation, i, PLearn::OnlineLearningModule::input_size, PLASSERT, PLERROR, PLearn::TVec< T >::size(), PLearn::RBMLayer::size, PLearn::softplus(), PLearn::tabulated_softplus(), and PLearn::OnlineLearningModule::use_fast_approximations.

{
    PLASSERT( target.size() == input_size );

    PLERROR( "RBMWoodsLayer::fpropNLL(): not implemeted yet" );

    real ret = 0;
    real target_i, activation_i;
    if(use_fast_approximations){
        for( int i=0 ; i<size ; i++ )
        {
            target_i = target[i];
            activation_i = activation[i];
            ret += tabulated_softplus(activation_i) - target_i * activation_i;
            // nll = - target*log(sigmoid(act)) -(1-target)*log(1-sigmoid(act))
            // but it is numerically unstable, so use instead the following identity:
            //     = target*softplus(-act) +(1-target)*(act+softplus(-act))
            //     = act + softplus(-act) - target*act
            //     = softplus(act) - target*act
        }
    } else {
        for( int i=0 ; i<size ; i++ )
        {
            target_i = target[i];
            activation_i = activation[i];
            ret += softplus(activation_i) - target_i * activation_i;
        }
    }
    return ret;
}

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::fpropNLL ( const Mat targets,
const Mat costs_column 
) [virtual]

Reimplemented from PLearn::RBMLayer.

Definition at line 1100 of file RBMWoodsLayer.cc.

References PLearn::RBMLayer::activation, PLearn::RBMLayer::activations, PLearn::RBMLayer::batch_size, PLearn::fast_exact_is_equal(), i, PLearn::OnlineLearningModule::input_size, PLearn::TMat< T >::length(), PLASSERT, PLERROR, PLearn::RBMLayer::size, PLearn::softplus(), PLearn::tabulated_softplus(), PLearn::OnlineLearningModule::use_fast_approximations, and PLearn::TMat< T >::width().

{
    // computeExpectations(); // why?

    PLERROR( "RBMWoodsLayer::fpropNLL(): not implemeted yet" );

    PLASSERT( targets.width() == input_size );
    PLASSERT( targets.length() == batch_size );
    PLASSERT( costs_column.width() == 1 );
    PLASSERT( costs_column.length() == batch_size );

    for (int k=0;k<batch_size;k++) // loop over minibatch
    {
        real nll = 0;
        real* activation = activations[k];
        real* target = targets[k];
        if(use_fast_approximations){
            for( int i=0 ; i<size ; i++ ) // loop over outputs
            {
                if(!fast_exact_is_equal(target[i],0.0))
                    // nll -= target[i] * pl_log(expectations[i]);
                    // but it is numerically unstable, so use instead
                    // log (1/(1+exp(-x))) = -log(1+exp(-x)) = -softplus(-x)
                    nll += target[i] * tabulated_softplus(-activation[i]);
                if(!fast_exact_is_equal(target[i],1.0))
                    // nll -= (1-target[i]) * pl_log(1-output[i]);
                    // log (1 - 1/(1+exp(-x))) = log(exp(-x)/(1+exp(-x)))
                    //                         = log(1/(1+exp(x)))
                    //                         = -log(1+exp(x))
                    //                         = -softplus(x)
                    nll += (1-target[i]) * tabulated_softplus(activation[i]);
            }
        } else {
            for( int i=0 ; i<size ; i++ ) // loop over outputs
            {
                if(!fast_exact_is_equal(target[i],0.0))
                    // nll -= target[i] * pl_log(expectations[i]);
                    // but it is numerically unstable, so use instead
                    // log (1/(1+exp(-x))) = -log(1+exp(-x)) = -softplus(-x)
                    nll += target[i] * softplus(-activation[i]);
                if(!fast_exact_is_equal(target[i],1.0))
                    // nll -= (1-target[i]) * pl_log(1-output[i]);
                    // log (1 - 1/(1+exp(-x))) = log(exp(-x)/(1+exp(-x)))
                    //                         = log(1/(1+exp(x)))
                    //                         = -log(1+exp(x))
                    //                         = -softplus(x)
                    nll += (1-target[i]) * softplus(activation[i]);
            }
        }
        costs_column(k,0) = nll;
    }
}

Here is the call graph for this function:

real PLearn::RBMWoodsLayer::freeEnergyContribution ( const Vec unit_activations) const [virtual]

Computes $ -log(\sum_{possible values of h} exp(h' unit_activations))$ This quantity is used for computing the free energy of a sample x in the OTHER layer of an RBM, from which unit_activations was computed.

Reimplemented from PLearn::RBMLayer.

Definition at line 1258 of file RBMWoodsLayer.cc.

References d, PLearn::logadd(), n, n_trees, PLASSERT, PLearn::TVec< T >::resize(), PLearn::TVec< T >::size(), PLearn::RBMLayer::size, tree_depth, tree_energies, tree_free_energies, and use_signed_samples.

Referenced by freeEnergyContributionGradient().

{
    PLASSERT( unit_activations.size() == size );
    int n_nodes_per_tree = size / n_trees;
    tree_free_energies.resize(n_trees);
    tree_energies.resize(n_trees * (n_nodes_per_tree+1) );

    int offset=0;
    int sub_tree_size = n_nodes_per_tree / 2;
    int sub_root = sub_tree_size;
    real result = 0;
    real tree_energy = 0;
    real tree_free_energy = 0;
    real leaf_activation = 0;
    for( int t = 0; t<n_trees; t++ )
    {
        for( int n = 0; n < n_nodes_per_tree; n = n+2 ) // Looking only at leaves
        {
            // Computation energy of tree
            tree_energy = 0;
            sub_tree_size = n_nodes_per_tree / 2;
            sub_root = sub_tree_size;
            for( int d=0; d<tree_depth-1; d++ )
            {
                if( n < sub_root )
                {
                    tree_energy -= unit_activations[offset+sub_root];
                    sub_tree_size /= 2;
                    sub_root -= sub_tree_size + 1;
                }
                else
                {
                    if( use_signed_samples )
                        tree_energy -= -unit_activations[offset+sub_root];
                    sub_tree_size /= 2;
                    sub_root += sub_tree_size+1;
                }
            }
            
            leaf_activation = unit_activations[offset+n];
            // Add free energy of tree with activated leaf
            if( n == 0)
                tree_free_energy = tree_energy - leaf_activation;
            else
                tree_free_energy = -logadd( -tree_energy + leaf_activation, 
                                            -tree_free_energy );
            tree_energies[offset+t+n] = tree_energy - leaf_activation;

            // Add free_energy of tree with inactivated leaf
            if( use_signed_samples )
            {
                tree_free_energy = -logadd( -tree_energy - leaf_activation, 
                                            -tree_free_energy );
                tree_energies[offset+t+n+1] = tree_energy + leaf_activation;
            }
            else
            {
                tree_free_energy = -logadd( -tree_energy, -tree_free_energy );
                tree_energies[offset+t+n+1] = tree_energy;
            }
        }
        tree_free_energies[t] = tree_free_energy;
        result += tree_free_energy;
        offset += n_nodes_per_tree;
    }
    return result;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void PLearn::RBMWoodsLayer::freeEnergyContributionGradient ( const Vec unit_activations,
Vec unit_activations_gradient,
real  output_gradient = 1,
bool  accumulate = false 
) const [virtual]

Computes gradient of the result of freeEnergyContribution $ -log(\sum_{possible values of h} exp(h' unit_activations))$ with respect to unit_activations.

Optionally, a gradient with respect to freeEnergyContribution can be given

Reimplemented from PLearn::RBMLayer.

Definition at line 1327 of file RBMWoodsLayer.cc.

References PLearn::TVec< T >::clear(), d, PLearn::TVec< T >::fill(), freeEnergyContribution(), i, PLearn::logadd(), n, n_trees, PLASSERT, PLearn::TVec< T >::resize(), PLearn::safeexp(), PLearn::TVec< T >::size(), PLearn::RBMLayer::size, tree_depth, tree_energies, tree_free_energies, unit_activations_neg_gradient, unit_activations_neg_gradient_init, unit_activations_pos_gradient, unit_activations_pos_gradient_init, and use_signed_samples.

{
    PLASSERT( unit_activations.size() == size );
    unit_activations_gradient.resize( size );
    if( !accumulate ) unit_activations_gradient.clear();
    
    // This method assumes freeEnergyContribution() has been called before,
    // with the same unit_activations vector!!!
    
    int n_nodes_per_tree = size / n_trees;
    int offset=0;
    int sub_tree_size = n_nodes_per_tree / 2;
    int sub_root = sub_tree_size;
    real tree_energy = 0;
    real tree_energy_gradient = 0;
    real tree_energy_leaf_on_gradient = 0;
    real tree_energy_leaf_off_gradient = 0;

    // Fills in the internal variables tree_energies and tree_free_energies.
    // I have to do this because I can't assume the last time freeEnergyContribution was
    // called was with the same unit_activations...
    freeEnergyContribution(unit_activations);

    unit_activations_neg_gradient.resize(size);
    unit_activations_neg_gradient_init.resize(size);
    unit_activations_neg_gradient_init.fill(false);
    if( use_signed_samples )
    {
        unit_activations_pos_gradient.resize(size);
        unit_activations_pos_gradient_init.resize(size);
        unit_activations_pos_gradient_init.fill(false);
    }

    for( int t = 0; t<n_trees; t++ )
    {
        for( int n = 0; n < n_nodes_per_tree; n = n+2 ) // Looking only at leaves
        {
            // Computation energy of tree
            tree_energy = 0;
            sub_tree_size = n_nodes_per_tree / 2;
            sub_root = sub_tree_size;
            // First do things on log-scale
            tree_energy_leaf_on_gradient = -tree_energies[offset+t+n] + tree_free_energies[t];
            tree_energy_leaf_off_gradient = -tree_energies[offset+t+n+1] + tree_free_energies[t];
            tree_energy_gradient = logadd(tree_energy_leaf_on_gradient,
                                          tree_energy_leaf_off_gradient);
            for( int d=0; d<tree_depth-1; d++ )
            {
                if( n < sub_root )
                {
                    if( unit_activations_neg_gradient_init[offset+sub_root] )
                        unit_activations_neg_gradient[offset+sub_root] = 
                            logadd(tree_energy_gradient,
                                   unit_activations_neg_gradient[offset+sub_root]);
                    else
                    {
                        unit_activations_neg_gradient[offset+sub_root] = 
                            tree_energy_gradient;
                        unit_activations_neg_gradient_init[offset+sub_root] = true;
                    }
                        
                    sub_tree_size /= 2;
                    sub_root -= sub_tree_size + 1;
                }
                else
                {
                    if( use_signed_samples )
                    {
                        if( unit_activations_pos_gradient_init[offset+sub_root] )
                            unit_activations_pos_gradient[offset+sub_root] = 
                                logadd(tree_energy_gradient,
                                       unit_activations_pos_gradient[offset+sub_root]);
                        else
                        {
                            unit_activations_pos_gradient[offset+sub_root] = 
                                tree_energy_gradient;
                            unit_activations_pos_gradient_init[offset+sub_root] = true;
                        }
                    }
                    sub_tree_size /= 2;
                    sub_root += sub_tree_size+1;
                }
            }
            
            unit_activations_neg_gradient[offset+n] = 
                tree_energy_leaf_on_gradient;
            unit_activations_neg_gradient_init[offset+n] = true;

            if( use_signed_samples )
            {
                unit_activations_pos_gradient[offset+n] = 
                    tree_energy_leaf_off_gradient;
                unit_activations_pos_gradient_init[offset+n] = true;
            }
        }
        offset += n_nodes_per_tree;
    }

    // Go back to linear-scale
    for(int i=0; i<size; i++)
        unit_activations_gradient[i] -= output_gradient * safeexp( unit_activations_neg_gradient[i] );

    if( use_signed_samples )
        for(int i=0; i<size; i++)
            unit_activations_gradient[i] += output_gradient * 
                safeexp( unit_activations_pos_gradient[i] );
}

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::generateSample ( ) [virtual]

generate a sample, and update the sample field

Implements PLearn::RBMLayer.

Definition at line 64 of file RBMWoodsLayer.cc.

References PLearn::TVec< T >::clear(), PLearn::RBMLayer::expectation_is_up_to_date, local_node_expectation, n_trees, PLASSERT_MSG, PLCHECK_MSG, PLearn::RBMLayer::random_gen, PLearn::RBMLayer::sample, PLearn::RBMLayer::size, tree_depth, and use_signed_samples.

{
    PLASSERT_MSG(random_gen,
                 "random_gen should be initialized before generating samples");

    PLCHECK_MSG(expectation_is_up_to_date, "Expectation should be computed "
            "before calling generateSample()");

    sample.clear();

    int n_nodes_per_tree = size / n_trees;
    int node, depth, node_sample, sub_tree_size;
    int offset = 0;

    for( int t=0; t<n_trees; t++ )
    {
        depth = 0;
        node = n_nodes_per_tree / 2;
        sub_tree_size = node;
        while( depth < tree_depth )
        {
            // HUGO: Note that local_node_expectation is really
            // used as a probability, even for signed samples.
            // Sorry for the misleading choice of variable name...
            node_sample = random_gen->binomial_sample(
                local_node_expectation[ node + offset ] );
            if( use_signed_samples )
                sample[node + offset] = 2*node_sample-1;
            else
                sample[node + offset] = node_sample;

            // Descending in the tree
            sub_tree_size /= 2;
            if ( node_sample > 0.5 )
                node -= sub_tree_size+1;
            else
                node += sub_tree_size+1;
            depth++;
        }
        offset += n_nodes_per_tree;
    }
}

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::generateSamples ( ) [virtual]

Inherited.

Implements PLearn::RBMLayer.

Definition at line 110 of file RBMWoodsLayer.cc.

References b, PLearn::RBMLayer::batch_size, PLearn::TMat< T >::clear(), PLearn::RBMLayer::expectations_are_up_to_date, PLearn::TMat< T >::length(), local_node_expectations, n_trees, PLASSERT, PLASSERT_MSG, PLCHECK_MSG, PLearn::RBMLayer::random_gen, PLearn::RBMLayer::samples, PLearn::RBMLayer::size, tree_depth, use_signed_samples, and PLearn::TMat< T >::width().

{
    PLASSERT_MSG(random_gen,
                 "random_gen should be initialized before generating samples");

    PLCHECK_MSG(expectations_are_up_to_date, "Expectations should be computed "
            "before calling generateSamples()");

    PLASSERT( samples.width() == size && samples.length() == batch_size );

    //PLERROR( "RBMWoodsLayer::generateSamples(): not implemented yet" );
    samples.clear();

    int n_nodes_per_tree = size / n_trees;
    int node, depth, node_sample, sub_tree_size;
    int offset = 0;

    for( int b=0; b<batch_size; b++ )
    {
        offset = 0;
        for( int t=0; t<n_trees; t++ )
        {
            depth = 0;
            node = n_nodes_per_tree / 2;
            sub_tree_size = node;
            while( depth < tree_depth )
            {
                // HUGO: Note that local_node_expectation is really
                // used as a probability, even for signed samples.
                // Sorry for the misleading choice of variable name...
                node_sample = random_gen->binomial_sample(
                    local_node_expectations(b, node + offset ) );
                if( use_signed_samples )
                    samples(b,node + offset) = 2*node_sample-1;
                else
                    samples(b,node + offset) = node_sample;
                
                // Descending in the tree
                sub_tree_size /= 2;
                if ( node_sample > 0.5 )
                    node -= sub_tree_size+1;
                else
                    node += sub_tree_size+1;
                depth++;
            }
            offset += n_nodes_per_tree;
        }    
    }
}

Here is the call graph for this function:

void PLearn::RBMWoodsLayer::getConfiguration ( int  conf_index,
Vec output 
) [virtual]

Computes the conf_index configuration of the layer.

Reimplemented from PLearn::RBMLayer.

Definition at line 1447 of file RBMWoodsLayer.cc.

References PLearn::TVec< T >::clear(), getConfigurationCount(), i, PLearn::ipow(), j, PLearn::TVec< T >::length(), n_trees, PLASSERT, PLearn::RBMLayer::size, PLearn::TVec< T >::subVec(), tree_depth, and use_signed_samples.

{
    PLASSERT( output.length() == size );
    PLASSERT( conf_index >= 0 && conf_index < getConfigurationCount() );

    int n_conf_per_tree = ipow(2,tree_depth); 
    int conf_i = conf_index;
    int begin = 0;
    int current_node, sub_tree_size, tree_conf_i;
    output.clear();
    Vec output_i;
    for ( int i = 0; i < n_trees; ++i ) {
        output_i = output.subVec( begin, n_conf_per_tree-1 );
        tree_conf_i = conf_i % n_conf_per_tree;
        // Get current tree's configuration
        output_i.clear();
        current_node = (n_conf_per_tree-1)/2;
        sub_tree_size = current_node;
        for( int j=0; j < tree_depth; j++)
        {
            if( tree_conf_i < current_node + 1 )
            {
                output_i[current_node] = 1;
                sub_tree_size /= 2;
                current_node -= sub_tree_size+1;
            }
            else
            {
                if( use_signed_samples )
                    output_i[current_node] = -1;
                sub_tree_size /= 2;
                current_node += sub_tree_size+1;
            }
        }
        conf_i /= n_conf_per_tree;
        begin += n_conf_per_tree-1;
    }
}

Here is the call graph for this function:

int PLearn::RBMWoodsLayer::getConfigurationCount ( ) [virtual]

Returns a number of different configurations the layer can be in.

Reimplemented from PLearn::RBMLayer.

Definition at line 1438 of file RBMWoodsLayer.cc.

References PLearn::RBMLayer::INFINITE_CONFIGURATIONS, PLearn::ipow(), n_trees, and tree_depth.

Referenced by getConfiguration().

{
    real ret = ipow(ipow(2.0,tree_depth),n_trees);
    if( ret > INT_MAX )
        return INFINITE_CONFIGURATIONS;
    else
        return (int) round(ret);
}

Here is the call graph for this function:

Here is the caller graph for this function:

OptionList & PLearn::RBMWoodsLayer::getOptionList ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 51 of file RBMWoodsLayer.cc.

OptionMap & PLearn::RBMWoodsLayer::getOptionMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 51 of file RBMWoodsLayer.cc.

RemoteMethodMap & PLearn::RBMWoodsLayer::getRemoteMethodMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 51 of file RBMWoodsLayer.cc.

void PLearn::RBMWoodsLayer::makeDeepCopyFromShallowCopy ( CopiesMap copies) [virtual]

Member Data Documentation

Reimplemented from PLearn::RBMLayer.

Definition at line 151 of file RBMWoodsLayer.h.

Definition at line 177 of file RBMWoodsLayer.h.

Referenced by bpropUpdate(), build_(), and makeDeepCopyFromShallowCopy().

Definition at line 164 of file RBMWoodsLayer.h.

Referenced by computeExpectations(), and makeDeepCopyFromShallowCopy().

Definition at line 174 of file RBMWoodsLayer.h.

Referenced by computeExpectations(), and makeDeepCopyFromShallowCopy().

Definition at line 173 of file RBMWoodsLayer.h.

Referenced by build_(), computeExpectation(), fprop(), and makeDeepCopyFromShallowCopy().

Definition at line 182 of file RBMWoodsLayer.h.

Referenced by bpropUpdate(), build_(), and makeDeepCopyFromShallowCopy().

Definition at line 180 of file RBMWoodsLayer.h.

Referenced by bpropUpdate(), build_(), and makeDeepCopyFromShallowCopy().

Definition at line 171 of file RBMWoodsLayer.h.

Referenced by computeExpectations(), and makeDeepCopyFromShallowCopy().

Definition at line 170 of file RBMWoodsLayer.h.

Referenced by build_(), computeExpectation(), fprop(), and makeDeepCopyFromShallowCopy().

Definition at line 181 of file RBMWoodsLayer.h.

Referenced by bpropUpdate(), build_(), and makeDeepCopyFromShallowCopy().

Definition at line 179 of file RBMWoodsLayer.h.

Referenced by bpropUpdate(), build_(), and makeDeepCopyFromShallowCopy().

Definition at line 186 of file RBMWoodsLayer.h.

Referenced by freeEnergyContribution(), and freeEnergyContributionGradient().

Definition at line 185 of file RBMWoodsLayer.h.

Referenced by freeEnergyContribution(), and freeEnergyContributionGradient().

Definition at line 188 of file RBMWoodsLayer.h.

Referenced by freeEnergyContributionGradient().

Definition at line 190 of file RBMWoodsLayer.h.

Referenced by freeEnergyContributionGradient().

Definition at line 187 of file RBMWoodsLayer.h.

Referenced by freeEnergyContributionGradient().

Definition at line 189 of file RBMWoodsLayer.h.

Referenced by freeEnergyContributionGradient().

Indication that samples should be in {-1,1}, not {0,1}, at nodes where a left/right decision is made.

Other nodes are set to 0.

Definition at line 67 of file RBMWoodsLayer.h.

Referenced by bpropUpdate(), computeExpectation(), computeExpectations(), declareOptions(), fprop(), freeEnergyContribution(), freeEnergyContributionGradient(), generateSample(), generateSamples(), and getConfiguration().


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines