PLearn 0.1
Public Member Functions | Static Public Member Functions | Public Attributes | Static Public Attributes | Static Protected Member Functions | Private Types | Private Member Functions | Private Attributes
PLearn::TestImputations Class Reference

Generate samples from a mixture of two gaussians. More...

#include <TestImputations.h>

Inheritance diagram for PLearn::TestImputations:
Inheritance graph
[legend]
Collaboration diagram for PLearn::TestImputations:
Collaboration graph
[legend]

List of all members.

Public Member Functions

 TestImputations ()
 Default constructor.
int outputsize () const
 SUBCLASS WRITING: override this so that it returns the size of this learner's output, as a function of its inputsize(), targetsize() and set options.
void train ()
 *** SUBCLASS WRITING: ***
void computeOutput (const Vec &, Vec &) const
 *** SUBCLASS WRITING: ***
void computeCostsFromOutputs (const Vec &, const Vec &, const Vec &, Vec &) const
 *** SUBCLASS WRITING: ***
TVec< string > getTestCostNames () const
 *** SUBCLASS WRITING: ***
TVec< string > getTrainCostNames () const
 *** SUBCLASS WRITING: ***
virtual string classname () const
virtual OptionListgetOptionList () const
virtual OptionMapgetOptionMap () const
virtual RemoteMethodMapgetRemoteMethodMap () const
virtual TestImputationsdeepCopy (CopiesMap &copies) const
virtual void build ()
 Finish building the object; just call inherited::build followed by build_()
virtual void makeDeepCopyFromShallowCopy (CopiesMap &copies)
 Transforms a shallow copy into a deep copy.

Static Public Member Functions

static string _classname_ ()
static OptionList_getOptionList_ ()
static RemoteMethodMap_getRemoteMethodMap_ ()
static Object_new_instance_for_typemap_ ()
static bool _isa_ (const Object *o)
static void _static_initialize_ ()
static const PPathdeclaringFile ()

Public Attributes

int min_number_of_samples
 ### declare public option fields (such as build options) here Start your comments with Doxygen-compatible comments such as //!
int max_number_of_samples
 The maximum number of samples used to test imputations for a variable.
PPath mean_median_mode_file_name
 The Path of the file with those statistics for all the variables.
PPath tree_conditional_mean_directory
 The Path of the dircetory containing the tree conditional means computed for each variable.
PPath covariance_preservation_file_name
 The Path of the file with the train_set empirically observed covariances and means.
VMat reference_set_with_covpres
 The reference set corresponding to the index computed with the ball_tree, with the initial imputations.
VMat reference_set_with_missing
 The reference set corresponding to the index computed with the ball_tree, with missing values.
TVec< string > missing_indicators
 The vector of missing indicator field names to be excluded in the distance computation.

Static Public Attributes

static StaticInitializer _static_initializer_

Static Protected Member Functions

static void declareOptions (OptionList &ol)
 Declares the class options.

Private Types

typedef PLearner inherited

Private Member Functions

void build_ ()
 This does the actual building.
void build_ball_tree (int nb_neighbors)
void initialize ()
void computeMeanMedianModeStats ()
void computeTreeCondMeanStats ()
void computeCovPresStats ()
real covariancePreservationValue (int col)
void computeNeighborhoodStats (int nb_neighbors, int max_miss_neigbors)
void createHeaderFile ()
void getHeaderRecord ()
void updateHeaderRecord (int var_col)
void createOutputFile ()
void getOutputRecord (int var_col)
void updateOutputRecord (int var_col)
void endtestimputation (const char *msg,...)

Private Attributes

ExhaustiveNearestNeighborsball_tree
Mat ref_cov
Mat ref_mis
int train_length
int train_width
Vec train_input
TVec< string > train_names
PPath train_metadata
StatsCollector train_stats
 The stats_collector responsible for collecting train cost statistics during training.
int train_total
int train_missing
int train_present
PPath header_file_name
VMat header_file
Vec header_record
int to_deal_with_total
int to_deal_with_next
string to_deal_with_name
real to_deal_with_value
VMat test_samples_set
TVec< intindices
int test_length
int test_width
real mmmf_mean_err
real mmmf_median_err
real mmmf_mode_err
PPath tcmf_file_name
VMat tcmf_file
real tcmf_mean_err
Mat cvpf_cov
Vec cvpf_mu
real cvpf_mean_err
Vec knnf_input
Vec knnf_neighbors
Vec knnf_mean_cov_err
Vec knnf_mean_miss_err
Vec knnf_nmiss_value_count
Vec weights
WeightedDistanceweighted_distance_kernel
PPath output_file_name
VMat output_file
Vec output_record
TVec< string > output_names

Detailed Description

Generate samples from a mixture of two gaussians.

Definition at line 65 of file TestImputations.h.


Member Typedef Documentation

Reimplemented from PLearn::PLearner.

Definition at line 67 of file TestImputations.h.


Constructor & Destructor Documentation

PLearn::TestImputations::TestImputations ( )

Default constructor.

Definition at line 56 of file TestImputations.cc.

{
}

Member Function Documentation

string PLearn::TestImputations::_classname_ ( ) [static]

Reimplemented from PLearn::PLearner.

Definition at line 51 of file TestImputations.cc.

OptionList & PLearn::TestImputations::_getOptionList_ ( ) [static]

Reimplemented from PLearn::PLearner.

Definition at line 51 of file TestImputations.cc.

RemoteMethodMap & PLearn::TestImputations::_getRemoteMethodMap_ ( ) [static]

Reimplemented from PLearn::PLearner.

Definition at line 51 of file TestImputations.cc.

bool PLearn::TestImputations::_isa_ ( const Object o) [static]

Reimplemented from PLearn::PLearner.

Definition at line 51 of file TestImputations.cc.

Object * PLearn::TestImputations::_new_instance_for_typemap_ ( ) [static]

Reimplemented from PLearn::Object.

Definition at line 51 of file TestImputations.cc.

StaticInitializer TestImputations::_static_initializer_ & PLearn::TestImputations::_static_initialize_ ( ) [static]

Reimplemented from PLearn::PLearner.

Definition at line 51 of file TestImputations.cc.

void PLearn::TestImputations::build ( ) [virtual]

Finish building the object; just call inherited::build followed by build_()

Reimplemented from PLearn::PLearner.

Definition at line 114 of file TestImputations.cc.

{
    // ### Nothing to add here, simply calls build_().
    inherited::build();
    build_();
}
void PLearn::TestImputations::build_ ( ) [private]

This does the actual building.

Reimplemented from PLearn::PLearner.

Definition at line 124 of file TestImputations.cc.

References PLearn::endl().

{
/*
  for each variable with missing values in the train set(which is in this case the test set)
    - radomly choose up to n samples with a value in the variable
    - build a set with these samples replacing the value with missing
    - perform the various type of imputations and compute the errors
  valider meanmedianmode, treeconditionalmean  covariancepreservation, neighborhood
  create a Mat: width is #of variables with missing values
  row 0: nb_present
  row 1: mean/mode imputation from preprocessing/final_train_input_preprocessed.pmat.metadata/mean_median_mode_file.pmat
  row 2: median/mode imputation from preprocessing/final_train_input_preprocessed.pmat.metadata/mean_median_mode_file.pmat
  row 3: mode
  row 4: treeconditionalmean imputation from prep/data/targeted_ind_no_imp.vmat.metadata/TreeCondMean/dir/'field_names'/Split0/test1_outputs.pmat
  row 5: covariance preservation imputation from preprocessing/final_train_input_preprocessed.pmat.metadata/covariance_file.pmat
  row 6 to 24: (row - 4) * i neighbors imputation from neighborhood/test_train_imputed_with_covariance_preservation.pmat.metadata/neighborhood_file.pmat
  lire le train_set
*/
    int nb_neighbors=100;
    MODULE_LOG << "build_() called" << endl;
    if (train_set)
    {
        build_ball_tree(nb_neighbors*3);
        output_file_name = train_metadata + "/TestImputation2/output.pmat";
        for (int iteration = 1; iteration <= train_set->width(); iteration++)
        {
            cout << "In TestImputations, Iteration # " << iteration << endl;
            initialize();
            computeMeanMedianModeStats();
            computeTreeCondMeanStats();
            computeCovPresStats();
            computeNeighborhoodStats(nb_neighbors,nb_neighbors*3);
            train();
        }
        endtestimputation("In TestImputations::build_(): we are done here");
    }
}

Here is the call graph for this function:

void PLearn::TestImputations::build_ball_tree ( int  nb_neighbors) [private]

Definition at line 162 of file TestImputations.cc.

References PLearn::endl(), PLERROR, and PLearn::tostring().

{ 
   // initialize primary dataset
    cout << "initialize the train set" << endl;
    train_length = train_set->length();
    train_width = train_set->width();
    train_input.resize(train_width);
    train_names.resize(train_width);
    train_names << train_set->fieldNames();
    train_metadata = train_set->getMetaDataDir();
    weights.resize(train_width);
    weights.fill(1.0);
    for (int mi_col = 0; mi_col < missing_indicators.length(); mi_col++)
    {
        int train_col;
        for (train_col = 0; train_col < train_width; train_col++)
        {
            if (missing_indicators[mi_col] != train_names[train_col]) continue;
            weights[train_col] = 0.0;
            break;
        }
        if (train_col >= train_width)
            PLERROR("In TestImputations::build_ball_tree():: no field with this name in input dataset: %s", (missing_indicators[mi_col]).c_str());
    }
    weighted_distance_kernel = new WeightedDistance(weights);
/*
    if (!reference_set_with_covpres) PLERROR("In TestImputations::build_ball_tree() no reference_set_with_covpres provided.");
    if (!reference_set_with_missing) PLERROR("In TestImputations::build_ball_tree() no reference_set_with_missing provided.");
    ball_tree = new BallTreeNearestNeighbors();
    ball_tree->setOption("rmin", "1");
    ball_tree->setOption("train_method", "anchor");
    ball_tree->setOption("num_neighbors", "100");
    ball_tree->setOption("copy_input", "0");
    ball_tree->setOption("copy_target", "0");
    ball_tree->setOption("copy_weight", "0");
    ball_tree->setOption("copy_index", "1");
    ball_tree->setOption("nstages", "-1");
    ball_tree->setOption("report_progress", "1");
    ball_tree->setTrainingSet(reference_set_with_covpres, true);
    ball_tree->train();
    ref_cov = reference_set_with_covpres->toMat();
    ref_mis = reference_set_with_missing->toMat();
*/
    if (!reference_set_with_covpres) PLERROR("In TestImputations::build_ball_tree() no reference_set_with_covpres provided.");
    if (!reference_set_with_missing) PLERROR("In TestImputations::build_ball_tree() no reference_set_with_missing provided.");
    ball_tree = new ExhaustiveNearestNeighbors();
    ball_tree->setOption("num_neighbors", tostring(nb_neighbors));
    ball_tree->setOption("copy_input", "0");
    ball_tree->setOption("copy_target", "0");
    ball_tree->setOption("copy_weight", "0");
    ball_tree->setOption("copy_index", "1");
    ball_tree->setOption("nstages", "-1");
    ball_tree->setOption("report_progress", "1");
    ball_tree->distance_kernel = weighted_distance_kernel;
    ball_tree->setTrainingSet(reference_set_with_covpres, true);
    ball_tree->train();
    ref_cov = reference_set_with_covpres->toMat();
    ref_mis = reference_set_with_missing->toMat();
/*
ExhaustiveNearestNeighbors(
# bool: Whether the kernel defined by the 'distance_kernel' option should be
# interpreted as a (pseudo-)distance measure (true) or a similarity
# measure (false). Default = true.  Note that this interpretation is
# strictly specific to the class ExhaustiveNearestNeighbors.
kernel_is_pseudo_distance = 1  ;

# Ker: Alternate name for 'distance_kernel'.  (Deprecated; use only so that
# existing scripts can run.)
kernel = *1 ->DistanceKernel(
n = 2 ;
pow_distance = 0 ;
optimized = 0 ;
is_symmetric = 1 ;
report_progress = 0 ;
specify_dataset = *0 ;
cache_gram_matrix = 0 ;
data_inputsize = -1 ;
n_examples = -1  )
 ;

# int: Number of nearest-neighbors to compute.  This is usually called "K".
# The output vector is simply the concatenation of all found neighbors.
# (Default = 1)
num_neighbors = 1  ;

# bool: If true, the output contains a copy of the found input vector(s).
# (Default = false)
copy_input = 0  ;

# bool: If true, the output contains a copy of the found target vector(s).
# (Default = true)
copy_target = 1  ;

# bool: If true, the output contains a copy of the found weight.  If no
# weight is present in the training set, a weight of 1.0 is put.
# (Default = true)
copy_weight = 0  ;

# bool: If true, the output contains the index of the found neighbor
# (as the row number, zero-based, in the training set.)
# (Default = false)
copy_index = 0  ;

# Ker: An optional alternative to the Euclidean distance (DistanceKernel with
# n=2 and pow_distance=1).  It should be a 'distance-like' kernel rather
# than a 'dot-product-like' kernel, i.e. small when the arguments are
# similar, and it should always be non-negative, and 0 only if arguments
# are equal.
distance_kernel = *1 ->DistanceKernel(
n = 2 ;
pow_distance = 0 ;
optimized = 0 ;
is_symmetric = 1 ;
report_progress = 0 ;
specify_dataset = *0 ;
cache_gram_matrix = 0 ;
data_inputsize = -1 ;
n_examples = -1  )
 ;
*/

}

Here is the call graph for this function:

string PLearn::TestImputations::classname ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 51 of file TestImputations.cc.

void PLearn::TestImputations::computeCostsFromOutputs ( const Vec input,
const Vec output,
const Vec target,
Vec costs 
) const [virtual]

*** SUBCLASS WRITING: ***

This should be defined in subclasses to compute the weighted costs from already computed output. The costs should correspond to the cost names returned by getTestCostNames().

NOTE: In exotic cases, the cost may also depend on some info in the input, that's why the method also gets so see it.

Implements PLearn::PLearner.

Definition at line 645 of file TestImputations.cc.

{}
void PLearn::TestImputations::computeCovPresStats ( ) [private]

Definition at line 434 of file TestImputations.cc.

References PLearn::isfile(), PLearn::VMat::length(), PLERROR, PLearn::pow(), PLearn::ProgressBar::update(), and PLearn::VMat::width().

{
    if (!isfile(covariance_preservation_file_name)) PLERROR("In TestImputations::computeCovPresStats() a valid covariance_preservation_file path must be provided.");
    VMat cvpf_file = new FileVMatrix(covariance_preservation_file_name);
    int cvpf_length = cvpf_file->length();
    int cvpf_width = cvpf_file->width();
    if (cvpf_length != train_width + 1)
        PLERROR("In TestImputations::computeCovPresStats() there should be %i records in the cvp file, got %i.", train_width + 1, cvpf_length);
    if (cvpf_width != train_width)
        PLERROR("In TestImputations::computeCovPresStats() train set and cvp width should be the same, got %i.", cvpf_width);
    //cvpf_file = new FileVMatrix(covariance_preservation_file_name);
    cvpf_cov.resize(train_width, train_width);
    cvpf_mu.resize(train_width);
    for (int cvpf_row = 0; cvpf_row < train_width; cvpf_row++)
    {
        for (int cvpf_col = 0; cvpf_col < train_width; cvpf_col++)
        {
            cvpf_cov(cvpf_row, cvpf_col) = cvpf_file->get(cvpf_row, cvpf_col);
        }
    }
    for (int cvpf_col = 0; cvpf_col < train_width; cvpf_col++)
    {
        cvpf_mu[cvpf_col] = cvpf_file->get(train_width, cvpf_col);
    }
    cvpf_mean_err = 0.0;
    ProgressBar* pb = new ProgressBar( "computing the covariance preservation imputation errors for " + to_deal_with_name, test_length);
    for (int test_row = 0; test_row < test_length; test_row++)
    {
        test_samples_set->getRow(test_row, train_input);
        cvpf_mean_err += pow(to_deal_with_value - covariancePreservationValue(to_deal_with_next), 2);
        pb->update( test_row );
    }
    delete pb;
    cvpf_mean_err = cvpf_mean_err / (real) test_length;
    //TODO check the formul
    //cvpf_mean_stddev = sqrt(cvpf_mean_err);

}

Here is the call graph for this function:

void PLearn::TestImputations::computeMeanMedianModeStats ( ) [private]

Definition at line 377 of file TestImputations.cc.

References PLearn::isfile(), PLearn::VMat::length(), PLERROR, PLearn::pow(), PLearn::ProgressBar::update(), and PLearn::VMat::width().

{
    if (!isfile(mean_median_mode_file_name)) PLERROR("In TestImputations::computeMeanMedianModeStats() a valid mean_median_mode_file path must be provided.");
    VMat mmmf_file = new FileVMatrix(mean_median_mode_file_name);
    int mmmf_length = mmmf_file->length();
    int mmmf_width = mmmf_file->width();
    if (mmmf_length != 3) PLERROR("In TestImputations::computeMeanMedianModeStats() there should be exactly 3 records in the mmm file, got %i.", mmmf_length);
    if (mmmf_width != train_width) PLERROR("In TestImputations::computeMeanMedianModeStats() train set and mmm width should be the same, got %i.", mmmf_width);
    real mmmf_mean = mmmf_file->get(0, to_deal_with_next);
    real mmmf_median = mmmf_file->get(1, to_deal_with_next);
    real mmmf_mode = mmmf_file->get(2, to_deal_with_next);
    mmmf_mean_err = 0.0;
    mmmf_median_err = 0.0;
    mmmf_mode_err = 0.0;
    ProgressBar* pb = new ProgressBar( "computing the mean, median and mode imputation errors for " + to_deal_with_name, test_length);
    for (int test_row = 0; test_row < test_length; test_row++)
    {
        to_deal_with_value = test_samples_set->get(test_row, to_deal_with_next);
        mmmf_mean_err += pow(to_deal_with_value - mmmf_mean, 2);
        mmmf_median_err += pow(to_deal_with_value - mmmf_median, 2);
        mmmf_mode_err += pow(to_deal_with_value - mmmf_mode, 2);
        pb->update( test_row );
    }
    delete pb;
    mmmf_mean_err = mmmf_mean_err / (real) test_length;
    mmmf_median_err = mmmf_median_err / (real) test_length;
    mmmf_mode_err = mmmf_mode_err / (real) test_length;
    //TODO check the formul
    //mmmf_mean_stddev = sqrt(mmmf_mean_err);
    //mmmf_median_stddev = sqrt(mmmf_median_err);
    //mmmf_mode_stddev = sqrt(mmmf_mode_err);

}

Here is the call graph for this function:

void PLearn::TestImputations::computeNeighborhoodStats ( int  nb_neighbors,
int  max_miss_neigbors 
) [private]

Definition at line 492 of file TestImputations.cc.

References PLearn::is_missing(), PLERROR, PLearn::pow(), and PLearn::ProgressBar::update().

{
    knnf_input.resize(train_width);
    knnf_neighbors.resize(nb_neighbors+max_miss_neigbors);
    knnf_mean_cov_err.resize(nb_neighbors);
    knnf_mean_miss_err.resize(nb_neighbors);
    knnf_nmiss_value_count.resize(nb_neighbors);
    knnf_mean_cov_err.clear();
    knnf_mean_miss_err.clear();
    knnf_nmiss_value_count.clear();
    ProgressBar* pb = new ProgressBar( "computing the neighborhood imputation errors for " + to_deal_with_name, test_length);
    for (int test_row = 0; test_row < test_length; test_row++)
    {
        test_samples_set->getRow(test_row, train_input);
        for (int test_col = 0; test_col < train_width; test_col++)
        {
            if (test_col == to_deal_with_next) knnf_input[test_col] = covariancePreservationValue(test_col);
            else if (is_missing(train_input[test_col])) knnf_input[test_col] = covariancePreservationValue(test_col);
            else knnf_input[test_col] = train_input[test_col];
        }
        ball_tree->computeOutput(knnf_input, knnf_neighbors);
        real knnf_sum_cov_value = 0.0;
        real knnf_sum_miss_value = 0.0;
        int  knnv_value_count = 0;
        for (int knnf_row = 0; knnf_row < knnf_neighbors.size() && knnv_value_count<nb_neighbors; knnf_row++)
        {
            real knnf_value = ref_mis((int) knnf_neighbors[knnf_row], to_deal_with_next);
            if(!is_missing(knnf_value))
            {
                knnf_sum_miss_value += knnf_value;
                knnf_nmiss_value_count[knnv_value_count]+=1;
                knnf_mean_miss_err[knnv_value_count] += pow(to_deal_with_value - (knnf_sum_miss_value / (knnv_value_count+1)), 2);
                knnv_value_count += 1;
            }
            if (!is_missing(knnf_value) && knnf_row<nb_neighbors)
            {
                knnf_sum_cov_value += knnf_value;
                knnf_mean_cov_err[knnf_row] += pow(to_deal_with_value - (knnf_sum_cov_value / (knnf_row+1)), 2);
            }else if(knnf_row<nb_neighbors){
                knnf_value = ref_cov((int) knnf_neighbors[knnf_row], to_deal_with_next);
                if (is_missing(knnf_value))
                    PLERROR("In TestImputations::computeNeighborhoodStats(): missing value found in the reference with covariance preserved at: %i , %i",
                            (int) knnf_neighbors[knnf_row], to_deal_with_next);
                knnf_sum_cov_value += knnf_value;
                knnf_mean_cov_err[knnf_row] += pow(to_deal_with_value - (knnf_sum_cov_value / (knnf_row+1)), 2);
            }
        }
        pb->update( test_row );  
    }
    delete pb;
    knnf_mean_cov_err/=test_length;
    knnf_mean_miss_err/=knnf_nmiss_value_count;
}

Here is the call graph for this function:

void PLearn::TestImputations::computeOutput ( const Vec input,
Vec output 
) const [virtual]

*** SUBCLASS WRITING: ***

This should be defined in subclasses to compute the output from the input.

Reimplemented from PLearn::PLearner.

Definition at line 644 of file TestImputations.cc.

{}
void PLearn::TestImputations::computeTreeCondMeanStats ( ) [private]

Definition at line 411 of file TestImputations.cc.

References PLearn::isfile(), PLERROR, PLearn::pow(), and PLearn::ProgressBar::update().

{
    tcmf_file_name = tree_conditional_mean_directory + "/" + to_deal_with_name + "/Split0/test1_outputs.pmat";
    if (!isfile(tcmf_file_name)) 
        PLERROR("In TestImputations::computeTreeCondMeanStats(): The '%s' file was not found in the tcf directory.",tcmf_file_name.c_str());
    tcmf_file = new FileVMatrix(tcmf_file_name);
    int tcmf_length = tcmf_file->length();
    if (tcmf_length < train_length) 
        PLERROR("In TestImputations::computeTreeCondMeanStats(): there are only %d records in the tree conditional output file. We need %d.",tcmf_length,train_length);
    tcmf_mean_err = 0.0;
    ProgressBar* pb = new ProgressBar( "computing the tree conditional mean imputation errors for " + to_deal_with_name, test_length);
    for (int test_row = 0; test_row < test_length; test_row++)
    {
        to_deal_with_value = test_samples_set->get(test_row, to_deal_with_next);
        tcmf_mean_err += pow(to_deal_with_value - tcmf_file->get(indices[test_row], 0), 2);
        pb->update( test_row );
    }
    delete pb;
    tcmf_mean_err = tcmf_mean_err / (real) test_length;
    //TODO check the formul
    //tcmf_mean_stddev = sqrt(tcmf_mean_err);
}

Here is the call graph for this function:

real PLearn::TestImputations::covariancePreservationValue ( int  col) [private]

Definition at line 473 of file TestImputations.cc.

References PLearn::is_missing().

{
    real cvpf_sum_cov_xl = 0;
    real cvpf_sum_xl_square = 0;
    for (int cvpf_col = 0; cvpf_col < train_width; cvpf_col++)
    {
        if (cvpf_col == col) continue;
        if (is_missing(train_input[cvpf_col])) continue;
        cvpf_sum_cov_xl += cvpf_cov(cvpf_col, col) * (train_input[cvpf_col] - cvpf_mu[cvpf_col]);
        cvpf_sum_xl_square += (train_input[cvpf_col] - cvpf_mu[cvpf_col]) * (train_input[cvpf_col] - cvpf_mu[cvpf_col]);
    }
    real cvpf_value;
    if (cvpf_sum_xl_square == 0.0) cvpf_value = cvpf_mu[col];
    else cvpf_value = cvpf_mu[col] + cvpf_sum_cov_xl / cvpf_sum_xl_square;
    return cvpf_value;
}

Here is the call graph for this function:

void PLearn::TestImputations::createHeaderFile ( ) [private]

Definition at line 546 of file TestImputations.cc.

References PLearn::endl(), and PLERROR.

{ 
    cout << "in createHeaderFile()" << endl;
    for (int train_col = 0; train_col < train_width; train_col++)
    {
        train_stats = train_set->getStats(train_col);
        train_total = (int)train_stats.n();
        train_missing = (int)train_stats.nmissing();
        train_present = train_total - train_missing;
        if (train_missing <= 0.0) header_record[train_col] = 0.0;                       // no missing, noting to do.
        else if (train_present < min_number_of_samples){
            header_record[train_col] = -1.0; // should not happen
            PLERROR("In TestImputations::createHeaderFile: train_present(%d) < min_number_of_samples (%d) for variable %d(%s)",
                    train_present,min_number_of_samples,train_col,train_set.fieldName(train_col).c_str());
        }
        else header_record[train_col] = 1.0;                                            // test imputations
    }
    header_file = new FileVMatrix(header_file_name, 1, train_names);
    header_file->putRow(0, header_record);
}

Here is the call graph for this function:

void PLearn::TestImputations::createOutputFile ( ) [private]

Definition at line 601 of file TestImputations.cc.

References PLearn::tostring().

{
    output_names.resize(6,knnf_mean_cov_err.size()+knnf_mean_miss_err.size()
                        + knnf_nmiss_value_count.size());
    output_names[0] = "test_length";
    output_names[1] = "mean";
    output_names[2] = "median";
    output_names[3] = "mode";
    output_names[4] = "tree_cond";
    output_names[5] = "cov_pres";
    for (int knnf_row = 0; knnf_row < knnf_mean_cov_err.size(); knnf_row++)
    {
        output_names.append("KNN_COV_" + tostring(knnf_row+1));
    }
    for (int knnf_row = 0; knnf_row < knnf_mean_cov_err.size(); knnf_row++)
    {
        output_names.append("KNN_MISS_" + tostring(knnf_row+1));
    }
    for (int knnf_row = 0; knnf_row < knnf_nmiss_value_count.size(); knnf_row++)
    {
        output_names.append("KNN_NB_MISS_" + tostring(knnf_row+1));
    }

    output_record.clear();
    output_file = new FileVMatrix(output_file_name, train_width, output_names);
    for (int train_col = 0; train_col < train_width; train_col++)
        output_file->putRow(train_col, output_record);
}

Here is the call graph for this function:

void PLearn::TestImputations::declareOptions ( OptionList ol) [static, protected]

Declares the class options.

Reimplemented from PLearn::PLearner.

Definition at line 63 of file TestImputations.cc.

References PLearn::OptionBase::buildoption, covariance_preservation_file_name, PLearn::declareOption(), max_number_of_samples, mean_median_mode_file_name, min_number_of_samples, missing_indicators, reference_set_with_covpres, reference_set_with_missing, and tree_conditional_mean_directory.

{

    declareOption(ol, "min_number_of_samples", &TestImputations::min_number_of_samples,
                  OptionBase::buildoption,
                  "The minimum number of samples required to test imputations for a variable.");
    declareOption(ol, "max_number_of_samples", &TestImputations::max_number_of_samples,
                  OptionBase::buildoption,
                  "The maximum number of samples used to test imputations for a variable.");
    declareOption(ol, "mean_median_mode_file_name", &TestImputations::mean_median_mode_file_name,
                  OptionBase::buildoption,
                  "The Path of the file with those statistics for all the variables.");
    declareOption(ol, "tree_conditional_mean_directory", &TestImputations::tree_conditional_mean_directory,
                  OptionBase::buildoption,
                  "The Path of the dircetory containing the tree conditional means computed for each variable.");
    declareOption(ol, "covariance_preservation_file_name", &TestImputations::covariance_preservation_file_name,
                  OptionBase::buildoption,
                  "The Path of the file with the train_set empirically observed covariances and means.");
    declareOption(ol, "reference_set_with_covpres", &TestImputations::reference_set_with_covpres,
                  OptionBase::buildoption,
                  "The reference set corresponding to the index computed with the ball_tree, with the initial imputations.");
    declareOption(ol, "reference_set_with_missing", &TestImputations::reference_set_with_missing,
                  OptionBase::buildoption,
                  "The reference set corresponding to the index computed with the ball_tree, with missing values.");
    declareOption(ol, "missing_indicators", &TestImputations::missing_indicators,
                  OptionBase::buildoption,
                  "The vector of missing indicator field names to be excluded in the distance computation.");

    inherited::declareOptions(ol);
}

Here is the call graph for this function:

static const PPath& PLearn::TestImputations::declaringFile ( ) [inline, static]

Reimplemented from PLearn::PLearner.

Definition at line 113 of file TestImputations.h.

:
    //#####  Protected Member Functions  ######################################
TestImputations * PLearn::TestImputations::deepCopy ( CopiesMap copies) const [virtual]

Reimplemented from PLearn::PLearner.

Definition at line 51 of file TestImputations.cc.

void PLearn::TestImputations::endtestimputation ( const char *  msg,
  ... 
) [private]

Definition at line 284 of file TestImputations.cc.

References PLERROR, and PLWARNING.

                                                           {
    va_list args;
    va_start(args,msg);
    getHeaderRecord();
    for (int train_col = 0; train_col < train_width; train_col++)
    {
        if (header_record[train_col] == 1.0) 
            PLWARNING("No all variable done!!!");
        else if (header_record[train_col] == 2.0){
            getOutputRecord(train_col);
            if(output_record[100]==0.0)
                PLWARNING("Element %d,%d is at zero in the output file. Meaby this variable was not treated",train_col,100);
        }
    }
    PLERROR(msg,args);
}
void PLearn::TestImputations::getHeaderRecord ( ) [private]

Definition at line 567 of file TestImputations.cc.

{ 
    header_file = new FileVMatrix(header_file_name, true);
    header_file->getRow(0, header_record);
}
OptionList & PLearn::TestImputations::getOptionList ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 51 of file TestImputations.cc.

OptionMap & PLearn::TestImputations::getOptionMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 51 of file TestImputations.cc.

void PLearn::TestImputations::getOutputRecord ( int  var_col) [private]

Definition at line 630 of file TestImputations.cc.

{ 
    output_file = new FileVMatrix(output_file_name, true);
    output_record.resize(output_file->width());
    output_file->getRow(var_col, output_record);
}
RemoteMethodMap & PLearn::TestImputations::getRemoteMethodMap ( ) const [virtual]

Reimplemented from PLearn::Object.

Definition at line 51 of file TestImputations.cc.

TVec< string > PLearn::TestImputations::getTestCostNames ( ) const [virtual]

*** SUBCLASS WRITING: ***

This should return the names of the costs computed by computeCostsFromOutputs.

Implements PLearn::PLearner.

Definition at line 646 of file TestImputations.cc.

References PLearn::TVec< T >::append().

{
    TVec<string> result;
    result.append( "MSE" );
    return result;
}

Here is the call graph for this function:

TVec< string > PLearn::TestImputations::getTrainCostNames ( ) const [virtual]

*** SUBCLASS WRITING: ***

This should return the names of the objective costs that the train method computes and for which it updates the VecStatsCollector train_stats.

Implements PLearn::PLearner.

Definition at line 652 of file TestImputations.cc.

References PLearn::TVec< T >::append().

{
    TVec<string> result;
    result.append( "MSE" );
    return result;
}

Here is the call graph for this function:

void PLearn::TestImputations::initialize ( ) [private]

Definition at line 300 of file TestImputations.cc.

References PLearn::endl(), PLearn::is_missing(), PLearn::isfile(), PLearn::manual_seed(), PLERROR, PLearn::shuffleElements(), and PLearn::ProgressBar::update().

{
    
    // initialize the header file
    cout << "initialize the header file" << endl;
    train_set->lockMetaDataDir();
    header_record.resize(train_width);
    header_file_name = train_metadata + "/TestImputation2/header.pmat";
    cout << "header_file_name: " << header_file_name << endl;
    if (!isfile(header_file_name)) createHeaderFile();
    else getHeaderRecord();
    
    // choose a variable to test imputations for
    cout << "choose a variable to test imputations for" << endl;
    to_deal_with_total = 0;
    to_deal_with_next = -1;

    for (int train_col = 0; train_col < train_width; train_col++)
    {
        if (header_record[train_col] != 1.0) continue;
        to_deal_with_total += 1;
        if (to_deal_with_next < 0) to_deal_with_next = train_col;
    }
    cout << "total number of variable left to deal with: " << to_deal_with_total << endl;
    if (to_deal_with_next < 0)
    {
        train_set->unlockMetaDataDir();
        // reviewGlobalStats();
        endtestimputation("In TestImputations::initialize() we are done here");
    }
    cout << "next variable to deal with: " << train_names[to_deal_with_next] << "("<<to_deal_with_next<<")"<<endl;
    to_deal_with_name = train_names[to_deal_with_next];
    updateHeaderRecord(to_deal_with_next);
    train_set->unlockMetaDataDir();
    
    // find the available samples with non-missing values for this variable
    train_stats = train_set->getStats(to_deal_with_next);
    train_total = (int)train_stats.n();
    train_missing = (int)train_stats.nmissing();
    train_present = train_total - train_missing;
    indices.resize((int) train_present);
    int ind_next = 0;
    ProgressBar* pb = new ProgressBar( "Building the indices for " + to_deal_with_name, train_length);
    for (int train_row = 0; train_row < train_length; train_row++)
    {
        to_deal_with_value = train_set->get(train_row, to_deal_with_next);
        if (is_missing(to_deal_with_value)) continue;
        if (ind_next >= indices.length()) 
            PLERROR("In TestImputations::initialize() There seems to be more present values than indicated by the stats file");
        indices[ind_next] = train_row;
        ind_next += 1;
        pb->update( train_row );
    }
    delete pb;
    
    // shuffle the indices.
    manual_seed(123456);
    shuffleElements(indices);
    
    // load the test samples for this variable
    if (indices.length() > max_number_of_samples) test_length = max_number_of_samples;
    else if (indices.length() < min_number_of_samples)
        PLERROR("TestImputations::initialize() Their is less examples(%d) for the variable %s then the min_number_of semples(%d)",
                indices.length(),to_deal_with_name.c_str(),min_number_of_samples);
    else test_length = indices.length();
    test_width = train_width;
    test_samples_set = new MemoryVMatrix(test_length, test_width);
    pb = new ProgressBar( "Loading the test samples for " + to_deal_with_name, test_length);
    for (int test_row = 0; test_row < test_length; test_row++)
    {
        train_set->getRow(indices[test_row], train_input);
        test_samples_set->putRow(test_row, train_input);
        pb->update( test_row );
    }
    delete pb;
}

Here is the call graph for this function:

void PLearn::TestImputations::makeDeepCopyFromShallowCopy ( CopiesMap copies) [virtual]
int PLearn::TestImputations::outputsize ( ) const [virtual]

SUBCLASS WRITING: override this so that it returns the size of this learner's output, as a function of its inputsize(), targetsize() and set options.

Implements PLearn::PLearner.

Definition at line 643 of file TestImputations.cc.

{return 0;}
void PLearn::TestImputations::train ( ) [virtual]

*** SUBCLASS WRITING: ***

The role of the train method is to bring the learner up to stage==nstages, updating the stats with training costs measured on-line in the process.

TYPICAL CODE:

  static Vec input;  // static so we don't reallocate/deallocate memory each time...
  static Vec target; // (but be careful that static means shared!)
  input.resize(inputsize());    // the train_set's inputsize()
  target.resize(targetsize());  // the train_set's targetsize()
  real weight;
  
  if(!train_stats)   // make a default stats collector, in case there's none
      train_stats = new VecStatsCollector();
  
  if(nstages<stage)  // asking to revert to a previous stage!
      forget();      // reset the learner to stage=0
  
  while(stage<nstages)
  {
      // clear statistics of previous epoch
      train_stats->forget(); 
            
      //... train for 1 stage, and update train_stats,
      // using train_set->getSample(input, target, weight);
      // and train_stats->update(train_costs)
          
      ++stage;
      train_stats->finalize(); // finalize statistics for this epoch
  }

Implements PLearn::PLearner.

Definition at line 579 of file TestImputations.cc.

References PLearn::endl(), and PLearn::isfile().

Here is the call graph for this function:

void PLearn::TestImputations::updateHeaderRecord ( int  var_col) [private]

Definition at line 573 of file TestImputations.cc.

{ 
    header_file->put(0, var_col, 2.0);
    header_file->flush();
}
void PLearn::TestImputations::updateOutputRecord ( int  var_col) [private]

Definition at line 637 of file TestImputations.cc.

{ 
    output_file->putRow(var_col, output_record);
    output_file->flush();
}

Member Data Documentation

Reimplemented from PLearn::PLearner.

Definition at line 113 of file TestImputations.h.

Definition at line 151 of file TestImputations.h.

The Path of the file with the train_set empirically observed covariances and means.

Definition at line 85 of file TestImputations.h.

Referenced by declareOptions().

Definition at line 180 of file TestImputations.h.

Definition at line 182 of file TestImputations.h.

Definition at line 181 of file TestImputations.h.

Definition at line 164 of file TestImputations.h.

Definition at line 163 of file TestImputations.h.

Definition at line 165 of file TestImputations.h.

Definition at line 171 of file TestImputations.h.

Definition at line 183 of file TestImputations.h.

Definition at line 185 of file TestImputations.h.

Definition at line 186 of file TestImputations.h.

Definition at line 184 of file TestImputations.h.

Definition at line 187 of file TestImputations.h.

The maximum number of samples used to test imputations for a variable.

Definition at line 79 of file TestImputations.h.

Referenced by declareOptions().

The Path of the file with those statistics for all the variables.

Definition at line 81 of file TestImputations.h.

Referenced by declareOptions().

### declare public option fields (such as build options) here Start your comments with Doxygen-compatible comments such as //!

The minimum number of samples required to test imputations for a variable.

Definition at line 77 of file TestImputations.h.

Referenced by declareOptions().

The vector of missing indicator field names to be excluded in the distance computation.

Definition at line 91 of file TestImputations.h.

Referenced by declareOptions().

Definition at line 174 of file TestImputations.h.

Definition at line 175 of file TestImputations.h.

Definition at line 176 of file TestImputations.h.

Definition at line 191 of file TestImputations.h.

Definition at line 190 of file TestImputations.h.

Definition at line 193 of file TestImputations.h.

Definition at line 192 of file TestImputations.h.

Definition at line 152 of file TestImputations.h.

Definition at line 153 of file TestImputations.h.

The reference set corresponding to the index computed with the ball_tree, with the initial imputations.

Definition at line 87 of file TestImputations.h.

Referenced by declareOptions().

The reference set corresponding to the index computed with the ball_tree, with missing values.

Definition at line 89 of file TestImputations.h.

Referenced by declareOptions().

Definition at line 178 of file TestImputations.h.

Definition at line 177 of file TestImputations.h.

Definition at line 179 of file TestImputations.h.

Definition at line 172 of file TestImputations.h.

Definition at line 170 of file TestImputations.h.

Definition at line 173 of file TestImputations.h.

Definition at line 168 of file TestImputations.h.

Definition at line 167 of file TestImputations.h.

Definition at line 166 of file TestImputations.h.

Definition at line 169 of file TestImputations.h.

Definition at line 156 of file TestImputations.h.

Definition at line 154 of file TestImputations.h.

Definition at line 158 of file TestImputations.h.

Definition at line 161 of file TestImputations.h.

Definition at line 157 of file TestImputations.h.

Definition at line 162 of file TestImputations.h.

The stats_collector responsible for collecting train cost statistics during training.

This is typically set by some external training harness that wants to collect some stats.

Reimplemented from PLearn::PLearner.

Definition at line 159 of file TestImputations.h.

Definition at line 160 of file TestImputations.h.

Definition at line 155 of file TestImputations.h.

The Path of the dircetory containing the tree conditional means computed for each variable.

Definition at line 83 of file TestImputations.h.

Referenced by declareOptions().

Definition at line 189 of file TestImputations.h.

Definition at line 188 of file TestImputations.h.


The documentation for this class was generated from the following files:
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Defines