PLearn 0.1
|
#include <SDBVMat.h>
This class is specialized for processing an ICBC database with the following fields: "policy_start_date"; "policy_end_date"; "bodily_injury_incurred"; "property_damage_incurred"; "accident_death_incurred"; "collision_lou_incurred"; "comprehensive_incurred"; "roadstar_incurred";
The output has at least 1 value and at most 8 values (number of outputs = targetname=="ALL"?5:1 + int(use_roadstar) + int(add_claims_sum_column) + int(rescale_by_interval)). targetname can be "ALL" or one of the 5 other targets (excluding "roadstar_incurred"), or sum_all_but_BI. There are targetname=="ALL"?5:1 + int(use_roadstar) "*_incurred" targets + possibly one target corresponding to the sum of all the "*_incurred" targets (if add_claims_sum_column==true) and possibly one "weight" if rescale_by_interval==true. If targetname=="sum_all_but_BI" then there is only one target, which is the sum of all 4 KOLs excluding Bodily Injury.
The weight is computed as follows: duration = Year(policy_end_date)-Year(policy_start_date) weight = duration^2
The 5 + int(use_roadstar) targets are computed as follows: _incurred_target = 0.001 * *_incurred; if (rescale_by_interval) _incurred_target /= duration if (rescale_by_start_date) _incurred_target /= smoothed *_incurred monthly mean for month m and year y where m and y are the month and year of the policy start date of the contract. (PS: for more information on the smoothed monthly means, read the comments in PLearn/Databases/ICBCdatabases.h) if is_missing(X_incurred_target) then X_incurred_target = 0
where Year is the date given in units of years (with fractional part), so that in the common case where the difference is one year, the weight is 1 and there is no normalization of the targets. Note that the target outputs are in UNITS OF 1000 DOLLARS PER YEAR.
This "weight" is the weight that should be given to the squared loss (f(x)-normalized_target)^2. This comes from the following reasoning:
We are to learn a function f(x) that predicts the incurred amounts for ONE YEAR, but the (x,target) pairs in the data may cover more or less than one year. So the loss should be (f(x)*duration - target)^2 where duration is in units of years. To make it more "standard" we instead consider the weighted squared loss, since (f(x)*duration - target)^2 = duration^2 * (f(x) - target/duration)^2 = w * (f(x) - normalized_target)^2
When add_claims_sum_column==true sum of targets = sum of all the *_incurred targets after they have been rescaled
PS: * Year is the date given in units of years (with fractional part), so that in the common case where the difference is one year, the weight is 1 and there is no normalization of the targets. Note that the target outputs are in UNITS OF 1000 DOLLARS PER YEAR. When rescale_by_interval==true, the "weight" is the weight that should be given to the squared loss (f(x)-normalized_target)^2. This comes from the following reasoning: We are to learn a function f(x) that predicts the incurred amounts for ONE YEAR, but the (x,target) pairs in the data may cover more or less than one year. So the loss should be (f(x)*duration - target)^2 where duration is in units of years. To make it more "standard" we instead consider the weighted squared loss, since (f(x)*duration - target)^2 = duration^2 * (f(x) - target/duration)^2 = w * (f(x) - normalized_target)^2
typedef SDBVMField PLearn::SDBVMFieldICBCTargets::inherited [private] |
PLearn::SDBVMFieldICBCTargets::SDBVMFieldICBCTargets | ( | Schema | schema, |
bool | use_roadstar, | ||
bool | add_claims_sum_column, | ||
bool | rescale_by_interval, | ||
bool | rescale_by_start_date, | ||
Mat & | start_date_rescaling_values, | ||
const string & | targetname = "ALL" |
||
) |
Definition at line 953 of file SDBVMat.cc.
References reference_start_date_month_, reference_start_date_year_, rescale_by_start_date_, and start_date_rescaling_values_.
: inherited(0,VMField::Continuous), use_roadstar_(use_roadstar), add_claims_sum_column_(add_claims_sum_column), rescale_by_interval_(rescale_by_interval), rescale_by_start_date_(rescale_by_start_date), start_date_rescaling_values_(start_date_rescaling_values), targetname_(targetname), start_date_(schema("policy_start_date")), end_date_(schema("policy_end_date")), bodily_injury_incurred_((targetname_=="ALL" || targetname_=="bodily_injury_incurred") ? schema("bodily_injury_incurred") : FieldPtr()), property_damage_incurred_((targetname_=="ALL" || targetname_=="sum_all_but_BI" || targetname_=="property_damage_incurred") ? schema("property_damage_incurred") : FieldPtr()), accident_death_incurred_((targetname_=="ALL" || targetname_=="sum_all_but_BI" || targetname_=="accident_death_incurred") ? schema("accident_death_incurred") : FieldPtr()), collision_lou_incurred_((targetname_=="ALL" || targetname_=="sum_all_but_BI" || targetname_=="collision_lou_incurred") ? schema("collision_lou_incurred") : FieldPtr()), comprehensive_incurred_((targetname_=="ALL" || targetname_=="sum_all_but_BI" || targetname_=="comprehensive_incurred") ? schema("comprehensive_incurred") : FieldPtr()), bodily_injury_count_((targetname_=="ALLcounts" || targetname_=="bodily_injury_count") ? schema("bodily_injury_count") : FieldPtr()), property_damage_count_((targetname_=="ALLcounts" || targetname_=="all_counts_but_BI" || targetname_=="property_damage_count") ? schema("property_damage_count") : FieldPtr()), accident_death_count_((targetname_=="ALLcounts" || targetname_=="all_counts_but_BI" || targetname_=="accident_death_count") ? schema("accident_death_count") : FieldPtr()), collision_lou_count_((targetname_=="ALLcounts" || targetname_=="all_counts_but_BI" || targetname_=="collision_lou_count") ? schema("collision_lou_count") : FieldPtr()), comprehensive_count_((targetname_=="ALLcounts" || targetname_=="all_counts_but_BI" || targetname_=="comprehensive_count") ? schema("comprehensive_count") : FieldPtr()), bodily_injury_severity_((targetname_=="ALLseverities" || targetname_=="bodily_injury_severity") ? schema("bodily_injury_severity") : FieldPtr()), property_damage_severity_((targetname_=="ALLseverities" || targetname_=="all_severities_but_BI" || targetname_=="property_damage_severity") ? schema("property_damage_severity") : FieldPtr()), accident_death_severity_((targetname_=="ALLseverities" || targetname_=="all_severities_but_BI" || targetname_=="accident_death_severity") ? schema("accident_death_severity") : FieldPtr()), collision_lou_severity_((targetname_=="ALLseverities" || targetname_=="all_severities_but_BI" || targetname_=="collision_lou_severity") ? schema("collision_lou_severity") : FieldPtr()), comprehensive_severity_((targetname_=="ALLseverities" || targetname_=="all_severities_but_BI" || targetname_=="comprehensive_severity") ? schema("comprehensive_severity") : FieldPtr()) // roadstar_incurred_((use_roadstar && (targetname_ == "ALL" || targetname == "roadstar_incurred")) ? // schema("roadstar_incurred") : FieldPtr()) { reference_start_date_year_ = rescale_by_start_date_ ? (int) start_date_rescaling_values_[0][0] : 0; reference_start_date_month_ = rescale_by_start_date_ ? (int) start_date_rescaling_values_[0][1] : 0; }
void PLearn::SDBVMFieldICBCTargets::convertField | ( | const SDBWithStats & | sdb, |
const Row & | theRow, | ||
const Vec & | outputField | ||
) | const [virtual] |
Given a database row, convert the appropriate parts to a (preallocated) output vector of the correct width (given by fieldWidth). Replace MISSING_VALUEs by missing_values_mapping.
Implements PLearn::SDBVMField.
Definition at line 1014 of file SDBVMat.cc.
References accident_death_count_, accident_death_incurred_, add_claims_sum_column_, bodily_injury_count_, bodily_injury_incurred_, collision_lou_count_, collision_lou_incurred_, comprehensive_count_, comprehensive_incurred_, PLearn::SDBVMField::convertMissing(), end_date_, PLearn::endl(), PLearn::SDBVMSource::getValue(), i, if(), PLearn::is_missing(), PLearn::PDate::month, n, PLERROR, property_damage_count_, property_damage_incurred_, reference_start_date_month_, reference_start_date_year_, rescale_by_interval_, rescale_by_start_date_, start_date_, start_date_rescaling_values_, targetname_, PLearn::FieldValue::toDate(), and PLearn::PDate::year.
{ int i = 0; PDate start_date = start_date_.getValue(sdb,row).toDate(); PDate end_date = end_date_.getValue(sdb,row).toDate(); // ******* BIG HACK TO GET AROUND A NASTY INCOMPREHENSIBLE BUG **** // (with the value read from file at this point not corresponding // to the value as seen otherwise in the file for the date of certain records) if (start_date.year<1970) start_date.year = end_date.year-1; // **************************************************************** real normalization = 0.001; real duration = (end_date - start_date)/365.25; if (rescale_by_interval_) { if (is_missing(duration) || duration<=0) { cout << "start_date = " << start_date << endl; cout << "end_date = " << end_date << endl; PLERROR("SDBVMFieldICBCTargets: incorrect dates"); } normalization = 0.001/duration; } if (rescale_by_start_date_) { int row_index = (start_date.year - reference_start_date_year_ - 1) * 12 + 12 - reference_start_date_month_ + start_date.month; if (targetname_=="ALL" || targetname_=="bodily_injury_incurred") { if (start_date_rescaling_values_[row_index][2] == 0) PLERROR("Trying to divide by zero"); output[i++] = real(bodily_injury_incurred_.getValue(sdb,row))*normalization / start_date_rescaling_values_[row_index][2]; } if (targetname_=="ALL" || targetname_=="property_damage_incurred") { if (start_date_rescaling_values_[row_index][3] == 0) PLERROR("Trying to divide by zero"); output[i++] = real(property_damage_incurred_.getValue(sdb,row))*normalization / start_date_rescaling_values_[row_index][3]; } if (targetname_=="ALL" || targetname_=="accident_death_incurred") { if (start_date_rescaling_values_[row_index][4] == 0) PLERROR("Trying to divide by zero"); output[i++] = real(accident_death_incurred_.getValue(sdb,row))*normalization / start_date_rescaling_values_[row_index][4]; } if (targetname_=="ALL" || targetname_=="collision_lou_incurred") { if (start_date_rescaling_values_[row_index][5] == 0) PLERROR("Trying to divide by zero"); output[i++] = real(collision_lou_incurred_.getValue(sdb,row))*normalization / start_date_rescaling_values_[row_index][5]; } if (targetname_=="ALL" || targetname_=="comprehensive_incurred") { if (start_date_rescaling_values_[row_index][6] == 0) PLERROR("Trying to divide by zero"); output[i++] = real(comprehensive_incurred_.getValue(sdb,row))*normalization / start_date_rescaling_values_[row_index][6]; } if (targetname_=="sum_all_but_BI") { if (start_date_rescaling_values_[row_index][3] == 0 || start_date_rescaling_values_[row_index][4] == 0 || start_date_rescaling_values_[row_index][5] == 0 || start_date_rescaling_values_[row_index][6] == 0) PLERROR("Trying to divide by zero"); output[i++] = (real(property_damage_incurred_.getValue(sdb,row))/ start_date_rescaling_values_[row_index][3] + real(accident_death_incurred_.getValue(sdb,row))/ start_date_rescaling_values_[row_index][4] + real(collision_lou_incurred_.getValue(sdb,row))/ start_date_rescaling_values_[row_index][5] + real(comprehensive_incurred_.getValue(sdb,row))/ start_date_rescaling_values_[row_index][6])*normalization; } } else { // no rescale by start date if (targetname_=="ALL" || targetname_=="bodily_injury_incurred") output[i++] = real(bodily_injury_incurred_.getValue(sdb,row))*normalization; if (targetname_=="ALL" || targetname_=="property_damage_incurred") output[i++] = real(property_damage_incurred_.getValue(sdb,row))*normalization; if (targetname_=="ALL" || targetname_=="accident_death_incurred") output[i++] = real(accident_death_incurred_.getValue(sdb,row))*normalization; if (targetname_=="ALL" || targetname_=="collision_lou_incurred") output[i++] = real(collision_lou_incurred_.getValue(sdb,row))*normalization; if (targetname_=="ALL" || targetname_=="comprehensive_incurred") output[i++] = real(comprehensive_incurred_.getValue(sdb,row))*normalization; if (targetname_=="sum_all_but_BI") output[i++] = (real(property_damage_incurred_.getValue(sdb,row))+ real(accident_death_incurred_.getValue(sdb,row))+ real(collision_lou_incurred_.getValue(sdb,row))+ real(comprehensive_incurred_.getValue(sdb,row)))*normalization; //counts if (targetname_=="ALLcounts" || targetname_=="bodily_injury_count") output[i++] = real(bodily_injury_count_.getValue(sdb,row)); if (targetname_=="ALLcounts" || targetname_=="all_counts_but_BI" || targetname_=="property_damage_count") output[i++] = real(property_damage_count_.getValue(sdb,row)); if (targetname_=="ALLcounts" || targetname_=="all_counts_but_BI" || targetname_=="accident_death_count") output[i++] = real(accident_death_count_.getValue(sdb,row)); if (targetname_=="ALLcounts" || targetname_=="all_counts_but_BI" || targetname_=="collision_lou_count") output[i++] = real(collision_lou_count_.getValue(sdb,row)); if (targetname_=="ALLcounts" || targetname_=="all_counts_but_BI" || targetname_=="comprehensive_count") output[i++] = real(comprehensive_count_.getValue(sdb,row)); //severities if (targetname_=="ALLseverities" || targetname_=="bodily_injury_severity") { int n = int(real(bodily_injury_count_.getValue(sdb,row))); output[i++] = n>0? real(bodily_injury_incurred_.getValue(sdb,row)) / n: 0; } if (targetname_=="ALLseverities" || targetname_=="all_severities_but_BI" || targetname_=="property_damage_severity") { int n = int(real(property_damage_count_.getValue(sdb,row))); output[i++] = n>0? real(property_damage_incurred_.getValue(sdb,row)) / n: 0; } if (targetname_=="ALLseverities" || targetname_=="all_severities_but_BI" || targetname_=="accident_death_severity") { int n = int(real(accident_death_count_.getValue(sdb,row))); output[i++] = n>0? real(accident_death_incurred_.getValue(sdb,row)) / n: 0; } if (targetname_=="ALLseverities" || targetname_=="all_severities_but_BI" || targetname_=="collision_lou_severity") { int n = int(real(collision_lou_count_.getValue(sdb,row))); output[i++] = n>0? real(collision_lou_incurred_.getValue(sdb,row)) / n: 0; } if (targetname_=="ALLseverities" || targetname_=="all_severities_but_BI" || targetname_=="comprehensive_severity") { int n = int(real(comprehensive_count_.getValue(sdb,row))); output[i++] = n>0? real(comprehensive_incurred_.getValue(sdb,row)) / n: 0; } //severity weights if (targetname_=="ALLseverities") output[i++] = real(bodily_injury_count_.getValue(sdb,row))>0? 1 : 0; if (targetname_=="ALLseverities" || targetname_=="all_severities_but_BI") { output[i++] = real(property_damage_count_.getValue(sdb,row))>0? 1 : 0; output[i++] = real(accident_death_count_.getValue(sdb,row))>0? 1 : 0; output[i++] = real(collision_lou_count_.getValue(sdb,row))>0? 1 : 0; output[i++] = real(comprehensive_count_.getValue(sdb,row))>0? 1 : 0; } } if (add_claims_sum_column_) { if (targetname_=="ALL") output[i] = output[0] + output[1] + output[2] + output[3] + output[4]; else output[i] = output[0]; i++; } if (rescale_by_interval_) output[i] = duration; // weight that should be given to squared loss // REPLACE MISSING VALUES BY ZEROS convertMissing(output); }
virtual int PLearn::SDBVMFieldICBCTargets::fieldWidth | ( | ) | const [inline, virtual] |
Width that the field occupies in the VMat.
Implements PLearn::SDBVMField.
Definition at line 983 of file SDBVMat.h.
{ if (targetname_=="ALL") return 5+int(use_roadstar_)+int(add_claims_sum_column_)+int(rescale_by_interval_); else if (targetname_=="sum_all_but_BI") return 1; else if (targetname_=="ALLcounts") return 5; else if (targetname_=="all_counts_but_BI") return 4; else if (targetname_=="ALLseverities") return 10; else if (targetname_=="all_severities_but_BI") return 8; else return 1+int(use_roadstar_)+int(add_claims_sum_column_)+int(rescale_by_interval_);}
Definition at line 1018 of file SDBVMat.h.
Referenced by convertField().
Definition at line 1012 of file SDBVMat.h.
Referenced by convertField().
Definition at line 1001 of file SDBVMat.h.
Referenced by convertField().
Definition at line 1016 of file SDBVMat.h.
Referenced by convertField().
Definition at line 1010 of file SDBVMat.h.
Referenced by convertField().
Definition at line 1019 of file SDBVMat.h.
Referenced by convertField().
Definition at line 1013 of file SDBVMat.h.
Referenced by convertField().
Definition at line 1020 of file SDBVMat.h.
Referenced by convertField().
Definition at line 1014 of file SDBVMat.h.
Referenced by convertField().
SDBVMSource PLearn::SDBVMFieldICBCTargets::end_date_ [protected] |
Definition at line 1009 of file SDBVMat.h.
Referenced by convertField().
Definition at line 1017 of file SDBVMat.h.
Referenced by convertField().
Definition at line 1011 of file SDBVMat.h.
Referenced by convertField().
Definition at line 1007 of file SDBVMat.h.
Referenced by convertField(), and SDBVMFieldICBCTargets().
Definition at line 1006 of file SDBVMat.h.
Referenced by convertField(), and SDBVMFieldICBCTargets().
Definition at line 1002 of file SDBVMat.h.
Referenced by convertField().
Definition at line 1003 of file SDBVMat.h.
Referenced by convertField(), and SDBVMFieldICBCTargets().
Definition at line 1008 of file SDBVMat.h.
Referenced by convertField().
Definition at line 1004 of file SDBVMat.h.
Referenced by convertField(), and SDBVMFieldICBCTargets().
string PLearn::SDBVMFieldICBCTargets::targetname_ [protected] |
Definition at line 1005 of file SDBVMat.h.
Referenced by convertField().
bool PLearn::SDBVMFieldICBCTargets::use_roadstar_ [protected] |