This course provides a unifying introduction to statistical modeling of multidimensional data through the framework of probabilistic graphical models, together with their associated learning and inference algorithms.

Teacher: Simon Lacoste-Julien, Office: D-11, Mila (6666 rue Saint-Urbain, 2nd floor, enter by reception)

Office hours: Friday 16h00-17h00 – in Cube 1 right by the Mila reception

TA: Jose Gallego, Office: Cube 1 right by the Mila reception

Office hours: Thursday 13:00-14:00

Tuesday 14h30-16h30 - Mila Agora, 6650 rue Saint-Urbain (ground floor)

Friday 14h00-16h00 - Mila Agora

Probability review

Maximum likelihood estimation

Linear regression, logistic regression, Fisher discriminant

K-means, EM, Gaussian mixtures

Directed and undirected graphical models

Exponential family, information theory

Gaussian networks

Factor analysis

Sum-product algorithm, HMM, junction tree

Approximate inference: sampling, variational methods

Estimation of parameters in graphical models

Bayesian methods

Model selection

Homework (40%) – 5 homework | homework logistics below

Project (30%) – project report to hand in + poster presentation on Dec 17th | detailed info about projects

Final exam (30%) – take-home exam, after poster presentation - due Dec 23rd

The prerequisites are previous coursework in linear algebra, multivariate calculus, and basic probability and statistics. There will be programming for the assignments, so familiarity with some matrix-oriented programming language will be useful (we will use Python with numpy).

The course will follow the (unpublished) manuscript

*An Introduction to Probabilistic Graphical Models*by Michael I. Jordan that will be made available to the students (but do not distribute!).Supplementary references:

For very detailed and rigorous reference: Probabilistic Graphical Models: Principles and Techniques by Daphne Koller and Nir Friedman. Referred as KF in outline below.

See Part I of the Deep Learning book by Ian Goodfellow, Yoshua Bengio and Aaron Courville for a very gentle review of applied maths useful for this class. Chapter 5 contains a useful presentation of machine learning basics. Referred as DL in outline below.

Another classical book, with a more Bayesian perspective than Mike's book, but at least completed, is Pattern Recognition and Machine Learning, by Chris Bishop.

The homework is to be handed online in Studium before

**the beginning of the class**(Tuesday) on the due date. The detailed instructions for the code handin-logistics can be found in the news in Studium.**Collaboration policy**: you can collaborate with colleagues while working on the homework, but you need to write your own independent write-up. And if you have collaborated with others on a question,*you need to credit the help of your colleagues by specifying them in the write-up*(proper acknowledgment is good practice for you have for academia later).**Late homework policy**:You have a budget of 6 late days that you can spend on the 5 homework. To use these days, you need to

*declare it*by filling the appropriate option in the Google form to hand in with the assignment.The late day penalty will be the following (as deadlines are on Tuesday):

handed in Tuesday after beginning of class: 10% penalty (or 1 day late used)

handed in Wednesday: 20% penalty (or 1 day late used)

handed in Thursday: 40% penalty (or 2 days late used)

handed in Friday: 80% penalty (or 3 days late used)

handed in later: you won't get any credit (or you have to have used some of your late days to reduce the number of days counted late)

handed in Monday: 100% penalty (or 4 days late used)

handed in Tuesday (one week later): 100% penalty (or 5 days late used)

**No assignment accepted more than one week late**

Below is a draft detailed outline that will be updated as the class goes on. For now it is the outline recopied from the Fall 2018 version of this class, with the links to the relevant old scribbled notes that will be updated with the new ones gradually. The related chapters in Mike's book are given (but note that they do not exactly correspond with the class content), and also sometimes pointers to the Koller and Friedman book (KF), the Deep Learning book (DL) or the Bishop's book (B). Related past ‘‘scribe notes’’ from the class that I taught in Paris are given for now, and will be updated with the scribe notes as I get them (if I get some).

Date | Topics | Related chapters Scribbled notes | Scribe notes | Homework milestones |

Sept 3 | Set-up & overview | intro slides lecture1 | Isabela Albuquerque (Fa17) lecture1.pdf source | |

Sept 6 | Probability review | 2.1.1 DL: 3 (nice and gentle) KF: 2.1 (more rigorous) lecture2 | William Léchelle (Fa16) lecture2.pdf source | |

Sept 10 | Parametric models Frequentist vs. Bayesian | 5 lecture3 | Philippe Brouillard and Tristan Deleu (Fa17) lecture3.pdf source | Hwk 1 out (hwk 1 source) |

Sept 13 | Bayesian (cont.) Maximum likelihood | lecture4 | Philippe Brouillard and Tristan Deleu (Fa17) lecture4.pdf | |

Sept 17 | MLE (cont.) Statistical decision theory | 1.3 in Bickel & Doksum Bias-variance tradeoff: 7.3 in Hastie's book lecture5 | Sébastien Lachapelle (Fa17) lecture5.pdf source | |

Sept 20 | Properties of estimators | old lecture6 | same as lecture5 | |

Sept 24 | Linear regression Logistic regression | DL: 7.1 (l2, l1-reg.) 6, 7 old lecture7 | Zakaria Soliman (Fa16) lecture6.pdf source | Hwk 1 due Hwk 2 out |

Sept 27 | Optimization Logistic regression (cted) + IRLS | 7 DL: 4.3 Boyd's book old lecture8 | MVA lecture2 | |

Oct 1 | Gen. classification (Fisher) Derivative tricks for Gaussian MLE Kernel trick (skipped) K-means | Matrix Diff. book 10, 11 lecture9 2017 (kernel trick) old lecture9 | MVA lecture3 | |

Oct 4 | GMM and EM | 10,11 old lecture10 | ||

Oct 8 | Graph theory Directed graphical models | 2 old lecture11 | MVA lecture4 | Hwk 2 due Hkw 3 out |

Oct 11 | DGM (cont.) Undirected graphical models | 2 old lecture12 | ||

Oct 15 | UGM (cont.) Inference: elimination alg. | 3 old lecture13 | ||

Oct 18 | Sum-product alg. Max-product junction tree | 4, 17 old lecture14 | MVA lecture7 | |

Oct 22 | Break: look at projects | |||

Oct 25 | Break: look at projects | |||

Oct 29 | HMM and EM | 12 old lecture15 | Hwk 3 due Hwk 4 out | |

Nov 1 | Class cancelled: look at projects | |||

Nov 5 | Class cancelled: look at projects | |||

Nov 8 | Information theory Max entropy Duality | 19 old lecture16 | MVA lecture5 | |

Nov 12 | MaxENT duality Exponential families | 8 (KL geometry: old lecture16) lecture17 2017 | MVA lecture6 MVA lecture8 | Project: team formed |

Nov 15 | Sampling | 21 old lecture18 | ||

Nov 19 | MCMC sampling | 21 (variance reduction: see old lecture18 2017) old lecture19 | ||

Nov 22 | Non-parametric models: Gaussian processes Dirichlet processes | 25 old lecture20 slides GP book DP tutorial | ||

Nov 26 | MCMC (cont.) Gibbs sampling | lecture21 | Hwk 4 due Hwk 5 out | |

Nov 29 | Variational methods Estimation in graphical models | Bishop: 10.1 old lecture22 9 (skipped parts of: old lecture22 2017) | ||

Dec 3 | Bayesian methods Model selection | 5, 26 old lecture23 | MVA lecture10 | Project: 1 page progress report due |

Dec 6 | (last lecture) Gaussian networks Factor analysis, PCA, CCA (Kalman filter) VAE | old lecture24 13 old lecture17 Fa2016 14 , (15) old lecture18 Fa2016 VAE – DL: 20.10.3 | ||

Dec 10 | No lecture this week (NeurIPS) work on your project! | |||

Dec 17 | Poster presentation Time and place TBC (most likely 1:00pm-4:00pm) | Hwk 5 due Take-home final out | ||

Dec 23 | Project report due Take-home final due (online) | |||