# Part I: theory of structured prediction

## Lecture 6 – 2/2 – Theory: generalization error bounds

• lecture6 scribbles

• recording

• Topics:

• Generalization error bound: VC dimension, Rademacher complexity

• Structured prediction generalization error bounds (factor graph complexity)

• Extra: non-parametric methods, kernel trick, RKHS

• Pointers:

• VC dimension / Rademacher complexity for binary case: see slides from presentation by John Shawe-Taylor at MLSS 2009.

• VC dimension definition: slide 38

• generalization error bound for binary classification with VC dimension: slide 46

• generalization error bound with Rademacher complexity: slide 87

• Structured prediction generalization bound: Corinna Cortes, Vitaly Kuznetsov, Mehryar Mohri, Scott Yang, “Structured Prediction Theory Based on Factor Graph Complexity”, NIPS 2016.

• Sidenote: relationship between constrained and regularized / penalized formulations:

• Kernel methods and RKHS:

• See the RKHS entry on Wikipedia.

• See a machine learning summer school tutorial on kernel methods by Arthur Gretton here. His first lecture gives an introduction to the RKHS.

## Lecture 7 – 2/4 – Kernels and RKHS

• recording

• Topics:

• Review of kernel trick, representer's theorem

• RKHS: intuition from spectral theorem + formalization

• Mercer's theorem and infinite dimensional SVD

• Pointers:

• See the RKHS entry on Wikipedia.

• See the nice Spring 2013 class “machine learning with kernel methods” by Jean-Philippe Vert. Below I give pointers to the 2013 version of the slides:

• slide 16: finite space example

• slide 55: representer's theorem

• slide 150: Mercer's theorem

• slide 156: subset of L2 viewpoint (see slide 163 for more info)

# Part II: convex optimization and structured prediction

## Lecture 9 – 2/11 – convex optimization

• lecture9 scribbles

• recording

• Topics:

• Finish going through the NIPS 2017 paper “On Structured Prediction Theory with Calibrated Convex Surrogate Losses”

## Lecture 12 – 2/23 – structured SVM optimization (II) - small QP

• lecture12 scribbles

• recording

• Topics:

• generic approach using duality to get small QP

• variational inequality perspective of saddle points

• small QP formulation for structured SVM

• examples of efficient loss-augmented inference: word alignment

## Lecture 21 – 4/1 – catalyst

• Topics:

• General acceleration scheme: catalyst

• Non-convex optimization

• submodular optimization

• Pointers:

## Poster session – 4/29

• 1:30 pm – 3:30 pm in Gather.Town