From Basic Machine Learning models to Advanced Kernel Learning

Course description

Statistical learning is about the construction and study of systems that can automatically learn from data. With the emergence of massive datasets commonly encountered today, the need for powerful machine learning is of acute importance. Examples of successful applications include effective web search, anti-spam software, computer vision, robotics, practical speech recognition, and a deeper understanding of the human genome.

This course gives an introduction to this exciting field. In the first part, we will introduce basic techniques such as logistic regression, multilayer perceptrons, nearest neighbor approaches, both from a theoretical and methodological point of views. In the second part, we will focus on more advanced techniques such as kernel methods, which is a versatile tool to represent data, in combination with (un)supervised learning techniques that are agnostic to the type of data that is learned from. The learning techniques that will be covered include regression, classification, clustering and dimension reduction. We will cover both the theoretical underpinnings of kernels, as well as a series of kernels that are important in practical applications. Finally we will touch upon topics of active research, such as large-scale kernel methods and the use of kernel methods to develop theoretical foundations of deep learning models.


The grading of the class will be done with (i) one final exam, (ii) two homeworks.

Reading material


Lectures are scheduled from 9:45-11:15 on Mondays, and from 8:15-9:45 on Thursdays. The first part of the class will recap basic supervised learning techniques, theory, and algorithms. Lecture notes or slides will be updated here on the fly.

You can download the slides for all lectures of the second part of the class (advanced kernel methods) here! Each lecture corresponds to a range of slides. Slides are frequently updated. Please let us know if you spot typos!

Homework 1. The first homework (data) is due by Monday, November 16, 2023. The report can be written in English or in French.

Homework 2. The second homework is due by Thursday, January 12, 2023 at 23:59PM. Students must submit a single PDF to the GradeScope plateform. Please follow the instructions to create an account on GradeScope and to submit the homework. For questions about the Homework please send an email to "", the object of the email must start with the following code: 4V527W.

# Date Room Teacher Topic
1 25/09/2022 PG Supervised learning basics
2 28/09/2022 PG Linear regression
3 02/10/2022 PG Linear regression
4 05/10/2022 PG Logistic regression
5 09/10/2022 PG Logistic regression and Maximum likelihood
6 16/10/2022 PG Maximum likelihood and K-Nearest Neighbors
7 19/10/2022 PG K-Nearest Neighbors (notebook)
8 23/10/2022 PG Multilayer Perceptrons
9 26/10/2022 JM Positive definite kernels, slides 1-23
10 6/11/2022 JM RKHS I, slides 25-55
11 9/11/2022 MA RKHS II, slides 25-55
12 13/11/2022 JM Kernel tricks, slides 56-82
13 16/11/2022 MA Kernel Ridge regression, slides 83-103
14 20/11/2022 JM Suplementary material: Rademacher complexity
15 23/11/2022 MA Support Vector Machines, slides 134-165
16 27/11/2022 JM Kernel PCA, slides 166-202, Video I
17 30/11/2022 MA Kernel K-means, kernel CCA, slides 166-202, Video II
18 04/12/2022 JM Kernel Jungle, slides 203-337, Video III
19 07/12/2022 MA kernel for biological sequences, slides 445-531, Video IV
20 11/12/2022 JM Kernel mean embeddings I, slides 1-30 (Homework 2)
21 14/12/2022 MA Kernel mean embeddings II, slides 31-57
23 18/12/2023 JM Large scale learning with kernels, slides 586-628
24 21/12/2023 MA Fondations of deep learning from a kernel point of view, slides 630-672
24 11/01/2023 MA Revisions: Kernel K-means, kernel CCA
# End/01/2023 Exam