Mathematics of Algorithmic Regularization in Deep Learning

The University of Manchester and IIT Kharagpur are happy to announce that the call for applications for the Joint Doctoral Program is now open. This is a time bound PhD program which needs to be completed in 4 years. 3 positions with full scholarships are available for the program. Additional positions with partial funding may be available.

Supervisors
The University of Manchester: Anirbit Mukherjee
IIT Kharagpur: Swagato Sanyal

Applications open: NOW
Closes: 6 March 2022
Eligibility information and how to apply: see here for full details
Qualification: PhD

Four-year study plan:

Year 1
In the first year, the student gets up to speed with statistical learning theory. If the student doesn’t have the required background then doing this will require the student to start on courses in stochastic optimization, measure-theoretic probability, high-dimensional probability (like the book by Vershynin) and partial differential equations.

Hopefully by the middle of the year the student would have studied some basic examples of algorithmic regularization and would have proven some theorems on extending them to cases beyond the textbook examples. We look to explore 2 natural extensions : to algorithms which are not just (stochastic) gradient descent and to simple non-linear predictors.

By the end of the first year the student would start to get familiar with at least a couple of papers on double descent and Deep Operator Nets : two specific themes which will define the eventual goal of the thesis on algorithmic regularization.

Year 2
In the second year the student continues to deeply study aspects of P.D.Es (focus on Navier-Stokes), stochastic optimization (focus on stochastic accelerated methods) and high-dimensional probability (focus on theorems around empirical processes).

The student starts to explore the cases of algorithmic regularization with more non-trivial non-linear predictors and things like proofs of Rademacher complexity with dropout. (We want to understand if these proofs of dropout can be written for nets of depth 3 or more.) By the end of the year, the student would have hopefully proven some non-trivial examples in both these themes.

Year 3
In the third year, we start to wholly focus on our most ambitious projects like trying to prove algorithmic regularization on single neural nets and product of neural nets – and more importantly for the specific loss functions on product of nets which are adapted to solve differential equations. We shall try to prove (or disprove!) double-descent on some such setup!

Year 4
The first half of the last year is likely to be spent in the student preparing for his/her job applications and appearing for the interviews. The second half will mostly go into polishing up the advanced projects from the last year and just writing up the thesis.

Related