Theories of Deep Learning

Module aims

State a theorem about representation, training dynamics, or generalisation with its hypotheses, and refuse empirical slogans as substitutes.

Course outline

1. 01 Approximation by networks 25h

   Learning result State a universal-approximation claim with architecture hypotheses.

   Formative check Four-option check

2. 02 Loss landscapes (taught grain) 25h

   Learning result Describe a landscape result the synopsis proves and a spurious-critical-point caveat.

   Formative check Counterexample

3. 03 Implicit bias and training dynamics 25h

   Learning result Name an implicit-bias slogan for a named algorithm and model class.

   Formative check Written derivation

4. 04 Generalisation analyses 25h

   Learning result Apply a bound only under the measure and complexity hypotheses given.

   Formative check Written derivation

5. 05 What experiments do not prove 20h

   Learning result Give an empirical success that the current theorem does not explain.

   Formative check Four-option check

6. 06 Synthesis 30h

   Learning result Write one theorem-plus-hypothesis argument without converting test accuracy into a proof.

   Formative check Written derivation

Learning results

Quote an approximation or implicit-bias result with function-class and training-model hypotheses, and give a setting where a naive claim fails.

Subject skills

Universal approximation at taught grain, NTK/mean-field language as in the synopsis, generalisation bounds that are actually proved there.

Transferable skills

Keep empirical plots out of the theorem statement; name the training model.