CAS 751: Information-Theoretic Methods in Trustworthy Machine Learning

The interplay between information theory and computer science is a constant theme in the development of both fields. This course discusses how techniques rooted in information theory play a key role in
(i) understanding the fundamental limits of classical high-dimensional problems in machine learning and
(ii) formulating emerging objectives such as privacy, fairness, and interpretability.

The course begins with an overview of f-divergences and data-processing inequalities, two important concepts in information theory, and then delves into central and local differential privacy and algorithmic fairness.

No background in information theory is required, but some knowledge of machine learning, statistics, and probability (equivalent to undergraduate courses in these topics) is needed.

Office Hours

Anonymous Contact
Here

Course Outline & Policy
Outline 2025

Course Schedule

Lecture Date Lecture Note References and Readings
Sept 3 Why “Trustworthy” machine learning? Watch this and this; Lecture 0, Review Probability
Sept 10 f-divergences 1 Ch 7 of PW2023, Lecture 1
Sept 17 f-divergences 2 and SDPI Ch 33 of PW2023
Sept 24 Foundations of DP Sections 1.4–1.6 of this and Sections 2, 3.1–3.2 of this
Oct 1 Properties of DP Sections 1.4–1.6 of this and Sections 2, 3.1–3.2 of this
Oct 8 Approximate DP and composition theorems Appendix A of this, this, and this blog post
Oct 15 Advanced composition and private gradient descent Advanced composition proof and ML primer
Oct 22 Mid-term  
Oct 29 Private SGD and Renyi DP Private SGD, Optimal RDP-to-DP conversion
Nov 5 Local DP Trust models
Nov 12 Statistical estimation under LDP Sec 31.1 of WP23 and this
Nov 19 A non-comprehensive exposition of fairness criteria in ML ProPublica: Machine Bias
Nov 26 Algorithmic fairness 1  
Dec 3 Algorithmic fairness 2  
Dec 10 Presentations