CAS 751: Information-Theoretic Methods in Trustworthy Machine Learning

The interplay between information theory and computer science is a constant theme in the development of both fields. This course discusses how techniques rooted in information theory play a key role in
(i) understanding the fundamental limits of classical high-dimensional problems in machine learning and
(ii) formulating emerging objectives such as privacy, fairness, and interpretability.

The course begins with an overview of f-divergences and data-processing inequalities, two important concepts in information theory, and then delves into central and local differential privacy and algorithmic fairness.

No background in information theory is required, but some knowledge of machine learning, statistics, and probability (equivalent to undergraduate courses in these topics) is needed.

Office Hours

Anonymous Contact
Here

Course Outline & Policy
Outline 2026

Course Schedule

Lecture Date Lecture Note References and Readings
Sept 9 Why “Trustworthy” machine learning? Watch this and this; Lecture 0, Review Probability
Sept 16 f-divergences 1 Ch 7 of PW2023, Lecture 1
Sept 23 f-divergences 2 and SDPI Ch 33 of PW2023, Lecture 2
Sept 30 Foundations of DP Sections 1.4–1.6 of this and Sections 2, 3.1–3.2 of this, Lecture 3
Oct 7 Properties of DP Sections 1.4–1.6 of this and Sections 2, 3.1–3.2 of this, Lecture 4
Oct 14 Mid-term  
Oct 21 Approximate DP and composition theorems Appendix A of this, this, and this blog post, Lecture 5
Oct 28 Advanced composition and private gradient descent Advanced composition proof and ML primer, Lecture 6 (incomplete)
Nov 4 Private SGD and Renyi DP Private SGD, Optimal RDP-to-DP conversion, Lecture 7: advanced composition (complete version)
Nov 11 Private ML Lecture 8
Nov 18 Local DP and Statistical estimation under LDP Trust models, Sec 31.1 of WP23 and this, Lecture 9
Nov 25 A non-comprehensive exposition of fairness criteria in ML ProPublica: Machine Bias
Dec 2 Algorithmic fairness  
Dec 9 Presentation