Course Overview

The objective of this course is to introduce students to state-of-the-art algorithms in large-scale machine learning and distributed optimization, in particular, the emerging field of federated learning. Topics to be covered include but are not limited to:


Comparison with Related Courses


There will be no required textbooks. Students are expected to read the research papers covered in each lecture.


We will use Piazza for class discussions. We strongly encourage students to post on this forum rather than emailing the course staff directly (this will be more efficient for both students and staff). Students should use Piazza to:

The course Academic Integrity Policy must be followed on the message boards at all times. Do not post or request homework solutions! Also, please be polite.

Grading Policy

Grades will be based on the following components:

Collaboration Policy

Group studying and collaborating on problem sets are encouraged, as working together is a great way to understand new material. Students are free to discuss the homework problems with anyone under the following conditions: Students are encouraged to read CMU's Policy on Cheating and Plagiarism.


Date Lecture Readings Announcements
Mon, 1 Feb Intro and Logistics [Slides]

Wed, 3 Feb SGD in Machine Learning [Slides]
Fri, 5 Feb OH

HW1 release
Mon, 8 Feb SGD in Neural Network Training, Momentum and Adaptive Methods [Slides]

Wed, 10 Feb SGD Convergence Analysis [Slides]

Fri, 12 Feb Math Quiz Review

Mon, 15 Feb Distributed Synchronous SGD [Slides]

Wed, 17 Feb Lecture Canceled

Fri, 19 Feb

HW1 due, HW2 release
Mon, 22 Feb Asynchronous SGD, AdaSync, Hogwild [Slides]

Wed, 24 Feb Break Day; No Class

Fri, 26 Feb

Mon, 1 Mar Local-update SGD [Slides]
Wed, 3 Mar Adacomm, Elastic Averaging, Overlap SGD [Slides]

Fri, 5 Mar

Mon, 8 Mar First Quiz

Wed, 10 Mar Quantized SGD, AdaQuant [Slides]

Fri, 12 Mar

Sun, 14 Mar

HW2 due
Mon, 15 Mar Federated Learning Intro [Slides]

Wed, 17 Mar Data Heterogeneity in Federated Learning[Slides] HW3 release
Fri, 19 Mar Midsemester Break; No classes

Mon, 22 Mar Computational Heterogeneity in Federated Learning [Slides]

Wed, 24 Mar Fairness in FL [Slides]
Fri, 26 Mar

Mon, 29 Mar Client Selection and Importance Sampling [Slides]
Wed, 31 Mar Robustness in FL [Slides]
Fri, 2 Apr

Mon, 5 Apr Break Day; No classes

Wed, 7 Apr Second Quiz

Fri, 9 Apr

HW3 due
Mon, 12 Apr Privacy and Security in FL [Slides]

Wed, 14 Apr Personalized and Multi-task Learning [Slides]

HW4 release

Fri, 16 Apr

Mon, 19 Apr Lecture Canceled
Wed, 21 Apr Decentralized SGD [Slides]

Fri, 23 Apr

Mon, 26 Apr Hyperparameter Optimization [Slides]

Wed, 28 Apr Review Lecture [Slides]

Fri, 30 Apr

Mon, 3 May No class

Wed, 5 May Third Quiz

Fri, 7 May

HW4 due