Course Overview

The objective of this graduate-level seminar course is to introduce students to algorithms in large-scale computing and machine learning. You will read, present and critique a curated set of research papers from both theory and systems. Each class will comprise of presentation and discussion of two research papers.

The first half of the course will cover distributed computing frameworks and scheduling and load balancing policies used in them. In the context of distributed storage, we will discuss coding-theoretic techniques used to improve availability and repair failed nodes. The second half of the course will focus on machine learning infrastructure. We will cover distributed SGD, federated learning and hyper-parameter tuning.

For more information - Course Syllabus


None. Basic knowledge of probability and linear algebra is encouraged

Grading Policy

Grades will be based on the following components:

You are allowed to take 3 grace days for this course. You can take at the most 1 grace day for an assignment.

Tentative Schedule

Date Lecture Speakers Homeworks
Mon, 26th Aug Logistics and Overview of the Topics

Wed, 28th Aug Probability Review

Mon, 2nd Aug Labor Day; No Class

Wed, 4th Sept Queueing Intro [Slides];

Scheduling for Parallel Computing [Slides]
HW1 Release
Mon, 9th Sept GCC Workshop on Effective Presentations [Slides1] [Slides2]

Wed, 11th Sept Grid Computing

Sribhuvan Sajja

Dhruva Kaushal

Mon, 16th Sept Tail at Scale

Lecture: Straggler Replication [Slides]
Varsha Narsing

HW1 Due

Wed, 18th Sept Sparrow

Attack of Clones
Wenting Chang

Samuel Nelson
HW2 Release
Mon, 23rd Sept Coding theory Intro [Slides]

Erasure Coded Storage [Slides]


Wed, 25th Sept Erasure Coded Storage [Slides]

Speeding up ML using Codes

Jaidev Singh
Mon, 30th Sept Rateless Codes [Slides]

Gradient Coding

Sweta Priyadarshi
HW2 Due
Wed, 2nd Oct Convergence Analysis of SGD (Chapter 4 only)

Survey of SGD methods

Varun Nagaraj Rao

Mon, 7th Oct Quiz 1

Wed, 9th Oct Invited talk on Coded Computing/Storage Saurabh Kakekodi [Slides]

Jack Kosaian [Slides]
HW3 Release

Mon, 14th Oct DistBelief

ImageNet Classification
Hun Namkung

Yuchen Wang

Wed, 16th Oct HogWild Paper

Slow and Stale Gradients Paper [Slides]

Sanghamitra Dutta

Mon, 21st Oct PipeDream

Stale Synchronous Parallel
Xiang yan

Jason Huang

HW3 Due

Wed, 23rd Oct Elastic Averaging SGD Paper

Cooperative SGD [Slides]
Yae Jee Cho

Jianyu Wang

Mon, 28th Oct AdaComm

Federated Learning Paper
Hao Liang

Soham Deshmukh

Wed, 30th Oct Fed Prox

Multi-task Learning
Yucheng Yin

Yixuan Lin
HW4 Release

Mon, 4th Nov Quiz 2

Wed, 6th Nov TernGrad Paper

Model Compression
Bhumi Bhanushali

Kathan Mehta

Mon, 11th Nov ATOMO

Deeptha Kumar

Ching-yi Lin

HW4 Due

Wed, 13th Nov MATCHA

Fed Learning with non-IID data
Swati Ravichandran

Akash Hegde

HW5 Release
Mon, 18th Nov MAB Intro; Bayesian Opt Intro (Guest talk) [Slides] [Tutorial]


Wed, 20th Nov HyperBand

Neural Architecture Search
Karan Hebbar

Tylor Vuong

Fri, 22th Nov

HW5 Due

HW6 Release

Mon, 25th Nov Spearmint Paper

Parallel Bayesian Opt
Daksha Shrivastava

Shreyas Chaudhari

Wed, 27th Nov Thanksgiving break; No class

Mon, 2nd Dec No Lecture: Extra office hours

HW6 Due

Wed, 4th Dec Quiz 3