95-865: Unstructured Data Analytics (Spring 2019 Mini 3)

Mon-Tue Jan 14-15

Reminder: Sections A3 and B3 meet Mondays and Wednesdays; Section C3 meets Tuesdays and Thursdays

Lecture 1: Course overview, basic text processing, and frequency analysis
[slides]

HW1 released (check Canvas)!

Lecture 2: Basic text analysis demo, co-occurrence analysis
[slides]
[Jupyter notebook (basic text analysis)]
[Jupyter notebook (co-occurrence analysis)]

Recitation 1: Basic Python review
[Jupyter notebook]

No class due to MLK Jr. Day (even though Tuesday is not a holiday, to keep the three sections synchronized, there will be no class on Tuesday for Section C3)

Lecture 3: Finding possibly related entities, PCA
[slides]
[Jupyter notebook (PCA)]

Causality additional reading:
[Computational and Inferential Thinking, "Causality and Experiments" chapter]

PCA additional reading (technical):
[Abdi and Williams's PCA review]

Recitation 2: More practice with PCA
[Jupyter notebook]

HW1 due 11:59pm, HW2 released

Lecture 4: t-SNE
[slides]
[required reading: "How to Use t-SNE Effectively" (Wattenberg et al, Distill 2016)]
[Jupyter notebook (MDS, Isomap, t-SNE)]

Python examples for dimensionality reduction:
[scikit-learn example (PCA, Isomap, t-SNE, and many other methods)]

Additional dimensionality reduction reading (technical):
[Isomap webpage]
[Simon Carbonnelle's t-SNE slides]
[t-SNE webpage]
["An Analysis of the t-SNE Algorithm for Data Visualization" (Arora et al, COLT 2018)]

Class cancelled due to polar vortex

Lecture 5 (this is not a typo): Introduction to clustering, k-means, Gaussian mixture models
[slides]
[Jupyter notebook (dimensionality reduction: images)]
[Jupyter notebook (dimensionality reduction: drug consumption data)]

Recitation 3: t-SNE (recorded video is in Canvas - look for "UDAP 01/25 Recitation", which is the same recitation material that was presented 1 week earlier for the policy version of UDA)
[slides]
[Jupyter notebook]

Additional clustering reading (technical):
[see Section 14.3 of the book "Elements of Statistical Learning" on clustering]

Lecture 6: Clustering and clustering interpretation demo, automatic selection of k with CH index
[slides]
[Jupyter notebook (clustering)]

Additional clustering reading (technical):
[see Section 14.3 of the book "Elements of Statistical Learning" on clustering]

Python cluster evaluation:
[scikit-learn cluster evaluation metrics (many, including CH index)]

Lecture 7: Hierarchical clustering, topic modeling
[slides]
[Jupyter notebook (hierarchical clustering)]

Additional reading (technical):
[see Section 14.3.12 "Hierarchical Clustering" of the book "Elements of Statistical Learning"]
[David Blei's general intro to topic modeling]

Recitation 4: Quiz review session

Lecture 8: Topic modeling (wrap-up; note that this material is fair game for the quiz), introduction to predictive analytics (this is the official start to the second half of the course on predictive data analysis), nearest neighbors, evaluating prediction methods
[slides]
[Jupyter notebook (topic modeling)]

Lecture 9: Model validation, decision trees/forests
[slides]
[Jupyter notebook (prediction and model validation)]

Mid-mini quiz (same time/place as recitation); in case of space issues, we do have an overflow room booked (HBH 1002)

HW2 due 11:59pm, HW3 released

Lecture 10: Introduction to neural nets and deep learning
[slides]
[Jupyter notebook (intro to neural nets and deep learning)]

Mike Jordan's Medium article on where AI is at (April 2018):
["Artificial Intelligence - The Revolution Hasn't Happened Yet"]

Pre-recorded recitation for support vector machines, ROC curves (this material is also fair game for the final exam despite us not having a live recitation for it)
[video recording is in Canvas ("UDAP 02/15 Recitation" -- you may have to go to "Page 2")]
[slides]
[Jupyter notebook]

Video introduction on neural nets:
["But what *is* a neural network? | Chapter 1, deep learning" by 3Blue1Brown]

Additional reading:
[Chapter 1 "Using neural nets to recognize handwritten digits" of the book Neural Networks and Deep Learning]

Lecture 11: Image analysis with CNNs (also called convnets)
[slides]
[Jupyter notebook (CNN demo)]

CNN reading:
[Stanford CS231n Convolutional Neural Networks for Visual Recognition]

Recitation 5: Final exam review

Lecture 12: Time series analysis with RNNs
[slides]
[Jupyter notebook (RNN demo)]

LSTM reading:
[Christopher Olah's "Understanding LSTM Networks"]

Lecture 13: Interpreting what a deep net is learning, other deep learning topics, wrap-up
[slides]
[fashion_mnist_cnn_model.h5 (pre-trained convnet needed for demo)]
[Jupyter notebook (interpreting a CNN)]

Gary Marcus's Medium article on limitations of deep learning and his heated debate with Yann LeCun (December 2018):
["The deepest problem with deep learning"]

Videos on learning neural nets (warning: the loss function used is not the same as what we are using in 95-865):
["Gradient descent, how neural networks learn | Chapter 2, deep learning" by 3Blue1Brown]
["What is backpropagation really doing? | Chapter 3, deep learning" by 3Blue1Brown]

Recent heuristics/theory on gradient descent variants for deep nets (technical):
["Don't Decay the Learning Rate, Increase the Batch Size" by Smith et al (ICLR 2018)]
["Parallelizing Stochastic Gradient Descent for Least Squares Regression: Mini-batching, Averaging, and Model Misspecification" by Jain et al (JMLR 2018)]

Some interesting reads (technical):
["Understanding deep learning requires rethinking generalization" by Zhang et al (ICLR 2017)]
["Relational inductive biases, deep learning, and graph networks" by Battaglia et al (2018)]

No class

HW3 due 11:59pm

Final exam, 4:30pm-5:50pm HBH A301