95-865: Unstructured Data Analytics (Fall 2018 Mini 2)

Lecture 1: Course overview, basic text processing and frequency analysis
[slides]

HW1 released (check Canvas)!

Lecture 2: Basic text analysis demo, co-occurrence analysis
[slides]
[Jupyter notebook (spaCy)]
[Jupyter notebook (co-occurrences)]

Recitation 1: Basic Python review
[Jupyter notebook]

Lecture 3: Finding possibly related entities, PCA, Isomap
[slides]
[Jupyter notebook (PCA)]
[Jupyter notebook (multidimensional scaling)]

Causality additional reading:
[Computational and Inferential Thinking, "Causality and Experiments" chapter]

PCA additional reading (technical):
[Abdi and Williams's PCA review]

Lecture 4: t-SNE
[slides]
[required reading: "How to Use t-SNE Effectively" (Wattenberg et al, Distill 2016)]
[Jupyter notebook (t-SNE)]

Python examples for dimensionality reduction:
[scikit-learn example (PCA, Isomap, t-SNE, and many other methods)]

Additional dimensionality reduction reading (technical):
[Isomap webpage]
[Simon Carbonnelle's t-SNE slides]
[t-SNE webpage]
["An Analysis of the t-SNE Algorithm for Data Visualization" (Arora et al, COLT 2018)]

Recitation 2: t-SNE
[Jupyter notebook]

Lecture 5: Introduction to clustering, k-means, Gaussian mixture models
[slides]

HW1 due 4:30pm (start of class), HW2 released!

Additional clustering reading (technical):
[see Section 14.3 of the book "Elements of Statistical Learning" on clustering]

Python cluster evaluation:
[scikit-learn cluster evaluation metrics (many, including CH index)]

Lecture 6: Clustering and clustering interpretation demo, automatic selection of k with CH index
[slides]
[Jupyter notebook (clustering)]

Recitation 3: Quiz review session

Lecture 7: Hierarchical clustering, topic modeling
[slides]
[Jupyter notebook (hierarchical clustering)]
[Jupyter notebook (Latent Dirichlet Allocation)]

Additional reading (technical):
[see Section 14.3.12 "Hierarchical Clustering" of the book "Elements of Statistical Learning"]
[David Blei's general intro to topic modeling]

Lecture 8 (this is not a typo): Introduction to predictive analytics, nearest neighbors, evaluating prediction methods, decision trees
[slides]
[Jupyter notebook (prediction and model validation)]

Lecture 9: Intro to neural nets and deep learning
[slides]
[Jupyter notebook (intro to neural nets and deep learning)]

Mike Jordan's Medium article on where AI is at (April 2018):
["Artificial Intelligence - The Revolution Hasn't Happened Yet"]

The Atlantic's article about Judea Pearl's concerns with AI not focusing on causal reasoning (May 2018):
["How a Pioneer of Machine Learning Became One of Its Sharpest Critics"]

HW2 due 4:30pm (start of class); HW3 released

Video introduction on neural nets:
["But what *is* a neural network? | Chapter 1, deep learning" by 3Blue1Brown]

Additional reading:
[Chapter 1 "Using neural nets to recognize handwritten digits" of the book Neural Networks and Deep Learning]

Adelaide Recitation 5: Support vector machines (SVM's), cross validation, decision boundaries, ROC curves
[slides (support vector machines)]
[Jupyter notebook]

Thanksgiving: no class

Lecture 10: Image analysis with CNNs (also called convnets)
[slides]
[Jupyter notebook (CNN demo)]

CNN reading:
[Stanford CS231n Convolutional Neural Networks for Visual Recognition]

Lecture 11: Time series analysis with RNNs, roughly how learning a deep net works (gradient descent and variants)
[slides]
[Jupyter notebook (RNN demo)]

LSTM reading:
[Christopher Olah's "Understanding LSTM Networks"]

Videos on learning neural nets (warning: the loss function used is not the same as what we are using in 95-865):
["Gradient descent, how neural networks learn | Chapter 2, deep learning" by 3Blue1Brown]
["What is backpropagation really doing? | Chapter 3, deep learning" by 3Blue1Brown]

Recent heuristics/theory on gradient descent variants for deep nets (technical):
["Don't Decay the Learning Rate, Increase the Batch Size" by Smith et al (ICLR 2018)]
["Parallelizing Stochastic Gradient Descent for Least Squares Regression: Mini-batching, Averaging, and Model Misspecification" by Jain et al (JMLR 2018)]

Recitation 4: Word embeddings as an example of self-supervised learning
[slides]
[Jupyter notebook]
Important: You do not need to know the technical details of the demo (also, for this class you do not need to know Pandas)

Lecture 12: Interpreting what a deep net is learning, other deep learning topics, wrap-up
[slides]
[fashion_mnist_cnn_model.h5 (pre-trained convnet needed for demo)]
[Jupyter notebook (interpreting a CNN)]

Gary Marcus's Medium article on limitations of deep learning and his heated debate with Yann LeCun (December 2018):
["The deepest problem with deep learning"]

HW3 due 4:30pm (start of class)

Some interesting reads (technical):
["Understanding deep learning requires rethinking generalization" by Zhang et al (ICLR 2017)]
["Relational inductive biases, deep learning, and graph networks" by Battaglia et al (2018)]

No class

Recitation 5: Final exam review session

Final exam 1pm-4pm HBH 1002