skip to page content SBU
Carnegie Mellon University
Machine Learning for Problem Solving
95-828 - Spring 2017

Home
Syllabus
Assignments
Notes

Tentative Syllabus (download as pdf)

Disclaimer: This is an ambitious list of topics that I aim to cover in this course. I will adjust the pace depending on the progress of and the feedback from the students in class. As such, it is possible that only some subset of these topics will end up being covered. HW and exams will be adjusted accordingly.


Date

Lectures and Readings

Out
/ Due

   

1/17

 

 

 

 

1/19

Lecture 1: Intro to ML

  • What is ML? ML applications
  • Machine learning paradigms
    • Supervised learning (classification, regression, feature selection)
    • Unsupervised learning (density estimation, clustering, dimensionality reduction)
  • Data mining concepts and tasks
    • Association rules, similarity search, cluster analysis, outlier analysis
  • Basic data types
    • (Mixed) attribute data, text, time series, sequence, network data
  • The problem solving process:
    • Business/project understanding, data understanding through EDA, data preparation, modeling, evaluation, deployment

Readings:

  • Witten & Frank       Chapter 1.1-1.3
  • Provost & Fawcett    Chapter 2

PART I: PRELIMINARY ANALYSIS AND DATA PREPARATION

   

1/19

 

 

1/24

Lecture 2: Exploratory Data Analysis

  • Getting to know your data
  • Data types
  • Attribute types
  • Data quality issues
  • Data visualization
    • Histogram, Kernel Density Estimation
    • Charts, plots, infographics
  • Correlation analysis

Readings:

  • Aggarwal         Chapter 2
  • Witten & Frank    Chapter 2
  • Hastie     Chapter 6.6.1



1/26

 

 

1/31

Lecture 3: Data Preparation

  • Feature creation
  • Data cleaning
    • Missing, inaccurate, duplicate values
  • Data transformation
    • Feature type conversion
    • Discretization
    • Normalization / Standardization
  • Data reduction
    • Feature and record selection
    • Principal Component Analysis
    • Multidimensional scaling
    • Manifold learning (Isomap, LLE)

Readings:

HW1 out

 

 

 

 

 

 

 


PART II: SUPERVISED LEARNING


2/2

 

2/7

Lecture 4: Linear Models

  • Linear Regression
  • Robust Regression
  • Sparse Linear Models
    • Feature subset selection: revisited
    • Shrinkage methods: ridge regression and Lasso
    • Principal components regression, Partial least squares

Readings:

  • ISLR (James, Witten, Hastie, Tibshirani)      Chapter 3.1, 3.2, 3.3, 3.4
  • ISLR (James, Witten, Hastie, Tibshirani)      Chapter 6.1, 6.2.1, 6.2.2, 6.3.1, 6.3.2
Other readings:
  • Hastie                Chapter 3.1-3.4, 4.4
  • Shalizi               Chapter 2, 11
  • Murphy              Chapters 1.4, 7.1-7.5, 13.3-13.5
  • Provost & Fawcett     Chapter 4
  • Witten & Frank        Chapter 7.5

   

2/7

2/9

Lecture 5: Logistic Regression and Generalized Models

  • Logistic Regression
  • Generalized Linear Models (GLMs)
  • Generalized Additive Models (GAMs)
    • Basis expansions
    • Generalizations, shape functions

Readings:

Other readings:
  • Hastie     Chapter 9.1, 9.3, 9.6
  • Shalizi     Chapter 12



2/14

Lecture 6: Model Selection

  • What is a good model?
  • Overfitting
  • Decomposition of error
  • Bias-Variance tradeoff
  • Cross Validation
  • Regularization
  • Information Criteria (AIC, BIC, MDL)

Readings:

  • Hastie                Chapter 7.1 - 7.10
  • Provost & Fawcett     Chapter 5

   

2/16

Lecture 7: Model Evaluation

  • Performance measures for Machine Learning
  • Creating baseline methods for comparison
  • Visualizing model performance

Readings:

  • Witten & Frank        Chapter 5
  • Provost & Fawcett     Chapter 7, 8, 11
  • Shalizi               Chapter 3, 10


2/21


2/23

Lecture 8: Tree-based Methods

  • Classification trees
  • From trees to rules
  • Missing values and pruning
  • Regression trees

Readings:

  • Hastie               Chapter 9.2
  • Witten & Frank        Chapter 4.3-4.4, 6.1-6.2
  • Provost & Fawcett     Chapter 3
  • Shalizi               Chapter 13
  • Murphy              Chapter 16.2
HW1 due
HW2 out



Project proposal due

 

   

2/28

Lecture 9: Support Vector Machines

  • SVM intuition, formulation, and the dual
  • Slack variables, Hinge loss
  • The Kernel trick
    • Kernel SVM
    • Kernel Logistic Regression
    • Kernel PCA

Readings:



3/2

3/7

Lecture 10: Instance-based Learning

  • Kernel Density Estimation
  • k-Nearest Neighbor Classifier
  • Kernel Regression
  • Locally-Weighted Linear Regression

Readings:

  • Hastie            Chapter 6.1-6.3, 6.6.1-6.6.2
  • Murphy           Chapter 1.4.1-1.4.3, 14.7
  • Shalizi            Chapter 7.1, 7.5

   

3/7

 

Lecture 11: Ensemble Learning

  • Combining multiple models
  • Bagging
  • Random Forests

Readings:

  • Witten & Frank       Chapter 8
  • Hastie              Chapter 10.1, 15, 16
  • ISL-with R             Chapter 8.2

 

 

 

 


3/9

Midterm Exam (in class)


3/13-17

Spring Break; No Classes


HW2 due HW3 out

   

3/21

 

Lecture 11 (cont.): Ensemble Learning

  • Bagging and Random Forests (Review)
  • Boosting

Readings:

  • Witten & Frank       Chapter 8
  • Hastie              Chapter 10.1, 15, 16
  • ISL-with R             Chapter 8.2

 

 

 

 


PART III: UNSUPERVISED AND SEMI-SUPERVISED LEARNING

   

 

3/23

 

3/28

Lecture 12: Clustering

  • Distance functions
  • Hierarchical clustering
  • k-means clustering
  • Kernel k-means clustering
  • k-medians clustering
  • Mixture models
  • The EM algorithm

Readings:

  • Witten & Frank        Chapter 6.8
  • ISLR (James, Witten, Hastie, Tibshirani)      Chapter 10.3
  • Provost & Fawcett     Chapter 6, 12 (part)



3/30

Lecture 13: Association Rules

  • Applications
  • Frequent itemsets
  • Association rule generation
  • Interesting patterns

Readings:

  • Witten & Frank        Chapter 4.5, 6.3
  • Provost & Fawcett     Chapter 12
  • Aggarwal             Chapter 4, 5.4

 

4/4

 

4/6

Lecture 14: Outlier Analysis

  • Definition and types of outliers
  • Challenges
  • Different types of detection techniques
    • Clustering, depth, and distance based techniques
    • Density-based techniques: LOF and LOCI
  • Ensemble methods: feature bagging, iForest
  • High-dimensional approaches

Readings:

  • Witten & Frank        Chapter 7.5
  • Aggarwal             Chapter 8, 9.4, 9.5

   

 

4/11

 

Lecture 15: Semi-supervised Learning

  • Assumptions (smoothness, cluster, manifold)
  • Semi-supervised learning
    • Self-training
    • Generative methods
    • Graph-based methods
    • Co-training

Readings:

Project midway report due

HW3 due
HW4 out


PART IV: LEARNING WITH OTHER DATA TYPES


4/13

Lecture 16: Unstructured Data: ML for Text

  • Representing text
  • Named entity extraction
  • Novelty and first-story detection
  • Topic models
  • Applications

Readings:

  • Witten & Frank        Chapter 9.5, 9.6
  • Provost & Fawcett     Chapter 10
  • Aggarwal             Chapter 13

   

 

4/18

 

4/25

Lecture 17: Dependent Data: ML for Time Series

  • Time series preparation and similarity
  • Trends and Anomalies
  • Forecasting with ARMA, ARIMA models
    • De-trending and seasonal components
  • Change-point detection
    • Monitoring the learning process: SPC algorithm
    • CUSUM, Minimum MSE
  • Multi-variate forecasting with VAR

Readings:

  • Aggarwal         Chapter 14
  • Shalizi           Chapter 21

   

4/25

4/27

Lecture 18: Dependent Data: ML for Networks

  • Transductive learning
  • Learning in networks with and without attributes
  • Graph-regularized classification

Readings:

 

 

 

 


5/2

5/4

Project Presentations I    (today's presenters return final report on 5/4)

Project Presentations II   (today's presenters return final report on 5/2)




HW4 due


Last modified by Leman Akoglu, Mar 2017