ITCS 6156/8156: Machine Learning

ITCS 6156/8156: Machine Learning
Spring 2023

Time and location: Tue, Thu 4:00 – 5:15pm, Bioinformatics 105

Instructor & TAs: Razvan Bunescu Akarsh Pokkunuru

Office: Woodward 210F Zoom & Woodward 231

Office hours: Mon, Fri 4:00 – 5:00pm Tue, Thu 11:00 – 12:00pm

Email: razvan.bunescu @ uncc edu apokkunu @ uncc edu

Instructor & TAs:	Razvan Bunescu	Akarsh Pokkunuru
Office:	Woodward 210F	Zoom & Woodward 231
Office hours:	Mon, Fri 4:00 – 5:00pm	Tue, Thu 11:00 – 12:00pm
Email:	razvan.bunescu @ uncc edu	apokkunu @ uncc edu

Recommended free texts:

Pattern Recognition and Machine Learning by Christopher Bishop. Springer, 2007.

The Elements of Statistical Learning: Data Mining, Inference, and Prediction by T. Hastie, R. Tibshirani, & J. H. Friedman. Springer Verlag, 2009 - 2017

Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville. MIT Press, 2016.

Mathematics for Machine Learning by Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong. Cambridge University Press, 2020.

Recommended free course on Mathematics for Deep Learning and Data Science:

Linear Algebra module.

Calculus module.

Probability and Statistics module.

Course description:
This course will introduce fundamental concepts, techniques, and algorithms underlying the theory and practice of machine learning (ML). Major ML models and techniques that we aim to cover include: perceptrons, linear regression, logistic regression, gradient descent, Support Vector Machines, k-nearest neighbors, decision tres, ensemble methods, k-Means clustering, neural networks, and generative adversarial networks. The description of the formal properties of the algorithms will be supplemented with motivating applications in a wide range of areas including natural language processing, computer vision, bioinformatics, or music analysis.

Prerequisites:
Students are expected to be comfortable with programming in Python, data structures and algorithms (ITSC 2214), and have basic knowledge of linear algebra (MATH 2164), calculus, and statistics. Relevant background material will be made available on this website throughout the course.

Lecture notes:

Syllabus & Introduction with Perceptrons
- Hand notes from lectures on Jan 12 and Jan 17.
Linear algebra and optimization in NumPy and SciPy
- Hand notes from lecture on Jan 24.
- Tutorials on NumPy and SciPy.
  - Broadcasting explained.
- Complete NumPy examples: notebook and PDF.
- In class NumPy examples: notebook and PDF.
- Matplotlib tutorial and examples:
  - A non-convex surface: visualization in 3D and code.
  - Athens houses: visualization and code.
Linear regression, ridge regression, and Lasso
- Hand notes from lectures on Jan 26, Jan 31 and Feb 2.
Maximum Likelihood and Maximum A Posteriori estimation: Sections 2.1 and 2.2
- Hand notes from lectures on Feb 2 and Feb 7.
Gradient Descent algorithms
- Hand notes from lectures on Feb 9 and Feb 14, and GD example.
- An overview of gradient descent optimization algorithms, Sebastian Ruder, CoRR 2016
Kernel methods:
- Perceptrons and Kernels
  - Hand notes from lectures on Feb 16 and Feb 21.
  - Large Margin Classification Using the Perceptron Algorithm, Yoav Freund and Robert E. Schapire, Machine Learning 1999
  - New ranking algorithms for parsing and tagging: Kernels over Discrete Structures, and the Voted Perceptron, Michael Collins and Nigel Duffy, ACL 2002
- Support Vector Machines
  - Hand notes from lectures on Mar 7, Mar 9, and Mar 14.
  - LibSVM applet demonstrating SVMs.
- Gaussian Processes
Logistic regression
- Hand notes from lectures on Mar 14 and Mar 16.
Machine learning and optimization in PyTorch
- Hand notes from lectures on Mar 21 and Mar 23.
- PyTorch tutorials.
- PyTorch examples and linear regression.
Feed-Forward Neural Networks and Backpropagation
- Hand notes from lectures on Mar 23, Mar 28, and Mar 30.
- Andrew Ng's introduction to deep learning from 2012 IPAM Graduate Summer School: video and PDF slides
- Andrej Karpathy: "Yes you should understand backprop"
Representation Learning I:
- Auto-encoders
  - Hand notes from lectures on Mar 30 and Apr 4.
- PCA, PCA whitening, and ZCA whitening
- k-Means clustering
  - Hand notes from lecture on Apr 14.
  - Clustering algorithms in scikit-learn.
- Word Embeddings
  - Hand notes from lecture on Apr 20.
Convolutional Neural Networks
- Hand notes from lecture on Apr 14, Apr 18, and Apr 20.
- Andrej Karpathy's notes on CS231n: Convolutional Neural Networks for Visual Recognition.
Recurrent Neural Networks and Attention
- Hand notes from lectures on Apr 25 and Apr 27.
Transformer: Self-Attention Networks
- Hand notes from lecture on Apr 27.
- Jay Alammar's Illustrated Transformer
- HuggingFace course section on How do Transformers work?
- Chapter 9 (section 7) in J & M on Deep Learning Architectures for Sequence Processing
Nearest Neighbor methods
- Feature Selection
Decision Trees
- Classification And Regression Trees (CART), slides by Alexandra Chouldechova @ CMU.
  - Decision trees in scikit-learn.
- Bias-Variance Decomposition, Bagging, and Boosting, slides by Tom Dietterich @ Oregon State.
  - Ensemble methods in scikit-learn.
Ensemble Methods
- Boosted decistion trees and Random Forests

Homework assignments^1,2:

Assignment 0 on background material:
- Probability theory, calculus, linear algebra
- Python
  - Skeleton code and data
Assignment 1 on Perceptrons:
- Theory
- Implementation using NumPy
  - Skeleton code and data
Assignment 2 on Linear regression (normal equations):
- Theory
- Implementation
  - Skeleton code and data
Assignment 3 on Linear regression (gradient descent):
- Theory
- Implementation
  - Skeleton code and data
Assignment 4 on Kernel Perceptrons and SVMs.
- Theory
- Implementation: (1) Kernel Perceptrons, (2) Text Classification, and (3) Digit Recognition.
  - Skeleton code and data
Assignment 5 on Logistic regression in NumPy and PyTorch.
- Theory
- Implementation: (1) LR in Numpy and (2) LR in Pytorch.
  - Skeleton code and data
Assignment 6 on Neural Networks in NumPy and PyTorch.
- Theory
- Implementation: (1) NNs in Numpy and (2) NNs in Pytorch.
  - Skeleton code and data
Assignment 7 on PCA and Auto-Encoders
- Theory
- Implementation
  - Skeleton code and data
Assignment 8 on CNNs for digit classification
- Skeleton code and data
Assignment 9 on RNNs for sentiment analysis
- Skeleton code and data

Paper presentation:

Final project:

Tips for choosing a project topic:
- UT Austin
- Stanford
Project report guidelines

Supplemental ML materials:

A Course in Machine Learning by Hal Daume III
Machine Learning by Tom Mitchell. McGraw Hill, 1997
Machine Learning @ Coursera video lectures and exercises.
Machine Learning @ Stanford video lectures and exercises.
Deep Learning course sequence @ Coursera.
Dive into Deep Learning by Zhang, Lipton, Li, and Smola. Amazon, 2019.

Background reading materials:

Python programming:
- Python for Everybody
- Python lecture
Probability and statistics:
- Basic probability theory (pp. 12-19) in Pattern Recognition and Machine Learning.
- Chapter 3 in DL textbook on Probability and Information Theory.
- Chapters 1-5 in Probability & Statistics for Engineers & Scientists.
- Nathaniel E. Helwig's Introduction to Probability Theory
- Statistical Inference book, Casella and Berger, 2001.
- Seeing Theory: A visual introduction to probability and statistics, Kunin et al., 2018.
Linear Algebra:
- Chapter 2 in DL textbook on Linear Algebra.
- Chapter 2 on Linear Algebra in Mathematics for Machine Learning.
- Inderjit Dhillon's Linear Algebra Background
- Gilbert Strang's Introduction to Linear Algebra
- Petersen et al.'s The Matrix Cookbook
- Mike Brookes' Matrix Reference Manual
Calculus:
- Basic properties for derivatives, integrals, exponentials, and logarithms.
- Chapter 4.3 in DL textbook on Numerical Computation.
- Gilbert Strang's Calculus texbook.

Machine learning software:

PyTorch Deep learning in Python

PyTorch tutorial from University of Amsterdam.

Scikit-learn Machine learning in Python
Hugging Face Transformers