ITCS 6156/8156: Machine Learning

Spring 2023

**Time and location:** Tue, Thu 4:00 – 5:15pm, Bioinformatics 105

Instructor & TAs: | Razvan Bunescu | Akarsh Pokkunuru |
||
---|---|---|---|---|

Office: | Woodward 210F | Zoom & Woodward 231 | ||

Office hours: | Mon, Fri 4:00 – 5:00pm | Tue, Thu 11:00 – 12:00pm | ||

Email: | razvan.bunescu @ uncc edu | apokkunu @ uncc edu |

This course will introduce fundamental concepts, techniques, and algorithms underlying the theory and practice of machine learning (ML). Major ML models and techniques that we aim to cover include: perceptrons, linear regression, logistic regression, gradient descent, Support Vector Machines, k-nearest neighbors, decision tres, ensemble methods, k-Means clustering, neural networks, and generative adversarial networks. The description of the formal properties of the algorithms will be supplemented with motivating applications in a wide range of areas including natural language processing, computer vision, bioinformatics, or music analysis.

Students are expected to be comfortable with programming in Python, data structures and algorithms (ITSC 2214), and have basic knowledge of linear algebra (MATH 2164), calculus, and statistics. Relevant background material will be made available on this website throughout the course.

- Syllabus & Introduction with Perceptrons
- Linear algebra and optimization in NumPy and SciPy
- Hand notes from lecture on Jan 24.
- Tutorials on NumPy and SciPy.
- Broadcasting explained.

- Complete NumPy examples: notebook and PDF.
- In class NumPy examples: notebook and PDF.
- Matplotlib tutorial and examples:
- A non-convex surface: visualization in 3D and code.
- Athens houses: visualization and code.

- Linear regression, ridge regression, and Lasso
- Maximum Likelihood and Maximum A Posteriori estimation: Sections 2.1 and 2.2
- Gradient Descent algorithms
- Hand notes from lectures on Feb 9 and Feb 14, and GD example.
- An overview of gradient descent optimization algorithms, Sebastian Ruder, CoRR 2016

- Kernel methods:
- Perceptrons and Kernels
- Hand notes from lectures on Feb 16 and Feb 21.
- Large Margin Classification Using the Perceptron Algorithm, Yoav Freund and Robert E. Schapire, Machine Learning 1999
- New ranking algorithms for parsing and tagging: Kernels over Discrete Structures, and the Voted Perceptron, Michael Collins and Nigel Duffy, ACL 2002

- Support Vector Machines
- Hand notes from lectures on Mar 7, Mar 9, and Mar 14.
- LibSVM applet demonstrating SVMs.

- Gaussian Processes

- Perceptrons and Kernels
- Logistic regression
- Machine learning and optimization in PyTorch
- Hand notes from lectures on Mar 21 and Mar 23.
- PyTorch tutorials.
- PyTorch examples and linear regression.

- Feed-Forward Neural Networks and Backpropagation
- Hand notes from lectures on Mar 23, Mar 28, and Mar 30.
- Andrew Ng's introduction to deep learning from 2012 IPAM Graduate Summer School: video and PDF slides
- Andrej Karpathy: "Yes you should understand backprop"

- Representation Learning I:
- Auto-encoders
- PCA, PCA whitening, and ZCA whitening
- k-Means clustering
- Hand notes from lecture on Apr 14.
- Clustering algorithms in scikit-learn.

- Word Embeddings
- Hand notes from lecture on Apr 20.

- Convolutional Neural Networks
- Hand notes from lecture on Apr 14, Apr 18, and Apr 20.
- Andrej Karpathy's notes on CS231n: Convolutional Neural Networks for Visual Recognition.

- Recurrent Neural Networks and Attention
- Transformer: Self-Attention Networks
- Hand notes from lecture on Apr 27.
- Jay Alammar's Illustrated Transformer
- HuggingFace course section on How do Transformers work?
- Chapter 9 (section 7) in J & M on Deep Learning Architectures for Sequence Processing

- Nearest Neighbor methods
- Decision Trees
- Classification And Regression Trees (CART), slides by Alexandra Chouldechova @ CMU.
- Decision trees in scikit-learn.

- Bias-Variance Decomposition, Bagging, and Boosting, slides by Tom Dietterich @ Oregon State.
- Ensemble methods in scikit-learn.

- Classification And Regression Trees (CART), slides by Alexandra Chouldechova @ CMU.
- Ensemble Methods
- Boosted decistion trees and Random Forests

- Assignment 0 on background material:
- Assignment 1 on Perceptrons:
- Assignment 2 on Linear regression (normal equations):
- Assignment 3 on Linear regression (gradient descent):
- Assignment 4 on Kernel Perceptrons and SVMs.
- Theory
- Implementation: (1) Kernel Perceptrons, (2) Text Classification, and (3) Digit Recognition.

- Assignment 5 on Logistic regression in NumPy and PyTorch.
- Theory
- Implementation: (1) LR in Numpy and (2) LR in Pytorch.

- Assignment 6 on Neural Networks in NumPy and PyTorch.
- Theory
- Implementation: (1) NNs in Numpy and (2) NNs in Pytorch.

- Assignment 7 on PCA and Auto-Encoders
- Assignment 8 on CNNs for digit classification
- Skeleton code and data

- Assignment 9 on RNNs for sentiment analysis

- Tips for choosing a project topic:
- Project report guidelines

- A Course in Machine Learning by Hal Daume III
- Machine Learning by Tom Mitchell. McGraw Hill, 1997
- Machine Learning @ Coursera video lectures and exercises.
- Machine Learning @ Stanford video lectures and exercises.
- Deep Learning course sequence @ Coursera.
- Dive into Deep Learning by Zhang, Lipton, Li, and Smola. Amazon, 2019.

- Python programming:
- Probability and statistics:
- Basic probability theory (pp. 12-19) in Pattern Recognition and Machine Learning.
- Chapter 3 in DL textbook on Probability and Information Theory.
- Chapters 1-5 in Probability & Statistics for Engineers & Scientists.
- Nathaniel E. Helwig's Introduction to Probability Theory
- Statistical Inference book, Casella and Berger, 2001.
- Seeing Theory: A visual introduction to probability and statistics, Kunin et al., 2018.

- Linear Algebra:
- Chapter 2 in DL textbook on Linear Algebra.
- Chapter 2 on Linear Algebra in Mathematics for Machine Learning.
- Inderjit Dhillon's Linear Algebra Background
- Gilbert Strang's Introduction to Linear Algebra
- Petersen et al.'s The Matrix Cookbook
- Mike Brookes' Matrix Reference Manual

- Calculus:
- Basic properties for derivatives, integrals, exponentials, and logarithms.
- Chapter 4.3 in DL textbook on Numerical Computation.
- Gilbert Strang's Calculus texbook.

- PyTorch Deep learning in Python
- PyTorch tutorial from University of Amsterdam.
- Scikit-learn Machine learning in Python
- Hugging Face Transformers