CS 6830: Machine Learning
Fall 2016

---

Time and Location: Mon, Wed, Fri 10:45 – 11:40am, ARC 101
Instructor: Razvan Bunescu
Office: Stocker 341
Office Hours: Mon, Fri 9:30 – 10:00am, or by email appointment
Email: bunescu @ ohio edu

Textbook:

Pattern Recognition and Machine Learning by Christopher Bishop. Springer, 2007.

Recommended Supplementary Texts:

Machine Learning by Tom Mitchell. McGraw Hill, 1997

Pattern Classification by Richard O. Duda, Peter E. Hart, & David G. Stork. Wiley-Interscience, 2001

The Elements of Statistical Learning: Data Mining, Inference, and Prediction by T. Hastie, R. Tibshirani, & J. H. Friedman. Springer Verlag, 2009

Course description:
Machine Learning is concerned with the design and analysis of algorithms that enable computers to automatically find patterns in the data. This introductory course will give an overview of the main concepts, techniques and algorithms that are relevant for the theory and practice of machine learning. The course will cover the fundamental topics of classification, regression and clustering, starting with simple learning models such as perceptrons, decision trees and logistic regression, and ending with more advanced models including Support Vector Machines, Conditional Random Fields and Bayesian Networks. The description of the formal properties of the algorithms will be supplemented with motivating applications in a wide range of areas including natural language processing, computer vision, bioinformatics and music analysis.

Prerequisites:
The students are expected to be comfortable with programming and to exhibit a basic level of mathematical dexterity. Relevant background material in linear algebra, probability theory and information theory will be made available during the course.

Lecture notes:

1. Syllabus & Introduction
2. Regression with Linear Models
   • An overview of gradient descent optimization algorithms, Sebastian Ruder, 2016
   • Animations of Gradient Descent Algorithms, Alec Radford, 2014
3. Fisher Linear Discriminant
4. Perceptrons and Kernels
   • Large Margin Classification Using the Perceptron Algorithm, Yoav Freund and Robert E. Schapire, Machine Learning 1999
5. Support Vector Machines
   • An Introduction to Support Vector Machines and Other Kernel-based Learning Methods, Nello Cristianini and John Shawe-Taylor [Available online through library.ohiou.edu]
   • Support Vector Machines [Trends and Controversies] , Marti Hearst, Susan Dumais, Edgar Osuna, John Platt, Bernhard Scholkopf, IEEE Intelligent Systems, 13(4), 1998
   • A Tutorial on Support Vector Machines for Pattern Recognition, Christopher J. C. Burges, Data Mining and Knowledge Discovery 1998
6. Nearest Neighbor Methods
7. Feature Selection
   • JMLR Special Issue on Variable and Feature Selection , Isabelle Guyon and Andre Elisseeff (editors), JMLR, 2003
   • Gene Selection for Cancer Classification using Support Vector Machines for Pattern Recognition, Guyon, Weston, and Barnhill, Machine Learning, v46, 2002
8. Decision Trees
9. Naive Bayes
   • Naive Bayes and Logistic Regression, new chapter in Tom Mitchell, Machine Learning, 2005
10. Logistic Regression
    • A Maximum Entropy Approach to Natural Language Processing, Adam Berger, Vincent Della Pietra and Stephen A. Della Pietra, Computational Linguistics, 1996
11. Hidden Markov Models
    • Markov Models, chapter 9 in Chris Manning and Hinrich Schutze, Foundations of Statistical Natural Language Processing, MIT Press. Cambridge, MA, May 1999
    • A tutorial on Hidden Markov Models and selected applications in speech recognition, Lawrence R. Rabiner, Proceedings of the IEEE 77 (2), 1989
12. Conditional Random Fields
    • An Introduction to Conditional Random Fields for Relational Learning, Charles Sutton and Andrew McCallum, 2007
13. PCA, Clustering

Homework Assignments:

• Assignment 1
  • Data
• Assignment 2
• Assignment 3
• Assignment 4

Final Project:

• Project suggestions
• Project report guidelines

Other online reading materials:

• James H. Martin's Introduction to probabilities
• Jason Eisner's equestrian Introduction to probabilities
• Inderjit Dhillon's Linear Algebra Background
• Mike Brookes' Matrix Reference Manual
• Strang's Video Lectures on Linear Algebra
• Convex Optimization, Stephen Boyd and Lieven Vandenberghe, Cambridge University Press 2004

Machine learning software:

• Weka Data Mining Software in Java
• scikit-learn Machine Learning in Python
• SVM^light Implementation of SVMs in C
• LIBSVM Implementation of SVMs in C++ and Java
• MALLET Java implementations of logistic regression, HMMs, linear chain CRFs, and other ML models.
• LibSVM applet demonstrating SVMs.
• k-Nearest Neighbor short animated video, by Antal van den Bosch