Optimization for Machine Learning and Data Science (Topics in Computer Science)

ITCS 6010/8010 - Fall 2023

In Fall 2023, we are offering a course on optimization targeted towards applications in machine learning and in data science. The targeted audience for this course are master’s and Ph.D. students in computer science, mathematics, data science and electrical engineering.

Time: Mondays & Wednesdays, 2:30 pm - 3:45 pm
Location UPDATED: Aug 21 & 23 on Zoom at https://charlotte-edu.zoom.us/j/95526959172, from Week 2 in person.
Mondays in Woodward 140, Wednesdays in CHHS 159 (College of Health and Human Services Bldg 159)

Course Summary

Optimization formulations and algorithms play a central role in modern data science, statistics and machine learning. This graduate-level course is an introduction to the basics of continuous optimization, with an emphasis on techniques that are most relevant to data science and machine learning and are amenable to large-scale implementations. We also discuss basic convergence properties and computational aspects of relevant optimization techniques. The relationship and applications of optimization to common machine learning models such as (generalized) principal component analysis, sparse and logistic regression, regularized empirical risk minimization and the training of deep neural networks is another main focus of this course. In addition to theory and algorithms, common optimization modeling software and computer solvers will be introduced to students.

Learning Objectives:

  • Learn how to apply optimization algorithms to fit, train and evaluate machine learning models
  • Understand the fundamental statistical and computational challenges in models motivating optimization
  • Identify and understand the differences between linear, convex, non-convex, smooth and non-smooth optimization problems to select appropriate solution algorithms
  • Develop a solid understanding of the basics of continuous optimization
  • Understand the trade-offs and limitations of different numerical optimization methods
  • Learn how to work with optimization modeling software and solvers.
  • Understand research papers at the intersection of optimization and data science/ machine learning
  • Gain practical experience implementing state-of-the-art customized optimization algorithms

Syllabus

OptMLDS_Syllabus

Tentative list of course topics

Data Models and Applications in Machine Learning

  • Unsupervised learning: Principal component analysis, $k$-means clustering
  • Supervised learning: Logistic regression, sparse regression, matrix completion, recommender systems, computer vision
  • Training of deep neural networks
  • Optimal transport: Image matching, shape registration

Algorithms and Theory of Continuous Optimization

  • Basics of convex analysis & optimization
  • Multidimensional calculus: gradients, optimality conditions
  • Gradient and descent methods
  • Accelerated gradient
  • Stochastic gradient descent
  • Coordinate descent
  • Proximal gradient
  • Subgradient methods
  • Second-order methods
  • Variable metric techniques
  • Conditional gradient
  • Minimax Optimization