Session I: Data Science & Statistical Learning

Topics

• The Statistical Learning Framework

• Preprocessing of Data

• Linear and Ridge Regression

Workbook with example of one-dimensional function: wave_regressions.ipynb, wave_regressions.py
Worked out notebooks: wave_regressions_sol.ipynb, wave_regressions_sol.py

• Sparse Regression

Workbook with regression on Boston housing data: boston_regression.ipynb, boston_regression.py
Worked out notebooks: boston_regression_sol.ipynb, boston_regression_sol.py

Notes of session

• PDF

References

These are further recommended readings for a deeper dive into the topics we covered:

Data Science as a science

• John Tukey’s 1962 paper on “The Future of Data Analysis”
• David Donoho’s 2015 paper on “50 Years of Data Science”

Learning Theory

• Chapters 1,2 and 5 of Shai Shalev-Shwartz and Shai Ben-David. “Understanding machine learning: From theory to algorithms”, Cambridge University Press, 2014 (PDF available).
  
• Sections 2.1-2.2 of Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. “An Introduction to Statistical Learning”, Vol. 112. Springer, 2013 (PDF available)

Preprocessing of data

• Chapter 7 of Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. “An Introduction to Statistical Learning”, Vol. 112. Springer, 2013 (PDF available)
• Chapter 3 and 4 of Andreas C. Müller, and Sarah Guido. “Introduction to machine learning with Python: a guide for data scientists”, O’Reilly, 2016. This reference focusses more on the practical aspects.

Linear, Ridge and Sparse Regression

• Chapter 6 of Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. “An Introduction to Statistical Learning”, Vol. 112. Springer, 2013 (PDF available)
• Chapter 9 of Shai Shalev-Shwartz and Shai Ben-David. “Understanding machine learning: From theory to algorithms”, Cambridge University Press, 2014 (PDF available).
• Chapter 10 of Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. “Foundations of Machine Learning”, MIT press, 2018 (PDF available).
  A very exhaustive reference for everything regarding sparse regression is the book:
• Trevor Hastie, Robert Tibshirani, and Martin Wainwright. "Statistical Learning with Sparsity: the LASSO and Generalizations, CRC press, 2015 (PDF available)

Model Validation, Cross-Validation

• Chapter 11 of Shai Shalev-Shwartz and Shai Ben-David. “Understanding machine learning: From theory to algorithms”, Cambridge University Press, 2014 (PDF available)
• Chapter 7 of Trevor Hastie, Robert Tibshirani, and Jerome Friedman, “The Elements of Statistical Learning: Data Mining, Inference, and Prediction”, Springer, 2009 (PDF available)
• Chapter 5 of Müller, Andreas C. and Sarah Guido. “Introduction to machine learning with Python: a guide for data scientists”, O’Reilly, 2016.