ITCS 4111/5111: Introduction to Natural Language Processing

Spring 2022

Natural Language Processing (NLP) is a branch of Artificial Intelligence concerned with developing computer systems that can process or generate natural language. This course will introduce fundamental tasks in NLP, including tokenization, word representations, text classification, syntactic and semantic parsing, and coreference resolution. Machine learning (ML) based techniques will be used in a number of NLP applications such as sentiment classification, information extraction, and named entity linking. Overall, the aim of this course is to equip students with an array of tools and techniques that they can use to solve known NLP tasks, as well as new types of NLP problems.

Students are expected to be comfortable with programming in Python, data structures and algorithms (ITSC 2214), and have basic knowledge of linear algebra (MATH 2164), statistics, and formal languages (regular and context free grammars). Knowledge of machine learning will be very useful, though not strictly necessary. Relevant background material will be made available on this website throughout the course.

- Syllabus & Introduction
- Python for programming, linear algebra, and visualization
- Python lecture
- Python tutorial
- NumPy tutorial
- Matplotlib tutorial

- Tokenization: From text to sentences and tokens
- Regular expressions
- Text classification using Naive Bayes
- Logistic regression
- Hand notes from lectures on Feb 15 and 17: one, two, three, four, five, six, and GD example.
- Hand notes from lecture on Feb 22: one, two, three, four, five, and six
- Slides 1 to 13 from CS 4156 lecture on Intro to ML
- Slides 1 to 24 (LR) and slide 27 (LR + L2 regularization) from CS 4156 lecture on Logistic Regression
- Slides 1 to 21 from CS 4156 lecture on Gradient Descent

- Manual annotation for NLP
- Word meanings; Sparse vs. dense representations of words
- Sequence labeling for POS tagging and NE recognition
- Hand notes from lecture on Mar 31: one.
- Part-of-Speech Tagging Guidelines for the Penn Treebank Project, Beatrice Santorini, Technical Report, 1990
- Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics?, Christopher D. Manning, 2011
- State-of-the-art in POS tagging from NLP-progress by Sebastian Ruder

- Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs)
- N-grams and Neural models for Language Modeling and Sequence Processing
- Hand notes from lecture on Apr 14 on designing an RNN that outputs parity bits.
- All Our N-gram are Belong to You

- Machine translation, Sequence-to-sequence models and Attention
- Transformer: Self-Attention Networks
- Hand notes from lecture on Apr 26: one and two.
- Jay Alammar's Illustrated Transformer
- HuggingFace course section on How do Transformers work?
- Chapter 9 (section 7) in J & M on Deep Learning Architectures for Sequence Processing

- Energy and Policy Considerations for Deep Learning in NLP, Strubell et al., ACL 2019.
- Coreference resolution
- Stanford 224N slides on coreference, anaphora, and models.
- List of commands for using NeuralCoref in SpaCy.
- Notebook with coref examples in SpaCy.

- Syntax, generative grammars, and syntactic parsing

- The One Hundred Year Study on Artificial Intelligence (AI100), the 2021 Report
- On the Opportunities and Risks of Foundation Models

- Assignment 0 on Python lists and strings.
- Assignment 1 on Word distributions.
- Assignment 2 on Wikipedia processing with regular expressions.
- Assignment 3 on Sentiment Analysis with Naive Bayes.
- Assignment 4 on Sentiment Analysis with Logistic Regression and engineered features.
- Assignment 5 on corpus acquisition and annotation.
- Assignment 6 on Vector Representations of Words.
- Assignment 7 on CRFs for NE Recognition.
- Viterbi question can be seen here.
- Skeleton code and data.

- Assignment 8 on RNNs for Sentiment Classification.
- Skeleton code and data.
- Educational cluster instructions.

- Assignment 9 on Transformer-based models for NLP.

- Resources and guidelines for the final project
- Tips for choosing a project topic:
- Project report guidelines

- Probability and statistics:
- Nathaniel E. Helwig's Introduction to Probability Theory
- Introductory Statistics book on OpenStax.
- Statistical Inference book, Casella and Berger, 2001.

- Linear Algebra:
- Gilbert Strang's Introduction to Linear Algebra
- Strang's Video Lectures on Linear Algebra
- Inderjit Dhillon's Linear Algebra Background
- Petersen et al.'s The Matrix Cookbook
- Mike Brookes' Matrix Reference Manual

- Natural language processing:
- spaCy
- AllenNLP
- Spark NLP
- Stanza
- brat rapid annotation tool
- Hugging Face Transformers

- Machine learning: