ITCS 4111/5111: Introduction to Natural Language Processing
Fall 2023
Time and Location: Tue, Thu 2:30 – 3:45pm, HHS 376
Instructor & TAs: |
|
Razvan Bunescu |
|
Sivani Josyula |
|
Manogna Chennuru |
Office: |
|
Woodward 210F |
|
Zoom & Burson 239B |
|
Zoom & Burson 239B |
Office hours: |
|
Tue, Thu 4:00 – 5:00pm |
|
Tue, Wed 10:00 – 11:00am |
|
Thu, Fri 10:00 – 11:00am |
Email: |
|
rbunescu @ charlotte edu |
|
sjosyul2 @ charlotte edu |
|
mchennu2 @ charlotte edu |
Recommended Texts (PDF available online):
Speech and Language Processing (3rd edition draft), by Daniel Juraksfy and James E. Martin. 2023.
Natural Language Processing, by Jacob Eisenstein. 2019.
Course description:
Natural Language Processing (NLP) is a branch of Artificial Intelligence concerned with developing computer systems that can analyze or generate natural language. This course will introduce fundamental linguistic analysis tasks, including tokenization, word representations, text classification, syntactic and semantic parsing, and coreference resolution. Machine learning (ML) based techniques will be introduced, ranging from Naive Bayes and logistic regression to Transformer-based language models, which will be used in a number of NLP applications such as sentiment classification, information extraction, or question answering. Overall, the aim of this course is to equip students with an array techniques and tools that they can use to solve known NLP tasks, as well as new types of NLP problems.
Prerequisites:
Students are expected to be comfortable with programming in Python, data structures and algorithms (ITSC 2214), and have basic knowledge of linear algebra (MATH 2164), statistics, and formal languages (regular and context free grammars). Knowledge of machine learning will be very useful, though not strictly necessary. Relevant background material will be made available on this website throughout the course.
Lecture notes:
- Syllabus & Introduction
- Python for programming, linear algebra, and visualization
- Tokenization: From text to sentences and tokens
- Regular expressions
- Strengths and Weaknesses of Language Models
- Application development using GPT and Llama-2 through the Chat completion API
- Text classification using Naive Bayes
- Hand notes from lecture on Sep 26: LM inference one and two; Naive Bayes example; and Bayes rule.
- Hand notes from lecture on Sep 28: Text classification task; and Naive Bayes model.
- Logistic regression
- Biases vs. fairness and rationality in NLP models
- Manual annotation for NLP
- Brat rapid annotation tool
- Word meanings; Sparse vs. dense representations of words
- Hand notes from lecture on Oct 31: one and two.
- Hand notes from lecture on Nov 2.
- Hand notes from lecture on Nov 7: one and two.
- Visualization of word embeddings with TensorFlow Embedding Projector.
- N-grams and Neural models for Language Modeling and Sequence Processing
- Machine translation, Sequence-to-sequence models and Attention
- Transformer: Self-Attention Networks
- Language Models: Pretraining and Fine-tuning
- Language Models: Prompting, In-context Learning, Chain of Thought, Instruct Tuning, RLHF
- Coreference resolution
- Syntax, constituency parsing, dependency parsing
Homework assignments1,2:
1
The code for assignment 7 is based on an assignment from the CS224n course at Stanford on NLP with Deep Learning.
Final project:
Background reading materials:
- Python programming:
- Probability and statistics:
- Linear Algebra:
- Calculus:
Supplemental readings:
- Training language models to follow instructions with human feedback, Ouyang et al., NeurIPS 2022
- Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task, Li et al., ICLR 2023.
- Theory of Mind May Have Spontaneously Emerged in Large Language Models, Michal Kosinski, Stanford 2023.
- Are Emergent Abilities of Large Language Models a Mirage?, Schaeffer et al., DeployableGenerativeAI 2023.
- What’s the Meaning of Superhuman Performance in Today’s NLU?, Tedeschi et al., ACL 2023.
Tools and packages:
- Natural language processing:
- Machine learning: