ITCS 4101: Introduction to Natural Language Processing
Fall 2024

---

Time and Location: Tue, Thu 1:00 – 2:15pm, Woodward 140

Instructor & TA:		Razvan Bunescu		Justin Smith
Office:		Woodward 410G		Burson 239B
Office hours:		Tue, Thu 5:30 – 6:30pm		Wed, Fri 12:00 – 1:00pm
Email:		razvan.bunescu @ charlotte edu		jsmit840 @ charlotte edu

Recommended Texts (PDF available online):

Speech and Language Processing (3rd edition draft), by Daniel Juraksfy and James E. Martin. 2024.

Natural Language Processing, by Jacob Eisenstein. 2019.

Course description:
Natural Language Processing (NLP) is a subdomain of Artificial Intelligence whose focus is on the development and study of computer systems that process or generate natural language. This course will introduce fundamental linguistic analysis tasks, including tokenization, word representations, syntactic parsing, semantic parsing, and coreference resolution. Machine learning (ML) based techniques will be introduced, ranging from Naive Bayes and logistic regression to Transformer-based language models, which will be used in a number of NLP applications such as sentiment classification, information extraction, or question answering. Overall, the aim of this course is to equip students with an array techniques and tools that they can use to solve known NLP tasks, as well as new types of NLP problems.

Prerequisites:
Students are expected to be comfortable with programming in Python, data structures and algorithms (ITSC 2214), and have basic knowledge of linear algebra (MATH 2164) and statistics (STAT 2122). Knowledge of machine learning will be very useful, though not strictly necessary. Relevant background material will be made available on this website throughout the course.

Lecture notes:

1. Syllabus & Introduction
2. Python for programming, linear algebra, and visualization
   • Python lecture
   • Examples from lecture on Aug 23
   • Python tutorial
   • NumPy tutorial
   • Matplotlib tutorial
3. Tokenization: From text to sentences and tokens
   • Hand notes from lecture on Sep 3: paper and whiteboard.
   • Examples from lecture on Sep 3.
4. Regular expressions
   • Examples from lectures on Sep 5 and 10: pdf and ipynb
   • More examples of regular expressions: pdf and ipynb
   • Regular expressions in Python with re
   • Regular expressions in UNIX with sed
5. Strengths and Weaknesses of Language Models
   • Durably reducing conspiracy beliefs through dialogues with AI, Costello et al., Science, September 2024.
   • AI can help humans find common ground in democratic deliberation, Tessler et al., Science, October 2024.
6. Application development using LLMs through the Chat completion API
   • Notebooks with examples using GPT, Llama, and Gemini.
7. Building LLM-powered applications with LangChain and AutoGen
   • Multi-agent examples and Jupyter notebooks from lecture on Sep 19.
   • Compound AI Systems, Zaharia et al., Feb 18, 2024.
8. Text classification using Naive Bayes
   • Notes from lecture on Sep 24: one, two and three.
   • Notes from lecture on Sep 26: one, two and three.
   • Notes from lecture on Oct 1.
9. Logistic regression
   • Chapter 5 on logistic regression from the textbook.
   • Notes and sampling code from lecture on Oct 8.
   • Notes from lectures on Oct 10, Oct 17, and Oct 22.
10. Biases vs. fairness and rationality in NLP models
    • Hand notes from lecture on Oct 24: one and two.
    • A Survey on Bias and Fairness in Machine Learning, Mehrabi et al., ACM Computing Surveys, 2021.
    • Benchmarking Cognitive Biases in Large Language Models as Evaluators, Koo et al., Findings of ACL, 2024
    • Challenging the appearance of machine intelligence: Cognitive bias in LLMs and Best Practices for Adoption, Talboy and Fuller, CoRR 2023.
    • Capturing Failures of Large Language Models via Human Cognitive Biases, Jones and Steinhardt, NeurIPS 2022.
11. Manual annotation for NLP
    • Brat rapid annotation tool
12. Word meanings; Sparse vs. dense representations of words
    • Hand notes from lecture on Nov 5: one, two, and three.
    • Hand notes from lecture on Nov 12.
    • Hand notes from lecture on Nov 14: one, two, and three.
    • Chapter 6 on vector semantics and embeddings from the textbook.
    • Visualization of word embeddings with TensorFlow Embedding Projector.
13. N-grams and Neural models for Language Modeling and Sequence Processing
    • Hand notes from lecture on Nov 19.
    • Hand notes from lecture on Nov 21: one, two, and three, and four.
    • Chapter 7 on neural networks and neural language models from the textbook.
    • Chapter 8 on RNNs and LSTMs from the textbook.
    • All Our N-gram are Belong to You
14. Machine translation, Sequence-to-sequence models and Attention
15. Transformers and Pretrained Language Models
    • Chapter 9 in J & M on Deep Learning Architectures for Sequence Processing.
    • Chapter 10 in J & M on Transformers and Pretrained Language Models.
    • The original Transformer paper Attention is all you need, by Vaswani et al., NIPS 2017.
16. Large Language Models: Pretraining, Fine-tuning, In-context Learning, Chain of Thought, Instruct Tuning, RLHF

Homework assignments¹:

• Assignment 0 on Python lists and strings.
  • Skeleton code.
• Assignment 1 on Word distributions.
  • Skeleton code and data.
• Assignment 2 on Wikipedia processing with regular expressions.
  • Skeleton code and data.
• Assignment 3 on Question Answering on semi-structured restaurant data using the chat completion API with GPT, LLama, or Gemini.
  • Skeleton code and data.
• Assignment 4 on LangChain for conversational assistants.
  • Skeleton code and data.
• Assignment 5 on Sentiment Analysis with Naive Bayes.
  • Skeleton code and data.
• Assignment 6 on Sentiment Analysis with Logistic Regression and engineered features.
  • Skeleton code and data.
• Assignment 7 on corpus acquisition, annotation, feature engineering, and evaluation.
  • Skeleton code and data.
• Assignment 8 on Vector Representations of Words.
  • Skeleton code and data.
• Assignment 9 on RNNs for Sentiment Classification.
  • Skeleton code and data.
  • Educational cluster instructions.
• Assignment 10 on Transformer-based models for NLP.
  • Skeleton code and data.

¹ The code for assignment 8 is based on an assignment from the CS224n course at Stanford on NLP with Deep Learning.

Background reading materials:

• Python programming:
  • Google's Crash Course on Python, free for UNCC students
  • W3 Schools' Python tutorial
  • Microsoft's Python for beginners
  • Python for Everybody
  • Python lecture
• Probability and statistics:
  • Basic probability theory (pp. 12-19) in Pattern Recognition and Machine Learning.
  • Chapter 3 in DL textbook on Probability and Information Theory.
  • Parts 1 and 2 in CS109: Probability for Computer Scientists course reader.
  • Statistical Inference book, Casella and Berger, 2001.
  • Seeing Theory: A visual introduction to probability and statistics, Kunin et al., 2018.
• Linear Algebra:
  • Chapter 2 in DL textbook on Linear Algebra.
  • Chapter 2 on Linear Algebra in Mathematics for Machine Learning.
  • Inderjit Dhillon's Linear Algebra Background
  • Gilbert Strang's Introduction to Linear Algebra
  • Petersen et al.'s The Matrix Cookbook
  • Mike Brookes' Matrix Reference Manual
• Calculus:
  • Basic properties for derivatives, integrals, exponentials, and logarithms.
  • Chapter 4.3 in DL textbook on Numerical Computation.
  • Gilbert Strang's Calculus texbook.

Supplemental readings:

• Training language models to follow instructions with human feedback, Ouyang et al., NeurIPS 2022
• Out-of-Distribution Detection and Selective Generation for Conditional Language Models, Ren et al., ICLR 2023
• Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task, Li et al., ICLR 2023.
• Theory of Mind May Have Spontaneously Emerged in Large Language Models, Michal Kosinski, Stanford 2023.
• Are Emergent Abilities of Large Language Models a Mirage?, Schaeffer et al., DeployableGenerativeAI 2023.
• What’s the Meaning of Superhuman Performance in Today’s NLU?, Tedeschi et al., ACL 2023.

Tools and packages:

• Natural language processing:
  • spaCy
  • brat rapid annotation tool
• Machine learning:
  • PyTorch Deep learning in Python
  • PyTorch tutorial from University of Amsterdam.
  • Scikit-learn Machine learning in Python
  • Hugging Face ML models and datasets
• Open-source platforms for LLM-based web apps:
  • Chatbot UI
  • Chainlit
  • Streamlit