Schedule

• Course Introduction
• Logistics

• Word embeddings and vector semantics

• Jurafsky & Martin Chapter 6

• Word embeddings and vector semantics (cont.)
• Sparse representations
• Dense representations

• Jurafsky & Martin Chapter 6

• Distributed Representation of Words and Phrases and their Compositionality (Mikolov et al., 2013) [link]
• Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013) [link]
• Word2vec Explained- deriving Mikolov et al.'s negative-sampling word-embedding method (Goldberg and Levy, 2014) [link]

HW 1 out

• Deriving the gradient of Word2vec
• Evaluation of word embeddings

• Jurafsky & Martin Chapter 6
• Distributed Representation of Words and Phrases and their Compositionality (Mikolov et al., 2013) [link]

• N-Gram Language Models
• Smoothing
• Evaluation of Language Models

• Jurafsky & Martin Chapter 7

• Neural network basics
• Autograd

• The Matrix Calculus You Need For Deep Learning (Terence Parr and Jeremy Howard) [link]
• Little book of deep learning (François Fleuret) - Ch 3, 4

• Auto Grad
• Building blocks of Deep Learning for Language Modeling
• CNNs

• Goldberg Chapter 9

• CNNs (contd.)
• RNNs
• Task specific neural network architectures
• Machine translation

• Understanding LSTM Networks (Christopher Olah) [link]
• Eisenstein, Chapter 18

• Neural Machine Translation and Sequence-to-sequence Models- A Tutorial (Graham Neubig) [link]

• RNNs (contd.)
• Training sequence models
• Machine translation (contd.)

• Statistical Machine Translation (Koehn) [link]
• Neural Machine Translation and Sequence-to-sequence Models- A Tutorial (Graham Neubig) [link]
• Learning to Align and Translate with Attention (Bahdanau et al., 2015) [link]
• Luong et al. (2015) Effective Approaches to Attention-based Neural Machine Translation [link]
• Attention is All You Need (Vaswani et al., 2017) [link]
• Illustrated Transformer [link]

Project teams due on 02/08

HW1 due

• Attention
• Transformers

• Neural Machine Translation and Sequence-to-sequence Models- A Tutorial (Graham Neubig) [link]
• Learning to Align and Translate with Attention (Bahdanau et al., 2015) [link]
• Luong et al. (2015) Effective Approaches to Attention-based Neural Machine Translation [link]
• Attention is All You Need (Vaswani et al., 2017) [link]
• Illustrated Transformer [link]

HW 2 out

• Transformers (contd.)
• Language modeling with Transformers
• Transfer Learning

• Illustrated Transformer [link]
• Attention is All You Need (Vaswani et al., 2017) [link]
• The Annotated Transformer (Harvard NLP) [link]
• GPT-2 (Radford et al., 2019) [link]

• Transfer Learning (contd.)
• Objective functions for pre-training
• Encoder-decoder pretrained models
• Architecture and pretraining objectives

• The Illustrated BERT, ELMo, and co. (Jay Alammar) [link]
• BERT- Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2018) [link]
• GPT-2 (Radford et al., 2019) [link]
• T5- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2020) [link]
• BART- Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (Lewis et al., 2019) [link]
• What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (Wang et al, 2022) [link]

• Decoding and generation
• Large language models and impact of scale
• In-context learning and prompting

• The Curious Case of Neural Text Degeneration (Holtzman et al., 2019) [link]
• How to generate text- using different decoding methods for language generation with Transformers [link]
• Scaling Laws for Neural Language Models (Kaplan et al., 2020) [link]
• Training Compute-Optimal Large Language Models (Hoffmann et al., 2022) [link]
• GPT3 paper - Language Models are Few-Shot Learners (Brown et al., 2020) [link]

• Guest Lecture 1: TBD

HW 2 due

• Post-training
• Supervised Finetuning
• Instruction Following

• Multitask Prompted Training Enables Zero-Shot Task Generalization (Sanh et al., 2021) [link]
• Scaling Instruction-Finetuned Language Models (Chung et al., 2022) [link]
• Are Emergent Abilities of Large Language Models a Mirage? (Sha et al., 2023) [link]
• Emergent Abilities of Large Language Models (Wei et al., 2022) [link]

Project proposals due 03/06

• Guest Lecture 2: TBD

• Post-training
• Reinforcement learning from Human Feedback
• Alignment

• Training language models to follow instructions with human feedback (Ouyang et al., 2022) [link]
• Fine-Tuning Language Models from Human Preferences (Ziegler et al., 2019) [link]
• Direct Preference Optimization- Your Language Model is Secretly a Reward Model (Rafailov et al., 2023) [link]
• RLAIF- Scaling Reinforcement Learning from Human Feedback with AI Feedback (Lee et al., 2023) [link]

• Post-training (contd...)

HW 3 out

• Guest Lecture 3: TBD

• Retrieval Augmented Generation (RAG)

• Guest Lecture 4: TBD

• RAG continued, Intro to Agent-based systems

• Project presentations session 1

Final project presentations

• Project presentations session 2

Final project presentations, HW 3 due on 4/27,
Final project reports due on 4/30