Schedule

• Course Introduction
• Logistics

• Word embeddings and vector semantics

• Jurafsky & Martin Chapter 6

• Word embeddings and vector semantics (cont.)
• Sparse representations
• Dense representations

• Jurafsky & Martin Chapter 6

• Distributed Representation of Words and Phrases and their Compositionality (Mikolov et al., 2013) [link]
• Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013) [link]
• Word2vec Explained- deriving Mikolov et al.'s negative-sampling word-embedding method (Goldberg and Levy, 2014) [link]

• Deriving the gradient of Word2vec
• Evaluation of word embeddings

• Jurafsky & Martin Chapter 6
• Distributed Representation of Words and Phrases and their Compositionality (Mikolov et al., 2013) [link]

HW1 out

• N-Gram Language Models
• Smoothing
• Evaluation of Language Models

• Jurafsky & Martin Chapter 7

• Neural network basics
• Autograd

• The Matrix Calculus You Need For Deep Learning (Terence Parr and Jeremy Howard) [link]
• Little book of deep learning (François Fleuret) - Ch 3, 4

• Auto Grad
• Building blocks of Deep Learning for NLP
• CNNs

• Goldberg Chapter 9

• CNNs (contd.)
• RNNs
• Task specific neural network architectures
• Machine translation

• Understanding LSTM Networks (Christopher Olah) [link]
• Eisenstein, Chapter 18

• Neural Machine Translation and Sequence-to-sequence Models- A Tutorial (Graham Neubig) [link]

• RNNs (contd.)
• Training sequence models
• Machine translation (contd.)

• Statistical Machine Translation (Koehn) [link]
• Neural Machine Translation and Sequence-to-sequence Models- A Tutorial (Graham Neubig) [link]
• Learning to Align and Translate with Attention (Bahdanau et al., 2015) [link]
• Luong et al. (2015) Effective Approaches to Attention-based Neural Machine Translation [link]
• Attention is All You Need (Vaswani et al., 2017) [link]
• Illustrated Transformer [link]

Project teams due on 02/09

• Attention
• Transformers

• Neural Machine Translation and Sequence-to-sequence Models- A Tutorial (Graham Neubig) [link]
• Learning to Align and Translate with Attention (Bahdanau et al., 2015) [link]
• Luong et al. (2015) Effective Approaches to Attention-based Neural Machine Translation [link]
• Attention is All You Need (Vaswani et al., 2017) [link]
• Illustrated Transformer [link]

• Transformers (contd.)
• Language modeling with Transformers
• Transfer Learning

• Illustrated Transformer [link]
• Attention is All You Need (Vaswani et al., 2017) [link]
• The Annotated Transformer (Harvard NLP) [link]
• GPT-2 (Radford et al., 2019) [link]

HW 1 due / HW 2 out

• Transfer Learning (contd.)
• Objective functions for pre-training
• Encoder-decoder pretrained models
• Architecture and pretraining objectives

• The Illustrated BERT, ELMo, and co. (Jay Alammar) [link]
• BERT- Pre-training of Deep Bidirectional Transformers for Language Understanding (Devlin et al., 2018) [link]
• GPT-2 (Radford et al., 2019) [link]
• T5- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2020) [link]
• BART- Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (Lewis et al., 2019) [link]
• What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (Wang et al, 2022) [link]

• Decoding and generation
• Large language models and impact of scale
• In-context learning and prompting

• The Curious Case of Neural Text Degeneration (Holtzman et al., 2019) [link]
• How to generate text- using different decoding methods for language generation with Transformers [link]
• Scaling Laws for Neural Language Models (Kaplan et al., 2020) [link]
• Training Compute-Optimal Large Language Models (Hoffmann et al., 2022) [link]
• GPT3 paper - Language Models are Few-Shot Learners (Brown et al., 2020) [link]

• Language Models for Code

• Efficient Training of Language Models to Fill in the Middle (Bavarian et al., 2022) [link]
• A Survey on Large Language Models for Code Generation (Jiang et al., 2024) [link]
• Code Llama- Open Foundation Models for Code (Roziere et al., 2023) <a href="https://arxiv.org/pdf/2308.12950", target="_blank">[link]</a>
• L2CEval- Evaluating Language-to-Code Generation Capabilities of Large Language Models (Ni et al., 2024) [link]

HW 2 due

• Post-training
• Supervised Finetuning
• Instruction Following

• Multitask Prompted Training Enables Zero-Shot Task Generalization (Sanh et al., 2021) [link]
• Scaling Instruction-Finetuned Language Models (Chung et al., 2022) [link]
• Are Emergent Abilities of Large Language Models a Mirage? (Sha et al., 2023) [link]
• Emergent Abilities of Large Language Models (Wei et al., 2022) [link]

HW2 due on 03/06

Project proposals due 03/08;

• Data for Languge Models
• Data curation methods
• Impact of data on LLMs
• Data-centric NLP

• OLMo- Accelerating the Science of Language Models [link]
• DOLMA- An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research [link]
• Organize the Web- Constructing Domains Enhances Pre-Training Data Curation [link]
• Molmo and PixMo- Open Weights and Open Data for State-of-the-Art Vision-Language Models [link]

• Post-training
• Reinforcement learning from Human Feedback
• Alignment

• Training language models to follow instructions with human feedback (Ouyang et al., 2022) [link]
• Fine-Tuning Language Models from Human Preferences (Ziegler et al., 2019) [link]
• Direct Preference Optimization- Your Language Model is Secretly a Reward Model (Rafailov et al., 2023) [link]
• RLAIF- Scaling Reinforcement Learning from Human Feedback with AI Feedback (Lee et al., 2023) [link]

• Post-training (contd...)

HW 3 out

• Evaluation of natural language generation systems
• LLM evaluations

• Retrieval Augmented Generation (RAG)

• NLP in Robotics

• tagE- Enabling an Embodied Agent to Understand Human Instructions, (Sarkar et al, 2023) [link]
• DoRO- Disambiguation of Referred Object for Embodied Agents (Pramanick et al, 2022) [link]
• Enabling Human-Like Task Identification From Natural Conversation (Pramanick et al, 2020) [link]

• RAG continued, Intro to Agent-based systems

• Project presentations session 1

Final project presentations

• Project presentations session 2

Final project presentations, HW 3 due on 4/28,
Final project reports due on 5/1