Schedule

• Course introduction
• Logistics
• Transformers - high level overview

• Attention is all you need (2017) [link]

• Optimization, backpropagation, and training

• Deep Feedforward Networks. Ian Goodfellow, Yoshua Bengio, & Aaron Courville (2016). Deep Learning, Chapter 6.5. [link]
• An overview of gradient descent optimization algorithms [link]
• A Gentle Introduction to Torch Autograd [link]
• Autograd Mechanics [link]

• Word embeddings
• Tokenization

• Distributed Representations of Words and Phrases and their Compositionality (2013) [link]
• GloVe- Global Vectors for Word Representation (2014) [link]
• BPE - Neural Machine Translation of Rare Words with Subword Units (2016) [link]
• SentencePiece- A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (2018) [link]

• Transformers
• Implementation details

• Attention is all you need (2017) [link]
• The Annotated Transformer [link]

• Positional Information - Absolute - Relative - ROPE - ALiBi
• Multi-Query Attention
• Grouped Multi-Query Attention
• Inference
• KV caching
• Encoder/decoder-only vs encoder-decoder

• Train Short, Test Long- Attention with Linear Biases Enables Input Length Extrapolation [link]
• RoFormer- Enhanced Transformer with Rotary Position Embedding [link]
• Fast Transformer Decoding- One Write-Head is All You Need [link]

Tips on choosing a project [slides]

HW1 out [link]

• Transfer Learning

• ELMo, Deep Contextualized Representations [link]
• ULMFit, Universal Language Model Fine-tuning for Text Classification [link]
• BERT, Pre-training of Deep Bidirectional Transformers for Language Understanding [link]
• ELECTRA, Pre-training Text Encoders as Discriminators Rather Than Generators [link]

Project teams due. [Team submission form]

• Model architecture and training objectives - Encoder-decoder, decoder-only - UL2 / FIM

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer [link]
• UL2- Unifying Language Learning Paradigms (2022) [link]
• What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022) [link]
• BART, Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension [link]

• Scale
• Compute analysis in transformers

• Scaling Laws for Neural Language Models [link]
• Training Compute-Optimal Large Language Models [link]

• Scaling laws and GPT-3
• Few-shot Learning
• Prompting
• In-context learning

• Language Models are Few-shot Learners (2020) [link]
• Rethinking the Role of Demonstrations- What Makes In-Context Learning Work? (2022) [link]
• Data Distributional Properties Drive Emergent In-Context Learning in Transformers (2022) [link]

• Prompting
• Emergence
• Reasoning
• Instruction tuning

• Chain of Thought Prompting Elicits Reasoning in Large Language Models [link]
• Tree of Thoughts- Deliberate Problem Solving with Large Language Models [link]
• The curious case of neural text degeneration (2020) [link]
• Training language models to follow instructions with human feedback (2022) [link]
• Multitask Prompted Training Enables Zero-Shot Task Generalization (2021) [link]
• Finetuned Language Models Are Zero-Shot Learners (2021) [link]
• Scaling Instruction-Finetuned Language Models (2022) [link]
• Tree of Thoughts- Deliberate Problem Solving with Large Language Models (2023) [link]

• Adaptation
• Reinforcement Learning for language models fine-tuning

• Fine-Tuning Language Models from Human Preferences (2019) [link]
• Learning to Summarize with Human Feedback (2020) [link]
• InstructGPT - Fine-Tuning Language Models from Human Instructions (2021) [link]
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (2022) [link]
• Direct Preference Optimization- Your Language Model is Secretly a Reward Model (2023) [link]

• Parameter-Efficient Transfer Learning for NLP (2019) [link]
• Prefix-Tuning- Optimizing Continuous Prompts for Generation (2021) [link]

HW 1 due (10/8)

• Challenges and Opportunities of Building Open LLMs

• What Language Model to Train if You Have One Million GPU Hours (2022) [link]
• Dolma- Trillion Token Open Corpus for Language Model Pretraining (2023) [link]
• Llama 2- Open Foundation and Fine-Tuned Chat Models (2023) [link]
• Pythia- A Suite for Analyzing Large Language Models Across Training and Scaling (2023) [link]
• BLOOM- A 176B-Parameter Open-Access Multilingual Language Model (2022) [link]
• Scaling Language Models- Methods, Analysis & Insights from Training Gopher (2022) [link]

• Parameter efficient fine tuning

• Parameter-Efficient Transfer Learning for NLP (2019) [link]
• Prefix-Tuning- Optimizing Continuous Prompts for Generation (2021) [link]
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022) [link]
• LoRA- Low-Rank Adaptation of Large Language Models [link]
• Scaling Down to Scale Up- A Guide to Parameter-Efficient Fine-Tuning [link]
• Efficient Transformers- A Survey [link]

• Guest lecture by Tushar Khot
  Reasoning with (De)Composition

• Hey AI, Can You Solve Complex Tasks by Talking to Agents? [link]
• Toolformer - A Tool-Augmented Language Model (2022) [link]
• Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback (2023) [link]
• ReAct- Synergizing Reasoning and Acting in Language Models [link]

10/20 Project proposal due

• Modular deep learning
• Mixture of experts

• Modular Deep Learning (2022) [link]
• A Review of Sparse Expert Models in Deep Learning (2022) [link]
• Outrageously Large Neural Networks- The Sparsely-Gated Mixture-of-Experts Layer (2017) [link]
• Switch Transformers- Scaling to Trillion Parameter Models (2021) [link]

• Retrieval augmented language models

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) [link]
• Improving language models by retrieving from trillions of tokens (2021) [link]
• REPLUG, Retrieval-Augmented Black-Box Language Models [link]

• Build an Ecosystem, Not a Monolith

• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022) [link]
• Exploring and Predicting Transferability across NLP Tasks (2020) [link]
• Editing Models with Task Arithmetic (2022) [link]

• Modeling long sequences
• Hierarchical and graph-based methods
• Recurrence and memory

• Higher-order Coreference Resolution with Coarse-to-fine Inference (2018) [link]
• Entity, Relation, and Event Extraction with Contextualized Span Representations (2020) [link]
• Memorizing transfomers (2022) [link]
• Hierarchical Graph Network for Multi-hop Question Answering [link]
• Compressive Transformers for Long-Range Sequence Modelling (2020) [link]
• Efficient transformers - A survey (2022) [link]

HW 2 due

• Modeling long sequences
• Sparse attention patterns
• Approximating attention
• Hardware aware efficiency

• Longformer- The Long-Document Transformer (2020) [link]
• BigBird - Transformers for Longer Sequences (2020) [link]
• Performer - Rethinking Attention with Performers (2021) [link]
• Reformer - The Efficient Transformer (2020) [link]
• Long T5 - Efficient Text-To-Text Transformer for Long Sequences (2022) [link]

• Training approaches for long sequences
• Hardware aware efficiency
• Societal considerations and impacts of foundation models

• FlashAttention - Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) [link]
• PRIMERA- Pyramid-based Masked Sentence Pre-training for Multi-document Summarization (2022) [link]
• Peek Across- Improving Multi-Document Modeling via Cross-Document Question-Answering (2023) [link]
• What's in my big data? (2023) [link]
• Red Teaming Language Models with Language Models (2022) [link]
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (2022) [link]

• Vision transformers
• Diffusion models

• An Image is Worth 16x16 Words- Transformers for Image Recognition at Scale (2020) [link]
• Training data-efficient image transformers & distillation through attention (2021) [link]
• Denoising Diffusion Probabilistic Models (2020) [link]

• Final project presentations -- session 1

• Towards Large Foundation Vision Models

• Scaling Vision Transformers to 22 Billion Parameters (2023) [link]
• From Sparse to Soft Mixtures of Experts (2023) [link]
• Scaling Vision Transformers (2021) [link]

• PaLI-X- On Scaling up a Multilingual Vision and Language Model [link]
• PaLI- A Jointly-Scaled Multilingual Language-Image Model [link]

• Final project presentations -- session 2

12/9 HW 3 due

• Foundation Models for Code and Math

• Evaluating Large Language Models Trained on Code (2021) [link]
• Solving Quantitative Reasoning Problems with Language Models (2022) [link]
• StarCoder- May the source be with you! (2023) [link]

• Program Synthesis with Large Language Models (2021) [link]
• Show Your Work- Scratchpads for Intermediate Computation with Language Models (2021) [link]

• Moved to 12/1 (see above)

12/18 Final project report due