Introduction

This course provides a deep dive into modern Natural Language Processing (NLP), focusing on neural approaches and their applications. The curriculum spans both foundational neural concepts and cutting-edge developments in the field.
The course begins with core neural network concepts in NLP, covering word embeddings, sequence modeling, and attention mechanisms. Students will gain a strong understanding of these building blocks while learning their practical implementations.
Building on these foundations, we explore transformer architectures and their evolution, including landmark models like BERT, GPT, and T5. The course examines how these models enable sophisticated language understanding and generation through pre-training and transfer learning. The latter portion covers contemporary advances: Large Language Models (LLMs), multi-modal integration, parameter-efficient fine-tuning, and model compression. We’ll analyze the capabilities and limitations of current systems while discussing emerging research directions.
Through lectures, hands-on assignments, and projects, students will gain both theoretical understanding and practical experience implementing modern NLP systems. The course emphasizes reproducible experimentation and real-world applications.

Prerequisites

This course requires:

  • Strong programming skills in Python and prior exposure to libraries such as numpy, PyTorch, or TensorFlow
  • Familiarity with probability and statistics, and linear algebra
  • Prior exposure to machine learning concepts through courses like CPSC 481 (Intro to Machine Learning) or CPSC 470 (Artificial Intelligence)

Important Note: This course assumes familiarity with fundamental machine learning concepts like gradient descent, neural networks, and backpropagation. Although we review these concepts, students without prior ML/AI coursework should first take an introductory ML or AI course before enrolling. The course material builds heavily on these concepts from day one, and there will not be time to cover these basics in detail. If unsure about prerequisites, please consult the instructor before enrolling.

Learning Resources

Textbook

  • Dan Jurafsky and James H. Martin. Speech and Language Processing (2024)
  • Yoav Goldberg. A Primer on Neural Network Models for Natural Language Processing

We will also using papers from major conferences in the field including ACL, EMNLP, NAACL, ICLR, NeurIPS, etc.

Communication

We use Canvas, Ed and email for main announcements. For questions about the course, discussions about material, and faciliatating discussions for projects between students, we will mainly use Ed Discussion.

Grading

Final grades will be comprised of:

  • 22%: Assignments, which includes both written and coding problem sets
  • 40%: Two Midterm Exams (20% each), in person, closed book
  • 8%: Participation and quizzes
  • 30%: Final projects, including a project proposal (5%), project final presentation (15%), project final report (15%), code and reproducibility checklist (5%)
  • Grading for graduate students: Graduate students will need to incorporate a novelty element and a more in-depth literature review in their final projects

AI Assistant policies

The use of AI tools (including but not limited to ChatGPT, Claude, GPT-4o, etc) for coursework is regulated as follows:

Permitted Uses:

  • Writing assistance: Grammar checks, style improvements, and clarity enhancements
  • Learning tool: Exploring concepts and asking questions while studying

Prohibited Uses:

  • Generating code solutions for assignments
  • Completing homework problems
  • Answering quiz/exam questions
  • Producing project deliverables
  • Using GitHub Copilot or similar coding assistants

Required Disclosure:

You must document any AI assistance in your submissions by:

  • Specifying which parts received AI assistance
  • Explaining how the AI was used
  • Including relevant prompts/interactions

Academic Integrity:

Using AI tools beyond these guidelines constitutes an Honor Code violation. When in doubt, consult the instructor before using AI tools.

Late submissions

You can still submit your assignment after the deadlines for up to 5 days. You will, however, receive partial credit for late submissions. Every late day will result in 10% deduction in full credit for that assignment

Note: Late days can only be used on the assignments, and not on the project proposal or the final report and the presentation.

Grading for graduate students

Grading components for graduate students will be the same as undergraduate students. The only difference is the following:

For class projects we expect graduate students to work on a research problem (The project should propose either a novel research, a novel investigation of existing methods, an extension of prior work for a specific purpose, or a new application.). Graduate student projects are also expected to have a more thorough literature review component in their final project report.

Class project (30%)

Students must complete a final research project on a topic of their choice related to the class. The students should team up with other students and the team size is limited to 2 to 3 students. If you don’t choose a team you will be randomly assigned a team mate. Invidiual projects are allowed only in exceptional cases and by providing reasonable justification.

  • 5%: proposal
    • Students should submit a 1 page proposal for their project. The proposal should state and motivate the problem, and position the proposed project within related work. The project proposal should also include a brief description of the approach as well as the experimental plan (e.g., baselines, datasets, etc) to validate the effectiveness of the approach. Here are some ideas on types of projects.:
    • For undergraduate students the project could be reimplementation of an exsiting method, a new user-facing application that uses NLP models for a new problem, a comprehensive survey into a subtopic of interest, deeper investigation of a paper and providing further insights by conducting additional experiments, or novel reseach.
    • For graduate students the project should include a component of novelty. E.g., it could propose a novel research, a novel investigation of existing methods, an extension of prior work for a specific purpose, or a new application.
  • 5%: Progress report
    • Students should submit a 1 page progress report for their project. The progress report should include the current status of the project, the literature review, the challenges faced, and the plan for the remaining part of the project.
  • 10%: Final project report
    • 2-4 (no more than 4 pages) page conference format report (e.g., NeurIPS) detailing the project motivation, related work, proposed approach, results, and discussion. You can think of this as a short conference paper. Negative results will not be penalized, but should be accompanied with detailed analysis of why the proposed methods didn’t work and provide some additional insights into the problem. 
    • References and appendix won’t count towards the page limit
  • 8%: Final project presentation
    • 5 minute in person in-class presentations
  • 2%: Code and reproducibility checklist
    • Your project code should be clean, readable, with clear running instructions, and the results should be fully reproducible. We will provide a reproducibility checklist that should be returned.

Integrity

Academic integrity requires that students at Yale acknowledge all of the sources that inform their coursework. Most commonly, this means (a) citing the sources of any text or data that you include in papers and projects, and (b) only collaborating with other students or using AI composition software in ways that are explicitly endorsed by the assignment. Yale’s dedication to academic integrity flows from our two primary commitments: supporting research and educating students to contribute to ongoing scholarship. A safe and ethical climate for research demands that previous authors and artists receive credit for their work. And learning requires that you do your own work. Conventions for acknowledging sources vary across disciplines, and instructors should instruct you in the forms they expect; they should also delineate which forms of collaboration among students are permitted. But ultimately it is the student’s responsibility to act with integrity, and the burden is on you to ask questions if anything about course policies is unclear.

Diversity statement

We embrace and celebrate diversity, understanding that the richest learning experiences come from the exchange of ideas among individuals from varied backgrounds, cultures, and perspectives. We uphold a commitment to mutual respect and open-mindedness, encouraging each participant to both share their unique insights and actively listen to others. Recognizing that learning is a collaborative and evolving process, we foster an inclusive environment where constructive criticism is welcomed, mistakes are embraced as opportunities for growth, and every student is both a teacher and a learner. Our goal is to cultivate a dynamic, respectful, and inclusive classroom environment.