Practical assessment (as per §10 (6) of the "Rahmen-PO für Mono-Masterstudiengänge 2021"): small research project on a self-chosen project in teams of 2; brief report (graded) and a short presentation (ungraded).

Large Language Models (LLMs) are one of the most exciting technologies of our time. Following the "Attention is All You Need" paper in 2017, the progress of LLMs has been rapid and today ChatGPT reaches hundreds of millions of users per month. The aim of this lecture is to take a detailed look into all stages involved in building, evaluating and serving a large language model. The first part of the lectures focuses on architecture components (dense and Mixture-of-Experts, different variants of attention) and how to do efficient pre-training. This involves optimization, how to parallelize training, scaling laws and data curation. In the second half, we will look into the evaluation of LLMs, how to perform inference efficiently, and cover later stages of the training pipeline like post-training (with supervised and reinforcement-learning based approaches) to align model behavior with human preferences and tasks. A particular focus is on principled engineering: implementing relevant algorithms and architectures from scratch, designing scaling experiments across data and architecture dimensions, and investigating problems that inevitably arise during training (instabilities, suboptimal hardware utilization). Students incrementally build a production-style LLM training framework over the course of the semester as part of the weekly exercises.

This is a fast-paced advanced lecture. Students attending this lecture should fulfill the prerequisites listed below (see "Literature").

-

Voraussetzungen:

Prerequisites:
Knowledge in PyTorch, deep learning and a solid understanding of mathematics are expected (e.g., by taking the lectures ML4103 "Deep Learning" and ML4101 "Mathematics and Machine Learning").