HW2: Complete Instructor Notebook

HW2: Complete Instructor Notebook

Cell 1 — markdown

CS 639 Spring 2026 — HW 2: Transformers

Cell 2 — markdown

Section 0: Setup & Data

Cell 3 — markdown

Make sure you are using GPU. the output of the cell below shoud be: Using device: cuda

Cell 4 — code

Cell 5 — code

Cell 6 — markdown

Oh look at that! We are going to train a mini LLM to speak like Shakespearean characters, and tokenize each character

Cell 7 — code

Cell 8 — code

Cell 9 — markdown

Section 1: Building Blocks (Provided)

Cell 10 — code

Cell 11 — code

Cell 12 — code

Cell 13 — markdown

Section 2: Part A — Core Transformer Implementation (50 pts)

Cell 14 — markdown

A1: Scaled Dot-Product Attention (15 pts) [Code]

Cell 15 — code

Cell 16 — code

Cell 17 — markdown

A2: Multi-Head Attention (15 pts) [Code]

Cell 18 — code

Cell 19 — code

Cell 20 — markdown

A3: Transformer Block (15 pts) [Code]

Cell 21 — code

Cell 22 — code

Cell 23 — markdown

A4: Training and Validation (5 pts) [Both]

Cell 24 — code

Cell 25 — markdown

The moment we’ve been waiting for.. lets start training! (Note: this training will take ~11 minutes on the free T4 GPU)

Cell 26 — code

Cell 27 — code

Cell 28 — code

Cell 29 — markdown

If your implementation is correct, you should see something resembling Shakespearean dialogue!

Cell 30 — code

Cell 31 — markdown

Section 3: Part B — Ablation Studies (20 pts)

Cell 32 — code

Cell 33 — markdown

B1: Varying Number of Attention Heads (10 pts) [Writeup]

Cell 34 — code

Cell 35 — code

Cell 36 — markdown

B2: Varying Number of Layers (10 pts) [Writeup]

Cell 37 — code

Cell 38 — code

Cell 39 — markdown

B3: Analysis (4 pts) [Writeup]

Cell 40 — markdown

Section 4: Part C — Tokenization & Positional Encoding (20 pts)

Cell 41 — markdown

C1: Character-Level vs. Word-Level Tokenization (10 pts) [Writeup]

Cell 42 — code

Cell 43 — code

Cell 44 — code

Cell 45 — code

Cell 46 — code

Cell 47 — markdown

C2: Rotary Positional Embeddings — RoPE (10 pts) [Both]

Cell 48 — code

Cell 49 — code

Cell 50 — code

Cell 51 — code

Cell 52 — code

Cell 53 — code

Cell 54 — code

Cell 55 — code

Cell 56 — code