README

Content locked

The catalog is public. Reading this content requires approved access.

CS 639 Assignment 5: Alignment

Google Colab Setup

Instructions

What the Notebook Already Provides

Success Criteria

Model and Task Configuration

Submission

Part A: Core Alignment Implementation

A1: Response-Only LM Loss (10 points) [Code]

A2: Sequence Log-Probabilities (10 points) [Code]

A3: Last-Token Pooling (10 points) [Code]

A4: Reward Model Forward Pass (10 points) [Code]

A5: Bradley–Terry Preference Loss (10 points) [Code]

A6: Preference Accuracy (10 points) [Code]

A7: Training and Validation (10 points) [Both]

Part B: Alignment Experiments

B1: Varying the Amount of Unsupported Preference Data (5 points) [Writeup]

B2: Varying Preference Noise (5 points) [Writeup]

Part C: Direct Preference Optimization — Comparison and Failure Analysis

C1: SFT vs. Reward-Model Reranking vs. DPO (5 points) [Writeup]

C2: DPO Loss and Training (25 points) [Both]

Final Reminder