Lecture 5: Natural Language Processing

Content locked

The catalog is public. Reading this content requires approved access.

What is NLP?

Why is it Hard?

Approaches to NLP

A Brief History

Language Models

Basic Idea

Training the Model

The Markov Assumption

n-gram Models

k=0: Unigram Model

k=1: Bigram Model

k=n-1: n-gram Model

Sampling from Language Models

Simple “Generative AI” from Letter Bigram

Sampling Shakespeare

n-gram Training Issues

Issue 1: Multiply Tiny Numbers

Issue 2: n-grams with Zero Probability

Evaluating Language Models

How do we know we’ve done a good job?

Extrinsic Evaluation

Intrinsic Evaluation: Perplexity

Representing Words

Traditional Representation: One-Hot Vectors

Smarter Representations

Training Word Embeddings

Word2vec

Word Embedding Properties

Beyond “Shallow” Embeddings

Transformers

Pretrained Models

Further NLP Tasks

Practice Problems

Key Takeaways