sp26-cs544 cs544
Lecture 24: Spark Internals and Performance Lecture 24: Spark Internals and Performance
Content locked
The catalog is public. Reading this content requires approved access.
Sign in to readLecture 24: Spark Internals and Performance
Overview
Learning Objectives
Schema Inference
With Schema Inference
Without Schema Inference
Collecting Data
When Collecting Works Well
When Collecting Fails
Caching
Persist and Cache Basics
Storage Levels
MEMORY_ONLY
MEMORY_ONLY_SER
DISK_ONLY
Replication (2x Levels)
Grouping
Logical View: Groups and Aggregates
Physical Execution on Partitions
Normal vs. Hash Partitioning
Optimization 1: Partial Aggregates
Optimization 2: Partition Coalescing
Optimization 3: Parquet Bucketing
Joining
Broadcast Hash Join (BHJ)
Shuffle Sort Merge Join (SMJ)
Network I/O: SMJ vs. BHJ
Seeing Join Type with Explain
Join Hints
Conclusion
Source