Lecture 20: MapReduce and Spark

Content locked

The catalog is public. Reading this content requires approved access.

Overview

Learning Objectives

Data Lakes

Review: Data Warehouses

From Data Warehouse to Data Lake

Hadoop MapReduce

SQL vs. MapReduce

Mappers

Reducers

Multiple Reducers

SQL to MapReduce Mapping

MapReduce with HDFS and Data Locality

Pipelines: Chaining MapReduce Jobs

Spark

Intermediate Data: MapReduce vs. Spark

Data Lineage: Transformations and Actions

Optimization

Partitions

Tasks

Repartitioning

Narrow vs. Wide Transformations

Caching

Spark SQL and DataFrames

Spark SQL

DataFrame API

DataFrames: Pandas vs. Spark

Deployment

Conclusion

Source