Lecture 19: Hadoop Ecosystem

Content locked

The catalog is public. Reading this content requires approved access.

Overview

Learning Objectives

Motivation for Distributed Storage

Layered Storage Systems

What If Your Data Is Too Big?

Partitioning

Cross-Partition Challenges

Server Failures and Replication

Motivation for System Redesign

Google Architecture and the Hadoop Ecosystem

Hadoop File System (HDFS)

Partitioning and Replication

DataNodes Store File Blocks

How Blocks Are Stored Locally

Replication Across DataNodes

Replication Across Racks

Heartbeats and Failure Detection

Input/Output: The NameNode/Worker Architecture

Reading a File

Writing a File

Read/Write Amplification

Tradeoffs: Replication Factor and Block Size

Data Locality

Load Balancing

I/O Load Balancing

Storage Balancing

HDFS Storage Balancing Strategies

Conclusion

Source