Lecture 25: Spark MLlib

Content locked

The catalog is public. Reading this content requires approved access.

Overview

Learning Objectives

Machine Learning Overview

Models and Predictions

Learning from Data

Train/Validation/Test Split

Training and Predicting APIs

Training

Predicting

Data Representation

Features Column

Spark MLlib Terminology

Transformer

Estimator

Pipelines

Spark MLlib Packages

Decision Trees

Structure and Predictions

Ensemble Methods

Tree Methods vs. Deep Learning

Training Decision Trees

Evaluating Tree Quality: Impurity

Splitting Nodes

When to Stop Splitting

The PLANET Algorithm

Hybrid Approach

Step 1: Compute Equi-Depth Histograms

Step 2: Split Points — In-Memory vs. Distributed

Step 3: Parallel Splitting

Physical Layout

Handling Small Nodes (In-Memory Build)

Handling Big Nodes (Distributed Build)

Conclusion

Source