README

P5 (3% of grade): Spark

Overview

Corrections/Clarifications

AI Usage

Setup

Docker Containers

Jupyter Container

VS Code users

Part 1: Spark Programming Models

Q1: How many problems are there with a cf_rating of at least 1600, having private_tests, and a name containing “_A.” (Case Sensitive)? Answer using all three Spark APIs.

Q2: How many correct PYTHON3 solutions are from CODEFORCES?

Q3: How many problems are of easy/medium/hard difficulty?

Q4: What tables/views are in our warehouse?

Part 2: Performance Analysis

Q5: Does the query plan for a GROUP BY on solutions data need to shuffle/exchange rows if the data is pre-bucketed?

Q6: Does caching make it faster to compute averages over a subset of a bigger dataset?

Part 3: Spark and Generative AI

Q7: How many JAVA solutions are there?

Q8: What is the maximum memory limit in bytes?

Part 4: Spark and Traditional Machine Learning

Q9: Do the problems with a missing cf_rating appear more or less challenging than other problems?

Q10: How does tree depth impact the quality of predictions?

Submission

Tester