Deck 08: Spark

Deck 08: Spark

Q1. What is the difference between these Spark caching levels?

Q2. Is data that is cached at the DISK_ONLY level serialized?

Q3. What is df.cache() a shortcut for?

Q4. What is the difference between these to Spark caching levels, and what are the tradeoffs?

Q5. What is the difference between normal partitioning and hash partitioning in Spark?

Q6. What is the idea of partial aggregates in Spark?

Q7. One optimization that is part of adaptive query execution (AQE) in Spark is “partition coallescing”.  What is it?

Q8. What does bucketBy do when Spark is writing parquet files?

Q9. What does SMJ stand for?

Q10. What does BHJ stand for?

Q11. How does a distributed JOIN work with BHJ?

Q12. How does a distributed JOIN work with SMJ?

Q13. Is this deterministic?

Q14. Consider Spark’s DecisionTreeRegressionModel and DecisionTreeRegressor.  Which one is fitted?

Q15. dt = DecisionTreeRegressor(featuresCol=”????”, labelCol=”????”)

Q16. If we have a fitted model in Spark, how do we used it to make predictions?

Q17. A Spark “Pipeline” object is an unfit model, consisting of multiple transformers and ending in an estimator.  When we call “fit” on it, we get a fitted PipelineModel.  What does it contain?

Q18. The traditional decision tree training algorithm considers splits on every column, for every possible split between rows (a threshold on a column is a split).  Where does Spark consider splits?

Q19. What is an “equi-depth” histogram?

Q20. If df is a Spark DataFrame, what are two ways you can make it appear as a view in Hive (for the purpose of upcoming SQL queries)?

Q21. If you have a query, say “SELECT * FROM data”, how can you run it with Spark SQL?  Assume your Spark session is in a variable “spark”.

Q22. What type does spark.sql(“QUERY”) return?

Q23. How do you write a Spark DataFrame (say df) as a Hive table?

Q24. What does spark.sql(“SHOW TABLES”) do?

Q25. If tv is a table OR view in Hive, how can you get the corresponding Spark DataFrame?

Q26. What kind of Spark operation is .count()?

Q27. How are grouping and windowing similar and different?

Q28. Say you want to filter the results of a Spark SQL query based on a value computed with a windowing function.  Why is this tricky, and what are some solutions?

Q29. Say you run this:

Q30. If you want to select the “x” column of a Spark DataFrame df, add 1 to it, and call the result column y, how would you do that?

Q31. Spark gives an error about a file already existing when you run this:

Q32. Why might you want bigger Spark partitions?

Q33. Why might you want smaller Spark partitions?

Q34. What are two categories of Spark transformations?

Q35. If df is a Spark DataFrame, how would we start/stop caching it?

Q36. What does RDD stand for, in the context of Spark?

Q37. How do you create a SparkData corresponding to a CSV at location PATH?

Q38. How can you see the number of RDD partitions for a Spark DataFrame df?

Q39. Say you run this:

Q40. What are the strengths and weaknesses of hash partitioning?

Q41. In MapReduce, what parameters do map and reduce functions take?

Q42. A reduce task might have the reduce function called several times.  What can we say about the keys across these multiple calls for a single task?

Q43. What determines the number of output files in a MapReduce job?

Q44. What are some performance advantages Spark has over MapReduce?

Q45. If you have a regular Python list, how can you convert it into a Spark RDD?

Q46. What is the relationship between actions, operations, and transformations?

Q47. A Spark task: