Deck 11: Cloud

Deck 11: Cloud

Q1. Is BigQuery a data warehouse or a data lake?

Q2. The parquet file format was inspired by the ______ format developed at Google, which was eventually replaced internally at Google with the _______ format.

Q3. When a value in a column is repeated many times consecutively, we can record just the value and count (instead of many copies of the value).  This is called ________ encoding.

Q4. When a column contains large strings, and the same values show up multiple times, we can replace those strings with shorter numeric codes.  We can use the numeric codes to lookup the original strings as needed.  This is called ______ encoding.

Q5. What is something that a GPL open-source license requires that is not required by MIT/BSD/Apache?

Q6. What is a type of intellectual property the Apache open-source license focuses on that the MIT and BSD licenses do not talk much about?

Q7. What are some factors you might consider when choosing a GCS bucket region?

Q8. What is the GCP marketplace?

Q9. How can you activate/deactivate your GCP credentials on a GCP VM?

Q10. What does this let you do?

Q11. You run this:

Q12. If you want to do a correlated cross join against a field C containing an array of structs, how do you first “expand” the arrays so each entry becomes its own row?

Q13. What are some examples of external tables we can register in BigQuery?

Q14. If column C is an array type, how do you get the Nth item from every array using BigQuery?

Q15. What can we call if we want to get a pandas DataFrame from a BigQuery query?  What about a geopandas GeoDataFrame?

Q16. Say you have a query like this:

Q17. What is the difference between ST_CENTROID_AGG(geo_column) and ST_CENTROID(geo_column)?

Q18. If you have floats long and lat, what can you call in BigQuery to convert this to a geographic point that can be used with other spatial functions?

Q19. In BigQuery, what are two reasons to do train/test splitting yourself (for ML) instead of letting BigQuery do it?

Q20. In BigQuery, what must be passed to ML.WEIGHTS(…)?

Q21. In BigQuery, what must be passed to ML.PREDICT(…)?

Q22. In BigQuery, what must be passed to ML.EVALUATE(…)?

Q23. In BigQuery, how do we do non-linear transformations on our features?

Q24. In BigQuery, how do you handle OneHot encoding?

Q25. What is a BigQuery slot?

Q26. What are two BigQuery billing models?

Q27. Which BigQuery billing model has priority access to resources (i.e., first “right of refusal”)?

Q28. What do on-demand and capacity billing charge for in BigQuery?

Q29. What does it mean that a BigQuery task is preemptible?

Q30. What does EC2 stand for, and what is it?

Q31. What is usually more expensive to rent in the cloud, 1 VM for 100 hours, or 100 VMs for 1 hour?

Q32. What is the difference between IaaS and PaaS?

Q33. Considering CSVs and parquets: when do we need to infer the schema?

Q34. What does cloud “lock in” mean?

Q35. What is transactions processing?

Q36. What is a “tenant”, and what is the difference between multi-tenant hosts and single-tenant hosts?

Q37. What is analytics processing?

Q38. What is a cloud cost you typically pay when VMs are powered off, but not deleted?

Q39. What does OLTP stand for?

Q40. What is a typical geographic hierarchy in clouds?

Q41. What does OLAP stand for?

Q42. What is ingress/egress in terms of cloud resources, and what is more expensive?

Q43. What is a data warehouse?

Q44. You can either pay a fixed price to get a VM (with high chances of success), or you can pay a fluxuating price (usually cheaper), and potentially get pre-empted (VM is taken away so cloud can use resources for something of higher priority).

Q45. What does ETL stand for?

Q46. For analytics work, what are some tradeoffs between storing data in a database vs. storing it in some files (e.g., parquet files)?

Q47. What does it mean for a format to be “column oriented”?

Q48. What is the orientation of CSVs and parquet files?

Q49. What is the tradeoff between text encoded and binary encoded formats?

Q50. Compression works by looking for repetition over some window, say 32 KB.  What would be the advantage and disadvage of increasing the window size?

Q51. What does snappy prioritize, high speed, or high compression ratios?

Q52. What is the difference between a data warehouse and data lake?

Q53. What is elasticity?