Afaque Ahmad
I’m Afaque, Solutions Architect at Databricks. I've cracked offers @ Apple, Uber & Atlassian. Previously, I was a Senior Data Engineer at QB/McKinsey & Urban Company (formerly UrbanClap). This channel was created out of my endeavor for teaching and simplifying complex data engineering concepts and, in doing so, perhaps offer a fresh perspective on how we approach them.
Watch out videos on my channel for best-in-class videos on Apache Spark, complex SQL, Advanced Python, and, emerging topics in the area of data engineering.
LinkedIn: www.linkedin.com/in/afaque-ahmad-5a5847129/
GitHub: github.com/afaqueahmad7117
Contact Email: afaque.ahmad@theseniorde.com
Afaque Ahmad
Here's a brand new ~6 hour video on Delta Lake. I can promise you that this will be the most comprehensive video that you'll ever watch on Delta Lake. Here's are the 44 topics I've covered:
- Challenges with Data Lakes
- Lack of ACID Support (ACID Explained)
- Atomicity
- Consistency
- Isolation
- Durability
- Lack of UPDATE, MERGE, DELETE Operations
- Data Reliability & Quality Issues
- Lab Architecture on Azure
- Lab Setup on Azure
- DML Operations on Delta Tables (Lab)
- Uncovering the Delta Log (_delta_log)
- How Delta Lake Computes the Latest State
- How Delta Lake's Transaction Log Scales
- Pessimistic Concurrency Control
- Optimistic Concurrency Control
- Time Travel & Versioning
- Schema Validation
- Schema Evolution
- Converting Parquet to Delta
- Managed & External Tables
- Deletion Vectors (Copy on Write vs. Merge on Read)
- Cloning Delta Tables
- Shallow Clone Explained
- Deep Clone Explained
- Shallow Clone Lab
- Deep Clone Lab
- CTAS (CREATE TABLE AS SELECT) vs. Deep Clone
- The Small File Problem
- Optimization Techniques
- OPTIMIZE Command & Bin Packing Algorithm
- OPTIMIZE Lab
- Root Causes of the Small File Problem
- Manual OPTIMIZE
- Optimize Write Explained
- Optimize Write Lab
- Auto Compaction
- VACUUM Command
- VACUUM Lab
- ZOrdering
- ZOrdering Lab
- Liquid Clustering
- Liquid Clustering Lab
- How to Choose Liquid Clustering Columns
Go ahead, level up your knowledge, link to the video here: https://youtu.be/8IjCyvyAPpM
3 months ago | [YT] | 46
View 7 replies