Data Engineering.
Build scalable data pipelines — ingestion, transformation, orchestration, and performance tuning across batch and streaming.
Beginner
Start here — no prior experience needed
Data Engineering Fundamentals
Build a complete ingest-transform-serve pipeline in fifty lines, then break it the three ways real pipelines break.
File Formats: CSV, JSON, Parquet
Measure the same dataset as CSV, JSON and Parquet — size, query time, column pruning — and see why a columnar format changes what queries are affordable.
Idempotency and Safe Retries
Every pipeline gets run twice. Four patterns that make that harmless — overwrite, delete-insert, merge, and deduplication on a natural key.
Batch Pipelines and Time Windows
Logical time versus wall-clock time, high-water marks that lose rows, and backfills that finish — the three things that decide whether a nightly job is correct.
Partitioning and Physical Layout
Partition pruning measured, the cardinality mistake that creates 50,000 tiny files, and why sorting inside a partition often beats adding another one.
Intermediate
For developers with core concepts down
Data Quality Checks That Run Every Load
Six checks worth writing, quarantine instead of dropping, thresholds against a trailing average, and the reconciliation test that catches what column checks cannot.
Incremental Loads and Change Data Capture
Four ways to extract only what changed, why deletes are the hard part, and building SCD type 2 history from a CDC stream that arrives out of order.
Orchestration Patterns
Task granularity, dependencies that skip rather than fail, retries with backoff, and the sensor pattern that waits for data instead of hoping it arrived.
Advanced
Production-grade patterns for experienced engineers
Streaming: Exactly-Once Semantics
Simulate a consumer crash and watch at-least-once produce duplicates, then fix it three ways — atomic offset commits, idempotent keys, and deduplication on read.
Schema Evolution and Data Contracts
Which schema changes are safe, why backward and forward compatibility are different questions, and enforcing a contract in CI so a breaking change fails the producer's build.