System design

Solution

OLTP CDC → raw/bronze files → typed silver (MERGE on order_id) → gold at one grain (order or line — pick one) → a serving warehouse or marts for BI. Orchestrate freshness; do not transform 2 TB in the orchestrator. Page on freshness and volume, not only “job succeeded.”

Solution

At-least-once plus an idempotent sink (MERGE on event_id / offset+partition). True EOS is a specific Kafka + sink setup. Saying “Kafka is exactly-once” without a key is a fail.

Solution

Type-2 dimension with valid_from/valid_to. Facts join as-of the event timestamp. Type-1 overwrite changes history. Say which product wants.

Solution

Dual-write CSV trees. Pick Iceberg (or UniForm) with a single writer. Two writers is how the copies lie.

Solution

Separate raw PII, tokenize or mask in silver, grant only the columns they are allowed, audit the extract. Slack SELECT * from bronze is the incident.

Solution

Bounded date chunks, a smaller pool, catchup off, and idempotent partition replace. Unpausing catchup from 2019 is the stampede.

Solution

Sub-100ms point lookups, per-user online features, or a product API that is not SQL. Then you need a keyed store or a feature/online path — and a contract with the batch grain.

Solution

Tags / chargeback on compute and storage, query tags from dbt/jobs, and an owner on every gold table. Architecture without FinOps is a slide.

Solution

Debezium/DMS → Kafka/MSK → sink that MERGEs into Iceberg on the primary key, watermark on LSN/time, dead-letter for poison rows. A nightly dump is not this design.

Solution

Tenant_id on every fact, row policies or separate schemas for large tenants, and never a shared “admin” role that can SELECT * across tenants. Compaction and clustering on tenant + date.

Solution

If the decision can wait an hour, batch is cheaper and simpler. Streaming is for operational actions (fraud, inventory, live ops). Do not stream a CFO dashboard to look modern.

Solution

Cross-region replica or object-lock, a metastore backup, and a restore drill. “S3 is durable” is not a runbook. Know RPO/RTO in hours, not adjectives.