Apache Flink

Solution

Processing time windows close on wall clock. Late Kafka events fall in the wrong window. Event time plus watermarks is the honest window. If you cannot explain watermarks, you are not ready to ship the job.

Solution

Checkpoints are automatic recovery. Savepoints are operator-triggered, versioned points for deploy and resize. You do not “take a checkpoint” for a migration; you take a savepoint.

Solution

A transactional or idempotent sink and a business key. Checkpoint barriers without a sink contract are at-least-once in practice. Dual-write without MERGE is a duplicate table.

Solution

Flink for low-latency keyed state and true event-time. Spark for micro-batch ETL that happens to read Kafka. If the SLA is warehouse minutes, you may not need either — Dynamic Tables / DLT might be enough.

Solution

Name the key, salt or pre-aggregate, or isolate the skew. Adding task managers without fixing the key just moves the hotspot. Same answer as Spark skew.

Solution

Late events vs completeness. A 10s watermark closes windows fast and drops late data (or side-outputs it). 10 minutes waits and uses more state. The product owner picks the lateness, not the framework default.

Solution

Checkpoint is automatic recovery state. Savepoint is an operator-triggered, versioned snapshot for upgrade/rewrite. You take a savepoint before a job graph change that would break checkpoints.

Solution

End-to-end is the weakest sink. The MERGE must be idempotent on a primary key. “Flink is exactly-once” without a sink contract is marketing.

Solution

TTL on state, tighter windows, incremental checkpoints, and whether you stored the whole event instead of an aggregate. Scaling TaskManagers helps after you stop keeping forever.

Solution

Flink SQL if the window is standard and you want a table API. DataStream when you need custom timers, side outputs, or a sink the SQL connector cannot express. Do not write DataStream to look senior.

Solution

It buckets into 09:00 and lies. Event-time windows + watermarks put counts on 03:00. Dashboards that use processing time after an outage show a fake spike.