LinkedIn data interview pack

Streaming

Consumer groups, rebalances, and why compaction is wrong on an event topic.

SQL

A feed metric that must not double-count impressions after a join to a member dimension.

hard · SQL

When do you use a covering index / clustering key versus rewriting the query?

If the query already filters a selective column and still scans everything, storage layout (partition, cluster, index) is the lever. If it selects * and joins at the wrong grain, rewrite first. Do not add a clustering key to hide a CROSS JOIN.

easy · Spark

Narrow vs wide transformation. Why does that decide runtime?

Narrow (map, filter) stays on the same partition. Wide (groupBy, join) shuffles over the network. Shuffle is the expensive part. If you can filter and project before a join, you cut shuffle bytes. That is the interview.

easy · Spark

foreach on the driver vs foreachPartition.

Side effects belong on executors, idempotently. A driver for-loop over collect() is the anti-pattern.

medium · Recent signal, 2026-08-22

Your consumer group rebalances every few minutes during peak. What do you check first?

max.poll.interval.ms vs processing time, cooperative sticky assignment, and whether a slow partition (hot key) is triggering leave/join. Adding consumers without fixing poll interval often makes lag worse.

easy · Kafka

Why can two consumers in the same group not share one partition?

A partition has a single offset per group. Throughput scales with partitions, not with extra consumers parked on the same partition. To add parallelism, add partitions (and accept the rebalance) or split topics by domain.

easy · Kafka

Compaction vs deletion retention. When do you use a compacted topic?

Compacted topics keep the latest value per key (CDC changelog, config, ktables). Time retention drops old messages regardless of key (raw events). Using compaction on a clickstream deletes history you still need for sessionization.

easy · Kafka

max.poll.interval vs a slow sink.

If processing exceeds the interval, the member is kicked and the group rebalances. Shrink the work per poll, speed the sink, or raise the interval with eyes open.

hard · Kafka

linger.ms and batch.size — what are you trading?

Latency for throughput. A firehose with linger 0 wastes RPCs. Correctness is acks/ISR, not linger.

medium · Kafka

Schema registry compatibility in CI.

A producer adding a required field will kill old consumers. Fail the PR on a BACKWARD/FORWARD break you did not mean.

hard · Kafka

acks=all and a producer still duplicates. How?

Retries after a timeout when the broker actually committed. Idempotent producer (enable.idempotence) plus a key. acks=all is durability, not exactly-once at the consumer.

medium · Kafka

Schema Registry BACKWARD vs FULL. What can a producer do on Monday?

BACKWARD: consumers with the new schema can read old data — typically add optional fields. FULL needs both directions. A required field without a default breaks consumers. CI should check compatibility, not Slack.

easy · Apache Flink

Event time vs processing time. When does the dashboard lie?

Processing time windows close on wall clock. Late Kafka events fall in the wrong window. Event time plus watermarks is the honest window. If you cannot explain watermarks, you are not ready to ship the job.

easy · Databricks

A shuffle-heavy join OOMs the worker. Name three levers before you “just scale the cluster.”

Broadcast the small side if it fits. Repartition both on the join key. AQE / skew join hints. Filter before the join. If it is still huge, rethink the grain — you may be exploding a many-to-many. Scaling the cluster last is the expensive answer.

hard · Trino

SELECT * on a 400-column fact. What happens?

Workers shuffle the world. Column prune in the SQL and in the file format. Trino is not magic; it is a distributed scan.

easy · Trino

Trino does not store the table. So what does it store, and why do people still page you?

It stores the query engine’s metadata/catalog config and the plan. The data lives in Iceberg/Hive/Postgres. People page you because a federated scan of prod Postgres looked like a warehouse.

easy · Trino

SELECT * on a 400-column Iceberg fact in Trino. What happens to the plan?

It still reads every column from Parquet footers/row groups you need — column pruning helps only if you list columns. Teach BI to select metrics. SELECT * is a cost bug.

hard · System design

Exactly-once from Kafka into a warehouse. What do you actually promise?

At-least-once plus an idempotent sink (MERGE on event_id / offset+partition). True EOS is a specific Kafka + sink setup. Saying “Kafka is exactly-once” without a key is a fail.