AWS Data Engineer Interview Questions cheat sheet
S3, Glue, Redshift, Kinesis, EMR, and Lambda questions with the service trade-offs AWS data interviews expect you to name.
S3 and storage
What consistency guarantees does S3 provide?- Since December 2020, S3 gives strong read-after-write consistency for all operations including overwrites and list, in every region, at no extra cost. That removed the need for workarounds like EMRFS consistent view and S3Guard. List operations are still eventually consistent only in the sense of cross-region replication, which is a different mechanism.
How do you choose S3 storage classes for a data lake?- Standard for active data, Intelligent-Tiering when access patterns are unknown since it moves objects automatically for a small monitoring fee per object, Glacier Instant Retrieval for archives needing millisecond access, and Deep Archive for compliance retention. Watch minimum storage durations — 30 days for Standard-IA, 180 for Deep Archive — which make small short-lived objects cost more, not less.
How should you lay out prefixes and partitions in a data lake?- Hive-style key=value prefixes such as dt=2026-03-01 so Glue and Athena can prune partitions. S3 now scales to at least 5,500 GET and 3,500 PUT requests per second per prefix and partitions internally, so the old random-hash-prefix advice is obsolete. The real constraint is file size — aim for 128 MB to 1 GB Parquet files.
What is the difference between S3 versioning, replication, and Object Lock?- Versioning keeps prior object versions so overwrites and deletes are recoverable, and it is a prerequisite for the other two. Replication copies objects to another bucket or region asynchronously for resilience or locality. Object Lock enforces WORM retention so even an administrator cannot delete within the retention period, which is what compliance auditors actually ask for.
How do you secure data in an S3-based lake?- Block Public Access at the account level, bucket policies plus IAM roles rather than long-lived keys, SSE-KMS with a customer-managed key when you need key-level audit and revocation, and VPC gateway endpoints so traffic never leaves the AWS network. Add Lake Formation for table and column grants when many teams share the same buckets.
Ingestion and streaming
Compare Kinesis Data Streams, Firehose, and MSK.- Data Streams is a low-latency shard-based stream you consume yourself, with 24-hour default retention extendable to 365 days. Firehose is a fully managed delivery pipeline to S3, Redshift, or OpenSearch with buffering and format conversion but no replay. MSK is managed Apache Kafka, chosen when you need the Kafka ecosystem, existing connectors, or portability.
How does sharding work in Kinesis Data Streams?- Each shard handles 1 MB per second or 1,000 records per second in, and 2 MB per second out shared across consumers unless you use enhanced fan-out, which gives each consumer its own 2 MB per second. Records with the same partition key go to the same shard, so a hot key throttles regardless of total capacity. On-demand mode scales shards automatically.
What is DMS used for and what is its main limitation?- Database Migration Service does one-time loads and ongoing change data capture from operational databases into S3, Redshift, or Kafka. The limitations to name are DDL change handling, which often needs a manual full reload, and its lack of transformation capability — it moves data, and modelling must happen downstream.
When is Lambda the right tool for data processing?- Event-driven, short, stateless work: reacting to an S3 put, light enrichment, routing, or triggering another job. Its limits are a 15-minute maximum execution, 10 GB memory, and 10 GB of ephemeral storage, so anything requiring a large shuffle or long scan belongs in Glue or EMR. Also watch concurrency limits throttling downstream databases.
How do you handle exactly-once delivery into S3 from a stream?- You generally get at-least-once, so make the write idempotent: derive the object key deterministically from the batch identifier or the source offsets so a retry overwrites the same key rather than adding a duplicate file. Then make the table format handle it — Iceberg or Delta commits give atomic visibility of a batch.
Processing with Glue and EMR
What are the components of AWS Glue?- The Data Catalog as a Hive-compatible metastore shared by Athena, EMR, and Redshift Spectrum; crawlers that infer schema and register partitions; Glue ETL jobs running managed Spark; Glue Studio for visual authoring; and Glue Data Quality for rule-based checks. The Catalog is the piece other services depend on, so treat it as the governance surface.
What is a Glue DynamicFrame and why does it exist?- A DynamicFrame is Glue's schema-flexible wrapper over a Spark DataFrame that tolerates inconsistent records, tracking a choice type when a field has multiple types and letting you resolve it later. Use it at ingestion where schemas are messy, then call toDF() and use plain Spark for the heavy transformation, because DataFrames get the full Catalyst optimizer.
When would you choose EMR over Glue?- EMR when you need cluster control — specific Spark, Hive, or Flink versions, custom native libraries, long-running clusters, deep spot instance cost optimization, or non-Spark frameworks. Glue when you want serverless per-job billing (with a 1-minute minimum) and tight Catalog integration without managing infrastructure. EMR Serverless sits between the two.
What are Glue job bookmarks and where do they fail?- Bookmarks persist processed state so a rerun skips already-processed S3 objects or JDBC rows. They fail when files are overwritten in place rather than added, when the bookmark key column is not monotonically increasing, and when a job partially succeeds — which is why you should still design idempotent writes rather than treating bookmarks as your only defense.
How do you cut EMR cost meaningfully?- Spot instances for task nodes with on-demand core and primary nodes, instance fleets with diverse types to reduce interruption, managed scaling, and transient clusters per job rather than always-on. Then attack the data side: columnar formats, partition pruning, and right-sized files, since compute time is usually wasted on scanning too much.
Redshift and analytics
Explain distribution styles in Redshift.- KEY co-locates rows with the same value on the same slice so joins on that key avoid redistribution; ALL replicates a small dimension to every node; EVEN round-robins; AUTO lets Redshift decide and switch as the table grows. Choosing KEY on a skewed column creates hot slices, which is the failure mode interviewers look for.
What is a sort key and how does it differ from an index?- A sort key controls physical row order, letting the zone maps (min/max per block) skip blocks that cannot match a filter — the columnar equivalent of an index without one being maintained separately. Compound sort keys favor the leading column; interleaved keys weight columns equally but are expensive to maintain and now rarely recommended.
What do VACUUM and ANALYZE do in Redshift?- VACUUM reclaims space from deleted rows and re-sorts unsorted regions; ANALYZE refreshes statistics for the planner. Modern Redshift runs both automatically in the background during low load, so the interview answer is that you monitor svv_table_info for unsorted percentage and stale statistics rather than scheduling them blindly.
Compare Redshift Spectrum, Athena, and Redshift Serverless.- Spectrum queries S3 from a Redshift cluster, joining external and internal tables and billed per terabyte scanned. Athena is a standalone serverless Trino-based engine over the Glue Catalog with the same per-scan billing. Redshift Serverless removes cluster management and bills in RPU-hours. Choose Athena for ad hoc lake queries and Redshift for repeated, concurrency-heavy BI.
How do you load data into Redshift efficiently?- COPY from S3 with multiple files so every slice loads in parallel, ideally a multiple of the slice count and each roughly 1 MB to 1 GB compressed. Avoid single-row INSERTs entirely. Use a staging table plus MERGE for upserts, and let COPY analyze compression on first load or set encodings explicitly with AZ64 for numerics.
How do you reduce Athena query cost?- Partition on the columns queries filter, convert to Parquet or ORC so only needed columns are read, compress with Snappy or Zstd, and avoid SELECT star. Athena bills per terabyte scanned with a 10 MB minimum per query, so many tiny files also cost more than their bytes. Partition projection avoids the crawler for predictable date layouts.
Orchestration, governance, and design
How do you orchestrate an AWS data pipeline?- Step Functions for service-to-service state machines with retries and error handling and no server to run, MWAA when teams already know Airflow and need its operators and backfills, and EventBridge for event-driven triggering. Glue workflows work for Glue-only pipelines. Justify the choice on team familiarity and cross-service breadth, not on features alone.
What does Lake Formation add over IAM and S3 policies?- Table, column, row, and cell-level grants over Glue Catalog objects, plus tag-based access control so permissions follow classification rather than paths, and cross-account sharing without copying data. IAM alone can only reason about prefixes, so a column-level rule is impossible. The cost is another permission model to keep aligned with IAM.
How do you monitor and alert on a pipeline in AWS?- CloudWatch metrics and alarms per job, structured logs going to CloudWatch Logs with metric filters for known error signatures, EventBridge rules catching Glue and Step Functions failure events into SNS, and CloudTrail for who changed what. Add data-level checks with Glue Data Quality, because a job that succeeds on empty input raises no alarm.
How would you design a cost-efficient data lake on AWS today?- S3 with Iceberg tables in the Glue Catalog (or S3 Tables for managed Iceberg), ingestion by Firehose or DMS into bronze, transformation with Glue or EMR Serverless, Athena for ad hoc and Redshift Serverless for BI concurrency, Lake Formation for grants, and lifecycle policies plus compaction jobs. The unifying point is one storage copy queried by several engines.
What is the difference between an IAM role, an instance profile, and a service role?- A role is a set of permissions assumable by a principal. An instance profile is the container that lets an EC2 instance assume a role. A service role is one an AWS service assumes on your behalf, such as Glue reading S3. The recurring interview point is that data jobs should assume roles rather than hold access keys, and role trust policies are where cross-account access breaks.
From DataLane — tutorials at/blog, practice SQL live in theplayground.