AWS

Solution

Ad-hoc and cheap scans: Athena on partitioned Parquet with a Glue catalog. Concurrent BI and predictable SLAs: Redshift (or Redshift Serverless). Glue is the job runner / catalog, not the warehouse. “Everything in Athena” dies on 40 concurrent dashboards.

Solution

Hive-style dt=YYYY-MM-DD (and maybe hour) on the columns you filter. Do not partition on high-cardinality ids (customer_id). Compact tiny files (target ~128–512 MB). Clustering / sort inside the file helps, but partition pruning is the first win.

Solution

Lambda is for minutes and megabytes, not Spark. Glue is managed Spark for batch ETL you do not want to babysit. EMR (or EKS + Spark) when you need custom images, long-running clusters, or cheaper reserved capacity. Say the timeout and the data size first.

Solution

SQS: decouple workers, no replay of a stream, competing consumers. Kinesis: managed stream, shards, AWS-native. MSK: Kafka protocol on AWS when you already think in topics and consumer groups. Do not put a 7-day clickstream on SQS.

Solution

A join on a column that is not the dist key, so every slice shuffles everything. Even in AUTO, a bad mental model (SELECT * fact JOIN huge dim ON a non-selective key) will not be saved by a wizard. Explain slice-local joins in one sentence.

Solution

Bucket policies are path-coarse. Lake Formation (or a catalog-centric grant model) is for column/row filters across Athena/Redshift Spectrum on the same Glue tables. If you only have two roles and one bucket, IAM is enough. Do not buy a governance product to hide a public bucket.

Solution

The default committer can list/rename badly on S3. Use a magic committer and an idempotent overwrite of a partition.

Solution

Untagged shared resources become a mystery bill. You should be able to say which DAG spent the credits.

Solution

DMS or Debezium to Kinesis/MSK, then MERGE into Iceberg/Delta/Redshift. A nightly dump is not CDC.

Solution

Close public access first. LF does not excuse a public prefix. Encryption and logging are next.

Solution

Did the query filter the partition column (not DATE(ts) wrapping)? Are there thousands of tiny files? Is SELECT * pulling 400 columns? Then look at partition projection vs Glue partitions that never got added.

Solution

Inventory (or S3 Batch) for discovery. LIST in a loop throttles and times out. Object creation events (SQS) for incremental ingest, not a nightly LIST.

Solution

When files are overwritten in place, clocks go backward, or you change the source path. A watermark you own in the lake is debuggable. Bookmarks are fine for append-only dumps you control.

Solution

Two copies of the fact “for convenience.” One lake table, engines as clients. Spectrum and Athena both scan S3; duplicating into a local Redshift table is a choice with a vacuum cost, not a default.

Solution

Buffer size/interval on Firehose, then a compaction job (Glue/Spark) to 128 MB+ objects. Query engines hate tiny files more than they hate a 60s buffer delay.

Solution

When many accounts/roles need column- and table-level grants on a shared catalog. A single team with two buckets can stay on IAM + bucket policy. LF is not a substitute for not putting PII in the key name.