AWS

Solution

Ad-hoc and cheap scans: Athena on partitioned Parquet with a Glue catalog. Concurrent BI and predictable SLAs: Redshift (or Redshift Serverless). Glue is the job runner / catalog, not the warehouse. “Everything in Athena” dies on 40 concurrent dashboards.

Solution

Hive-style dt=YYYY-MM-DD (and maybe hour) on the columns you filter. Do not partition on high-cardinality ids (customer_id). Compact tiny files (target ~128–512 MB). Clustering / sort inside the file helps, but partition pruning is the first win.

Solution

Lambda is for minutes and megabytes, not Spark. Glue is managed Spark for batch ETL you do not want to babysit. EMR (or EKS + Spark) when you need custom images, long-running clusters, or cheaper reserved capacity. Say the timeout and the data size first.

Solution

SQS: decouple workers, no replay of a stream, competing consumers. Kinesis: managed stream, shards, AWS-native. MSK: Kafka protocol on AWS when you already think in topics and consumer groups. Do not put a 7-day clickstream on SQS.

Solution

A join on a column that is not the dist key, so every slice shuffles everything. Even in AUTO, a bad mental model (SELECT * fact JOIN huge dim ON a non-selective key) will not be saved by a wizard. Explain slice-local joins in one sentence.

Solution

Bucket policies are path-coarse. Lake Formation (or a catalog-centric grant model) is for column/row filters across Athena/Redshift Spectrum on the same Glue tables. If you only have two roles and one bucket, IAM is enough. Do not buy a governance product to hide a public bucket.

Solution

The default committer can list/rename badly on S3. Use a magic committer and an idempotent overwrite of a partition.

Solution

Untagged shared resources become a mystery bill. You should be able to say which DAG spent the credits.

Solution

DMS or Debezium to Kinesis/MSK, then MERGE into Iceberg/Delta/Redshift. A nightly dump is not CDC.

Solution

Close public access first. LF does not excuse a public prefix. Encryption and logging are next.