All stacks · Cloud platforms
Glue, Kinesis, Redshift, S3, Athena, EMR, and the AWS data platform.
AWSS3, Glue, Kinesis, Redshift, Athena, EMR, Lambda — a map of the AWS data services, what each is actually for, and the combinations that work in practice.
AWSState transitions versus scheduler hours, execution history versus task logs, and the specific signals that tell you which orchestrator your pipeline has outgrown.
AWSLambda is excellent glue and a terrible framework. Here are the limits that shape every design, the idempotency you must build yourself, and the point to stop.
AWSA production comparison of Redshift Serverless and Snowflake: how RPU-seconds differ from credits, where each one bends under concurrency, and what actually breaks in a migration.
AWSAthena bills you for bytes scanned. Here is the partition projection, Parquet layout, CTAS compaction, and workgroup guardrails I use to cut that number by 90 percent.
AWSWhen Kinesis Data Streams earns its shards, when Firehose is enough, and the buffering and partitioning patterns that keep S3 landings queryable.
AWSGlue vs EMR for production Spark: DPU pricing vs instance pricing, cold starts, library control, and the job size where teams switch.
Amazon RedshiftCOMPOUND sort keys and DISTKEY still decide scans on RA3. VACUUM and ANALYZE are not optional. Spectrum SELECT * is an S3 bill. Concurrency scaling will surprise you.
KafkaWhen to run Kafka (or MSK) versus Kinesis Data Streams: partitions vs shards, replay, multi-cloud, and the hidden cost of “managed.”
AWS is where most data platforms physically live, whether the warehouse on top is Redshift, Snowflake, or Databricks. S3 is the default lake, Glue the default catalog, and the ingestion, orchestration, and IAM services around them are the plumbing every AWS data engineer maintains.
These guides map the AWS data stack from a practitioner's view: S3 layouts that engines can prune, Glue and Athena without the small-file tax, Kinesis versus MSK for streams, Step Functions versus MWAA for orchestration, and the IAM patterns that keep security review short.
S3, IAM, Glue/Athena, and one ingestion path (Kinesis or DMS) cover the daily 80%. Lambda and Step Functions follow quickly. Everything else — EMR, Redshift, MSK — can be learned when a workload demands it.
Glue for serverless batch jobs where you want zero cluster management; EMR when you need custom libraries, long-running clusters, spot-instance economics at scale, or Spark versions Glue does not ship. Most teams outgrow Glue later than they expect.
Almost always small files, missing partitions, or JSON/CSV instead of Parquet. Partition by date, compact files toward 128 MB+, convert to Parquet with compression, and query cost typically drops by 10x or more.
Kinesis for AWS-native simplicity at modest scale — no brokers to run. MSK (managed Kafka) when you need the Kafka ecosystem, replay flexibility, or throughput economics past Kinesis pricing. Firehose is the right answer when the goal is just "stream to S3 in batches."
No — pick one warehouse. Redshift makes sense for heavy AWS-native integration and reserved-capacity pricing; if Snowflake is already the standard, adding Redshift mostly adds migration and governance overhead.
One email a week with our latest tutorials. No spam.
Newsletter signup is not live yet. Use the contact form if you want to be notified.