DataLane

All stacks · Cloud platforms

GCP

BigQuery, Dataflow, Pub/Sub, Dataproc, and Composer.

GCP cover

Related reading

About GCP

Google Cloud's data stack is the most opinionated of the big three: BigQuery at the center, Dataflow for unified batch and streaming, Pub/Sub for messaging, and increasingly Vertex AI alongside. Teams choose GCP for exactly this coherence — fewer assembly decisions, more building.

These guides cover the platform beyond BigQuery itself: Pub/Sub-to-BigQuery streaming architectures, Dataflow versus Spark trade-offs, Cloud Composer operations, Dataplex governance, and cost management across a serverless-first stack.

What you'll learn here

  • The reference architecture: Pub/Sub → Dataflow → BigQuery, and when to simplify it
  • Dataflow (Apache Beam) concepts and when Spark on Dataproc fits better
  • Cloud Composer (managed Airflow) sizing and operations
  • Governance with Dataplex and IAM patterns for data teams
  • Cost management in a serverless-first platform

Frequently asked questions

Is learning GCP worth it versus AWS?

AWS has more job volume; GCP punches above its weight in data roles specifically because BigQuery is so widely adopted, including by companies that otherwise run AWS. Skills transfer heavily — object storage, IAM, and orchestration concepts are the same shape everywhere.

Dataflow or Dataproc (Spark)?

Dataflow for fully serverless pipelines and unified batch/streaming under the Beam model, especially when you do not want cluster operations. Dataproc when your team already writes Spark, needs its ecosystem, or wants to reuse code across clouds.

Do I need Dataflow if BigQuery can do ELT?

Often not — Pub/Sub BigQuery subscriptions land streams directly into tables, and scheduled SQL or Dataform handles transformation. Dataflow earns its complexity for heavy per-event enrichment, joins across streams, or exactly-once stateful processing.

What is Dataplex for?

Lake and warehouse governance: cataloging assets across projects, data quality scans, lineage, and policy at the domain level. It matters once multiple teams share the platform — before that, plain IAM plus good dataset conventions is usually enough.

How does GCP pricing bite data teams?

BigQuery on-demand scans (fixable with partitioning and byte limits), Dataflow jobs left streaming with no traffic, and cross-region egress in multi-region architectures. Budgets plus alerts per project are ten minutes of setup that prevent most surprises.

New GCP posts, straight to your inbox

One email a week with our latest tutorials. No spam.

Newsletter signup is not live yet. Use the contact form if you want to be notified.

↑↓ navigate openesc close