Where Lambda Belongs in a Data Pipeline and Where It Quietly Becomes a Distributed System You Cannot Debug

The first Lambda in a data pipeline is always a good idea. A file lands in S3, a function validates the header row and moves it to the right prefix. Twenty lines of Python, no infrastructure, costs nothing. It works perfectly for a year.

The eleventh Lambda is where it goes wrong. By then the function that validated a header is calling another function that enriches records, which writes to a queue read by a function that loads a warehouse, which publishes an event that triggers a function that refreshes a dashboard. Nobody drew this. It grew. And the first time it breaks at 3 a.m. you discover that you have built a distributed system with no orchestrator, no run history, no retries you control, and no way to answer the only question that matters: did every record get processed exactly once?

I have built that system and I have dismantled it. This post is what I learned about where Lambda genuinely belongs in a data pipeline, the limits that quietly shape every design, and the specific signals that mean you have crossed from “glue” into “framework I now maintain.”

What Lambda is genuinely excellent at

Three jobs, and it is the best tool in AWS for all three.

Event routing. Something happened, decide what should react. An S3 ObjectCreated event that inspects the key and starts the right downstream process. An EventBridge rule that fans out. This is Lambda’s home turf: short, stateless, one decision.

Stateless record reshaping. Parse, flatten, tag, redact. A Firehose transformation Lambda that turns nested JSON into a flat record before Parquet conversion is exactly right — see the Firehose patterns post for how that composes. No state, no coordination, bounded input.

Cheap control-plane work. Calling an API to kick off a Glue job, refreshing a partition, posting to Slack when a check fails, writing an audit row. Work measured in seconds that happens thousands of times a day and would be absurd to run on a server.

The common thread: one input, one decision, no memory of anything before it. Every Lambda pathology I have debugged came from violating that sentence.

flowchart TD
  s3["S3 object created"] --> l1["Lambda: route and validate"]
  l1 --> good{"Valid?"}
  good -->|no| quar["Quarantine prefix + alert"]
  good -->|yes| sfn["Step Functions execution"]
  sfn --> glue["Glue or EMR job"]
  sfn --> load["Warehouse load"]
  sfn --> check["Data quality check"]
  check --> done["Publish completion event"]

Lambda decides. Something with run history executes. That boundary is the whole design.

The limits that shape the design

Memorize these, because every architecture decision is downstream of them.

• 15 minutes maximum execution. Hard.
• 10,240 MB maximum memory, and CPU scales with memory. A function at 1,769 MB gets roughly one full vCPU; below that you are on a fraction.
• 512 MB of /tmp by default, configurable to 10,240 MB.
• 6 MB synchronous request and response payload; 256 KB for asynchronous invocations and most event sources.
• 1,000 concurrent executions per region by default, shared across every function in the account.
• 50 MB zipped deployment package, 250 MB unzipped, or 10 GB as a container image.

The timeout is the one everyone quotes and the one that matters least. If your work does not fit in 15 minutes it obviously does not fit, and you knew that. The limits that actually cause incidents are the payload sizes and the concurrency ceiling, because they fail intermittently under load rather than immediately in testing.

Memory is the interesting lever because it is really a CPU lever. I have cut the cost of a JSON-heavy function by more than half by raising memory from 512 MB to 2,048 MB: four times the price per millisecond, but six times faster, and the 1 ms billing granularity means you keep the difference. Never tune Lambda memory by guessing. Run the function at 512, 1024, 1769, and 3008 MB against a real payload and pick the cheapest total.

Idempotency is your job, not Lambda’s

This is the section I wish someone had made me read.

Lambda’s retry behavior depends on the invocation type, and none of the options is “exactly once.”

• Synchronous (API Gateway, direct invoke): the caller retries or does not. Lambda does nothing.
• Asynchronous (S3, SNS, EventBridge): Lambda retries twice by default, with delays, then sends to a dead-letter destination if you configured one. If you did not, the event is gone.
• Poll-based (SQS, Kinesis, DynamoDB Streams): the event source mapping retries the batch until it succeeds or expires. For Kinesis that means a poison record blocks the shard until the record ages out of retention.

So the same S3 file can trigger your loader twice. The same Kinesis batch can be reprocessed after a partial failure. In a data pipeline, “processed twice” means “double-counted revenue,” and that is the bug that costs you credibility with the finance team forever.

The fix is an idempotency key and a conditional write. Not a “check if it exists then write” — that races. A single atomic operation:

import hashlib
import os
import boto3
from botocore.exceptions import ClientError

ddb = boto3.client("dynamodb")
TABLE = os.environ["IDEMPOTENCY_TABLE"]  # PK: idem_key, TTL attr: expires_at


def claim(idem_key: str, ttl_seconds: int = 86_400 * 7) -> bool:
    """Atomically claim a unit of work. False means someone already has it."""
    import time

    try:
        ddb.put_item(
            TableName=TABLE,
            Item={
                "idem_key": {"S": idem_key},
                "expires_at": {"N": str(int(time.time()) + ttl_seconds)},
            },
            # The whole point: fails if the key exists. No read-then-write race.
            ConditionExpression="attribute_not_exists(idem_key)",
        )
        return True
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise


def handler(event, context):
    for record in event["Records"]:
        bucket = record["s3"]["bucket"]["name"]
        key = record["s3"]["object"]["key"]
        etag = record["s3"]["object"]["eTag"]  # changes when content changes
        # Key on content identity, not on delivery identity.
        idem_key = hashlib.sha256(f"{bucket}/{key}/{etag}".encode()).hexdigest()

        if not claim(idem_key):
            print(f"skip duplicate {bucket}/{key} etag={etag}")
            continue

        load_file(bucket, key)

Two details that matter. The key is derived from content identity (bucket, key, ETag), not from the request ID or the event ID, because the same file redelivered is a duplicate while the same key with new content is not. And the TTL is long enough to cover your realistic replay window — seven days, not one hour, because the duplicate that hurts you arrives during an incident recovery, not during normal operation.

For SQS specifically, use partial batch responses so one bad record does not force redelivery of nine good ones:

{
  "FunctionName": "orders-loader",
  "EventSourceArn": "arn:aws:sqs:us-east-1:123456789012:orders-queue",
  "BatchSize": 100,
  "MaximumBatchingWindowInSeconds": 20,
  "FunctionResponseTypes": ["ReportBatchItemFailures"],
  "ScalingConfig": { "MaximumConcurrency": 40 },
  "comment": "Handler returns {'batchItemFailures': [{'itemIdentifier': msgId}]}"
}

ReportBatchItemFailures is one line of configuration and one return value, and it is the difference between “one poison message reprocesses 99 good ones forever” and “one poison message goes to the DLQ.” I have never seen a team enable it before their first incident.

Concurrency is a shared account resource

The default limit is 1,000 concurrent executions per region for all functions in the account combined. This is the sharpest edge in serverless data engineering and it is invisible until it cuts you.

Here is the shape of the incident. A backfill drops 400,000 files into S3. The routing Lambda scales to 1,000 concurrent instances in seconds. Every other Lambda in the account — the one that answers your API, the one that processes payments, the one that sends alerts — starts getting throttled, because there is no capacity left. The alerting Lambda cannot alert about the outage because it is part of the outage.

Two settings prevent this and you should set both:

Reserved concurrency caps a function and guarantees it that capacity. Set it on every data-plane function. A loader with reserved concurrency of 50 cannot consume the account.

Maximum concurrency on the event source mapping (for SQS) throttles the poller rather than the function, which avoids burning receive counts on invocations that will be throttled anyway.

There is a third consideration people forget: what is downstream. A Lambda that scales to 500 concurrent executions and each one opens a database connection has just opened 500 connections. Postgres will refuse them. Redshift will queue them. Snowflake will spin up warehouse clusters and bill you for the privilege. Lambda’s elasticity is only useful if everything it touches is equally elastic, and warehouses are not.

If your function writes to a warehouse, batch through a queue with a bounded concurrency and use a connection proxy, or write to S3 and load in bulk. Never let per-record Lambdas talk directly to a relational warehouse.

The point where it becomes a distributed system

Here are the five signals. When I see two of them, I stop adding Lambdas.

1. A function invokes another function. Direct Lambda-to-Lambda invocation is a function call with a network partition in the middle and no stack trace.
2. State lives in DynamoDB purely to coordinate steps. If you have a table whose rows mean “step 3 of 5 completed,” you have written an orchestrator.
3. Retry logic is in your handler. Sleeps, backoff loops, and if attempt < 3 inside a function that is already being retried by the platform. These compound in ways nobody can reason about.
4. You cannot answer “what happened to the file from Tuesday?” without grepping CloudWatch Logs Insights across four log groups.
5. A deploy of one function requires testing three others. That is a distributed monolith with extra latency.

The fix is not “use fewer Lambdas.” It is to move the workflow into something that has a workflow model — Step Functions for event-driven AWS-native flows, Airflow for scheduled data pipelines with dependencies. I compare those two directly in Step Functions vs Airflow. Lambda stays, as the thing that does one unit of work inside a step that has retries, timeouts, and a visible execution history.

That refactor is usually smaller than it sounds. The business logic in the handlers rarely changes. What changes is that the invoke calls and the coordination table disappear, replaced by a state machine definition you can read in one screen.

Observability: log for the incident, not for the demo

CloudWatch gives you invocation counts, duration, errors, and throttles. That tells you a function is unhappy. It does not tell you which file failed.

What I log from every data-plane Lambda, as structured JSON:

import json
import logging
import time

log = logging.getLogger()
log.setLevel(logging.INFO)


def emit(event_name: str, **fields):
    """One line of JSON per event. Queryable in Logs Insights."""
    log.info(json.dumps({"event": event_name, "ts": time.time(), **fields}))


def handler(event, context):
    started = time.time()
    key = event["Records"][0]["s3"]["object"]["key"]
    emit(
        "load.start",
        request_id=context.aws_request_id,
        s3_key=key,
        # Retry detection: same key, different request id, within minutes.
        remaining_ms=context.get_remaining_time_in_millis(),
    )
    try:
        rows = load_file(key)
        emit("load.ok", s3_key=key, rows=rows, ms=int((time.time() - started) * 1000))
    except Exception as exc:
        emit("load.fail", s3_key=key, error=type(exc).__name__, detail=str(exc)[:500])
        raise  # Re-raise so the platform's retry and DLQ actually engage

The raise at the end is not decoration. A Lambda that catches every exception and returns successfully has told the event source “this batch is done,” and your data is gone with a green metric. Swallowing exceptions in a data pipeline Lambda is the single most common cause of silent loss I have found in code review.

Two more things worth wiring up: a CloudWatch alarm on the function’s Throttles metric (not just Errors), and a dead-letter queue with an alarm on depth greater than zero. A DLQ nobody watches is a data loss archive.

Pitfalls

Catching exceptions to keep the dashboard green. Errors are the mechanism by which retries and DLQs work. A handler with a bare except: pass has disabled every reliability feature AWS gives you.

No reserved concurrency on data-plane functions. One backfill consumes the account limit and takes unrelated production systems with it. Set it on everything that reads a queue or an S3 event.

Direct connections from Lambda to a warehouse. Five hundred concurrent executions means five hundred connection attempts. Land in S3 and bulk load, or route through a bounded-concurrency queue.

Treating the 15-minute timeout as the design constraint. A job that takes 14 minutes today takes 16 next quarter. If runtime scales with data volume, Lambda is the wrong runtime — that work belongs in Glue or EMR.

Kinesis poison records with no bisect. By default a failing batch is retried until the records expire, blocking the shard. Set BisectBatchOnFunctionError, a MaximumRetryAttempts, and an OnFailure destination.

Layers as dependency management. Layers are convenient and they are also an invisible version pin shared across functions. I have debugged a “works in staging” failure that was one function pinned to layer version 3 and another to version 4. Use container images when dependencies are non-trivial.

FAQ

Is Lambda cheaper than running a small container?

For bursty and infrequent work, dramatically. For steady work it inverts fast: a function running continuously at 1 GB costs roughly an order of magnitude more than a Fargate task doing the same thing. My rough line is 25 to 30 percent duty cycle. Above that, price a container.

Can I run pandas or DuckDB in a Lambda?

Yes, and it is a genuinely good pattern for small-to-medium files. A 2 GB Parquet file can be aggregated in a 10 GB Lambda in under a minute, which beats spinning up Spark by a wide margin — the same argument I make in DuckDB for local pipelines. Use a container image, mount enough /tmp, and watch the memory ceiling.

How do I handle a file bigger than memory?

Stream it. boto3 gives you a file-like object from get_object()["Body"], and both pyarrow and csv can iterate it in chunks. If you cannot stream it because the work needs the whole dataset in memory, that is your signal to move to Glue.

What is the right batch size for an SQS-triggered Lambda?

Start at 10 with a batching window of a few seconds, and raise it only if per-invocation overhead dominates. Large batches amplify the cost of a single poison message unless you have ReportBatchItemFailures enabled. With it enabled, batches of 100 are reasonable.

Should Lambda write directly to my lake tables?

To S3, yes. To transactional table formats, be careful: concurrent Lambda writers to the same Iceberg or Delta table will collide on commit and retry, and at high concurrency you get a livelock. Land raw files from Lambda and do table commits from a single coordinated writer.

Does provisioned concurrency fix cold starts for data workloads?

It fixes latency, at the price of paying for idle capacity. For data pipelines, cold starts are almost never the problem — a 200 ms cold start on a job with a five-minute SLA is noise. Save provisioned concurrency for synchronous, user-facing paths.

What this means for your pipelines

Use Lambda for the jobs it is unbeatable at: routing events, reshaping records, and making small API calls. Keep each function to one input and one decision, make it idempotent with a conditional write, give it reserved concurrency, and let it throw exceptions so the platform’s retries mean something. Done that way, a Lambda is the cheapest and most reliable component in your architecture.

The discipline is refusing to let it become the architecture. Every time you are tempted to have one function call another, or to track “which step are we on” in a table, you are choosing to build an orchestrator with no UI, no run history, and no way to replay Tuesday. Move that coordination into Step Functions or Airflow and keep the functions as leaf nodes.

The test I use in design review is simple: can a new engineer, given a failed record, find out what happened to it in under five minutes without asking anyone? If the answer is yes, the Lambdas are glue. If the answer is “well, first you check this log group, then this DynamoDB table,” you have already built the distributed system you cannot debug — and the good news is that it is much easier to fix now than after the next eleven functions.