MLOps for Data Engineers: Feature Stores, Training Pipelines, and Where You Fit

MLOps looks intimidating from the outside — new tools, new vocabulary, new conference talks. From the inside, it is mostly data engineering with models in the middle. The failures that hurt in production are stale features, leaky joins, and silent schema changes. That is our job.

flowchart LR
  raw[Raw events] --> feat[Feature pipeline]
  feat --> store[Feature store / tables]
  store --> train[Training DAG]
  train --> registry[Model registry]
  store --> serve[Online or batch scores]
  registry --> serve
  serve --> mon[Drift + outcomes]

Three pipelines and a service. Features, training, scores, then drift.

The ML lifecycle, as pipelines

A production ML system is three pipelines and a service:

1. Feature pipeline — raw events to model inputs. This is ETL.
2. Training pipeline — features in, model artifact out. A DAG with retries, lineage, and a schedule or trigger.
3. Inference — batch scoring (another pipeline) or a real-time API in front of the model.
4. Monitoring — data drift is a quality problem wearing a lab coat.

The new vocabulary for a data engineer is the model registry and evaluation against labels. Everything else transfers. If you can ship the Airflow DAG and dbt tests, you can run this shape.

What a feature store actually solves

Not “a place to put more columns.” Two problems:

Train/serve consistency. If days_since_last_order is computed one way in the training SQL and a subtly different way in the serving API, the model silently underperforms. A feature store computes each feature once and serves that definition to both paths.

Point-in-time correctness. Training rows must use feature values as they were at prediction time. Joining today’s features onto last year’s events leaks the future into training. Accuracy looks great in the notebook and dies in production.

Tools: Feast (open source), Tecton (managed), or platform-native stores (Databricks, SageMaker). Small teams often skip the product and enforce both properties with careful dbt models. That is legitimate until the feature count and the online path make the duplication painful.

sequenceDiagram
  participant Event as Training event
  participant Feat as Feature table
  participant Join as PIT join
  Event->>Join: event_at, user_id
  Feat->>Join: valid_from, valid_to, value
  Join->>Event: value as-of event_at

Features as-of the event, not as-of now. This is the join that stops leakage.

Point-in-time joins: the skill that transfers

This is the join. Learn it once.

-- features as-of event time, not as-of now
select
    e.user_id,
    e.event_at,
    e.churned,
    f.orders_30d,
    f.days_since_last_order
from ml.training_events e
left join ml.user_features_history f
    on f.user_id = e.user_id
   and f.valid_from <= e.event_at
   and (f.valid_to is null or f.valid_to > e.event_at);

If user_features is a current-state snapshot only, you cannot train honestly. You need history (type-2, or a daily snapshot you can as-of). That is slowly changing dimensions, which you already know.

A cheap test that catches leaks:

-- future leak: feature row starts after the event
select count(*) as leaked
from ml.training_events e
join ml.user_features_history f
    on f.user_id = e.user_id
   and f.valid_from > e.event_at
where f.orders_30d is not null;

leaked must be 0 in CI. I treat this like a uniqueness test on a fact table. If you only remember one MLOps habit, remember this.

Train/serve parity tests

Same feature name, same SQL (or the same generated SQL). The failure mode is a Python function in the API and a slightly different CASE in dbt.

Pattern that works without a vendor:

1. One dbt model (or one Feast definition) per feature group
2. Batch training reads that table as-of
3. Online serve reads the latest row from the same model, or a materialization built from it
4. A daily job recomputes a sample of online features from the batch definition and diffs

def parity_ok(online: float, batch: float, atol: float = 1e-6) -> bool:
    return abs(online - batch) <= atol

Log mismatches. A 1-day off-by-one in days_since_last_order is enough to wreck a churn model.

The minimum viable MLOps stack

For a team’s first production model you need surprisingly little:

Piece	What to use
Orchestration	The scheduler you already run
Feature tables	dbt + type-2 or daily snapshots
Experiment + registry	MLflow
Batch inference	Scheduled job → warehouse table
Monitoring	Input quality checks + weekly outcomes

import mlflow

MODEL_URI = "models:/churn_model@production"


def score_day(snapshot_date: str) -> None:
    model = mlflow.pyfunc.load_model(MODEL_URI)
    features = load_features(snapshot_date)  # same SQL as training as-of
    scores = model.predict(features)
    write_to_warehouse(scores, "ml.churn_scores")

Batch scoring is another pipeline step. Idempotent on snapshot_date. Retries rewrite the same partition. Do not pass a dataframe through XCom — write a table, like any other task.

Online inference is an API in front of the same feature definition plus the registered artifact. If you do not have a latency SLO, stay on batch until someone can name the SLO.

Skip the feature-store product until you have two consumers (train and serve) that have already drifted once. Buy it to stop the second drift, not to look busy.

Training DAG habits that are just DE habits

• Pin data. Training run 2026-08-01 reads a snapshot or a time travel table, not “whatever gold is today.”
• Pin code and model flavor. Log the git sha and the MLflow run id.
• Split by time, not a random 80/20 on users who exist in the future. Random splits leak.
• Promote through stages. Staging scores a holdout. Production is a registry alias, not a filename on someone’s laptop.

ml.churn_scores   -- partition by score_date
ml.churn_labels   -- realized outcome, late arriving
ml.churn_eval     -- weekly join of scores to labels

Late labels are late-arriving facts. Your merge/upsert patterns still apply.

Monitoring: drift is a quality check

You already know freshness, nulls, uniqueness, and ranges — see Python data quality checks. Add two ML-specific checks:

1. Input drift — weekly distribution of top features vs the training snapshot (PSI or a simple KS on one or two columns is enough to start)
2. Outcome eval — predictions vs realized labels once labels exist

select
    score_date,
    avg(abs(score - churned::int)) as mae
from ml.churn_eval
where score_date >= current_date - 28
group by 1
order by 1;

-- fail the feature job if a group is older than the SLO
select
    feature_group,
    max(valid_from) as latest_valid_from,
    datediff('hour', max(valid_from), current_timestamp()) as lag_hours
from ml.user_features_history
group by 1
having datediff('hour', max(valid_from), current_timestamp()) > 26;

A silent schema change (new enum, unit change from cents to dollars) is more common than cinematic “concept drift.” Fail the feature job if the contract breaks. Do not wait for the data scientist to notice AUC.

flowchart LR
  scores[ml.churn_scores] --> eval[Weekly join]
  labels[ml.churn_labels] --> eval
  eval --> mae[MAE / outcome]
  feat[Feature snapshot] --> drift[Input drift]
  drift --> gate[Fail the feature job]
  mae --> gate

Scores meet late labels. Drift fails the feature job, not a Slack shrug.

Evals for the data platform, not just the model

Model metrics (AUC, log-loss) belong to the training job. Platform evals belong to you:

Eval	Fail if
PIT leak count	> 0
Train/serve parity sample	mismatch rate above a tiny epsilon
Feature freshness	latest valid_from older than the SLO
Score partition present	missing score_date for the run
Label join rate	sudden drop — labels pipeline broke

Run the leak query and a parity sample in the same CI as dbt tests. “The notebook looked good” is not a gate.

Where you fit (and why the pay is real)

The industry keeps discovering that models fail in production because of data: stale features, silent schema changes, drifted distributions, leaky joins. The people who fix those problems are data engineers who learned the ML vocabulary.

You do not need to become a researcher. You need to:

• Own feature tables and their history
• Make training snapshots reproducible
• Make scoring idempotent
• Put evals next to the DAG

That is a smaller step than the job titles suggest, and one of the sturdier career moves in the field right now. RAG and LLM transforms (RAG pipelines, LLMs in pipelines) are the same idea with different artifacts.

Pitfalls

Current-state features joined to historical events. Instant leakage.

Two definitions of the same feature. The store exists to prevent this; a wiki page does not.

Scoring from a pickle on a laptop path. Registry URI or it is not production.

Random train/test split on time-series users. The model “sees” tomorrow.

Watching only model AUC. The feature job can be empty and AUC still looks fine on last week’s leftover table.

Buying a full ML platform for one batch model. MLflow + dbt + the scheduler you have.

A first production slice

1. One batch model, one feature group, one score table.
2. Type-2 or daily snapshots so PIT joins are possible.
3. Leak test in CI.
4. MLflow registry, promote @production.
5. Idempotent daily score job.
6. Weekly join to outcomes.
7. Only then talk about online serve or a feature-store product.

FAQ

Do I need a feature-store product for one batch model? No. dbt plus type-2 history plus a leak test is enough until train and serve have drifted once.

Why did the notebook AUC look great and production die? You joined today’s features onto last year’s events. Point-in-time or it is leakage.

Can I train on a random 80/20 split? Not if users exist in the future of the split. Split by time.

Where should the pickle live? In a registry URI (models:/churn_model@production). A laptop path is not production.

What is train/serve parity? Same feature name, same SQL (or generated SQL). A daily sample that diffs online vs batch.

Is watching AUC enough? No. The feature job can be empty and last week’s leftover scores still produce an AUC.

What this means for data engineers

MLOps is not a new career so much as a new set of tables and a stricter join. Point-in-time correctness and train/serve parity are the work. Do those well and the model people look like geniuses. Skip them and everyone blames “the model.”