A production deep learning system is rarely a single script — it's an ML pipeline: a sequence of connected, often automated stages (data ingestion, preprocessing, training, evaluation, deployment) that together turn raw data into a deployed, monitored model.
Why Pipelines, Not One-Off Scripts
The end-to-end project lifecycle covered in the DL Project Development category is straightforward to run manually once — but a real production system needs to run these steps repeatedly and reliably: retraining on new data, re-evaluating, and redeploying, often on a schedule or triggered automatically. A pipeline formalizes this sequence into a reproducible, automatable structure rather than a collection of manually-run notebooks.
The Common Pipeline Stages
pipeline_stages = [
"data_ingestion", # pull latest raw data
"data_validation", # check schema, ranges, missing values -- fail fast on bad data
"preprocessing", # clean, transform, feature engineer
"training", # train the model
"evaluation", # compute metrics, compare to current production model
"conditional_deployment", # deploy only if the new model beats the current one
]Code — A Simple Pipeline Using a DAG-Style Orchestrator (Airflow-style)
# Conceptual structure -- an Airflow DAG defines stages and their dependencies
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
with DAG("model_retraining_pipeline", schedule_interval="@weekly", start_date=datetime(2026, 1, 1)) as dag:
ingest = PythonOperator(task_id="ingest_data", python_callable=ingest_data)
validate = PythonOperator(task_id="validate_data", python_callable=validate_data)
train = PythonOperator(task_id="train_model", python_callable=train_model)
evaluate = PythonOperator(task_id="evaluate_model", python_callable=evaluate_model)
deploy = PythonOperator(task_id="deploy_if_better", python_callable=conditional_deploy)
ingest >> validate >> train >> evaluate >> deploy # defines execution orderEach stage runs only after its dependencies complete successfully; a failure at any stage (e.g. data validation catching a schema problem) halts the pipeline before wasting compute on training against bad data, and alerts the team.
Why Data Validation as an Explicit Stage Matters
Placing a validation check immediately after ingestion — verifying schema, expected value ranges, and missing-value rates — catches upstream data problems before they silently propagate into a full training run, directly preventing the kind of wasted compute and misdiagnosed results warned about in Model Training.
Common Mistakes
- Running each pipeline stage manually and inconsistently rather than formalizing them into an automated, reproducible pipeline — this makes retraining error-prone and hard to audit as a project matures.
- Omitting an explicit data validation stage, allowing a schema change or data quality issue upstream to silently propagate into training, producing confusing, hard-to-diagnose downstream failures.
Interview Relevance
Q: "Why is an explicit data validation stage placed immediately after data ingestion in a production ML pipeline, rather than relying on later stages to catch problems?" Catching schema mismatches, unexpected value ranges, or missing-value spikes immediately after ingestion halts the pipeline before wasting significant compute on training against bad data — and produces a clear, immediate, easy-to-diagnose failure at the actual source of the problem, rather than a confusing downstream symptom (a training crash, or worse, a silently degraded model) discovered much later and harder to trace back to its root cause.
Practice Question
Why does automating an ML pipeline (rather than running each stage manually) become increasingly important as a model needs to be retrained regularly?
