From Notebook to Production: Building a Robust ML Pipeline with Airflow and CI/CD

Diagram of a production ML pipeline with Airflow orchestration and CI/CD

A Machine Learning model can work perfectly in a notebook and still be nowhere close to a system you can actually run in production.

The notebook mainly answers one question:

Can the model learn something useful from the available data?

Production raises many more.

What happens if the data arrives late? If its schema changes? If a step fails after partially writing its results? If a new version of the code is incompatible with an older model? If an inference run needs to be replayed? If the server restarts mid-processing? If an external dependency becomes temporarily unavailable?

On an ML system that runs daily, these situations aren’t exceptional. They’re part of the platform’s normal operation.

The pipeline described here has therefore gradually been structured around one principle: inference isn’t a script you run, it’s a chain of processing steps, each with explicit inputs, outputs, dependencies, and validity conditions.

This article isn’t about the model itself — it’s about the infrastructure that lets you run it reproducibly: ingestion, data validation, orchestration with Apache Airflow, environment isolation, CI/CD, deployment, and observability.

1. The notebook isn’t the unit of production

During the research phase, an ML workflow can be fairly linear: load the data → compute the features → load the model → predict → analyze the result

In that environment, the operator — the developer — is implicitly part of the system. They check that the data exists, rerun a cell after an error, inspect a suspicious distribution, delete an inconsistent intermediate file, or decide a run needs to be restarted.

None of this can be relied on in production.

The pipeline has to be able to determine, on its own:

whether its inputs are available
whether they’re valid
which steps need to run
which of them can be replayed
whether an existing output can be reused
whether an error is transient or blocking
whether the results can be published

Moving to production, then, is less about “automating the notebook” than about making explicit every decision that used to be made implicitly around it.

2. Breaking the pipeline down by responsibility

The first step was separating the processing into distinct responsibilities.

A simplified view of the pipeline looks like this:

Distribution of responsibilities across the pipeline's components

Among other things, this separation keeps an ingestion problem from being mistaken for an inference error, and keeps an incomplete result from getting published just because the Python process exited without an exception.

Every step has its own contract.

Inference, for example, shouldn’t have to determine whether the raw data is complete enough. It should receive data whose completeness has already been checked.

Likewise, publication shouldn’t decide whether a prediction looks valid. It should receive an artifact that has already cleared the relevant checks.

This organization introduces an important property: a step should never silently compensate for guarantees the previous step failed to provide.

3. Airflow and DAGs: orchestrating without absorbing business logic

Apache Airflow represents workflows as DAGs (Directed Acyclic Graphs).

A DAG describes a workflow: it represents the different tasks to run and the dependencies that determine their order.

For example:

acquisition → validation → preparation

The validation task can only run after acquisition, and preparation in turn depends on validation succeeding.

The term acyclic simply means these dependencies can’t form a loop: an upstream task can never end up depending on a downstream one.

A DAG, then, doesn’t represent the business logic itself. It mainly describes which tasks need to run, their dependencies, and the conditions governing their execution. Airflow then takes care of scheduling and orchestrating them.

It can handle, among other things:

dependencies
execution order
trigger conditions
retries
timeouts
required resources
execution states

Business logic stays, as much as possible, in independent Python components.

This distinction matters. A pipeline tightly coupled to Airflow quickly becomes hard to test or run outside the orchestrator. When Airflow tasks instead call components with explicit interfaces, those components can be run and tested on their own.

DAGs that map to operational boundaries

The whole system doesn’t necessarily have to be represented by a single DAG.

When several chains have distinct responsibilities and can be run or replayed independently, separating them can become a useful architectural choice.

A simplified view looks like this:

Three-DAG architecture: ingestion, inference and publication

Each DAG keeps one identifiable responsibility and produces a state the next component can consume. This separation also makes reasoning about recovery much easier.

A new inference run doesn’t necessarily require redoing ingestion if an already-validated dataset is available. Similarly, a publication can be replayed from already-validated predictions without recomputing the whole ML chain.

Errors, retries, and success conditions can therefore be handled at the level that actually corresponds to the processing involved.

Lightweight DAGs, independent components

Separating responsibilities doesn’t mean moving their implementation into the Airflow files, though.

An inference DAG, for example, can define the dependencies between several tasks:

preparation → inference → validation

but the corresponding processing stays implemented in dedicated Python components.

The DAG describes the orchestration.
The Python component does the actual work.

This separation, among other things, lets you test an inference run or a validation step without starting Airflow, and keeps the orchestrator from gradually turning into a container for all the application logic.

The goal isn’t to artificially multiply DAGs either.

Two processes don’t need to be split apart just because they could be. A boundary becomes worth drawing when it corresponds to a real operational responsibility: a different execution cadence, the ability to replay independently, a reusable intermediate state, or distinct success conditions.

The split, then, aims for a balance between two extremes:

a monolithic DAG, absorbing all the business logic
a multitude of DAGs, with no real operational boundary

In this architecture, Airflow controls when, in what order, and under what conditions the processing runs.

The application code defines what that processing actually does.

4. Depending on state, not just on a schedule

A schedule based purely on time is often not enough for a data pipeline.

It can be tempting, for example, to organize processing like this:

05:00 → ingestion
06:00 → inference
06:30 → publication

This setup, though, carries an implicit assumption: every previous step will have finished correctly within the expected window.

The fact that it’s 6 a.m. doesn’t guarantee the data inference needs is actually available.

Ingestion may have taken longer than expected, validation may have failed, or the expected data may simply not be available yet.

The real dependency, then, isn’t it's 6 a.m. — it’s dataset validated and available

Orchestration has to represent these dependencies explicitly. A downstream task becomes runnable once the state it depends on genuinely exists and satisfies the expected conditions — not simply because some theoretical time has been reached.

This distinction matters especially once several DAGs work together.

The inference DAG doesn’t depend just on the ingestion DAG’s theoretical completion time — it depends on the existence of a dataset that has actually cleared the required checks.

Likewise, publication depends on validated predictions, not simply on an inference task having finished.

The schedule obviously still has its uses for deciding when to check or trigger a process. It just doesn’t replace the guarantees that process actually needs to run.

This makes it possible to orchestrate the pipeline around actual system states and dependencies, rather than around a sequence of scripts assumed to finish at fixed times.

5. Idempotence as a property of the pipeline

A production pipeline has to be replayable. A task can fail after having:

downloaded the data
created part of its files
written a few records
computed some features
started a publication

If rerunning it assumes nothing ran before, every incident requires manual intervention.

Processing is therefore designed, as much as possible, to be idempotent:

run(x)
run(x)
run(x)

has to converge to the same final state as a single, correct run.

Depending on the component, this can involve:

deterministic writes
stable identifiers
controlled artifact replacement
transactions
temporary files followed by an atomic move
upsert operations
checking the existing state before modifying it

Idempotence fundamentally changes how incidents get handled. A retry stops being a potentially dangerous operation and becomes a normal recovery mechanism.

6. Retrying doesn’t mean hiding errors

Not every error, though, should be retried the same way.

A temporary network outage and a schema mismatch aren’t the same kind of incident. The first can legitimately be retried; the second should generally stop the pipeline.

Automatically retrying a structurally invalid input for hours doesn’t make the system more robust. It just delays the diagnosis.

The retry policy therefore distinguishes transient errors from structural ones.

This distinction also produces more useful alerts: an error that will probably resolve itself on the next retry doesn’t deserve the same priority as a broken contract on the input data.

7. Validating data before inference

An external service can respond successfully while still delivering unusable data. HTTP monitoring, then, isn’t enough.

Before letting the rest of the pipeline proceed, several properties can be checked:

presence of expected fields
types
required values
cardinalities
unknown categories
volumes
distributions
extreme values
consistency across multiple objects

This distinction is fundamental: a silent change in the data can be more dangerous than an outright failure. When a source stops responding, the pipeline stops.

When it keeps responding with a shifted distribution or a field whose meaning has changed, inference can keep producing results that are technically valid but statistically inconsistent.

8. From schema checks to distribution checks

Validation, then, doesn’t stop at the schema. Two datasets can have exactly the same columns and still represent very different populations.

The pipeline also monitors certain statistical properties of the inputs and can compare them against historical references.

Conceptually:

Diagram of data distribution controls

Not every deviation should obviously halt production — a variation can be perfectly legitimate.

The goal is to distinguish between:

structural invariants, whose violation must block the pipeline
strong anomalies, which need inspection
normal statistical variation, which just needs to be logged

This hierarchy avoids two extremes: checking nothing at all, or building a system so sensitive it stops at the slightest natural change in the data.

9. ML artifacts are versioned as one coherent set

A production model isn’t just a weights file. Its behavior also depends on everything that transforms the data before and after it runs.

A deployable artifact can conceptually bundle together:

model/
├── model
├── configuration
├── feature definition
├── categorical vocabulary
├── preprocessing metadata
├── calibration
└── evaluation metadata

The goal is to prevent a situation where:

the model belongs to one version
the categorical vocabulary belongs to another
the preprocessing corresponds to whatever code version is currently deployed on the server
the calibration belongs to yet another experiment

These combinations can be technically runnable while still producing incorrect results. Deployment, then, has to preserve compatibility between code, data, and ML artifacts.

10. CI: building and validating

The CI pipeline checks whatever properties it can before deployment. Depending on the component, this includes:

unit tests
integration tests
configuration validation
dependency checks
image builds
pipeline component checks
publishing the artifacts needed for deployment

The goal is to stop a bad change as early as possible.

Diagram of the continuous-integration validation steps

An image that can’t be built should never reach the server. An invalid configuration should be detected before Airflow ever starts.

This principle reduces how many errors get discovered in the most expensive environment there is: production.

11. CD: deploying a reproducible state

Deployment isn’t just about releasing a new version to production. It also has to guarantee that the deployed state is identifiable, versioned, and reproducible.

A production environment whose state depends on manual changes or on things that only exist on the server quickly drifts away from its sources of truth. The system, then, relies on identifiable artifacts and automated procedures.

The general principle is:

Diagram of continuous deployment ensuring a reproducible state

Infrastructure and service configuration are also described declaratively as much as possible. The server isn’t treated as the source of truth. It’s a target on which a state defined elsewhere has to be reproducible.

12. Why manual changes become dangerous

Fixing something directly on a server can feel efficient when a problem needs solving fast.

But if that change only exists on the server, production’s actual state starts drifting away from the state described by the sources of truth.

That change will disappear at the next deployment, once the state defined in the sources of truth gets reapplied.

For as long as that drift lasts, the state actually running in production can no longer be reconstructed solely from identified, versioned components.

Any lasting change has to be made in the code, the configuration, or the declarative infrastructure, then propagated to production through the deployment process.

Production has to remain a consequence of these sources of truth — never a parallel source of configuration in its own right.

13. Containerization alone doesn’t guarantee reproducibility

Docker lets you encapsulate a service together with its runtime environment:

Python version
libraries
system dependencies
the application itself
runtime configuration

This boundary lets the same version of a service be built, tested, and run with the same system and software dependencies.

But this reproducibility is limited to whatever the container actually encapsulates. A container guarantees neither the version of the data being used nor the compatibility between the model, its ML artifacts, and the transformations it depends on.

A perfectly versioned Docker image running a model that’s incompatible with its own preprocessing is still a broken system.

Containerization, then, provides an important reproducibility guarantee, but it’s not enough on its own to make an entire ML system reproducible.

14. Observability: knowing what actually happened

Automating a pipeline doesn’t, on its own, guarantee that its behavior is observable. Knowing that a task ran isn’t always enough to tell whether the processing actually produced the expected result.

Orchestration provides a first layer of observability:

task status
duration
retries
dependencies
logs
run history

But a task’s technical status isn’t always enough. A task can succeed and still produce zero results. An ingestion run can finish without raising an exception while producing an abnormally low data volume. An inference run can produce scores with an unusual distribution.

Observability, then, has to cover several levels and answer the questions that go with each one:

Infrastructure: Are the server and the required services available?
Process: Are the components running correctly?
Pipeline: Have all the expected steps completed?
Data: Is the input complete and consistent?
ML: Do the outputs still match the expected behavior?

This hierarchy keeps a process that exited without an error from being treated, on its own, as proof the system is working correctly.

15. Decoupling the ML pipeline from the product

Inference results aren’t exposed directly to the end client. Only a state that has passed the pipeline’s checks gets published to a dedicated backend layer.

This layer is the interface between two systems with different responsibilities and different life cycles:

Diagram separating ML computation from the final product

The ML pipeline can therefore be recomputing its results, temporarily unavailable, or being redeployed, without the app necessarily losing the last valid state that was published.

Conversely, evolving the product doesn’t require changing the inference pipeline.

This separation, then, lets the ML computation and the services that expose its results to users evolve and operate independently of each other.

16. Designing for failure rather than for the happy path

A pipeline’s happy path is usually easy to describe: data arrives, processing runs, results get published.

Robustness, though, is decided in the paths that deviate from that scenario.

For every step, you have to define not just what should happen when it succeeds, but also the state the system should be left in when it fails. An interruption shouldn’t leave that answer to chance.

Depending on the step, the expected behavior can differ:

automatically replaying an idempotent operation
stopping the chain when a contract is no longer respected
preventing dependent processes from starting
discarding a partial output
keeping the last valid state
producing enough information to diagnose the incident

The goal, then, isn’t to build a pipeline that never fails. It’s to make sure that when a step fails, it leads to a known, observable, and controlled state.

This ability to control failure states is often more important for the system’s robustness than shaving a few seconds off the happy path.

This pursuit of robustness doesn’t mean piling on infrastructure components, though. Every component you introduce has to address an identifiable operational problem and remove more complexity than it adds.

17. The pipeline’s operational invariants

In the end, the architecture can be summed up by a handful of invariants.

1. A downstream task never compensates for an invalid input. Data explicitly clears the required checks before being consumed by the next steps.

2. A replayed task must converge to a consistent state. Retries are part of the system’s normal operation and shouldn’t require systematic manual cleanup.

3. Technical success doesn’t guarantee a valid result. A process’s exit code is complemented by checks on the data and artifacts it produced.

4. A model is deployed together with everything it needs to run. Weights, configuration, preprocessing, vocabulary, and calibration all belong to the same logical version.

5. Publication happens after validation. A partial or inconsistent output never automatically replaces the last valid state.

6. Production isn’t a source of truth. Lasting changes have to be reproducible from the code, the configuration, and the automation.

7. Observability covers the entire chain. Infrastructure, orchestration, data, and ML outputs are distinct levels of control.

8. Dependencies are represented explicitly. A schedule doesn’t replace an availability contract between two processes.

9. The orchestrator doesn’t carry the business logic. DAGs coordinate the processing and its dependencies; their implementation stays in components that can be run and tested independently.

10. The product is decoupled from the ML computation. A temporary pipeline outage shouldn’t necessarily become an outage for the consuming service.

These invariants make it possible to reason about the pipeline independently of whatever algorithm happens to be in use.

18. Conclusion

Going from a notebook to a production ML system isn’t just a matter of dropping a Python script into a cron job or a container.

The nature of the problem changes. The model now has to run inside a chain capable of handling imperfect data, external dependencies, partial processing, retries, multiple artifact versions, and successive deployments.

Robustness no longer rests solely on the model working correctly. It also depends on far more operational properties:

processing only runs once its dependencies are genuinely satisfied
errors and recoveries lead to known, controlled states
code, configuration, and artifacts make it possible to reproduce the state actually deployed
results are checked before being exposed to the product, which stays decoupled from the ML computation

At this level of maturity, the question is no longer just:

“Does the model produce good predictions?”

It becomes:

“Can you explain, reproduce, and control the path that produced each result?”

It’s this ability to make dependencies, states, and validity conditions explicit that turns a chain of ML processing steps into a pipeline you can actually run in production.