Building an End-to-End ML Product as a Solo Developer: From Data to Mobile App

In Machine Learning, training a model is often only a small part of the problem.

You have to collect and structure the data, avoid temporal leakage, build a credible validation strategy, orchestrate the processing, deploy the models, monitor data quality, expose the results through a backend, and finally build the product that uses them.

This is exactly the kind of project I’ve been building over the past few years: a mobile product built around a predictive ranking system applied to time-based events with multiple candidates.

The project today consists of a Flutter app available on iOS and Android, a cloud backend, an automated processing infrastructure, and a complete Machine Learning pipeline spanning data ingestion, inference, and the production of scores and confidence levels.

This article deliberately doesn’t cover the specific business domain, the data used, the variables, or detailed model performance. The goal is different: to discuss the main engineering challenges involved in building such a system.

---

1. The problem: ranking rather than simply predicting

The ML problem here isn’t classic classification where each observation is independent.

Each event contains multiple candidates. The system has to produce a ranking relative to the other candidates within that same event.

Schematically:

This difference shapes a large part of the architecture.

Correctly predicting the absolute value associated with a candidate isn’t necessarily the same as correctly positioning it relative to the other candidates in the same event.

The system combines several modeling objectives to exploit complementary representations of the problem.

These objectives don’t impose the exact same representation of the problem. Combining them raises an architectural question: how do you exploit several models without simply averaging their predictions?

---

2. A multi-level ML architecture

Combining the models relies on a multi-level stacking architecture.

Its conceptual version can be represented like this:

The idea behind stacking is simple: the predictions produced by the first-level models become inputs to downstream models.

Implementing it correctly on temporal data is much less simple.

The models being combined aren’t necessarily homogeneous: several model families can coexist to exploit different representations of the candidates and their interactions.

---

3. The central constraint: never learning from the future

On temporal data, a classic random K-fold can produce extremely misleading results.

If an observation from 2025 contributes to training a model that generates a prediction for 2023, that prediction is no longer representative of what would actually have been possible in 2023.

For a single model, the problem already matters. For stacking, it becomes critical.

Imagine the first level learns on the whole dataset and then generates scores used to train the second level.

The second model indirectly has access to information coming from the target it’s trying to predict. The metrics become artificially good, without being representative of what’s actually achievable in production.

Strictly temporal OOF predictions

Each level is therefore trained using strictly temporal, future-only Out-of-Fold predictions.

The principle is:

For each period, the model can only use information that was available before it.

The predictions generated this way become the features for the next level. The process has to be repeated at every stage of the stack.

This costs much more than standard training, but it buys something essential: a much more realistic simulation of production conditions.

---

4. Separating ranking from confidence

The architecture explicitly separates the ranking problem from the confidence-estimation problem.

The first question is:

Which candidate should rank ahead of the others?

The second:

How much can we trust that ranking for a given use case?

These aren’t quite the same problem.

The main pipeline therefore produces the ranking, while a later stage analyzes various signals coming from the prediction system.

This stage can draw on many pieces of information from the system. As examples, it can pull in signals related to:

• scores produced at different levels
• dispersion across models
• gaps within the ranking
• characteristics of the resulting ranking
• agreement or disagreement between multiple signals

These are only general families — in practice, this representation can rely on a much larger feature space.

This layer produces a probability.

But a probability shown to a user needs to actually mean something.

A model that consistently predicts a probability of 0.80 when the corresponding event occurs only 60% of the time may still be useful for ranking, but its probabilities are poorly calibrated.

The calibration of the outputs is therefore measured, notably with the Brier Score and analyses across probability bins.

The goal then becomes:

Predicted probability ≈ observed frequency

This distinction between discrimination and calibration matters especially when the model’s outputs are used directly by the product.

---

5. The model is only part of the system

At this point, the problem is no longer just about model quality. A perfectly trained model is useless if:

• the daily data doesn’t arrive
• its schema changes silently
• a pipeline step fails
• the features are no longer computed the same way
• inference uses an incompatible version of the model
• the results aren’t published
• the app doesn’t know a new prediction exists

The ML chain is therefore treated as a production system, not a Python script run by hand.

Orchestrating processing with Airflow

Processing is orchestrated with Apache Airflow; the daily pipeline can be simplified like this:

Airflow provides:

• dependency management
• retries
• execution history
• controlled parallelization
• observability
• recovery after failure
• separation of the pipeline’s different responsibilities

An important property of the system is also its idempotence.

Rerunning a step after a failure shouldn’t produce an inconsistent state or arbitrarily duplicate data.

Idempotence is far less glamorous than a new ML model, but considerably more important once the system is running in production.

Data has to be monitored too

An outright failure is relatively easy to detect. The most dangerous problems are usually silent.

A source can keep responding with a valid HTTP status while having changed:

• a field
• a category
• a distribution
• a volume
• a value convention
• part of its schema

The pipeline can then keep running. And that’s exactly the problem. The pipeline therefore includes quality checks right at ingestion, comparing the data received against what the system considers normal.

This includes checks on:

• presence of the expected data
• structure
• unusual values
• unknown categories
• distributions
• volumes

The goal is to catch data anomalies before they turn into degraded ML metrics.

A distribution shift caught at ingestion is much easier to diagnose than a performance drop noticed several weeks later.

Reproducibility: versioning the model’s dependencies

Reproducibility requires versioning the transformations and information needed to use the model.

Categorical variables are a simple example.

During training, a model learns a representation tied to a given vocabulary. If that vocabulary is rebuilt differently at inference time, two categories can end up with different representations.

The pipeline therefore uses a versioned vocabulary, kept alongside the artifacts the model needs.

A model, in the end, isn’t just:

model.bin

It’s closer to:

model/
├── weights
├── configuration
├── categorical vocabulary
├── preprocessing information
├── calibration
└── metrics

Deployment has to treat this whole set as a single coherent unit.

Preserving consistency between training and production

Versioning the artifacts isn’t enough, though, if the data actually shown to the model differs between training and inference.

The same model, for example, might receive a variable built one way during training and computed with slightly different logic once deployed.

The pipeline can then keep running normally. Predictions get produced, no service goes down, and no obvious technical error shows up.

Yet the model is no longer being used under the conditions it was trained and validated on.

This kind of mismatch between how data is prepared during training and how it’s produced at inference is generally called train/serve skew.

It can show up at different points in the chain:

• feature definition or ordering
• transformations and normalization
• category encoding
• windows used for historical aggregations
• default values or handling of missing data
• reference-data versions
• how intermediate signals are produced or combined

The risk shows up in particular when the same piece of information is reconstructed through two different paths:

TRAINING ──► feature construction ──► model

INFERENCE ──► feature construction ──► model

Both paths are supposed to produce the same representation.

But if they rely on different implementations, configurations, or versions, they can gradually drift apart as the system evolves.

Reproducibility, then, isn’t just about being able to reload a model’s weights.

It also means preserving the contract between the model and the data it receives.

That means versioning transformations and schemas, sharing the same processing code where possible, and checking that the representations produced stay consistent between training and inference.

In a multi-level architecture, this constraint doesn’t only apply to features built directly from raw data.

A model’s outputs can become another model’s inputs. How those signals are produced is itself part of the contract to preserve.

Finally, this problem needs to be distinguished from data leakage. A system can perfectly respect temporal causality while still exhibiting train/serve skew.

The two problems answer different questions:

Data leakage: did the model use information it shouldn’t have known at the simulated point in time?

Train/serve skew: is the information provided in production constructed consistently with the information used during training?

The first question is about the validity of training and evaluation.

The second is about how faithfully the validated system matches the one actually running in production.

In both cases, a technically functioning pipeline can produce predictions without any visible application error. That’s exactly what makes these problems particularly important to control in an ML system running in production.

---

6. From the ML server to the cloud backend

Once predictions are produced and validated, they need to become accessible to the product.

Responsibilities are deliberately kept separate. The ML infrastructure handles the heavy processing. The cloud backend mainly handles:

• exposing results
• authentication
• user management
• access rights
• subscription logic
• syncing with mobile platforms

Validated predictions get published to a cloud database the app can consume. This separation, in particular, keeps the mobile app from depending directly on the inference server.

A temporary interruption of the ML pipeline therefore doesn’t necessarily mean the app becomes unavailable.

Mobile subscriptions are a distributed system in their own right

Selling on iOS and Android adds a subsystem that has almost nothing to do with Machine Learning anymore.

The user buys through Apple or Google, but the backend still has to reliably determine what access to grant them.

The system has to handle, among other things:

• subscription creation
• renewal
• expiration
• cancellation
• purchase restoration
• server notifications
• state synchronization
• device changes

The authorization logic can’t simply rely on a value sent by the app. The backend therefore keeps its own server-side view of entitlement state, kept in sync with information coming from the platforms.

This subsystem illustrates how far the engineering responsibilities of an initially ML-focused product can expand once it is operated as a real mobile application.

---

7. A Flutter app as the system’s last layer

The end client is built in Flutter to share a single codebase between iOS and Android.

The app has no knowledge of the ML pipeline’s complexity. It essentially receives data that’s already prepared:

Event
   │
   ├── ranked candidates
   ├── scores
   ├── confidence levels
   └── associated information

That’s deliberate.

The phone should neither reproduce the feature engineering nor run the full inference chain.

This separation also makes it possible to evolve the models without forcing a new app release every time the ML pipeline changes.

---

8. Automating deployment and operations

The infrastructure is designed to limit recurring manual operations.

The project relies on several automation building blocks around:

• Docker
• CI/CD
• automated deployments
• infrastructure as code
• service management
• monitoring

In a context where one person handles all the maintenance, automation becomes an architectural constraint: every recurring manual operation increases both the operating cost and the risk of error.

The goal, then, isn’t to multiply infrastructure components, but to reduce the interventions needed to keep the system running day to day.

---

9. From experimental model to production ML system

The difficulty of a production ML system isn’t limited to how well its models perform.

It also lies in the ability to guarantee reproducible training, valid predictions, data quality, and continuity across the whole chain.

In production, several properties need to be verifiable:

• training must be reproducible under controlled conditions
• a temporal prediction must be generated without using future information
• data anomalies or drift must be detectable
• a failed task must allow for a controlled recovery
• a previous version must be cleanly redeployable
• a partial pipeline run must be identifiable
• a temporary inference outage shouldn’t necessarily make the product unavailable
• the transformations used at inference must stay consistent with those used during training
• a new model version must be evaluated relative to the system already in place

This last point matters especially in an architecture made up of several models.

Distinguishing experimentation from improvement

Several model families, new variables, architectures, and combination strategies can be evaluated during development.

Some of these experiments don’t improve the system. That’s a normal part of a properly conducted experimental process.

Every candidate change is therefore subject to a validation protocol before it can be considered an improvement to the system:

A local improvement to a model isn’t necessarily an improvement to the product.

A slightly weaker model on its own can still improve an ensemble if its behavior complements the existing models.

Conversely, a model with good standalone metrics may provide no measurable benefit once integrated into the stack.

Evaluation therefore isn’t based on standalone performance alone. It also measures error correlation, complementarity between models, the statistical robustness of the gain, and its impact once reintegrated into the full pipeline.

That topic alone deserves its own article.

---

10. An architecture that spans several disciplines

This architecture brings several engineering disciplines together within a single system.

I designed and implemented every layer of this architecture, from the data/ML chain through to the backend and the mobile clients.

Designing a system at this scale alone forces architectural trade-offs and requires concentrating complexity only where it actually creates value.

This kind of end-to-end responsibility mainly requires mastering not just the individual components, but also how they interact.

• How does a feature-engineering decision affect inference?
• How do you version a model together with its preprocessing?
• How do you expose results without coupling the mobile app to the ML pipeline?
• How do you deploy a change without interrupting the service?
• How do you tell an application failure apart from a data problem?
• How do you evolve the model, the backend, and the mobile clients independently?

These integration questions represent a significant part of the engineering work behind a production ML system.

---

11. Conclusion

This architecture brings together data engineering, Machine Learning, MLOps, cloud backend, DevOps, and mobile development around a single production system.

Over the course of experimentation and deployment, the project has involved a broad technical stack: Python, LightGBM, CatBoost, DeepSets, Set Transformers, MLP and tabular Transformer architectures, PyTorch, Airflow, Docker, infrastructure as code, Firebase, and Flutter. But the real challenge lies less in any one of these pieces on its own than in integrating them into a coherent, operable chain.

Actually putting an ML system in front of users means dealing with the whole chain: data → feature engineering → experimentation → validation → training → calibration → inference → orchestration → monitoring → backend → product

It also means that a more complex model, a new feature, or a better local metric isn’t necessarily an improvement to the system.

In upcoming articles, I’ll go deeper into some of the problems covered here: temporal validation and stacking without data leakage, measuring diversity between models, understanding why apparent ML improvements disappear during validation, and monitoring a production data pipeline.