Does Model Diversity Actually Matter in an ML Ensemble?

Illustration of model diversity in a Machine Learning ensemble

Combining several models is common practice in Machine Learning. The intuition is simple: if different models don’t make exactly the same mistakes, combining them can let one compensate for some of the other’s weaknesses.

This idea easily leads to a seemingly reasonable rule: to improve an ensemble, diversify its models.

Changing algorithm family, objective function, or architecture should then produce different behaviors and enrich the ensemble. In practice, the relationship is far less direct.

Two very different architectures can produce almost the same errors. Conversely, a model strongly decorrelated from the others can simply be bad in a different way, without providing any complementarity that’s actually usable.

The question, then, isn’t just:

Are the models different?

but rather:

Do their differences let the ensemble solve the problem better?

The experiments presented in this article are deliberately simplified and anonymized. They illustrate the phenomena observed without describing the composition of any specific system.

1. Why look for diversity in the first place?

Suppose two models, each getting 20% of observations wrong. If their errors fall on exactly the same examples, combining them leaves little room to fix those failures:

Model A     ✓  ✓  ✗  ✓  ✗
Model B     ✓  ✓  ✗  ✓  ✗

Both systems fail in the same place.

Now imagine a different situation:

Model A     ✓  ✓  ✗  ✓  ✗
Model B     ✓  ✗  ✓  ✓  ✓

Individual performance can be similar, but the situation is far more interesting. When A fails, B sometimes succeeds. If these differences follow an exploitable pattern, combining the models can then produce a better decision than either model on its own.

This is one of the fundamental reasons ensembles are worth building: complementary errors across several models can let their combination handle certain hard cases better.

But this property doesn’t automatically follow from how many models are in the ensemble. It depends, in particular, on the structure of their errors.

2. Different models can end up learning almost the same thing

A first, natural way to pursue this diversity is to change the model.

You can change:

the objective function
the algorithm family
the architecture
the training constraints

The assumption is that by learning differently, the models will produce behaviors different enough to become complementary. Our experiments show this isn’t automatic.

Two variants from the same family, trained with different objectives, can still produce strongly correlated errors.

Changing the training strategy further doesn’t guarantee greater diversity either. Some candidates can become weaker while still failing on essentially the same observations.

The change, then, exists in the training procedure, much less so in the final behavior.

Changing how a model is optimized, then, doesn’t guarantee it will produce a genuinely different behavior.

3. A completely different architecture doesn’t guarantee more diversity either

You could push this reasoning further. If tweaking a variant of a model isn’t enough, why not use a radically different architecture?

It’s a natural strategy in an ensemble. Models relying on different mechanisms don’t build their representations the same way and don’t necessarily share the same inductive biases. You might therefore expect noticeably different behaviors.

To test this hypothesis properly, the models can be trained on the same information, so the comparison mainly isolates their architecture.

Yet in our experiments, two neural architectures built on very different principles produced strongly correlated errors. Despite different learning mechanisms, their behavior stayed very similar.

A different architecture can process the same information differently without producing a behavior different enough to enrich an ensemble.

Architectural diversity isn’t necessarily behavioral diversity.

4. Diversity is measured in behavior

If a model’s family or architecture isn’t enough to establish its complementarity, you have to look directly at what it produces. Several measurements then become useful.

Prediction correlation

Two models can produce very close scores, observation by observation. A strong correlation indicates that, overall, they rank the observations in a similar way.

But this measurement isn’t always enough. Two models can produce different numerical scores while ultimately making the same decisions.

Error correlation

So you can look directly at whether the models succeed and fail on the same events. For two models A and B, you can, for example, build:

A : 1  1  0  1  0  1
B : 1  0  0  1  1  1

where 1 is a success and 0 an error. Two models with similar average performance can therefore reveal very different behaviors.

Complementary successes

Another approach is to directly count the situations where a candidate succeeds while the reference models fail.

These events are especially interesting for an ensemble because they show that the candidate succeeds on some cases where the reference models fail.

But you also have to count the reverse: in how many situations does the candidate fail while the existing models succeed?

A candidate that brings new successes but introduces even more new errors isn’t necessarily a good source of diversity. Complementarity, then, isn’t just a matter of how many disagreements exist. What matters is what those disagreements actually contribute.

Prediction correlation, error correlation, disagreements, and complementary successes each describe a different facet of the models’ behavior.

None of these measurements, on its own, defines what “good” diversity for an ensemble is. Whether those differences are useful has to be evaluated separately.

5. Being different can simply mean being worse

This is probably one of the main traps when you deliberately try to decorrelate models. In our experiments, a model we made deliberately much simpler showed errors that were noticeably less correlated with the other candidates’.

At first glance, that’s exactly what you’re looking for. Its behavior was different. It could therefore have looked like an excellent candidate for enriching the ensemble. But its standalone performance was also noticeably weaker. Part of that decorrelation, then, could have come from a much less interesting phenomenon: the model wasn’t just making different errors — it was simply making more of them.

Low correlation, on its own, doesn’t let you tell an exploitable complementarity apart from noise or underfitting. A random model would be extremely different from the existing ones. That wouldn’t make it useful.

Performance and diversity therefore have to be considered together: maximizing one at the expense of the other doesn’t guarantee a better ensemble.

6. There’s a trade-off between quality and diversity

Candidates can be represented along two dimensions:

Diagram of the trade-off between model quality and diversity

The ideal candidate naturally seems to sit in the top right:

good enough performance on its own
different enough to bring complementary successes

But real experiments often land in the other zones.

A model can be good but redundant — it essentially reproduces the behavior of the models already in the ensemble.

It can be different but too weak — its decorrelation partly comes from additional errors.

It can also be both weak and redundant, in which case its value to the ensemble is limited.

Diversity, then, isn’t a goal independent of performance. Neither dimension, on its own, tells us whether a candidate will improve the ensemble.

7. Forcing diversity doesn’t guarantee creating information

This difficulty leads to a tempting experiment. If several models converge toward similar behaviors, why not deliberately force a new candidate to explore something else?

One option, for example, is to deliberately restrict the information the candidate has access to. The main models keep the full set of available information, while the new candidate is forced to work on a deliberately different subset.

The goal is to keep it from too easily reconstructing the same relationships as the existing models, and to push it toward exploiting other signals.

The candidate can no longer simply reconstruct the same solution from the full available signal. Its behavior can indeed become more different. But one fundamental constraint remains: removing information doesn’t create new information.

If what’s left still contains an exploitable, complementary signal, the strategy can pay off. Otherwise, the candidate simply gets weaker. Its decorrelation increases because it has less information to solve the problem correctly. You do get more statistical diversity — but not necessarily more useful diversity.

This experiment highlights an important distinction:

You can force a model to produce something different. You can’t force the data to contain a complementary signal.

8. Diversity metrics remain indicators, not proof

Prediction correlation, error correlation, complementary successes, or disagreements can all help identify interesting candidates.

But none of these measurements, on its own, proves that a model will actually improve the ensemble. That’s an important limitation. A candidate can show strong decorrelation and enough standalone performance to look promising.

The question, though, remains open:

Will the rest of the system actually be able to exploit this difference?

In our experiments, some candidates that looked interesting on intermediate metrics did change the system’s behavior once integrated.

But that change didn’t necessarily translate into an overall improvement. Some objectives improved slightly while others stayed flat or regressed. The diversity detected upstream was real, but it simply didn’t translate into a net gain for the ensemble.

Diversity metrics, then, have to stay exactly what they are: descriptive measurements of similarity or disagreement between models. They let you observe that a candidate behaves differently — not conclude that the difference will be useful once integrated into the system.

Whether that diversity is actually useful has to be evaluated separately, on the complete system.

9. The complete system is the real test of complementarity

In an ensemble made up of several levels, an intermediate model isn’t used only for its standalone performance. Its outputs become inputs to downstream components.

A candidate that’s slightly weaker when evaluated on its own can therefore still be worth keeping if its predictions let the system fix some of the other models’ errors.

Conversely, an excellent candidate can become almost useless if its behavior is too close to what the existing components already provide.

The real question then becomes:

Can the system actually exploit what this model does differently?

Validation, then, has to follow the actual path the information takes: new candidate → integration → re-evaluation → output of the complete system.

Several candidates can show different behaviors when studied in isolation, then bring no improvement once integrated into the ensemble.

Useful complementarity, then, can’t be inferred from the difference between models. It can only be observed through its effect on the complete system.

10. Conclusion

Diversity does play an important role in a Machine Learning ensemble. But pursuing it as a goal in its own right easily leads to the wrong conclusions.

Changing algorithms doesn’t guarantee different errors. Getting different errors doesn’t guarantee they’re complementary. And identifying apparent complementarity doesn’t guarantee the system can actually exploit it.

An ensemble, then, isn’t improved just because it contains more models, or because their architectures are more varied.

A new component only genuinely becomes worth adding to the ensemble once its integration actually improves the complete system.

Diversity, then, is a means, not an end. The question, in the end, isn’t how to make the models more different.

It’s whether their differences let the system better handle situations its current components already handle poorly.