One Average Score Hides a Dozen Failures.

Every AI leaderboard sells you the same thing: one number. A model scores 80% and you move on. The number feels like a verdict. It is not. It is an average, and an average is a blender.

Averaging exists to smooth over variance. That is its entire job. Feed it a model that is flawless on some tasks and hopeless on others, and it will hand you back a single reassuring figure that describes neither. The 80% does not mean the model gets four out of five things right everywhere. It can just as easily mean it gets everything right in eight categories and everything wrong in two.

Those two categories do not disappear. They get folded into the mean and stop being visible. You bought the number for confidence. What you actually bought was a place to hide the failures.

Picture a compliance benchmark spanning ten frameworks. The model aces GDPR, ISO 27001, and SOC 2. It is competent on PCI and NIST CSF 2.0. And it collapses on the EU AI Act, the Data Act, and DORA, scoring near zero on obligations it never learned to track.

Run the average. You might land at 80%. Publish that, and a buyer reads 80% as broadly reliable. The reality is that this model is production-ready for some regimes and actively dangerous for others. Deploy it against the EU AI Act on the strength of an 80% headline and you are not covered. You are exposed, and the number told you the opposite.

This is not a rounding quirk. It is what aggregation does by design. Composite scores are built to compress many results into one ranking, and researchers cataloging the problem note that most multi-task benchmarks aggregate every evaluation into a single central leaderboard. The compression is the feature. It is also the failure.

A student can pass a course with an 80% average because a strong final offsets a weak midterm. Compliance does not work that way. Nobody averages your regimes.

An auditor examining your EU AI Act posture does not credit your excellent GDPR mapping against your missing risk classifications. A DORA examiner does not care that your SOC 2 evidence is immaculate. Each framework is scored on its own terms, against its own obligations, by its own reviewer. The whole model of a headline average assumes strengths can pay down weaknesses. In GRC, they cannot. A gap in one regime is a gap in that regime, full stop.

That is why a single aggregate benchmark is the wrong instrument for compliance work before you even question the underlying data. It answers a question no auditor asks.

Imagine two systems both score 80 percent. One gets 80 across every category. The other gets 100 on policy summaries, 100 on definitions, 90 on timelines, and 30 on breach obligations. The average is the same. The procurement decision should not be.

That is why GRC benchmarks need framework, category, distribution, and coverage views. A tool that fails one obligation family can still look good in an aggregate. The buyer’s job is to find the category that would embarrass them in front of an auditor before the tool does.

This is not a Sorena opinion. It is the direction serious evaluation research has been moving for years.

Work on AI fairness evaluation argues plainly that reducing a complex property to a single number is inappropriate, because distinct concerns cannot be meaningfully averaged into one score, and that the reform is to expose multiple measurements rather than hide potential harms inside a single figure. The recommendation is to replace single-score leaderboards with benchmark suites that surface trade-offs instead of masking them.

Standards guidance points in the same direction. NIST's draft guidance on automated benchmark evaluations says automated benchmarks are not well-suited for all use cases, tells evaluators to check whether the benchmark reflects the intended use case, and recommends considering item-level results alongside aggregate statistics. The message is consistent: one number can be a headline, but it is not the answer.

If a benchmark is going to inform a compliance decision, one number is disqualifying on its own. Demand the breakdown. A benchmark you can trust for GRC has to show:

• A score per framework, so EU AI Act performance is never hidden behind GDPR performance
• A score per category within each framework, so a strong average cannot mask a dead obligation
• The distribution, not just the mean, so you can see where the model is confident and where it guesses
• Coverage stated explicitly, so a partial answer never gets counted as a complete one

This is exactly how Sorena's benchmark reports its results: per category, per framework, session by session, with coverage tracked as its own line rather than dissolved into an average. You are meant to see the weak spots, not have them smoothed away.

A vendor who will only quote you one number is either not measuring the categories or not showing you the ones that fail.

Treat every headline benchmark figure as a question, not a conclusion. When you see one aggregate number, the only correct response is: which categories built that average, and how far apart are they?

If the answer is a tight band where every framework lands near the mean, the average is honest and the model is broadly reliable. If the answer is a wide spread where a few strong categories carry a few failing ones, the average is fiction and the model is a specialist wearing a generalist's badge.

You cannot tell those two situations apart from the headline. They can produce the identical number. In consumer software that ambiguity is a nuisance. In compliance it is the difference between covered and exposed. The average will not tell you which one you are. The breakdown will. Insist on the breakdown.