Coverage Is the Number That Actually Matters.

Most people judge an AI answer the way they judge writing. Is it clear. Is it confident. Does it sound like someone who knows the subject. That instinct is fine for an email. It is dangerous for compliance.

Fluency measures how good an answer sounds. Accuracy measures whether the claims it makes are true. Both are worth having. Neither tells you the one thing an audit turns on: did the answer leave something out.

An answer can be perfectly written and perfectly true about the obligations it names, and still miss a third of the obligations that apply to you. Every sentence in it can survive review while the work still fails completeness review. The score that would have caught this is not on the scoreboard most teams use.

The metric has a name outside compliance. In machine learning, Google defines recall as the share of actual positives that were correctly classified: TP divided by TP plus FN. In information retrieval, NIST TREC evaluation uses recall and precision to ask the same practical question: how much of the relevant set did the system retrieve.

Precision asks a different question: of the things the system returned, how many were correct. You can be flawless on precision and still fail the job, because precision never penalizes the item you never mentioned. Recall is the metric that counts the miss.

Coverage is recall applied to obligations. Of everything the task required, what share did the system find, map, and address. In GRC that is not one metric among many. It is the metric that tells reviewers whether silent gaps remain.

In some settings, 60 percent is treated as partial credit. In compliance, 60 percent is a gap list.

An auditor does not usually average your controls into one forgiving score. They do not credit you for the 27 obligations you handled and quietly erase the 3 you missed if those 3 were applicable. They look for the gap, and the gap is where the finding can live. One unaddressed data transfer clause, one skipped breach notification timeline, one control that applied and was never mapped can be enough to matter.

This is what makes partial answers so misleading. A 60 percent answer does not look 60 percent finished. It looks done. The missing obligations do not announce themselves. They are absent, and absence reads as completeness until someone goes looking.

Coverage is a denominator problem. If a task has 30 expected obligations and the system finds 27, the answer may read beautifully and still be missing 10 percent of the work. Those three misses are not formatting issues. They are silent false negatives.

That is why benchmark buyers should ask for the expected obligation count, the found obligation count, the missed items, and the source passages behind each answer. Accuracy on the found items is not enough. In compliance, the item nobody found is often the one that creates the finding.

There are two ways to be wrong. You can include something that does not belong, or you can leave out something that does. Machine learning calls these false positives and false negatives, and they do not cost the same.

A false positive is noise. An obligation flagged that turns out not to apply wastes reviewer time. A false negative is an obligation that applied and was never surfaced at all. It costs nothing to produce and can be expensive to discover, because you may only discover it when a regulator, a customer, or an auditor does.

Optimizing for fluency and confidence tends to suppress exactly the wrong error. A model tuned to sound complete may not tell you what it does not know. It fills the gap with prose instead of flagging it. That is the failure mode compliance can least afford.

This is not a Sorena opinion. It is how governance frameworks define a trustworthy system. The NIST AI Risk Management Framework lists valid and reliable as the first characteristic of trustworthy AI, and treats reliability as correct operation under expected conditions over time, not correctness on only the easy cases.

Compliance frameworks work the same way. ISO/IEC 27001, NIST SP 800-53, GDPR, the EU AI Act, and the EU Data Act are written as requirements, controls, rights, duties, and conditions. For the obligations that apply, partial coverage does not become complete just because the covered items were handled well.

So the evaluation has to match the standard. If the standard demands completeness, the metric that grades your tool has to expose coverage. Anything else measures a quality the audit may appreciate but will not accept as a substitute for missing work.

Once coverage is the frame, most AI demonstrations get easier to judge. Ask three questions of any result.

What was the denominator. A coverage number means nothing without a complete, expert-defined list of what should have been found. Percentages against a vague target are decoration.

Were misses counted, not just errors. A benchmark that only tallies wrong statements is measuring precision. It says nothing about what was skipped. Coverage requires counting the obligations that never appeared.

Can the coverage be proven. An answer that claims completeness but cannot trace each obligation to a source is asking for trust the audit will not extend. This is why our benchmarks score coverage against auditor-defined checklists and require every point to be grounded in a primary source, so completeness is demonstrated rather than asserted.

You get the behavior you measure. Optimize for fluency and you get answers that sound finished. Optimize for accuracy alone and you get true statements about an incomplete set. Optimize for coverage and you get a system that is forced to show what it found and what it missed.

In most work, the polished answer is the better answer. In compliance, the complete answer is the only answer that clears. The question worth asking of any tool is not whether its output reads well. It is a colder one: of everything the task required, how much did it actually catch. Coverage is that number. It is the one a reviewer will check, so it is the one that matters.