---
title: "General-Purpose AI in GRC: Helpful or False Confidence?"
canonical_url: "https://www.sorena.io/resources/chatgpt-in-grc-false-confidence"
source_url: "https://www.sorena.io/resources/chatgpt-in-grc-false-confidence"
author: "Sorena AI Team"
description: "We ran an internal benchmark on 43 real GRC tasks scored by auditors. Sorena reached full checklist coverage in that test; the general-purpose baseline averaged 26%. Here is the method and data."
published_at: "2026-05-08"
updated_at: "2026-07-03"
keywords:
  - "general-purpose AI GRC"
  - "AI compliance benchmark"
  - "GRC AI accuracy"
  - "hallucinations compliance"
  - "Sorena benchmark"
  - "audit ready AI"
---
**[SORENA](https://www.sorena.io/)** - AI-Powered GRC Platform

[Home](https://www.sorena.io/) | [Solutions](https://www.sorena.io/solutions) | [Artifacts](https://www.sorena.io/artifacts) | [About Us](https://www.sorena.io/about-us) | [Contact](https://www.sorena.io/contact) | [Portal](https://app.sorena.io)

---

# General-Purpose AI in GRC: Helpful or False Confidence?

*Sorena AI Team | Benchmarks | 2026-05-08*

> AI can answer a GRC question in seconds. The dangerous part is not the speed. It is how confident a wrong answer sounds right up until an auditor proves it incomplete. So we measured it.

![Coverage comparison between Sorena AI and a general-purpose AI baseline across GRC tasks](https://cdn.sorena.io/cdn-cgi/image/width=1400,quality=88,format=auto/images/resources/chatgpt-in-grc-false-confidence.png)

## Key takeaways

- GRC does not fail because teams lack answers. It fails when teams believe they are done when they are not.
- Across 43 auditor-scored tasks in our January 2026 internal benchmark, Sorena AI reached full checklist coverage while the general-purpose baseline averaged 26%.
- The baseline produced 183 factual errors across the evaluation. Sorena produced none under the benchmark scoring rubric.
- Confidence without coverage is not progress. It is exposure you cannot see until an audit finds it.

## Why general-purpose AI feels useful in GRC

AI changed how teams approach governance, risk, and compliance. Questions that once took days now get answered in seconds. Drafts appear instantly. Summaries sound confident.

At the surface level, a general-purpose assistant genuinely helps. It can:

- Summarize policies and regulations
- Explain concepts in plain language
- Brainstorm controls and approaches
- Draft responses that sound reasonable

For early exploration or learning, that is valuable. It reduces friction and speeds up understanding. The problem starts when teams treat these drafts as finished work.

## What GRC actually requires

Real GRC work has non-negotiable requirements:

- Full coverage of applicable obligations
- Clear mapping between requirements and evidence
- Accurate applicability decisions
- Traceability to primary sources
- Outputs that stand up to audits, regulators, and customers

Confidence without coverage is not progress. It is a risk. A summary that reads well but silently drops half the obligations does not save you time. It hides the work you still owe.

## What the benchmark measured

To understand the real difference, we ran an internal benchmark of Sorena AI against a leading general-purpose AI across 43 independent, real-world GRC tasks, each reviewed by auditors who are domain experts. These were not toy examples. They included:

- [GDPR](/artifacts/eu/general-data-protection-regulation), [CCPA](/artifacts/us/california-consumer-privacy-act), and [CPRA](/artifacts/us/california-privacy-rights-act) privacy audits
- [EU AI Act](/artifacts/eu/artificial-intelligence-act), [Data Act](/artifacts/eu/data-act), and sustainability readiness
- Regulatory timelines and applicability analysis
- Framework crosswalks across ISO, NIST, PCI, ETSI, and IEC
- Clause-by-clause delta analyses
- Audit-ready compliance plans

Each task was scored against auditor-defined requirements in two independent passes, with coverage, accuracy, and factual errors tracked explicitly. The results describe that benchmark set and scoring rubric; teams should still validate performance on their own documents and workflows.

## A benchmark needs a method, not just a winner

**A credible GRC benchmark starts before anyone asks the model a question.** Define the task set, the expected source universe, the scoring rubric, the denominator for coverage, the baseline model and date, and who reviews the answer. Then publish the misses, not just the wins.

That matters because fluency hides failure. A general-purpose answer can sound plausible while skipping the obligation, failing to cite the source, or refusing to admit that coverage is partial. The useful benchmark question is not "which model sounds better?" It is "which system found the required obligations, cited them, and produced an answer an auditor can verify?"

## The results: answers versus execution

The pattern was consistent across the benchmark set.

**Sorena AI**

- Reached full coverage against the auditor checklists used in the evaluation
- Grounded answers in primary sources
- Flagged gaps instead of guessing
- Produced reviewable, source-linked outputs

**The general-purpose baseline**

- Typically covered a fraction of the required obligations
- Missed large portions of the auditor checklists
- Introduced factual errors and unverifiable claims
- Could not prove completeness or coverage

The gap was not marginal in this evaluation. The baseline often produced answers that sounded right on the surface but quietly missed critical obligations, timelines, or controls that auditors expect to see. In GRC, slow work can be fixed. Wrong work looks fast until it collapses, breaks trust, and sends teams backward into rework.

## The detailed benchmark data

Below is the full breakdown: coverage by category, and every one of the 43 sessions scored by two independent auditors. Sorena coverage is the copilot column; the baseline column is the average of both scoring passes.

Read the table as an audit-readiness check, not a beauty contest. The useful signal is whether the answer found the required obligations, cited the source, avoided factual errors, and left a reviewer enough evidence to verify the result.

Across 43 auditor-scored tasks, Sorena AI reached 100% coverage while the general-purpose baseline averaged 25%. Sorena produced 0 factual errors; the baseline produced 183.

Scores reflect independent verification against source documentation (two auditors per task, averaged across two review passes).

### Coverage by category

| Category | Tasks | Sorena AI | Baseline (avg) | Baseline errors |
| --- | ---: | ---: | ---: | ---: |
| Privacy Audit | 12 | 100% | 30% | 43 |
| AI Act Compliance | 6 | 100% | 28% | 20 |
| Regulatory Timeline | 3 | 100% | 18% | 17 |
| Sustainability Compliance | 9 | 100% | 21% | 53 |
| Employment Law | 2 | 100% | 18% | 3 |
| Technical Review | 11 | 100% | 28% | 47 |

### All 43 sessions

| # | Date | Category | Scenario | Summary | Sorena | ChatGPT | Factual errors | Auditor 1 (S/B/T) | Auditor 2 (S/B/T) | Sorena did well | ChatGPT issue | Evaluation note |
| ---: | --- | --- | --- | --- | ---: | ---: | ---: | --- | --- | --- | --- | --- |
| 1 | 2026-01-06 | Privacy Audit | Privacy Notice Audit - Global e-commerce retailer | Audit of a global e-commerce privacy notice against GDPR and CPRA/CCPA, focusing on transparency, retention, cross-border transfers, and user rights. | 100% | 38% | 5 | 32/16/32 | 95/25/95 | Verified the current US and EU/UK notices, mapped GDPR + CPRA requirements to exact policy text, and produced a gap list with remediation steps. | Couldn't validate the live notices, relied on outdated sources, and missed core disclosures (rights, request methods, categories). | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 32/32, baseline 16/32. Pass 2: Sorena 95/95, baseline 25/95. |
| 2 | 2026-01-06 | AI Act Compliance | AI Terms & Privacy Audit - AI lab | Audit of an AI lab's consumer terms and privacy policy for EU AI Act and GDPR, focusing on provider duties, transparency, and operational compliance. | 100% | 19% | 3 | 42/13/42 | 156/12/156 | Mapped GDPR accountability and EU AI Act GPAI duties (copyright/TDM, watermarking, transparency) into an audit-ready checklist with citations. | Stayed high-level, omitted key GDPR accountability and AI Act obligations, and leaned on secondary sources. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 42/42, baseline 13/42. Pass 2: Sorena 156/156, baseline 12/156. |
| 3 | 2026-01-06 | Privacy Audit | Privacy Policy Audit - Consumer device manufacturer | Privacy policy audit for a consumer device ecosystem, assessing GDPR/CPRA disclosures, retention clarity, transfers, and rights transparency. | 100% | 21% | 5 | 40/14/40 | 247/19/247 | Pinpointed GDPR/CPRA gaps like right-to-object and recipient disclosures, backed by precise citations and service-level retention details. | Missed multiple mandatory disclosures and raised a few misleading compliance concerns without grounding in the policy. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 40/40, baseline 14/40. Pass 2: Sorena 247/247, baseline 19/247. |
| 4 | 2026-01-06 | AI Act Compliance | Cloud Service Terms Audit - Major cloud provider | Contract-focused audit of cloud service terms and privacy notices for EU AI Act and GDPR coverage, including transfers, processor terms, and AI restrictions. | 100% | 19% | 4 | 49/14/49 | 205/19/205 | Delivered clause-level GDPR processor analysis and AI governance review, including transfer safeguards, AI service restrictions, and concrete next steps. | Skipped contract-specific privacy and AI requirements and made unsupported claims instead of verifying the source terms. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 49/49, baseline 14/49. Pass 2: Sorena 205/205, baseline 19/205. |
| 5 | 2026-01-06 | Regulatory Timeline | EUDR Timeline - Office equipment manufacturer | EU Deforestation Regulation (EUDR) workback plan for a paper supply chain, with due diligence milestones, evidence expectations, and reporting deadlines. | 100% | 19% | 9 | 30/9/30 | 207/17/207 | Built an EUDR workback plan anchored to the amended deadlines, product scope, and technical evidence requirements (geolocation, reporting, customs). | Got key dates and scope wrong and missed multiple mandatory EUDR obligations, making the timeline unsafe to rely on. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 30/30, baseline 9/30. Pass 2: Sorena 207/207, baseline 17/207. |
| 6 | 2026-01-06 | Regulatory Timeline | EUDR Timeline - Beverage multinational | EUDR compliance timeline for a global beverage supply chain, mapping commodity sourcing to scope, due diligence steps, and declaration deadlines. | 100% | 13% | 7 | 46/8/46 | 204/16/204 | Mapped EUDR obligations to a beverage supply chain with commodity/CN code examples, correct deadlines, and a step-by-step evidence plan. | Misstated go-live dates and scope and omitted many legal requirements and operational steps needed for execution. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 46/46, baseline 8/46. Pass 2: Sorena 204/204, baseline 16/204. |
| 7 | 2026-01-06 | Regulatory Timeline | EU Data Act Timeline - Connected appliance manufacturer | EU Data Act compliance timeline for a connected-appliance manufacturer, covering data access, sharing, trade secrets, and cloud switching requirements. | 100% | 26% | 1 | 28/11/28 | 33/4/33 | Provided a date-driven roadmap covering user access/sharing, trade secret safeguards, and cloud switching duties, with a practical workstream plan. | Covered basics but omitted critical obligations like cloud switching rules, gatekeeper restrictions, and Commission guidance milestones. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 28/28, baseline 11/28. Pass 2: Sorena 33/33, baseline 4/33. |
| 8 | 2026-01-06 | Privacy Audit | Privacy Policy Audit - Gaming platform | Privacy policy audit for a gaming platform, focusing on GDPR transparency and CPRA/CCPA disclosures for California residents. | 100% | 43% | 3 | 28/16/28 | 47/14/47 | Performed a requirement-by-requirement GDPR + CPRA audit including retention-per-category, request methods, and opt-out signal expectations. | Left out several California and GDPR specifics (submission methods, statutory disclosures) and offered less operational guidance. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 28/28, baseline 16/28. Pass 2: Sorena 47/47, baseline 14/47. |
| 9 | 2026-01-06 | AI Act Compliance | AI Terms & Privacy Audit - AI platform | Audit of an AI platform's terms and privacy policy for EU AI Act and GDPR readiness, emphasizing transparency, training boundaries, and provider vs deployer responsibilities. | 100% | 48% | 6 | 43/23/43 | 94/40/94 | Separated what the documents prove vs what's missing, covering rights tooling, child handling, security posture, and EU AI Act GPAI obligations. | Missed practical rights pathways and several AI transparency and child-safety requirements, including a document-reference error. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 43/43, baseline 23/43. Pass 2: Sorena 94/94, baseline 40/94. |
| 10 | 2026-01-06 | AI Act Compliance | Cloud Terms + DPA Audit - Cloud provider | Audit of cloud service terms and a data processing addendum for GDPR Article 28 and EU AI Act readiness, including key contractual caveats and deployer obligations (e.g., FRIA). | 100% | 33% | 5 | 145/49/145 | 190/63/190 | Mapped processor contract clauses and highlighted high-impact caveats (like pre-release scope exclusions), plus deployer AI Act duties such as FRIA. | Focused on broad GDPR alignment but missed key contractual caveats and most deployer-focused AI Act obligations. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 145/145, baseline 49/145. Pass 2: Sorena 190/190, baseline 63/190. |
| 11 | 2026-01-06 | AI Act Compliance | AI API Terms + Privacy Audit - Model API provider | Audit of an AI model API's terms and privacy policy for GDPR and EU AI Act requirements, focusing on data-use boundaries, retention, and developer obligations. | 100% | 28% | 1 | 45/14/45 | 88/22/88 | Clarified paid vs unpaid data-use boundaries, retention windows, and both GDPR + AI Act transparency duties for developers and deployers. | Skipped controller/legal-basis details and several concrete requirements, and included an incorrect AI Act citation. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 45/45, baseline 14/45. Pass 2: Sorena 88/88, baseline 22/88. |
| 12 | 2026-01-06 | Privacy Audit | Privacy Policy Audit - Global search platform | Privacy policy audit for a global search platform, assessing data categories, purposes, rights, transfers, retention, and opt-out tooling under GDPR and CPRA. | 100% | 25% | 4 | 38/12/38 | 105/20/105 | Grounded findings in policy text, covering legal bases, controller identity, opt-out tooling (GPC, ad settings), and actionable fixes. | Missed major requirements (cookies, minors, sources) and made unsupported claims contradicted by the policy. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 38/38, baseline 12/38. Pass 2: Sorena 105/105, baseline 20/105. |
| 13 | 2026-01-06 | Privacy Audit | Privacy Policy Audit - Social platform | Privacy policy audit for a social platform, focusing on disclosure completeness, legal bases, retention clarity, and rights mechanisms under GDPR and CPRA. | 100% | 22% | 5 | 50/19/50 | 92/5/92 | Retrieved and analyzed the current geo-dynamic policy plus the US regional notice, verifying sale/share, GPC handling, and rights workflows with quotes. | Couldn't access the current policy, relied on an outdated version, and missed essential CPRA disclosures and request mechanisms. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 50/50, baseline 19/50. Pass 2: Sorena 92/92, baseline 5/92. |
| 14 | 2026-01-06 | Privacy Audit | Privacy Statement Audit - Enterprise software vendor | Enterprise privacy statement audit for GDPR and CPRA, focusing on transparency obligations, retention, DSAR mechanics, and user rights coverage. | 100% | 47% | 2 | 70/20/70 | 38/25/38 | Completed a comprehensive GDPR + CPRA audit with verified opt-out mechanisms, DSAR timelines, and cookie/ePrivacy considerations. | Covered headline items but omitted several statutory details (marketing objection, sources, timelines) needed for a compliance-grade assessment. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 70/70, baseline 20/70. Pass 2: Sorena 38/38, baseline 25/38. |
| 15 | 2026-01-06 | AI Act Compliance | Product Terms + Privacy Audit - Enterprise cloud/vendor | Audit of enterprise product terms and privacy statements for EU AI Act and GDPR, focused on contractual commitments and shared responsibilities across the AI value chain. | 100% | 31% | 1 | 72/26/72 | 47/12/47 | Connected product terms, DPA expectations, and AI governance obligations, calling out what must be confirmed contractually vs operationally. | Provided a higher-level review and missed several contract-specific protections and practical compliance actions (breach timing, training safeguards). | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 72/72, baseline 26/72. Pass 2: Sorena 47/47, baseline 12/47. |
| 16 | 2026-01-06 | Privacy Audit | Privacy Statement Audit - Streaming service | Privacy statement audit for a streaming service, evaluating GDPR transparency and CPRA disclosures such as sharing, preference signals, and required policy structure. | 100% | 33% | 5 | 41/19/41 | 74/14/74 | Verified EU/UK lawful bases and transfer safeguards, and pinpointed California disclosure gaps (GPC, 12-month lists, non-discrimination). | Made incorrect claims about lawful bases, transfers, and CPRA disclosures and conflated DNT vs GPC. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 41/41, baseline 19/41. Pass 2: Sorena 74/74, baseline 14/74. |
| 17 | 2026-01-06 | Privacy Audit | Terms + Privacy Audit - Secure messaging app | Audit of a secure messaging app's terms and privacy disclosures for GDPR and CPRA, focusing on lawful bases, retention, rights, and audit-ready gaps. | 100% | 32% | 1 | 38/18/38 | 67/11/67 | Reviewed multiple relevant sources (policy, shop opt-out page, support guidance) and flagged Art. 27 representative and CPRA signal requirements. | Missed several requirements and confused organizational structure, leading to a misleading compliance conclusion. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 38/38, baseline 18/38. Pass 2: Sorena 67/67, baseline 11/67. |
| 18 | 2026-01-06 | Privacy Audit | Privacy Policy Audit - Music streaming service | Privacy policy audit for a music streaming service, reviewing GDPR/CPRA disclosures around data categories, sharing, international transfers, and rights. | 100% | 36% | 1 | 40/21/40 | 84/17/84 | Found and cited specific policy statements (sale/share posture) and assessed children's protections, authorized agents, and request methods. | Missed key disclosures and incorrectly claimed important statements were absent. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 40/40, baseline 21/40. Pass 2: Sorena 84/84, baseline 17/84. |
| 19 | 2026-01-06 | Privacy Audit | Privacy Policy Audit - Messaging platform | Privacy policy audit for a messaging platform under GDPR and CPRA, including transfers, retention, rights workflows, and required disclosures. | 100% | 56% | 1 | 45/18/45 | 28/20/28 | Covered GDPR + CPRA specifics, including timelines, 12-month disclosures, and nuanced cross-regulation considerations (ePrivacy, case law). | Omitted several mandatory CPRA/GDPR elements (non-discrimination, SPI scope, minors) and provided less actionable remediation. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 45/45, baseline 18/45. Pass 2: Sorena 28/28, baseline 20/28. |
| 20 | 2026-01-06 | Privacy Audit | Privacy Policy Audit - Short-form video platform | Privacy policy audit for a short-form video platform under GDPR and CPRA, focusing on disclosures, rights, ad legal bases, and cross-border processing. | 100% | 24% | 6 | 38/14/38 | 103/12/103 | Validated EEA/UK disclosures (consent for ads, complaint routes) and pinpointed California statutory gaps like required link text and SPI handling. | Missed several policy-specific disclosures and lacked statutory precision on opt-out and automated decision-making requirements. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 38/38, baseline 14/38. Pass 2: Sorena 103/103, baseline 12/103. |
| 21 | 2026-01-06 | Privacy Audit | Privacy Policy Audit - Social network | Privacy policy audit for a social network, evaluating GDPR and CPRA transparency items, user rights coverage, and retention disclosures. | 100% | 30% | 5 | 120/47/120 | 66/14/66 | Mapped the policy to GDPR + CPRA with clear statutory checkpoints (toll-free methods, 12-month lists, SPI limit-use) and actionable remediation. | Skipped critical California format and request-method requirements and left gaps in indirect-source and necessity disclosures. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 120/120, baseline 47/120. Pass 2: Sorena 66/66, baseline 14/66. |
| 22 | 2026-01-07 | Employment Law | Union Comparison - Swedish software developer | Comparison of Swedish unions and collective agreements for a full-time software developer, covering benefits, tradeoffs, and agreement coverage. | 100% | 20% | 1 | 45/10/45 | 52/9/52 | Compared unions and collective agreements with the practical details that drive decisions (time bank, sick pay layers, pensions, notice periods). | Stayed at a high level, missed core CBA differences, and included an incorrect benefit-duration claim. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 45/45, baseline 10/45. Pass 2: Sorena 52/52, baseline 9/52. |
| 23 | 2026-01-07 | Employment Law | Employment Contract Review - Sweden | Employment contract compliance review under Swedish law, identifying risk areas, missing mandatory elements, and practical remediation guidance. | 100% | 17% | 2 | 33/8/33 | 53/5/53 | Applied the right Swedish-law framework (CBA context, working time limits, deductions, sick pay) and separated real risks from non-issues. | Flagged clauses as violations without CBA context and missed several statutory requirements needed for a defensible review. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 33/33, baseline 8/33. Pass 2: Sorena 53/53, baseline 5/53. |
| 24 | 2026-01-07 | Technical Review | Security Guidelines Review - Connected products | Technical review of connected product security guidelines, identifying inconsistencies and aligning requirements to real regulatory regimes and standards. | 100% | 25% | 12 | 26/10/26 | 141/16/141 | Turned internal security guidance into an audit-ready, regulator-aligned checklist (CRA/RED/EN 303 645) with dates, scope boundaries, and citations. | Missed most regulatory specifics and produced multiple incorrect or vague suggestions that weaken the guideline. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 26/26, baseline 10/26. Pass 2: Sorena 141/141, baseline 16/141. |
| 25 | 2026-01-10 | Technical Review | Cybersecurity Conformity Planning - CE/CRA readiness | Cybersecurity conformity assessment planning for CE/RED readiness, including evidence artifacts, assessment steps, test strategy, and documentation expectations. | 100% | 37% | 5 | 149/65/149 | 114/35/114 | Produced a CE conformity assessment plan with harmonized-standard traceability (OJ entries, clause IDs) and concrete pass/fail test criteria. | Gave a conceptual plan but missed traceability details auditors need (OJ numbers, provision IDs, acceptance criteria) and had version inconsistencies. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 149/149, baseline 65/149. Pass 2: Sorena 114/114, baseline 35/114. |
| 26 | 2026-01-10 | Technical Review | IoT Security Crosswalk + Test Plan - Consumer IoT | Consumer IoT security crosswalk and test plan, mapping ETSI and NIST requirements into testable procedures and evidence lists. | 100% | 30% | 4 | 118/45/118 | 90/20/90 | Delivered a full ETSI EN 303 645 <-> NIST 8259A crosswalk with quote-level traceability and assessor-grade test techniques. | Provided a general mapping but lacked verifiable quote anchors, missed key control extractions, and suggested risky version/citation approaches. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 118/118, baseline 45/118. Pass 2: Sorena 90/90, baseline 20/90. |
| 27 | 2026-01-10 | Technical Review | FIPS 140 Delta Analysis - Cryptographic modules | Delta analysis of FIPS 140-1 vs FIPS 140-2 for cryptographic modules, highlighting changed requirements and assessment implications. | 100% | 41% | 1 | 118/54/118 | 58/21/58 | Captured clause-by-clause deltas with testable requirements, including numeric thresholds and CMVP-ready artifacts. | Covered the basics but missed many validation-critical details (authentication thresholds, DTR/IG references, level-specific RNG rules). | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 118/118, baseline 54/118. Pass 2: Sorena 58/58, baseline 21/58. |
| 28 | 2026-01-10 | Technical Review | FIPS <-> ISO Crypto Module Mapping | Crosswalk between FIPS and ISO/IEC cryptographic module requirements, mapping controls and clarifying evidence expectations for audits. | 100% | 34% | 1 | 110/42/110 | 62/19/62 | Mapped FIPS 140-2/140-3 to ISO 19790 with deep links to the SP 800-140x series and validation evidence expectations. | Delivered a partial crosswalk but omitted key precision items (verbatim section quotes, interface taxonomy, program documents). | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 110/110, baseline 42/110. Pass 2: Sorena 62/62, baseline 19/62. |
| 29 | 2026-01-10 | Technical Review | ISO 27001/27002 Migration Package - ISMS update | ISO 27001/27002 migration package from 2013 to 2022, covering control changes, reorganization themes, and statement of applicability updates. | 100% | 34% | 4 | 130/54/130 | 88/24/88 | Created a practitioner-ready migration kit: machine-readable change matrix, filled SoA examples, and a timeboxed transition plan backed by authoritative sources. | Missed key migration artifacts (Annex B baseline, CSV/filled SoA) and made a couple of unverified claims about control groupings. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 130/130, baseline 54/130. Pass 2: Sorena 88/88, baseline 24/88. |
| 30 | 2026-01-10 | Technical Review | NIST 800-53 <-> ISO 27001/27002 Mapping | Control mapping between NIST SP 800-53 Rev. 5 and ISO/IEC 27001:2022 Annex A to support alignment, crosswalks, and audit preparation. | 100% | 12% | 5 | 175/40/175 | 68/1/68 | Explained rev4 to rev5 changes and produced an auditor-friendly crosswalk to ISO with gaps, tests, and source-grounded rationale. | Relied too much on workbook references and provided fewer verifiable anchors and test/evidence details. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 175/175, baseline 40/175. Pass 2: Sorena 68/68, baseline 1/68. |
| 31 | 2026-01-10 | Technical Review | NIST CSF 1.1 to 2.0 Crosswalk | Crosswalk from NIST Cybersecurity Framework 1.1 to 2.0, highlighting changes and mapping structure to support transition planning. | 100% | 29% | 5 | 180/72/180 | 39/7/39 | Mapped CSF changes to practical transition steps and linked crosswalks to authoritative artifacts for traceability and automation. | Delivered a reasonable summary but provided fewer pointers to official mapping exports and less detail on profile/tier migration. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 180/180, baseline 72/180. Pass 2: Sorena 39/39, baseline 7/39. |
| 32 | 2026-01-10 | Technical Review | NIST 800-171 Rev. 3 Delta + CMMC Mapping | Clause-level delta analysis of NIST SP 800-171 Rev. 2 vs Rev. 3 with CMMC 2.0 mapping, identifying added objectives and assessment impact. | 100% | 18% | 4 | 170/30/170 | 79/14/79 | Produced audit-defensible deltas with examples, ODP governance, and a clear Rev. 3 to Rev. 2 to CMMC mapping approach. | Listed new requirements but missed assessment-method impacts and source traceability, and added a nonessential news citation. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 170/170, baseline 30/170. Pass 2: Sorena 79/79, baseline 14/79. |
| 33 | 2026-01-10 | Technical Review | OT Security Framework Crosswalk + Gaps (IEC 62443/NIST) | OT security framework crosswalk between IEC 62443 requirements and NIST SP 800-82 guidance, identifying gaps plus example tests and evidence. | 100% | 20% | 3 | 99/20/99 | 99/20/99 | Built an OT-safe IEC 62443 <-> NIST 800-82 crosswalk with verification methods, evidence, and a realistic gap model. | Covered high-level mapping but missed several nuanced gaps and lacked the same level of audit-defensible sourcing. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 99/99, baseline 20/99. Pass 2: Sorena 99/99, baseline 20/99. |
| 34 | 2026-01-10 | Technical Review | PCI DSS v3.2.1 to v4.0 Delta + Crosswalk | PCI DSS v3.2.1 to v4.0 delta analysis with crosswalks to NIST SP 800-53 Rev. 5 and ISO/IEC 27001:2022, including key changes and timelines. | 100% | 32% | 3 | 155/52/155 | 155/47/155 | Delivered a PCI DSS migration package with authoritative citations, crosswalks, and evidence-ready remediation guidance. | Missed several audit-defensibility elements (official artifacts, full quotes) and included an incorrect timeline claim. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 155/155, baseline 52/155. Pass 2: Sorena 155/155, baseline 47/155. |
| 35 | 2026-01-14 | Sustainability Compliance | EU Energy Efficiency Directive Readiness - IoT appliances | Readiness assessment for an EU IoT home-appliance manufacturer under the EU Energy Efficiency Directive, including obligations, exemptions, and a practical implementation plan. | 100% | 26% | 8 | 81/21/81 | 83/22/83 | Separated what is mandatory vs optional, identified applicability triggers, and produced an evidence-driven readiness roadmap with governance and reporting steps. | Missed or diluted multiple explicit requirements and produced several overconfident obligations without sufficient grounding. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 81/81, baseline 21/81. Pass 2: Sorena 83/83, baseline 22/83. |
| 36 | 2026-01-14 | Sustainability Compliance | ESPR + Digital Product Passport Readiness - Appliances | Readiness assessment for ESPR and Digital Product Passport obligations for an EU smart-appliance manufacturer, covering applicability, data requirements, and execution plan. | 100% | 22% | 2 | 122/25/122 | 112/26/112 | Mapped the expected DPP/ESPR obligations to concrete product, data, and supply-chain controls with implementation sequencing. | Left key requirements vague, missed multiple Sorena-identified constraints, and under-specified required artifacts and scope conditions. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 122/122, baseline 25/122. Pass 2: Sorena 112/112, baseline 26/112. |
| 37 | 2026-01-14 | Sustainability Compliance | EU Batteries Regulation Readiness - Embedded batteries | Readiness plan for EU Batteries Regulation obligations relevant to consumer appliances with embedded or supplied batteries, including labeling, due diligence, and reporting. | 100% | 27% | 6 | 147/35/147 | 112/34/112 | Provided a compliance-ready breakdown of obligations by battery type and role (producer/importer), with evidence deliverables and timeline discipline. | Overlooked several explicit obligations and mis-prioritized workstreams, creating compliance gaps for key battery-related duties. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 147/147, baseline 35/147. Pass 2: Sorena 112/112, baseline 34/112. |
| 38 | 2026-01-14 | Sustainability Compliance | EU CSDDD Readiness - Supply chain due diligence | Readiness assessment for EU corporate sustainability due diligence obligations for an EU-listed appliance manufacturer, including governance, risk mapping, and remediation. | 100% | 34% | 4 | 98/28/98 | 98/38/98 | Turned due diligence requirements into implementable controls: governance, policy, risk mapping, supplier engagement, grievance handling, and reporting. | Missed several explicit duties and introduced ambiguous guidance that would leave audit-critical evidence and controls incomplete. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 98/98, baseline 28/98. Pass 2: Sorena 98/98, baseline 38/98. |
| 39 | 2026-01-14 | Sustainability Compliance | EU CSRD/ESRS Compliance Plan - Listed appliance manufacturer | CSRD/ESRS compliance applicability and readiness plan for an EU-listed smart-appliance manufacturer, including reporting scope, materiality, assurance, and data controls. | 100% | 20% | 11 | 119/19/119 | 104/25/104 | Clarified applicability boundaries and produced a compliance program plan spanning governance, materiality, ESRS datapoints, assurance, and disclosure logistics. | Had multiple scope/timeline inconsistencies and missed high-impact requirements around reporting mechanics and assurance-readiness artifacts. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 119/119, baseline 19/119. Pass 2: Sorena 104/104, baseline 25/104. |
| 40 | 2026-01-14 | Sustainability Compliance | EU CSRD/ESRS Compliance Plan - Listed automotive manufacturer | CSRD/ESRS applicability and compliance plan for an EU-listed automotive manufacturer, including ESRS scope, phased timelines, and operational reporting readiness. | 100% | 25% | 5 | 72/19/72 | 100/23/100 | Provided a structured, evidence-driven program plan with clear scoping, sequencing, and accountability to operationalize ESRS reporting. | Missed multiple explicit requirements and introduced misleading simplifications that would create gaps in CSRD reporting readiness. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 72/72, baseline 19/72. Pass 2: Sorena 100/100, baseline 23/100. |
| 41 | 2026-01-14 | Sustainability Compliance | EU Green Claims Readiness - IoT appliances | Readiness assessment for EU green-claims compliance in marketing and product communications for an EU IoT appliance manufacturer. | 100% | 12% | 6 | 89/11/89 | 99/12/99 | Converted green-claims obligations into a practical substantiation workflow: claims inventory, evidence standards, governance, and review gates. | Overlooked key compliance requirements and provided under-scoped guidance that could increase greenwashing risk. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 89/89, baseline 11/89. Pass 2: Sorena 99/99, baseline 12/99. |
| 42 | 2026-01-14 | Sustainability Compliance | EU Packaging Waste EPR Readiness - Appliances | Packaging waste and EPR compliance readiness plan for an EU home-appliance manufacturer, covering registration, reporting, labeling, and operational controls. | 100% | 14% | 6 | 103/14/103 | 119/17/119 | Outlined a compliance-ready EPR program with country-by-country obligations, operational ownership, and reporting/evidence requirements. | Missed several explicit obligations and under-specified evidence and process controls required for multi-country EPR compliance. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 103/103, baseline 14/103. Pass 2: Sorena 119/119, baseline 17/119. |
| 43 | 2026-01-14 | Sustainability Compliance | EU Water Sustainability Readiness - IoT appliances | EU water-sustainability and water-efficiency compliance readiness plan for IoT appliances, including product efficiency, disclosures, and governance. | 100% | 14% | 5 | 145/29/145 | 137/12/137 | Translated water-efficiency obligations and expectations into actionable controls, product requirements, and evidence-backed readiness steps. | Left multiple requirements uncovered and included misleading generalizations that would not hold up in an audit. | Two-pass scoring against the same requirements per pass. Pass 1: Sorena 145/145, baseline 29/145. Pass 2: Sorena 137/137, baseline 12/137. |

*Auditor columns show Sorena/Baseline/Total requirements met in each independent review pass.*

### Notes and disclaimers

- Results based on internal evaluation conducted January 2026.
- ChatGPT (baseline) is OpenAI ChatGPT, used as a general-purpose AI comparison.
- All factual errors counted are from ChatGPT responses only.
- This evaluation focused on regulatory and compliance research tasks.
- Results may vary depending on specific use case and document types.
- Not a substitute for legal counsel or professional advice.

## Why this gap exists

This is not a failure of intelligence. It is a mismatch of purpose.

A general-purpose assistant is designed to generate helpful, conversational answers, optimize for fluency, and respond quickly with plausible output.

Sorena AI is designed to execute compliance work: track obligations explicitly, enforce coverage and completeness, ground every statement in a source, and produce outputs auditors can verify.

General-purpose AI can answer questions fluently. Sorena is built to execute GRC work with coverage, citations, and reviewable evidence. These are different jobs.

## The risk of false confidence

The most dangerous failure mode in GRC is not being wrong loudly. It is being wrong quietly. When teams lean on a general-purpose assistant for audit-critical work:

- Missing obligations are not obvious
- Partial answers look complete
- Errors are hard to detect
- Coverage cannot be proven

That creates false confidence. Teams believe the work is done until an audit, regulator, or customer proves otherwise. Visibility without execution is not progress. It is exposure.

## Where general-purpose AI still has a place

None of this means a general-purpose assistant has no role in GRC. It is useful for learning and education, early exploration, drafting ideas that will be validated later, and general assistance outside audit-critical workflows.

But when compliance must be complete, grounded, and defensible, execution matters more than answers. That is where Sorena AI is the system of record and the execution layer GRC requires.

## What actually works at scale

The benchmark points to a simple conclusion. GRC scales when:

- Humans make judgments and decisions
- Systems handle coverage, mapping, tracking, and evidence
- Execution is continuous, not episodic
- Outputs are verifiable by design

Do not take our word for it. Benchmark it yourself against your own tasks and see the difference in real GRC work.

## Frequently asked questions

**How was the benchmark scored?**

Each of the 43 tasks was scored against auditor-defined requirements in two independent passes by domain-expert auditors. Coverage is the share of required obligations explicitly and correctly addressed. Factual errors were tracked separately and counted only against the general-purpose baseline responses.

**What is the general-purpose baseline?**

The baseline is OpenAI ChatGPT, treated here as a leading general-purpose AI assistant used as a comparison point. All factual errors reported are from the baseline responses only. Results reflect an internal evaluation conducted in January 2026 and may vary by use case and document type.

**Does 100% coverage mean Sorena replaces auditors or legal counsel?**

No. Sorena handles coverage, mapping, tracking, and evidence so people can focus on judgment and risk decisions. The output is audit-ready and source-linked, but it is not a substitute for legal counsel or professional advice.

## Sources

- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework?ref=sorena.io)
- [Sorena AI Benchmarks](https://www.sorena.io/solutions/benchmarks)

## Keep reading

- [Your GRC Tool Tracks Work. Ours Does the Work.](/resources/grc-tool-tracks-work-ours-does-the-work.md)
- [AI Agents Following Malicious Instructions Is a Real Security Problem](/resources/ai-agents-malicious-instructions.md)
- [AI Is Not Free for the Planet. Using It to Power ESG Is the Minimum We Owe.](/resources/ai-esg-the-minimum-we-owe.md)


---

[Privacy Policy](https://www.sorena.io/privacy.md) | [Terms of Use](https://www.sorena.io/terms-of-use.md) | [DMCA](https://www.sorena.io/dmca.md) | [About Us](https://www.sorena.io/about-us.md)

(c) 2026 Sorena AB (559573-7338). All rights reserved.

Source: https://www.sorena.io/resources/chatgpt-in-grc-false-confidence.md