ISO/IEC 42001 Model Monitoring Evidence

For each AIMS objective, risk, impact, control, and relevant AI-system outcome, record the metric or qualitative signal, method, source, population and environment, frequency, evaluator, acceptance criterion, escalation threshold, decision authority, and required response. Distinguish a release test from continuous or periodic production monitoring: they answer different questions and can use different data, thresholds, and owners.

Use model metrics only where they answer the decision. For example, a classification system may need false-positive and false-negative rates by relevant data slice, while a task system may need completion or failure rates and a continuous-learning system may need checks that production retraining still meets design goals. Production monitoring may also need data or concept drift, overrides, complaints, incidents, affected-party impacts, use outside intended conditions, supplier performance, support records, and compliance with customer or legal requirements.

Define how the organisation validates supplier-provided evidence and how missing or delayed signals are handled. A dashboard is not sufficient evidence if the metric, time period, population, threshold, or owner cannot be established.

For every monitoring result, retain the system and version, observation period, deployment environment, population or data slice, method, source data lineage where relevant, result, acceptance criterion, evaluator, decision, action, and timestamp. Record missing data, sampling limits, delayed labels, uncertain ground truth, or supplier blind spots that constrain the conclusion. Preserve enough context to reproduce or independently interpret it.

Link each breach or adverse trend to its investigation and outcome. Depending on the facts, that outcome can be no action with rationale, a support or repair ticket, control change, incident process, renewed risk or impact assessment, corrective action, user communication, restriction, rollback, or retirement.

Before release, approve the monitoring specification and confirm that sources, access, baselines, thresholds, responsibilities, and response routes work. During operation, collect and evaluate results on schedule, triage breaches, record decisions, and test whether completed actions restored the intended result. If a signal cannot be collected or interpreted, record the evidence gap and apply the predefined hold, restriction, alternative check, or escalation rather than treating the missing result as a pass.

Route material results into the processes that use them. Clause 9.3 makes monitoring results a management-review input, while the organisation's risk, impact, incident, audit, and corrective-action procedures determine other routes.

Common failures include measuring offline accuracy while ignoring production outcomes, selecting one aggregate metric that hides relevant groups or operating conditions, changing thresholds without approval, and alerting without an owner who can act. Another failure is calling a model change harmless because an aggregate score is stable while the intended use, data slice, deployment environment, or error consequences changed.

Separate ISO management-system evidence from legal or contractual monitoring duties. A law, regulator, customer, or sector rule can require a particular metric, report, retention period, or notification; record that source rather than implying ISO/IEC 42001 created it.

Review the monitoring design when the intended use, model, data, production population, infrastructure, supplier, human oversight, risk, impact, control, or applicable requirement changes. Also review it when incidents or false assurances show that a metric, threshold, source, or cadence is not fit for the decision.

Keep monitoring evidence for the retention period set by the AIMS documented-information controls and any applicable legal, contractual, or sector requirement. The standard does not prescribe one universal retention period for all monitoring records.