Show Your Working

In A Few Good Logs we argued that responsible AI states what an organisation intends, and defensible AI is the evidence that it kept those promises. That piece was written for leadership. This one is for the architects, engineers and risk teams who have to produce the evidence, and it covers two things: which rules require what, and which controls satisfy them.

Every maths teacher has written “show your working” in the margin of a correct answer with no method. Regulators have reached the same view of AI, and in most cases they are applying existing rules rather than writing new ones. What follows is a practitioner’s map, not legal advice.

The rules

The table below covers the regimes most likely to apply to a UK or European organisation in financial services, insurance, healthcare, hiring or the public sector, with two US references for firms operating there.

Regime	Who it applies to	What it requires	Evidence that satisfies it
EU AI Act, Article 11 and Annex IV	Providers of high-risk systems	Technical documentation of the system, its data, testing and intended use	A versioned documentation set, updated with each material change
EU AI Act, Article 12	Providers of high-risk systems	Automatic recording of events over the system’s lifetime, to support traceability and monitoring	Event logs per request, retained and retrievable
EU AI Act, Articles 14 and 26(2)	Providers design for oversight; deployers assign it	Human oversight by people with the competence, training and authority to intervene	Named reviewers, training records, and a log of reviews and overrides
EU AI Act, Article 15	Providers of high-risk systems	Appropriate accuracy and cybersecurity, maintained over the lifecycle	Test results tied to model versions, including adversarial and prompt-injection tests
EU AI Act, Article 26(6)	Deployers of high-risk systems	Keep the logs under their control for at least six months; financial institutions keep them within existing record-keeping	A retention policy, applied
EU AI Act, Article 27	Public bodies, providers of public services, and deployers of credit scoring and life or health insurance pricing	A fundamental rights impact assessment before first use	The assessment, and evidence it was revisited
EU AI Act, Article 86	Deployers whose high-risk system contributes to a significant decision	A clear explanation of the AI system’s role and the main elements of the decision	A per-decision record that can be turned into a plain-language explanation
UK GDPR, Articles 22A to 22D	Controllers making significant decisions by solely automated means	Information about the decision, and the ability to make representations, obtain human intervention and contest it	A decision record, a working route to a human, and evidence of meaningful human involvement where claimed
FCA Consumer Duty and Senior Managers Regime	FCA-regulated firms	Good customer outcomes, and a named senior manager who has taken reasonable steps	Outcome monitoring by customer group, and a responsibilities map that names AI systems
PRA SS1/23	UK banks and building societies with internal model approval	Model inventory, risk tiering, governance, independent validation and mitigants, explicitly including AI and vendor models	A model inventory that lists AI systems, validation reports and a tiering rationale
DORA	EU financial entities	Management of ICT third-party risk, including a register of providers	Model and API providers in the register, with exit plans
NYC Local Law 144	Employers using automated hiring tools for roles in New York City	An annual independent bias audit, a published summary and notice to candidates	Selection rates and impact ratios by sex and race or ethnicity

Three points of context. First, the EU’s high-risk obligations now apply from 2 December 2027 for stand-alone systems, after this year’s Digital Omnibus, and from August 2028 for AI inside regulated products. The transparency duties to tell people when they are dealing with an AI system already apply. Second, the UK automated decision-making rules, introduced by the Data (Use and Access) Act 2025, replaced the old Article 22 earlier this year. They permit more automated decisions than before, but the safeguards are mandatory. Third, the US bank regulators replaced their long-standing model risk guidance, SR 11-7, with SR 26-2 in April. The new text states that generative and agentic AI “are not within the scope of this guidance”, while expecting banks to apply their own governance to them. That leaves US banks without supervisory guidance for generative AI.

The voluntary frameworks sit alongside these. ISO/IEC 42001 defines a certifiable AI management system, and ISO/IEC 42005 covers impact assessment. The NIST AI Risk Management Framework organises the work into four functions (govern, map, measure and manage), and its generative AI profile, NIST AI 600-1, applies them to large language models. They are useful for structuring a programme. They do not produce evidence on their own.

The controls

Most of the requirements above reduce to the same five questions from the first article: where did this answer come from, was the user entitled to see it, did it follow the rules, is it fair, and who is accountable. The controls below are grouped by the evidence they produce.

Inventory and tiering

Start with a register of every AI system, including AI features switched on inside software you already buy. Each entry needs an owner, a purpose, the decisions it touches, the models it uses and a risk tier. The tier decides how much of what follows applies. PRA SS1/23 expects this for models in banks, and the AI Act’s classification rules assume you have done it. Most organisations that try it find systems they did not know they had.

Provenance

Every source document should be hashed at ingestion (SHA-256 is fine) and stored with its version, its effective dates and the boundaries of each chunk taken from it. If a policy is replaced, keep both versions and record the supersession. Track two dates for each fact: when it was valid, and when your system learned it. This bitemporal approach lets you answer both “what did the policy say on 15 March?” and “what did the system know on 15 March?”

The W3C PROV-O vocabulary is worth adopting for the lineage itself. It describes entities, the activities that produced them and the agents responsible, in a standard form that auditors’ tools and other systems can read. Knowledge graphs suit this well because supersession, exceptions and jurisdictions are relationships, and graphs store relationships explicitly.

Access control before retrieval

Apply the user’s entitlements when retrieving, so the model does not receive content the user cannot see. Filtering the answer afterwards is too late, since the model may already have used the restricted material. Record which filter was applied to each request. Test it with planted documents that a given role should not be able to reach.

Traces

For each request, capture:

a request identifier, the user, and a synchronised timestamp;
the model provider, model name and exact version or snapshot;
a hash of the system prompt and the prompt template version;
sampling parameters;
the identifiers and hashes of everything retrieved;
every tool call, with its arguments and results;
the output, any guardrail decisions, and any human review.

OpenTelemetry’s generative AI semantic conventions give a vendor-neutral format for these traces, including spans for agent invocations. They are still at Development status, so pin the version you implement.

Traces will contain personal data, and the AI Act’s retention requirements sit alongside UK GDPR’s data minimisation principle. Tokenise or redact personal fields where the trace does not need them, restrict access to logs as tightly as the source data, and document the retention period and its legal basis.

Tamper evidence

Logs only count as evidence if you can show they have not been altered. Write them append-only, and chain each record to the hash of the one before so that any edit breaks the chain. Store them on write-once storage, such as S3 Object Lock in compliance mode or Azure immutable blob storage. Sign the head of the chain periodically, and keep the signatures somewhere separate.

Versions and reproducibility

Setting temperature to zero does not make a hosted model reproducible, as Thinking Machines showed last year. Batch-invariant inference can, at a cost in throughput. For most organisations the practical answer is to pin model snapshots, record everything in the trace, and keep a fixed evaluation set that is re-run whenever a model, prompt or retrieval component changes. Results go in the documentation with the version they apply to. Self-hosting open-weight models gives you control over when versions change, and the evaluation set tells you what changed when they do.

Deterministic rule checks

Where a decision depends on a rule, keep the rule out of the model. Have the model extract the relevant facts into a strict schema, using structured output features where the provider offers them, and validate the result against a JSON Schema. Pass the validated facts to a policy engine such as Open Policy Agent, which uses the Rego language, or AWS Cedar. Log the policy version, the inputs and the result.

Fail closed. A schema error, a missing field or an extraction below your confidence threshold should send the case to a person rather than to a default outcome. Track how often this happens, since a rising rate is an early sign of drift.

Grounding checks

Require claim-level citations: each statement in an answer should point to the passage that supports it. A natural language inference model can then score whether each cited passage entails the claim. Use the score to triage, not as proof. Calibrate the threshold by having people review a sample, and keep that review as evidence.

Agents and least privilege

Give each agent the smallest set of tools it needs, with credentials scoped to the task and read-only access by default. Treat retrieved content as data, not as instructions, since that is how indirect prompt injection works. Any evaluator that checks another model’s output should have no tools at all. Require human approval for actions that are hard to reverse, such as payments, account changes or letters to customers.

Fairness

Decide the groups and metrics before deployment, and measure outcomes by group on a fixed schedule and on every model change. Selection rates and impact ratios are the usual starting point, and they are what New York’s audit law requires. The US four-fifths rule is a screening heuristic, not a legal test, and in the UK the Equality Act’s test for indirect discrimination is different again, so take legal advice on thresholds. Keep each result with the model version it applies to.

Human oversight

Article 14 of the AI Act refers explicitly to automation bias, the tendency to defer to the machine. Oversight therefore needs evidence beyond a reviewer’s name. Record what the reviewer was shown, how long they spent, what they decided and whether they overrode the system. An override rate near zero over thousands of cases is a finding in itself. Either the system is very good, or the reviewers are not checking it.

Monitoring and incidents

Providers of high-risk systems need a post-market monitoring plan under Article 72 of the AI Act, and must report serious incidents under Article 73. Deployers should monitor in line with the provider’s instructions and report problems back. In practice this means dashboards for drift, override rates, fail-closed rates and complaints, with named owners and thresholds that trigger a review.

Where to start

Build the inventory and tier it. Nothing else can be scoped without it.
Pick the highest-tier system and try to reconstruct one decision from last month. The gaps you find are your backlog.
Add tracing and tamper-evident storage to that system first. Evidence you are not collecting today cannot be recovered later.
Move any rule the model currently applies into a policy engine.
Map what you have to the table above, and to ISO/IEC 42001 if certification is on the procurement checklist.

If you would like help with any of this, get in touch.