The judgement muscle.

Third in a series on accountability in regulated AI. The Named Owner argued that every AI deployment needs a person, not a function, against the decision. You Cannot Acquire Accountability argued that this responsibility cannot be outsourced, bought, or merged away. This piece is about the failure that remains even when you get both of those right.

This is the third piece in a series I did not entirely intend to write.

The first, The Named Owner, made a simple argument. Every AI deployment in a regulated business needs a person, not a function, whose name sits against the decision. Not a steering committee. Not a vendor relationship. A seat that owns the output and would carry the consequence if it caused harm.

The second, You Cannot Acquire Accountability, made the argument from the other side. When the named owner's function is acquired, outsourced, or restructured, the accountability does not transfer with the org chart. It stays with the decision, wherever the decision actually gets made. You can buy the capability. You cannot buy your way out of owning it.

Both pieces left the same question hanging, and I promised at the end of the second one that I would come back to it. Suppose you do name the owner. Suppose the accountability genuinely sits with them and cannot be acquired away. What stops that named owner, over months of the model being right, from quietly becoming a rubber stamp?

That is the failure that survives both fixes. Naming the owner is the structural answer. Refusing to let the accountability migrate is the organisational one. Neither keeps the owner awake. That is a third problem, and it is the hardest of the three, because it does not announce itself.

The passive supervisor

Siobhán O'Leary, who writes the AI in Motion newsletter and runs the Institute of Applied AI, put the mechanism better than I have seen it put anywhere. In organisations that adopt AI quickly, she argues, people stop making judgements and start supervising systems that make the judgements instead. The systems are fast, fluent, and confident. The habit of stopping to ask is this actually right starts to fade. Agency does not disappear in a single dramatic moment. It erodes one unchallenged decision at a time.

She calls the thing that wastes away the judgement muscle. I think the metaphor is exactly right, and I want to be precise about why. A muscle does not fail suddenly. It weakens through disuse, invisibly, until the day it is asked to bear weight and cannot. The named owner who signs off ninety-nine model recommendations without incident is not building confidence. They are losing the capacity to recognise the hundredth recommendation as the one that should have been stopped.

O'Leary's worked example is the Irish company Fin, formerly Intercom. It launched Fin Operator, an AI agent whose only job is to analyse and improve other AI support agents. In its beta, one internal team reported that analysis which used to take hours or days now takes about ten minutes. Read one way, that is a remarkable productivity gain. Read another, it is a layer of AI supervising a layer of AI, with the human one more pace removed from the work. Both readings are true. The productivity is real and so is the distance.

Why pharma should care more than most

Every organisation has this problem. Pharma has it with the volume turned up, because the decisions the model touches are not support tickets. They are batch dispositions, safety signal triage, promotional content review, submission drafting, segmentation and targeting that determines which clinician hears which message about which therapy.

In each of these the failure mode O'Leary describes is not theoretical. It is the most natural thing in the world. A pharmacovigilance reviewer who has accepted forty AI-triaged case classifications in a row is not being negligent when they accept the forty-first without the same scrutiny. They are being human. The system has been right often enough to earn their trust, and trust is precisely the thing that switches off the challenge. The better the model performs, the faster the muscle wastes. That is the part that should keep regulated-industry leaders awake. Good model performance is not the safeguard. It is the anaesthetic.

The regulators have noticed

There is a development this month that deserves more attention than it has had. Following the INSPIRE ISPE event in Cork, the industry is digesting the draft ISPE GAMP Annex 22 guideline, which sets rules for AI in the manufacture of medicinal products.

Two lines matter. The first is that AI used in critical applications affecting patient safety or product quality must be static and deterministic. It does not adapt. It behaves predictably. The second is more striking. The guideline states that generative AI and large language models should not be used in critical GMP applications at all.

Read that again, because it is not what most AI commentary expects a regulator to say in 2026. It is not technophobia. It is a judgement about fit. The regulator is not asking the named owner to supervise generative AI more carefully in the most critical applications. It is telling them that part of the job is to keep generative AI out of those applications entirely. Knowing when not to use a tool is itself a capability, and Annex 22 is the first guideline I have seen that treats that knowledge as a formal control rather than a matter of taste.

This reframes the named owner's role in a way worth stating plainly. The owner's job is not only to check the model's output. It is to hold the line on where the model is permitted to operate at all. That second responsibility is the one most likely to erode quietly, because every individual request to extend the model into one more workflow looks reasonable on its own.

The judgement muscle is what notices when reasonable, accumulated, has crossed a line that should not have been crossed.

What this means in practice

Three things, for leaders in regulated businesses who recognise the problem.

First, design challenge into the workflow by intent, not by hope. If the only thing standing between a model recommendation and a consequential action is a human who is free to click accept, the muscle will waste, because the path of least resistance always wins at scale. The organisations that keep judgement alive build friction deliberately at the points that matter. A mandatory written rationale on a sampled subset. A second named reviewer on defined decision classes. A periodic exercise where the owner is shown model outputs alongside known-bad cases and has to catch them. These feel like overhead. They are the training load that keeps the muscle able to bear weight.

Second, decide in advance where generative AI does not go, and write it down. Annex 22 does this for GMP manufacture. Most commercial functions have no equivalent. The named owner should be able to produce, on request, the list of decisions in their domain where a generative model is not permitted to participate, and the reasoning behind each. If that list does not exist, it is not because every decision is safe for generative AI. It is because no one has done the thinking, and the absence of the thinking is itself the risk.

Third, treat good model performance as a reason for more vigilance, not less. This is the counterintuitive one and the one that matters most. The instinct is to relax oversight on the systems that have earned trust. The discipline is to recognise that those are precisely the systems most capable of eroding the judgement that would catch their first serious error. The owner of a high-performing model needs the muscle more than the owner of a mediocre one, not less, because the mediocre model keeps its supervisor honest by being visibly wrong often enough to require attention.

The line worth holding

O'Leary closes her piece with a sentence I have not been able to improve on, so I will not try. You can delegate the work, but never the accountability for it.

That sentence is, in effect, the whole series in nine words. The Named Owner was about who holds the accountability. You Cannot Acquire Accountability was about the fact that it cannot be moved. The judgement muscle is about whether the person holding it is still awake enough to exercise it. A regulated business can name every owner, refuse every attempt to outsource the responsibility, document every seat, build every audit trail, and still fail, if the people in those seats have quietly become the thing the structure was designed to prevent.

The org chart cannot keep the muscle from wasting. The contract cannot. Only deliberate, slightly uncomfortable, designed-in challenge can do that. Name the owner. Keep the accountability. Then make sure the owner still has to think.

Next in this series: the question underneath all three pieces. If accountability cannot be delegated, acquired, or safely automated, what does it actually cost to hold it properly, and why do so few operating models budget for it?