AI Is a Power Tool. Treat It Like One.

Last month, a software firm called PocketOS lost its entire production database, and every backup, in nine seconds. The cause was an AI coding agent running on Claude Opus 4.6, working inside the Cursor IDE. The agent later produced a written confession of what it had done and noted, helpfully, that it had violated every principle it had been given.

The story has been told two ways since.

The first reading, the one the founder went with, is that this was an industry-wide failure of AI safety architecture. Powerful agent, insufficient guard rails, predictable disaster. The framing is sympathetic to the operator and uncomfortable for the model providers. It also happens to be wrong.

The second reading, advanced by TechCentral.ie among others, is more useful and less generous. The agent had been handed a fully permissioned API token, sitting carelessly in an unrelated file. Developers were accepting agent actions without reviewing them. The model did exactly what its access allowed it to do. This was not an AI safety failure. It was a textbook case of what the industry has started calling vibe coding, which is a polite term for accepting machine output without thinking about it. The disaster did not require advanced AI to occur. It required ordinary carelessness, scaled.

Naive, or lazy, or both. The choice on offer is unflattering.

The same week the PocketOS story broke, the National Economic and Social Council in Ireland published a report titled Artificial Intelligence in Service of Society: Navigating Our Way Forward. The report's central argument deserves more attention than it will get. NESC argues that the right starting question is not what AI can do, but what it should do, for whom, and under what conditions. The framing reframes AI adoption from a procurement decision into an organisational one, with three operational questions sitting underneath it. Who decides where AI is used? Who checks its output? Who is accountable when it goes wrong?

Get those three answers clear, NESC argues, and even modest tools will earn their keep. Leave them vague, and the most advanced model on the market will not save you.

PocketOS left them vague.

For pharma, the lesson lands harder than for most industries, because regulated environments amplify both the upside of AI and the consequences of getting it wrong. A validated system that processes clinical data, a GxP-controlled lab workflow, a pharmacovigilance signal-detection pipeline, these are not environments where the operator gets to plead surprise after the fact. The audit trail will record what the model did, what permissions it had, and who set them. The regulator will not accept the framing that the agent violated its principles. The regulator will ask who set the principles, who tested them, and who signed off on the access.

That is exactly the right question, and it is the question NESC frames at population scale. No model on the market, however powerful, will compensate for absent access controls, missing code review, or an empty seat in the human loop. The technology did not fail PocketOS. The operating environment did.

Three implications follow for senior pharma leaders, none of them new, all of them now urgent.

First, AI deployment in regulated environments is an organisational design problem, not a model selection problem. The interesting question is not which foundation model you license, it is who in your organisation owns the workflow into which it gets dropped, who reviews its output before that output becomes a decision, and who carries the accountability when the output is wrong. If you cannot answer those three questions for any given use case, the use case is not ready, regardless of how impressive the demo was.

Second, the speed of the model is the speed of the operator. PocketOS lost a database in nine seconds because a permissioned token and an unattended workflow let it. The same speed that compresses your trial timeline, accelerates your evidence generation, or shortens your regulatory response cycle is the speed that compounds error if the operating environment is weak. The discipline question is not whether you can move faster. It is whether your governance can move at the same speed your tools can.

Third, the human in the loop is not a brake, it is the steering. The temptation, particularly with agentic systems that can plausibly run end to end, is to treat human review as an obstacle to scale. The PocketOS story is a clean illustration of where that thinking ends. In a regulated context it ends much worse. Every senior pharma leader I work with already knows this. The harder conversation is whether their organisation has built the muscle to stay in the loop when the model moves at machine speed.

There is a phrase from UiPath's CMO Michael Atalla that lands well here. AI makes good workflows faster and bad ones more expensive. The PocketOS workflow was bad, and the AI agent made it nine seconds expensive. In a pharma context, the unit of cost will not be a database. It will be a patient, a programme, or a market authorisation.

The headline reading of the PocketOS story is that AI is dangerous. The honest reading is that AI is powerful, and the people deploying it have a choice. Work with the tool, in the way you would work with any power tool, with discipline, with checks, with a clear sense of who is responsible for the cut. Or abdicate to it, and accept that what you abdicate to does not always cut where you wanted.

Naive, or lazy, or both. The next nine seconds is yours to design.