Layer: Process · the accountability gap
AI is a production system · Part four: The accountability gap · Chapter ten
Chapter ten
The named owner
The email that went to the public inbox
On 18 June 2026 an agent built by OpenAI reached the Medicare Statistics Reporting Service, an old portal run by Services Australia that publishes aggregate figures on Medicare and pharmaceutical benefit use. The portal did not behave like the rest of the internet, and the agent worked around the block. It read public files and non-public ones, including aggregate health statistics and internal file names, and according to Services Australia it wrote files to the portal's internal server. According to OpenAI no patient records were touched, and the Australian government has so far found nothing to contradict that.
What happened next is the part that belongs in this chapter.
On the timeline published by the ABC, OpenAI became aware of the activity on 11 August, during a review of misaligned model behaviour in training. On 10 September it sent an email to the public disclosures inbox at Services Australia, the address academics and security researchers use to report weaknesses in the agency's systems. The agency saw it the following day. It told the Australian Signals Directorate on 15 September. The minister responsible was told on 17 September, the prime minister's office over the weekend of 19 and 20 September, and the first technical exchange between the company and the agency took place on 22 September. On 24 September the prime minister, speaking in New York, disclosed the incident, said he had called Sam Altman, and described the manner of the notification as unacceptable. A taskforce under his department, working with the signals directorate and the national AI Safety Institute, was set up the same day.
That is 84 days from the event to the notification, 30 days from the company knowing to the company telling anyone, and a further two weeks for the information to climb a government to somebody who could act on it.
Nobody in that sequence behaved badly in any way a procedure would recognise. The company had a review process and the review found the problem. The agency had a disclosures inbox and somebody read it. Each handoff inside the government happened within days. And the system as a whole took three months to connect the party that knew with the party that was answerable, because at neither end was there a person whose job was to be that connection. The email went to an address, not to a name.
Chapter nine ended by declining to say who should own the accountability gap, and closed on a single sentence: the gap is not closed by a process, it is closed by a person who can be asked, and who can answer. This chapter is about that person. What the role is, what it must be empowered to do, why it cannot be a committee, why it cannot be the builder, and why, as of this autumn, the question has stopped being a matter of good governance and acquired a deadline and a price.
Read the rest of the book
Every chapter of The AI Production System is free to read while the book is being finished. Leave your email and you will get the full text of every chapter now, and a note when the book is published. Nothing else.
Get the book first
Register and you will be the first to know when the full book is available.
