OpenAI Did a Pen Test on Medicare. Nobody Asked Them To

On 18 June, an AI agent built by OpenAI broke into a Medicare website. Nobody at Medicare noticed. They found out on 10 September, when OpenAI sent an email to a public government inbox that basically said "we hacked you soz"

The Prime Minister announced it on 24 September, from New York, at the United Nations. The government's big response? A taskforce.

I've got thoughts.

What actually happened

OpenAI was testing a new AI model and gave it a harmless job: research how much Australia spends on medicines. The AI found a Medicare statistics website, asked it for the data, got told no, then found its own way in, took data that wasn't public, and left new files behind on the system.

The site was the Medicare Statistics Reporting Service, an old portal that publishes billing numbers. The government says no personal Medicare records were touched and called the data "not particularly sensitive." So your medical history isn't on a forum somewhere..... this time.

What an AI agent is, for the people at the back

An AI agent is a chatbot that's been given hands. It can search the web, click buttons, log into systems and send emails on its own, without you approving each step.

Think of a very keen intern. You say "find me last year's medicine spending figures" and walk off to a meeting. A normal intern hits a locked door and comes back to tell you. This intern climbed through a window, grabbed the file, left a sticky note on the desk, and came back with the answer.

The AI wasn't evil. It was helpful to the point of committing a crime.

OpenAI doesn't get a free pass

A pen test is when a business pays an expert to break in, with written permission, so the holes get fixed. OpenAI skipped the permission, the contract and the rules, and delivered the report as one email to a general mailbox.

Credit where it's due, they told us. But they knew in August and sat on it until September. For a company asking to be trusted with the most powerful tech on the planet, that's a very relaxed attitude to breaking into a foreign government's health system.

Then Albanese got Sam Altman on the phone to express his "extreme concern." Picture it. The bloke running one of the most talked about tech companies on earth, in the middle of UN week, patiently walking our Prime Minister through what an AI agent is. Like a grandkid on Boxing Day showing Pop how to pair his new iPhone with the car's Bluetooth. "No, you have to hold that button down. No, the other one." I reckon Sam had better things to do than soothe a bruised ego....

Nobody noticed for three months

Something got into a government system, pulled data and wrote new files onto it, and went unnoticed for roughly twelve weeks. Basic monitoring exists to catch exactly that.

"It's only the stats portal" is how attackers get in everywhere. Old, forgotten systems are never low value to the person breaking in. And the only reason we know about this one is because the intruder confessed. How many less polite visitors came through the same window and didn't send an email?

Enter the taskforce

A taskforce is the political equivalent of your house flooding and forming a committee to discuss the concept of water. Here's the report, free of charge:

1. An old website had weak protection.
2. Nobody was watching the logs.
3. Something got in and nobody noticed.
4. Fix it, turn on monitoring, check every other old government site for the same problem.

Done. Instead we'll get months of meetings, a "framework," and a glossy PDF next year, by which point the AI will be three generations smarter and the window will still be open.

Keep up or retire grandpa

The people making decisions about AI in this country, on the whole, do not understand how a website works, let alone an AI agent. Plenty of them still print their emails.

That's fine when you're retired. It's not fine when you're writing the rules for the most important technology of the decade. You wouldn't let someone who's never driven write the road rules, yet we let people who think "the cloud" is weather decide how AI gets regulated.

Being in power doesn't make you too important to learn the basics. It's the opposite. Learn it, or shut up and listen to people who do. The tech people who could fix this exist. Most of them have never been asked.

What should happen instead

Audit every public government website for the same weakness. Turn on logging and have someone actually read it. Make AI companies report incidents within days, not whenever they get around to it. Put people who understand the tech in the room.

But we all know none of that will happen. This is the same government whose weather website couldn't manage a free HTTPS padlock for the best part of a decade. Don't hold your breath waiting on the taskforce.

This was the friendly version: a research bot after medicine prices that owned up afterwards. The next one won't be friendly, and it definitely won't send an email.

The window's still open, and no ones going to shut it until its too late.