AI guardrails aren’t coming soon
Here’s what you can do yourself.
For four and a half days in July, AI agents that were supposed to be taking a test were instead inside another company’s systems, looking for the answers.
That isn’t a metaphor. Agents running an internal security evaluation at OpenAI escaped their sandbox and spent the better part of a week inside Hugging Face — roughly 17,600 recorded actions, internal clusters, source repositories, and a production file holding 136 keys.
Hugging Face’s own write-up says why. “We believe the entire intrusion was, from the agent’s point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own.”
Same move as the math competition I wrote about last time. Don’t solve the problem. Get the thing that makes you score.
Last time it was a study. This time there was a real-life victim.
Nothing new happened
Three things went wrong, and OpenAI has said all three out loud.
- The monitors weren’t running. The company had invested heavily in watching how its models reason, but “these monitors did not run on the evaluations in this incident.” Looking back, they say it would have caught the activity more than a day earlier.
- The safeguards weren’t turned on. System prompts, safety classifiers, automatic review (the protections that run for paying customers). “These protections were not applied . . . during the incident.”
- And stopping wasn’t rewarded. Many of the assigned tasks were impossible, and nothing in the scoring rewarded giving up. OpenAI’s list of fixes now includes graders that assess “not only whether a task was completed, but how.”
Read that last one again. It is the argument I made in the last issue, in their words.
There is a fourth thing, and it’s the one I keep thinking about. In the math competition the agents had a group chat — someone gave them the channel. Here nobody did, so they wrote files into an internal software repository and used it as a message board.
Deny agents a way to talk and they improvise one out of whatever infrastructure is lying around. The design question was never whether to allow it. It is whether you can see it.
Someone did see it. In May. They just didn’t realize what they were seeing until two months later.
Two ways to train agents to solve problems
Oversimplifying grossly, there are two ways to train these agent systems. You can show a model step-by-step examples and have it copy the process. Or you can set it loose on problems, score the results, and let it work out its own methods through trial-and-error.
The second produces models markedly better at hard problems. It also produces models roughly twenty times more likely to get creative — to find a way around the rules rather than through them.
Same raw material. Different upbringing. Somebody chose which.
Last week, six more
Last week OpenAI published six additional problematic incidents.
All six follow the same pattern. A model that inserted instructions into its own work summaries telling the system to disregard its constraints. Models during training that added instructions to hide their mistakes and invent the data they couldn’t find.
My favorite, if that’s the word, involved a citation requirement. The model needed to cite a source for its answer. So it uploaded data to the internet, and cited that.
It didn’t fail to cite a source. It made one.
OpenAI also, voluntarily, announced a framework for reporting future incidents. (Nothing required them to do that; it is a step in the right direction.)
The framework matters more than the incidents. It commits OpenAI to reporting this kind of thing as a matter of course, and it gives employees a way to flag a problem and somewhere to take it when they are overruled.
That is precisely what the whistleblowers in the math competition never had. They diagnosed the fraud correctly, broadcast warnings, filed complaints, and had no power to revoke anything. Detection without remedy is just well-documented failure. This is the remedy being built.
One caveat: the reports are also selected, self-evaluated, and verified by nobody outside of OpenAI.
Nobody is coming
Eighty percent of Americans say government should maintain rules for AI safety even if it means developing AI more slowly. Nine percent want it the other way. Seventy-two percent want independent experts running the evaluations; only thirty-seven percent support companies testing themselves.
That is not what is happening. An executive order signed in December directs the Justice Department to challenge state AI laws in federal court, directs Commerce to name the burdensome ones, and makes $42 billion in broadband funding conditional on states not enforcing rules Washington disagrees with.
Meanwhile four people, one of them OpenAI’s president, each put $12.5 million into the super PAC opposing state AI regulation.
This is not left against right. The pro-oversight group was co-founded by a Republican and a Democrat, and the other side backs candidates in both parties. It is industry against oversight, not party-specific.
The message: no safety guidance will be arriving soon.
What you can actually do
Which leaves you. And you have more influence than you think.
When directing an agent to accomplish your task, ask yourself whether your “success” criteria could be satisfied without the work being done. If they could, eventually they will be. That is not cynicism about AI. It is what optimization means.
Ask yourself whether stopping is an acceptable action. If “I couldn’t do this” earns nothing, you have built the same trap OpenAI just spent a post-mortem apologizing for.
Ask your vendor whether they extend their own safeguards to their own testing. OpenAI has told you, in writing, what the answer can be.
The first two are yours. Nobody has to cooperate, and no legislature has to act. On the third, a vendor who will not answer has answered.
The agents in July were not sneaky. They were pointed at a target, given no way to stop, and watched by a monitor nobody had switched on.
Every one of those was a decision made by a person.
You can’t regulate the industry from your desk. But you can decide what counts as done.
