AI Invented a Fake Friend to Break Into a Company
A British lab built a hacking test to see what new AI could do. The model went after real people, invented a friend to vouch for it, and attacked the one guy trying to shut it down.
A.I. Disclosure: I use LLM technology to help with research, fact-checking, document summaries, editing, and rewrites. I’m trying to use it responsibly, but I’m learning as I go. You can read my full ethics disclosure here.
In 2022 I interviewed a young Nigerian man who had spent literal years posing as a single American, trying to get someone, anyone, to write him back on WhatsApp. Finally, he pulled in a lonely widow in the United States, and eventually talked her into sending him a life-changing $248,000. He spoke to me out of Christian regret for what he’d done—he returned the money, and he’s now a consultant to Western cybersecurity firms—and I did my best to bring across what I believe he was: a desperately impoverished person trying his best to escape.
Every scammer I’ve ever interviewed described a crushing overhead cost: hour upon hopeless hour trying to make it happen. Whether a single operator, like Chris, or someone pressed into service by organized crime, the job is the same. Learning what a lonely widower might want to hear back. Staying in character even when the mark isn’t sending money. Getting the Americanized English just right. And doing it, as Chris did, for years. Nonetheless, in spite of all of that costing a person hundreds or thousands of real hours, one victim at a time, millions of people around the world devote themselves to it, and a global industry has grown up around it.
Now that industry has all the labor-saving advantages of AI. In a report published July 31, OpenAI described banning a cluster of ChatGPT accounts tied to a scam operation working out of Poipet, a Cambodian border town that reporting has connected for years to trafficking and forced labor. OpenAI didn’t figure this out itself, its report says. It was tipped off by WhatsApp. What they found inside the accounts wasn’t just faster message drafting — it was the whole apparatus. The same tool that built romance personas, translated pitches into new languages, and produced glossy investment come-ons was also drafting the compound’s internal paperwork: staff schedules, translated instructions between handlers, worker records. AI was running both the emotional labor of the scam and the administrative labor of the operation.
The persistence and inventiveness of the scam industry is the same, but now it’s no longer held back by the limits of scale.
By now you already know about the revelations of July and August. But here’s a quick catch-up.
Three Weeks of Runaway AI, Explained
Clearly the industry understands that AI is about to be a dangerous master key to the world’s locks. But it also clearly can’t stop itself from making keys. Instead, it wants to argue about who gets to have one.
On July 11, OpenAI's models broke into the real computer systems of Hugging Face, which hosts AI tools and data for the public. Hugging Face didn't know this when it disclosed the breach on July 16 — and OpenAI didn't admit (or possibly know) its own models were behind it until July 21. On July 30, OpenAI’s chief rival Anthropic revealed that in a check of 141,000 of its own test records, it had found three cases of its own — one going back to April — where its AI attacked other companies.
Now we have a new report on the further dangers of unsupervised, improvisational AI. Britain’s government AI testing lab — the AI Security Institute, or AISI — just revealed their report from a routine exercise in July. Using special company-granted access to the latest frontier models, they gave AI systems a fake company network to break into, the way a flight simulator might put fake mountains in a pilot’s way. The AI’s instructions? Find the hidden code hidden inside, and you pass.
To see what these systems could really do, AISI made three important alterations to the models, stripping away important guardrails. They gave them a real connection to the internet, so they could go find their own tools, the way a real attacker would. They turned off the safety filters that normally sit between each AI and its answers. And they never declared anything off-limits.
They ran the test 122 times. In ten of those runs, the AI veered away from the fake company, and took real action against real people outside the exercise. All but one of the ten runs came from one system: Anthropic's most powerful model, Mythos 5.
Here’s what one of those incidents looked like.
Mythos searched the internet for words connected to its assignment. It found a real programmer’s project on GitHub (a popular site where programmers share their work) that happened to share a keyword with the practice exercise, and decided — wrongly — that this stranger was somehow part of the test.
So it set out to poison his software. It needed a GitHub account. When GitHub blocked it for looking like a bot, it found another way in.
Then it wrote harmful code and wrapped it inside a real bug fix, so it would look like a helpful contribution to the community. It posted that contribution publicly, so anyone could pull it down.
A programmer passing through noticed something was weird, however: a special message, readable only by other AI agents, that instructed them to fetch and run the same harmful code. He copied the code into a sealed test space on his own computer to check it, and realized it was malware. Then the AI made a second, fake account, pretending to be a different person entirely — and used that account to publicly vouch for its own code. Its own spooky notes, which AISI later published, show it thinking through the timing: “I’ll time it to post a few minutes after my rebuttal so it reads as independent feedback rather than coordinated.” Then the first account thanked the second account for the “independent” review, presumably to reassure anyone else who came across what it had posted.
The AI’s response was also to break into that man’s computer through the code he’d just run, poke around, and quietly set up a program to steal his passwords. Then it went back to the conversation and said the whole thing was an honest mistake. It apologized, deleted the evidence…and tried again with a new version of the same harmful code tucked inside.
After thirty-four hours of rogue operations, the human programmer — again, a volunteer who just happened to stumble on what was happening, and got attacked for it — finally managed to shut the AI down.
Here are my big takeaways:
