The Trap Closes on Anthropic
Dario Amodei spent January warning the world about AI companies with too much power. He spent February learning what happens when you try to limit your own.
For years, I’ve interviewed tech executives about the ethically dubious uses to which their creations could be put, and I’ve tended to get the same response: it’s not our job to head that off.
Take an interview I conducted with Anduril CEO Palmer Luckey a few years ago, as his company was building Lattice — a system that combined drone technology and facial recognition into a package designed to find and follow individuals. Built for the border, it could in theory be dropped into any city and produce a total surveillance system from nothing.
I asked him whether building new technology meant you also had a responsibility to build the ethics that go with it. That’s not his role, he told me. “I’m more of a technology guy than a policy guy.” When it comes to ethics, he said, “I think the way is not through technological control. It’s to have the political wherewithal to say, ‘Yes, we know the tech exists to do this. But you can’t. We’re not going to accept that as a society.’”
Luckey was operating under an assumption that, at the time, was reasonable: that when government got involved with technology, more restraint would follow. Democracy would supply the ethics that Silicon Valley declined to. He could pull hard toward capability, knowing the system would hold.
That assumption is gone. And its absence is exactly what has put Anthropic CEO Dario Amodei in the position he’s in this week.
Amodei is different from your typical tech CEO, and for him, that’s the problem. He is the rare tech leader who actually accepted the responsibility that Luckey and so many others refused. Last month, Anthropic published an 80-page constitution — its “model spec” — outlining what its AI is supposed to value and how it should conduct itself. Earlier this year, Amodei published a 20,000-word personal essay warning about what happens when AI companies accumulate too much power. He has been, for months, the most articulate critic of his own industry. He named the trap. And now he’s in it.
Defense Secretary Pete Hegseth has summoned Amodei to the Pentagon for what sources describe as a tense meeting over the terms for military use of Claude. “This is not a get-to-know-you meeting,” a senior Defense official told Axios. “This is a sh*t-or-get-off-the-pot meeting.” (Axios)
This is what the trap looks like when it closes.
The Pentagon’s specific demand: drop Anthropic’s internal restrictions on autonomous weapons and the mass surveillance of Americans. CNBC These are not radical safety positions. A generation ago, you couldn’t have hired a twenty-year-old into a defense contractor without this kind of commitment already in place. They are the bare minimum of what a company operating in a democracy might be expected to require of itself. The Pentagon is now treating these ethical standards as obstruction. “The problem with Dario is, with him, it’s ideological,” a senior Defense official told Axios. They meant it as a criticism. I consider it the most generous thing anyone could say about him.
Anthropic found out that its AI had been used in the operation that captured Venezuelan President Nicolás Maduro the same way the rest of us did: after the fact. (Bloomberg) The company didn’t know. The safeguards didn’t stop the deployment. They just generated customer-relationship friction afterward. That is the gap between the constitution and the contract. That is what the documents are actually worth when the economics get serious.
And here’s what that could cost Amodei.
