Why Do AI Leaders Believe They're Doing the Right Thing?
With billions invested and runaway AI breaking out all over, it's worth understanding a powerful quirk of the mind, and why it seems to have the tech industry by the throat.
The rogue AI fallout continues, and it’s teaching us how AI folks think.
To catch you up: OpenAI announced last week that its creation had attacked beloved AI lab Hugging Face, 10 days after Hugging Face reported the incident to law enforcement and to the public. (Reuters reported that the company didn’t realize it was responsible for perhaps a week.) This week it turned out OpenAI’s agents attacked other companies, too. Yesterday Anthropic announced that, prompted by the OpenAI episode, it looked back and figured out that in three instances, its experimental creations also broke out and attacked other companies. And today, according to Reuters, OpenAI is discovering that hey, hold on, wait a minute…our AI has probably been responsible for other attacks, too.
I spent yesterday looking at whether this stuff crosses OpenAI’s own red lines (the answer: probably) — and I spent some ink on what I consider the real issue at hand here, best summed up by a new petition from AI’s top people. The signatories include not just the chief scientists of companies like Google and Meta, it also includes OpenAI and Anthropic, who signed on as whole companies. It’s a startling read, because its spirit is basically “stop us before we do this again.” It begs the federal government to slow AI development down — please come down here and rein in us kids, recess is out of control — because “intense competitive pressure” means the companies more or less cannot help but plow ahead in spite of the obvious dangers.
That should have been the headline, it seems to me (and, in fairness, it did get some good coverage). But most of the publicity went to a more industry-friendly question: should AI models be open-weight or closed? Should you be able to download a model to your computer, tweak it as you like, and do whatever you want with it? Or should it be tightly controlled by the company who makes it, and rented to you under strict terms of service?
Jensen Huang, CEO of AI chip company Nvidia, used his first-ever post on X to publish a joint letter with two dozen other companies extolling the national-security, human-safety, and speedy-innovation virtues of open-weight models. The letter reads, in part:
Startups, established businesses, universities, and public institutions can build on advanced models without training one from scratch or paying frontier-model prices for every task. Open weights let every organization match the right model to the right job at the right cost, reserving frontier-scale capability for genuine frontier problems and running efficient, specialized models everywhere else.
-”Open Weights and AI Leadership”, July 24, 2026
Okay, sure, makes sense. As the letter points out, the open-source software revolution of the 1980s and 1990s now undergirds most modern technology. But it’s also important to understand that Huang’s support for open-weight models isn’t just a philosophical position. It’s a pretty handy one for his business, too. Because in an open-weight world, the “startups, established businesses, universities, and public institutions” the letter describes would all be new potential Nvidia customers.
Three Weeks of Runaway AI, Explained
On July 11, engineers at a New York company called Hugging Face — a kind of shared online lab space for AI developers — noticed something bad moving through their systems. It was looking for something. It was fast. It had credentials, and it used them to attempt tens of thousands of actions, some brilliant, some nonsense.
Anthropic’s Dario Amodei, meanwhile, has come out against the idea that AI should be broadly open-weight, because, he argues, it creates the opportunity to use AI to build, say, bioweapons far faster than we could put up an AI-generated defense against them. Also a reasonable argument — and he argues for standards and safety regulations around this stuff that I believe are obviously necessary. But this position — that closed AI models are safer for the world, and should thus be the norm — also dovetails conveniently with his closed-model company’s rumored IPO, which could arrive this fall, or sooner.
When I was first playing with AI as a research tool, I found that it was very easy to fall into the trap of prompting it to go out and find evidence to support my opinions. (These days I’ve learned to ask it for research in open-ended ways, rather than telling it my biases in advance. Does that solve the problem? Fucked if I know. Here’s my ethics statement.) I feel like that’s what we’re seeing with all the big names speaking out for or against open-weight models. Here’s a cheat sheet on them, and then I want to dig into a well-established but little-publicized psychological phenomenon that I think may explain what’s happening to me, and to them.
Musk v Altman: How Can Billionaires Claim Money Doesn't Motivate Them?
The essence of being a good trial attorney is the willingness — and I wish journalists had more of this — to awkwardly, almost psychopathically repeat a line of inquiry. And on Monday, Elon Musk’s attorney Steven Molo asked some version of the same question more than a dozen times in two hours. He varied the phrasing. He raised his voice once. He badgered, circled back, and let it hang.
The Signatories
Nvidia sells the hardware. Closed models concentrate chip buying among a handful of customers with enormous leverage over their supplier. Open weights turn every government, university, hospital system and mid-sized company on earth into a potential buyer. The letter’s stated principles — diffusion, sovereignty, security — are coherent, absolutely, and they also describe the most favorable possible market structure for Nvidia.
Meta gave away its open AI model Llama for years, fell behind, spent $14.3 billion on ScaleAI, hired Alexandr Wang, built its new, fanciest model as a closed model, and started selling access to it this month. It now has one of each. Industry support for open weights helps Meta no matter what Meta ships, and hurts labs whose only product is rented access. Mark Zuckerberg recently criticized closed models in the New York Times, warning that they would centralize power in a dangerous way. But note where Meta has never argued for openness: nobody can download Instagram’s ranking algorithm. That model isn’t open-weight.
OpenAI signed on to Huang’s letter, days after it admitted/discovered that its own models did the thing that make openness look necessary. It sells closed frontier access and also publishes open-weight models, so signing costs nothing. It also puts distance between OpenAI and its rival Anthropic, the company standing publicly alone against open-weight, and currently absorbing the industry’s hostility. That’s helpful at a moment when OpenAI has an uncomfortable question of its own to answer.
Anthropic sells only rented access to models you can’t download. A universal safety-testing mandate like CEO Dario Amodei is arguing for raises the cost of shipping a frontier model — a cost Anthropic already pays and its cheapest competitors don’t. Amodei almost certainly believes what he’s saying; he’s been saying it since before it was profitable. But that can be true and still serve his business interests.
Palantir doesn’t train a frontier model. It sells the layer that puts models to work inside a customer’s walls — AIP, Foundry, Ontology, Apollo — and its deepest customers are U.S. intelligence, defense, and critical-infrastructure agencies operating behind classification barriers. A closed frontier model rented from OpenAI or Anthropic reaches those customers as an API call to someone else’s cloud, which is the one thing a classified, air-gapped network is built to forbid. Three weeks before it signed Huang’s letter, Palantir announced a joint engine with Nvidia to run Nvidia’s open-weight Nemotron models inside exactly those environments, letting agencies fine-tune on their own data without the data ever leaving the perimeter. Open weights aren’t only a national-security position for Palantir. They also describe the only models it can run in the rooms where its business lives.
Microsoft owns roughly 27 percent of OpenAI, a stake it valued near $135 billion after last fall’s recapitalization, and OpenAI is contracted to spend $250 billion running on Microsoft’s Azure cloud. It is the largest financial backer of the closed-model company on earth, so what financial motive does it have to sign a letter for open weights? Azure now hosts an open-model catalog thousands of models deep, and every query Microsoft routes to a downloadable model on its own cloud is a query it doesn’t pay OpenAI or Anthropic to answer. In June it announced its own in-house models for the same reason. Microsoft rents the ground the models run on. It collects whether the model is open or closed, and open happens to cost it less.
Andreessen Horowitz manages one of the largest venture books in technology and has built a standing policy operation in Washington around a single phrase — “Little Tech” — under which it argues, in papers with subtle titles like “Asserting American Leadership in Open Source AI,” that downloadable models are what stop power from flowing to a few big labs. The firm led Mistral’s first round and holds positions across a portfolio of open-model and app-layer startups whose margins depend on cheap, swappable models underneath them. It also runs a program called Oxygen that hands those same companies access to more than 20,000 Nvidia GPUs in exchange for equity. Open weights lower the price of the one input every a16z company has to buy, and keep the closed labs from owning the stack the firm has bet on. The principle protects little tech, sure. It also protects the book.
Dell sells the boxes. Its “AI Factory with Nvidia” packages the hardware to run models on a company’s own floor, and every enterprise that downloads a model instead of renting one becomes a Dell customer — the Jensen Huang argument, one rung down the stack.
IBM ships its own open Granite models and sells watsonx and the consultants to install them. Open weights mean enterprise deployments on the client’s premises and billable hours; the service call on the open-weight model you’ve deployed at your office is their business.
Perplexity builds its search product on models it doesn’t own and competes with the closed labs whose APIs it would otherwise depend on. Open weights are what keep it from paying rent to the companies it’s trying to beat.
Y Combinator funds thousands of startups that can’t absorb frontier API bills. Cheap, downloadable models are portfolio life support — the same venture logic as a16z, spread across a far wider bet.
Mistral is a French lab already in the coalition line. Open weights are its whole product and its sovereignty pitch, and it’s backed by a16z and Nvidia — its signature closes a loop with two other names on the letter.
Mariana Minerals is an oddball on this petition. It’s a mining company. I have no idea what it’s doing here. It has no clear stake in whether models are open or closed; its presence mostly suggests how wide Nvidia cast the net.
Do these folks all believe, in their heart of hearts, that open-weight models are the only philosophically defensible option? Or are they just taking the position that best serves their businesses? Well, social psychology teaches us that both things can be true. Turns out that tension has a name, and that it explains a lot about our modern moment.
