Who gets to define AI safety?
It’s a strange thing to sell a technology while warning that it might destroy the world.
Over the last few months, I’ve watched the AI safety conversation spill out of my narrow corner of tech twitter and into the mainstream, arriving as warnings of impending doomsday. Those warnings increasingly depict AI as some omnipotent superintelligence beyond human influence. We spend so much time imagining what it might someday do on its own that the harm people are already enabling with it becomes an afterthought.
My frustration with doomers isn't that I think AI is entirely harmless or shouldn't be regulated at all. Quite the opposite. I actually do think there are serious risks that deserve consideration, but the doomer discourse flattens all nuance under the blanket statement “AI bad”, leaving people frightened without helping them understand what makes a particular use dangerous or what would prevent the harm. This framing of all AI being bad actively undermines that conversation.
The AI safety conversation I want is one that’s grounded in how these systems actually work, who controls them, and how we democratize access. Instead, we're getting apocalyptic narratives that distract from real accountability and potentially hand even more power to the companies we're supposed to be holding accountable. I don’t think their version of safety should shape what gets regulated, who gets to build AI, and which harms we’re willing to tolerate.
AI is made by humans
AI runs on systems people build. People decide what those systems can do and access.
An LLM is a neural network trained, initially, to predict the next token from the text preceding it. Further training shapes how it follows instructions and responds to feedback. An LLM’s ability to produce convincing language isn’t proof of consciousness. I find it deeply unsettling when companies encourage people to think of AI as conscious, and discussions of “AI welfare” and proposals for legal rights make me similarly uneasy, especially when it’s still unclear how we’ll hold the companies deploying these systems responsible. Corporations being treated like people under the law is bad enough; I’m wary of where extending that framing to software could lead.
There’s also a huge leap from model capabilities improving to their ability to act. Becoming more intelligent doesn’t automatically give a model access to weapons, money, or physical infrastructure. Any scenario in which AI acquires those things needs to explain how it gets there, including the decisions or failures that make it possible and which people and institutions are responsible.
Talking about “rogue agents” can obscure precisely those decisions. An agent may behave in ways its developers didn’t anticipate, but we still need to understand what allowed it to act and where it could have been stopped. Calling AI uncontrollable shouldn’t let the people building and deploying it off the hook.
The harm is already here
The AI safety conversation keeps returning to catastrophes that might happen someday as a result of AI gaining uncontrollable power. Meanwhile, the more urgent harm people are already enabling with these systems seems to attract far less attention. AI is already used for fraud and surveillance. The U.S. military has used AI to help plan and coordinate its bombing campaign in Iran. The Internet Watch Foundation has also documented AI being used to generate child sexual abuse material and manipulate images of real children. These safety problems deserve a response, not pontificating about a mass extinction event.
Whose values?
When a company talks about aligning AI with human values while supporting its use in military operations, I’m unsure whose values are doing the work? Where do the people on the receiving end of those operations fit into that definition of “safety”?
The language used seems to be about “aligning AI with human values.” I don’t know what that means. We all disagree about politics, religion, and morality. We don’t even agree on what counts as an acceptable level of risk. I am failing to see how calling a system “aligned” would resolve those disagreements or give a handful of companies the authority to decide whose values count.
Who benefits from the panic
Politicians across party lines seem to be repeating the industry’s most frightening claims without examining their assumptions, and much of the press keeps amplifying them. That leaves the public frightened and gives the companies making those claims more authority to prescribe the response. I find this troubling for three reasons.
- Fearmongering creates illiteracy. First, it leaves people less equipped to understand the technology. “AI bad” or “AI dangerous” tells us very little about how a system works, what makes a particular use dangerous, or where we could intervene. When the explanation begins and ends with an intelligence beyond human control, people are encouraged to defer to whoever claims to understand it. Existing technical illiteracy becomes an opportunity to cultivate dependence instead of an invitation to educate.
- Industry criticism collapses into “AI bad”. Second, it collapses legitimate criticism of the industry into opposition to the technology itself. I’m frustrated by how easily the concentration of wealth and power under capitalism gets treated as an indictment of scientific progress. AI could contribute to advances in medicine and make useful capabilities available to people who currently cannot afford them. Whether those benefits are widely shared or inaccessible depends on decisions about ownership, access, and public investment. Those are things we should be talking about and fighting over, not whether AI is inherently good or bad. Treating the inequalities of the current industry as inseparable from the technology concedes way too much to the companies that control it.
- The biggest labs get to write the rules. Third, this fear can help those same companies secure their position. If lawmakers accept the premise that AI poses an extraordinary danger without understanding how that danger arises, sweeping restrictions can start to look like the only responsible response. The largest labs then become both the source of the warnings and the experts consulted about what everyone else should be required to do.
That is where I worry about regulatory capture. If in the future world of AI regulation, staying compliant requires expensive audits or licenses that small teams cannot realistically keep up with, the biggest companies can absorb those costs while their competitors struggle to enter the market. For open source developers, requirements to control every downstream use or modification could make releasing a model prohibitively difficult. We could lose opportunities to study, adapt, and build on the technology independently, leaving more people dependent on a few proprietary providers.
If AI labs push for compliance requirements that smaller competitors can’t afford, are they protecting the public, or turning their economic advantage encoded as regulation?
AI safety deserves a real conversation
AI can cause devastating harm while doing exactly what people deployed it to do. It doesn’t have to become conscious, go “rogue” or escape human control. Human decisions enable that harm, whether they are deliberate or negligent, and the people making those decisions should be held accountable.
I’m not opposed to guardrails, but I am vehemently opposed to guardrails shaped by speculative fears and the commercial interests of the industry’s most powerful players. The people affected by these systems deserve a say in what safety means. The companies selling us AI shouldn’t get to define it for everyone else.