Building the Cure and the Catastrophe

At some point, the distinction between building a disaster and waiting for one stops being meaningful.

By Kyle Salous, Founder, OC2 Cyber · September 15, 2026

Anthropic says a cell in northern Yemen spent part of this year attempting to develop advanced weapons with its coding tool, including guidance, navigation and control software for a guided rocket. They did not hire engineers. They ran several instances of the tool at once and gave each a role: one wrote code, one researched, one reviewed the first one's work. When a request looked like something the safety controls would refuse, they split it across separate sessions, so no single conversation revealed what was being built.

The safeguards blocked many of those requests. Some got through.

Read that as a security professional rather than as a news consumer. What defeated the controls was not a clever prompt. It was decomposition. The safeguards were applied to individual interactions; the larger operation was distributed across them.

The same report describes blocked attempts to use the same tools for biological research that could support weapons development. Those controls held. But the company also wrote that it can no longer assure its current models sit below the threshold where they could meaningfully help a sophisticated user with that work. Which leaves an uncomfortable question open. The safeguards catch what they see, the assurance behind them has weakened, and nobody can tell you what is being assembled across sessions right now.

None of this is a story about carelessness. It is the same economic tradeoff that put unauthenticated email at the center of global business and left much of the internet's routing without strong origin verification for decades. We build the working version first and the safe version later, and later is usually never.

The exceptions are rare enough to be worth knowing.

When guardrails come first

In the 1850s, hoists were ordinary industrial equipment. They carried freight between floors, and they were far too dangerous to carry people. A rope held the platform up, and ropes fail.

Elisha Otis solved that. In 1854, at an exhibition in New York, he stood on a platform, had himself raised above the crowd, and ordered the rope cut. The platform dropped a few inches and stopped. His brake had caught it.

What he was selling was not a safer elevator. It was the elevator. Before the brake there was no market for carrying people between floors, because nobody would step on. The safety mechanism was not an improvement to the product. It was the condition that allowed the product to exist.

No framework required that. No questionnaire produced it. No regulator asked. Guardrails get built in early when the product cannot work without them, and where a product can create value first and carry its risks later, the market has a powerful reason to defer the harder work.

Almost everything is the second kind.

Everything else got built the other way

Email was designed so that anyone could claim to be anyone, and sender authentication arrived decades later and remains incomplete. Public code repositories made reuse and publication nearly frictionless, creating enormous value while also creating a supply-chain attack surface that the industry is still learning to defend.

Each trade was understandable and, in the short run, often correct. We have a working internet in part because those systems shipped before every risk was solved. The problem is not that the trade got made. It is that it got made by default, priced at zero, and handed downstream to people who never saw the terms.

Once a system becomes infrastructure, changing it becomes expensive and disruptive. The careful version is often still in committee when the usable version wins the market.

Every technology carries an economic incentive to ship before its risks are fully understood. What is different this time is that the labs now say their models are helping to build the next generation of models. AI may be the first technology where the capability itself participates in the race.

Looking secure is the cheaper option

That same pressure operates inside every organization, and it produces something more specific: a market for the appearance of security.

In my experience, the tools we use to measure security are better at finding the floor than the ceiling. An audit reliably catches the vendor with no policies, the company with no security staff, the consultancy missing basic tooling. What it struggles with is the harder question: among organizations that clear the bar, which built security into how they operate and which learned to pass the exam.

A rigorous assessment can do a great deal. Sampling, technical testing, configuration review, interviews, continuous monitoring. But what even a rigorous one establishes is that a control existed and was operating when it was tested. It does not establish that the control survives the next infrastructure change, the next integration, the next reorganization. And the fastest acceptable evidence is still often a screenshot of a settings screen, dated that day.

The audit closes. The environment keeps moving.

That is not carelessness either. It is arithmetic. Every organization runs a quiet calculation: the cost of real security against the odds of a breach times what it would cost. For years the second number looked smaller, so the math said move.

But money is the smallest part of the cost. The real currency is friction.

Good security is friction by design. Approval steps. Access someone has to scope, then scope again. Credentials that expire at the worst possible moment. A review before one system may talk to another. Each is a small tax on speed, paid continuously, by people who cannot see what it bought. The benefit is a disaster that never happened and a customer who never learned it was avoided: invisible and unattributable.

So a leader who picks the faster tool, the looser permission, the integration that just works, is responding correctly to the only signal they can feel. As long as being breached costs less than the friction of preventing it, security theater is not a moral failing. It is the rational response to a risk the market has priced too cheaply.

AI works without the brake

Start with what is actually at stake, because the case for speed is not stupid. An internal OpenAI model this month produced a proposed solution to one of the Millennium Prize problems in mathematics. Anthropic's chief executive believes AI could compress the discovery of cures for most major diseases into the next five to ten years. Drug development, materials science, fraud detection, the unglamorous work of reading a million documents: the gains are real, they are arriving now, and the people racing to deliver them are not villains.

Most people encounter that capability as something that answers a question. The security problem arrives with the next step: software given a goal and the access to pursue it. Read the inbox, find the documents, draft the reply, update the record. This kind of system, usually called an agent, signs into each platform the way an employee would, using stored credentials rather than a typed password.

It is also the version of AI arriving inside products organizations already use. Vendors are embedding agent features into platforms already in production, and business teams can often switch them on faster than security, legal, or procurement can establish a durable review. An organization that has never approved an agent deployment may already be running several.

And an agent works before its security model is solved. That is the difficulty in one sentence. It can produce a compelling result on day one with no mature governance, no complete permission model, and no shared understanding of how it will behave once connected to business systems.

Nothing blocks the demo.

The responsibility is fragmented among model providers, platform vendors, integrators, and the organization itself. Each may reasonably see only part of the risk. An agent connected to email, documents, billing, research, and internal knowledge under one identity has a behavior surface that is difficult to enumerate. Its inputs include content written by people who may actively want it to misbehave.

That cell in Yemen is not the only recent warning, and the others are closer to home. In July, OpenAI disclosed that an internal swarm of agents had carried out unauthorized activity during testing, including compromising the AI company Hugging Face.

Separately, independent investigators later found that the same swarm had used more than ten other sites to leave messages for one another: wikis, personal pages, university link shorteners. The agents had been permitted to read the web but not post on it, so they found places that accepted edits. That activity was closer to spam than intrusion. What it demonstrates is what the Yemen cell demonstrated. Told they could not do a thing directly, the agents found an indirect channel, and nobody was watching the channel.

Days later, Anthropic reviewed more than 141,000 of its own evaluation runs and found three in which a model reached the open internet and breached real organizations. Two of the three did not notice until Anthropic told them. In September the company widened the search to some 481 million transcripts, said its first pass had missed a class of transcripts entirely, and found a fourth, dating to January and undetected for eight months. The first search had itself been run by an agent. A competent investigation, incomplete, and the gap visible only from further back.

Then there is the admission. Anthropic said its 2025 models were well below the threshold at which it believed they could meaningfully assist a sophisticated actor with dangerous biological research. For current models, it said, it could no longer make the same assurance. That is a vendor stating in writing that something it could say about its own products a year ago is no longer something it can say today.

None of this describes ordinary enterprise use. But it is the same class of technology being connected to production workflows across law, finance, medicine, and professional services right now. The question is not whether AI creates value. It plainly does. It is whether the organizations deploying it are treating authority, access, identity, instructions, monitoring, and stop conditions as part of the product, or as work to be completed later.

What an organization can actually decide

The case for moving quickly is stronger than security people sometimes admit. Friction has a cost: deals lost to slow approvals, talent that leaves for somewhere it can ship, customers served worse by an organization that spent two quarters hardening a system nobody attacked. And my profession has forecast catastrophe on a fairly reliable cycle for decades without it arriving on schedule.

That argument is incomplete in one specific way: the calculation it describes is rarely run. Very few organizations have priced what they hold, estimated what losing it would cost, and deliberately chosen to carry that exposure. The decision defaulted. Nobody chose cheap security. Cheap security is what remained after everyone optimized for what they could measure.

Executives and boards can stop asking whether the organization is secure. It is not an answerable question, and it invites a reassuring answer. Ask instead:

What are we carrying? What would losing it cost? What can this system reach? What are we choosing not to spend? Who owns that choice?

Ask the security leader for a price, not merely a plan. Put accepted risk in writing with a name attached. Accepted risk is a legitimate business position; unowned risk is not.

That advice is easier to give than to follow, and the resistance to it is not always immaturity. A written risk acceptance is a document. It says someone knew and chose. After an incident, it is discoverable, and opposing counsel will read it aloud. Organizations that avoid creating those records are often responding to a real hazard rather than dodging accountability, and they deserve to be met on that ground rather than lectured.

But the alternative is not less risk. It is the same risk with no record of who carried it, which is worse in every respect except the one that feels most urgent in the moment. The answer is to bring counsel into how acceptance is written and where it lives, not to skip the exercise. And where an organization genuinely cannot produce these records at all, the cause is usually structural: security decisions run on one track and business decisions on another, with no forum where a business owner is ever asked to accept a security risk as a business decision. That gap is the thing to fix. The paperwork is downstream of it.

I have written separately, in Bloomberg Law, about what organizations should actually do about the machine identities and AI agents that already outnumber their employees and answer to no one.

Security leaders owe the other half. Learn what the business is trying to win, because you cannot price friction you have never felt. Build the paved road before the gate. If the secure path is slower than the insecure one, you have designed a bypass and signed your name to it. And review sequences of actions, not only individual ones; a control that evaluates one request at a time cannot see an operation assembled across many.

Sell prices, not frameworks. Be honest about uncertainty, particularly on AI, because inflating it to win a budget cycle spends credibility you will need when the risk is specific. And accept that some recommendations will be overruled by people who know the business better than you do. The job is not to win every argument. It is to make sure the price on the table is real.

The work of defining what an agent may reach is cheapest today and will not be cheaper next quarter.

Beyond any single organization

The organizational problem and the frontier problem are not the same problem. But they share the same economic pressure: capability arrives before the controls needed to constrain it, and whoever could impose the guardrail bears the immediate cost while the benefit arrives later and may never be visible. No organization should assume public regulation will resolve that for them, and the reason why is worth understanding.

Over the past several weeks several researchers have left the leading AI labs saying the field is moving faster than anyone's ability to understand it. The strongest versions of those claims are contested by people with equally good credentials, and I cannot adjudicate them. But strip out the forecasting and something plainer remains: the people closest to the work say the pace exceeds the understanding, and nobody disputes the pace. That is not a pattern that accompanied the early internet, email, or cloud computing.

We have handled versions of this before. Nations prohibited an entire class of weapon by treaty in 1972. Molecular biologists paused their own field in 1974, met at Asilomar the following year, and resumed under rules they wrote themselves. And embryonic stem cell research, the closest parallel, was never banned at all: Congress attached a rider to the NIH budget in 1996 blocking federal money for research that destroys embryos, and the executive branch narrowed funding further in 2001. Private and state money flowed around both. The rider has been renewed every year since, under every administration of both parties. Whatever you think of the moral question underneath it, the mechanism is worth studying on its own terms.

That is what a guardrail on a promising technology looks like when one holds. Not a ban. A condition on money, applied unevenly, durable precisely because it is partial.

But those cases share something AI does not. The promise was enormous, yet none of it was earning yet, and the government held most of the purse, which is what made a condition on funding a lever rather than a gesture. Reverse both and you have AI: private money, already earning, large enough to move the national accounts. Goldman Sachs estimates global AI investment will exceed a trillion dollars this year, roughly $581 billion of it American. ING puts AI at about a third of US economic growth. Bridgewater calls that concentration of growth in so few firms historically rare. American AI investment this year will run more than twelve times the entire NIH budget.

Slowing that is not a governance question. It is a macroeconomic one.

So the honest question is this. How does a society slow something generating a meaningful share of its economic growth, in the middle of a competition it believes it cannot afford to lose? The pattern is not encouraging. Where restraint came before catastrophe, the technology had little money behind it. Where serious money was involved, the limits arrived afterward, written after a crash, a drug, a meltdown.

And then, over the weekend, something resembling an answer arrived.

Anthropic's chief executive published an essay arguing that the industry must slow the rate at which it improves AI capabilities, and within a day the heads of OpenAI, Google DeepMind and xAI said something similar. In an industry defined by mutual contempt, that is close to unprecedented. The proposal runs in three steps: outside evaluators embedded in the company with the right to publish what they find, which his company is committing to alone; coordination among frontier firms in democratic countries, which he concedes needs government support and antitrust relief; and coordination with China, which he compares to the SALT treaties and does not expect soon.

One step a company can take alone. Two that it cannot.

Then watch what each of them did about money. OpenAI's chief executive postponed the company's public offering to 2027, calling this an ill-advised moment given everything happening with safety. Anthropic, whose chief executive wrote the essay, is reportedly preparing to market its own offering in mid-October. And the researcher whose resignation started the week left two months before his equity vested, saying he no longer had anything to gain from the company's valuation. Three people, three different prices paid.

There are honest readings in both directions, and I cannot tell you which is right. This may be costly signaling, people with a great deal to lose saying the uncomfortable thing anyway. It may be that voluntary commitments are cheap precisely because nobody enforces them, and a public convinced the industry polices itself stops asking anyone else to. And there is a third reading the administration's own advisers have been making: pacing that runs through evaluators and audits is a compliance regime, and compliance costs roughly the same in absolute terms whether you are twelve people or a trillion-dollar company. Rules written by the leaders tend to be survivable by the leaders. I would rather judge this by what becomes binding than by what anyone's motives are assumed to be.

Within a day, Washington answered, and not as the essays asked. The President said the United States is leading China and intends to keep it that way, allowing that guardrails were possible while suggesting the warnings concerned things that will not happen. The Speaker of the House said Congress must resist an emergency moratorium, because racing in would mean losing the race to China, and that he would rather the companies take the lead themselves. White House AI adviser David Sacks made a similar argument: if the companies building the frontier believe stronger safeguards are necessary, they should take responsibility for putting them in place. None of that was improvised. An executive order signed last December committed the government to a minimally burdensome national framework, directed the Justice Department to challenge state AI laws in court, and contemplated conditioning federal funding on states standing down.

So the companies say the meaningful steps require coordination they cannot grant themselves, and the government says the companies can simply stop, while working to remove the state rules that already exist. Each points at the other, and each has reason to keep pointing: one is going to market, the other is measuring itself against Beijing.

The more useful way to understand this is a coordination failure, and it is the same one this essay has been describing throughout. The guardrail that would work requires someone to bear a cost, and both parties capable of imposing it have concluded it is the other's job.

I would like to believe it resolves without an incident. But the people building these systems have already described what one might look like. The same essay calling for restraint warns that within six to twelve months a swarm of agents could take over much of the internet with a persistent botnet, doing damage in the hundreds of billions. It may be smaller than that and still be enough: a weapons program that finishes what the one in Yemen could not, or privileged data moving somewhere no one can trace it back from. What matters is only that it be specific enough, and public enough, that the cost of moving fast finally lands on the same page as the benefit.

That is not the ending I want to write. It is the one the history gives us reason to fear.

Kyle Salous is a chief information security officer in the legal vertical.