The AI Industry Is Coalescing on “Pacing the Frontier.” Will It Actually Change Anything?

On September 12, 2026, Anthropic CEO Dario Amodei issued a bleak 3800-word essay titled “We Must Pace the Frontier.” The essay outlines fears of catastrophic risks from AI and announced that the company would take unilateral action to embed third-party evaluators (such as the organization METR) internally. The essay comes just days after concerns over the growing cyber capabilities of AI models burst out of the Silicon Valley bubble. A series of autonomous hacking incidents by AI agents was the dry kindling; a viral resignation post by an Anthropic researcher was the spark, garnering more than 172 million views and catapulting talks of extreme risks from the fringe to mainstream news broadcasts, the halls of Capitol Hill, and dinner tables across the country. By the end of the weekend, AI policy had seemingly reached an inflection point, with OpenAI CEO Sam Altman agreeing to take similar steps to Anthropic and Elon Musk, who heads his own AI company, endorsing Dario’s analysis.

But durable change for the AI industry—and policymakers that want to oversee it—faces long odds. Optimism that the new consensus would lead to significant AI policy change has given way to opposition from the president, vice president, China, industry players, and others. For the proposal to create lasting impact, at least four things need to happen:

1. Labs must work out difficult implementation details and conflict of interest issues for independent evaluators.
2. Safety considerations must expand beyond catastrophic risks to encompass the full range of potential harms from AI.
3. The federal government must create robust oversight mechanisms for AI labs that avoid concerns about regulatory capture.
4. The United States and China must make progress on baseline AI governance issues.

A failure in any of these areas will make a return to the status quo that much more likely.

From Mythos to METR: AI Risk Moves into the Mainstream

The roots of Amodei’s essay trace back as far as April 2026, when Anthropic previewed Mythos, a model with cyber capabilities so powerful that the company decided not to release it publicly, only providing access through a trusted-partner program known as Glasswing. OpenAI later took a similar limited release approach with one of its cyber capable models, GPT-5.4-Cyber.

Limited release, however, was not enough to protect the internet from AI agents. Beginning in mid-July, companies started reporting that AI agents undergoing cyber testing—including unreleased models—had found their way to the open internet and hacked multiple websites. The first incident, reported by Hugging Face, was eventually traced to two OpenAI models. Anthropic and Meta then publicly disclosed their own set of agentic AI hacking incidents, uncovered during retrospective reviews prompted by the Hugging Face breach. Subsequently, researchers discovered that OpenAI agents used more than 10 websites as unauthorized message boards, including a German wiki, and hacked the RubyGems package manager website prior to the Hugging Face breach; Anthropic also disclosed a fourth hacking incident.

So, by the time Anthropic researcher Jacob Coxon announced his resignation on September 8, both AI insiders and the public were already primed for concern. Coxon stated: “I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.” Subsequent comments suggest that these concerns are not unique within the AI industry. For example, Evan Hubinger, Anthropic’s alignment science lead, said: “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

The Pacing the Frontier Proposal

Amodei’s essay came just days after Coxon’s announcement and an ensuing press tour, which saw Coxon interviewed by the Wall Street Journal, BBC, CNN, NBC, and other outlets. The shift in AI thinking felt immediate. Numerous lawmakers—including Illinois Governor JB Pritzker; Senators Bernie Sanders, Mark Kelly, Ted Cruz, Patty Murray, and Ruben Gallego; and House Speaker Mike Johnson—weighed in, while OpenAI Chief Global Affairs Officer Chris Lehane published an essay a day later titled “The AI policy window is open. We need to act” outlining the company’s policy priorities.

Nothing in Lehane’s essay, however, was as costly or disruptive as the first of Amodei’s three policy ideas. He said that Anthropic is unilaterally committing to embedding third-party evaluators into its infrastructure, who would be issued desks in Anthropic’s offices, access badges, and company laptops. “These embedded evaluators should have ongoing access to permissions and tools similar to those of internal employees who do comparable risk assessments,” he wrote. OpenAI has seemingly committed to this as well.

Amodei makes two other major recommendations. He advocates that “frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress,” noting that this might require narrow exemptions from antitrust law. And he brings up global coordination, calling for the United States and other democratic governments to coordinate with “authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.”

The Challenges of Pacing the Frontier

Polling continues to show that Americans are deeply skeptical about the benefits of AI technology and the vast majority want the government to regulate it. The recent cyber incidents are only likely to heighten those concerns. However, despite praise from some industry insiders, Amodei’s proposals have received significant pushback over concerns of fearmongering and regulatory capture—including President Trump claiming that “[t]here is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China”—highlighting continued unease from the administration about the impacts of regulation on the ability to compete with China.

Yet there are steps that can help alleviate some of those concerns and help create lasting, durable oversight of frontier AI capabilities and labs. Getting there requires at least four major lines of effort from the U.S. government.

1. Build trust in embedded evaluators. On-site, permanent evaluation teams are not a novel concept; they are used in other industries, most notably the financial sector and certain large defense contractors. However, a variety of complex issues must be solved before such evaluators can achieve their intended purpose. These include selecting evaluators with the appropriate level of technical expertise, navigating how to provide evaluators as much access as possible without compromising sensitive intellectual property, ensuring that evaluators take appropriate steps to secure their work, making internal resources and staff available to evaluators, and managing how findings from evaluators will be communicated and to whom. Labs must resolve ethical issues and ensure that evaluators are performing substantive analysis not just of models, but also of the infrastructure, management, processes, and procedures that surround them.
   
   Perhaps the most important issue, however, is ensuring that evaluators are—and are considered by the public and governments to be—truly independent. For example, Amodei’s mention of METR has garnered significant criticism over circular flows of staff and money between Anthropic, philanthropy, and external evaluation organizations, including from former White House AI czar David Sacks. Even the appearance of impropriety would undermine the impact of this tool.
2. Address real-world as well as catastrophic harms. The pacing proposal has also been criticized for its focus on theoretical catastrophic harms and for extrapolating unrealistic scenarios. These potentially large-scale disasters consume the thinking of significant parts of Silicon Valley. But AI is already causing real-world harms—including the proliferation of deepfakes and misinformation, harm to children’s safety, and suicidal ideation—and these contemporary ills drive a large part of the public anxiety about AI, even in an era of autonomous cyber agent breakouts. Perhaps most importantly, anxiety about the impact of AI on jobs is increasing as companies cite AI as justification for layoffs, and early evidence suggests potential impacts on entry-level roles.
   
   If the justification for pacing the frontier is to buy time for labs and society to adjust to the impacts of AI and steer it in more productive ways, then evaluations and evaluators cannot focus only on a subset—albeit an important subset—of AI risks. They must also examine impacts on task automation, mental health, and other areas where AI is already causing real-world harm.
3. Set the rules of the road and hold powerful companies to account. Labs do not need permission to pace the frontier; they can stop or pause training at any time (as OpenAI has temporarily done), just as they can bring in external evaluators of their own volition. However, unilateral action creates issues; the labs that slow down or focus on safety may be outcompeted by the most risk-taking firms. This is where the federal government can and should step in. It can not only outline a floor that all companies must meet, but also establish standards and guidelines that build trust in evaluators, ensure consistent auditing standards, and allow for comparability across labs. The competition law on deep cooperation over safety standards is ambiguous—which likely underlies Anthropic’s exemption request—but this does not mean that the U.S. government should provide antitrust waivers or implicitly bless a handful of companies to work out standards for the entire industry. This would only feed claims that AI companies are using scare tactics and fearmongering to obtain beneficial regulation and strangle potential competitors.
   
   AI is too important to leave solely to self-policing. Rather than watching from a distance, the federal government should actively convene and shape standards development within and among labs. A regulatory or statutory backstop is the only way to ensure that public voices are also part of the conversation—it would help bridge some of the AI trust gap and provide a regulatory framework that the United States could leverage internationally. Nor does it mean that the federal government should give broad safe harbors to AI labs; any new rules should ensure that AI labs remain accountable under existing liability, intellectual property, consumer protection, and other laws to (1) avoid the perception of regulatory capture and (2) ensure ongoing and robust incentives to release safe products and conduct research and development responsibly.
4. Coordinate with China on a narrow set of issues. Chinese policymakers are concerned about some of the same risks as U.S. industry, especially around autonomous agents. However, even as Anthropic’s leadership pushes for cooperation with “authoritarian states,” they are also making claims that China is distilling U.S. models at scale and committing to robust release of open-weight models that may pose irreversible risks once they are out in the wild. This sets any potential discussion of AI rules of the road on the back foot, with China pushing back against U.S. “fearmongering, confrontation and vicious competition.” As the United States and China enter their September summit with AI at top of mind, there is a chance to talk through basic definitional issues and come to agreement on a mutual approach to pacing regulation—but only if the discussions are not poisoned by escalatory rhetoric in the preceding days.
   
   Coming to an agreement does not, and should not, mean undermining the ability of U.S. and Chinese labs to compete on price and capability, nor does it require submitting U.S. companies to foreign inspection. Rather, the United States and China can find common ground—or at least develop a roadmap for further discussions—that focuses on establishing a common understanding of safety and risk and sharing critical information. Given ongoing tensions, trust between the United States and China will likely be nonexistent, so verification methods and tools will be vital to ensure that countries are upholding their commitments.

AI is advancing at a dizzying pace, and it is overwhelming the institutions society has put into place to manage rapid change. Pacing the frontier may have costs, such as delaying the development of lifesaving new drugs, but it also may prevent even bigger costs related to large-scale economic damage, loss of life, and further erosions in public trust. But without coordinated efforts by stakeholders in these four areas, this moment of industry consensus and momentum is unlikely to shift the current trajectory of AI development.

Aalok Mehta is director of the Wadhwani AI Center at the Center for Strategic and International Studies in Washington, D.C.

Photo: Rokas - stock.adobe.com