The AI Safety-Diplomacy Disconnect
The AI safety community has made an enormous push into policy circles over the past five years or so. Several organizations are singularly focused on preventing advanced AI from assisting political actors in coups, AI-generated novel bioweapons, international conflict, and artificial super intelligences that supplant human hands in authoring the world’s destiny. But despite unprecedented coordination of talent, funding, and research, the movement fails to generate the kind of policy shift that would reflect their own sense of urgency.
AI is showing signs of profound capabilities across domains, and there is no evidence that it is aligned. If we are truly on the eve of catastrophe, there will be plenty of blame to go around after the dust settles. Model developers created these AIs, investors enabled them, and markets bade them all do so. Governments, whom the AI safety movement implores to force a pause, actively encourage this. If officials won’t heed the call of alarm, what can all the ranks of AI policy professionals, evaluators, game theoreticians, forecasters, bloggers, and effective altruists do better?
Based on a few thousand hours of one humble practitioner’s experience with influential figures from China, the US, and many other countries: a few things.
- Stop underestimating the art of diplomacy.
- Stop overestimating the intuitive implications of evaluations.
- Focus more effort into the areas governments care about, and less into lost causes.
To the first point, many AI safety orgs and policy institutes have made efforts to improve international coordination on AI safety. Results have been mixed with some unforced errors. Dialogues have been very effective in sensitizing governments to AI topics and in promoting better communication between states. But on multiple occasions they also have taken the tradecraft of diplomacy far too lightly.
Startup culture is a good fit for mobilizing resources to get people working on important things. In the market, where you can move fast and break things, there is a lot of value in speed. Politics, especially international relations, doesn’t work that way. It takes time to build credibility and rapport. Bringing the counterparty towards an accord can put them at personal and professional risk and requires trust and credibility. This doesn’t come fast, and isn’t easily repaired once broken.
AI dialogues, especially US-China initiatives, have been incredible at gathering talented, culturally informed, and technically capable people together. Like many tech start ups, though, those people are almost all under 30 years old. They cannot realistically have acquired the social capital to replace their seniors in high level dialogues. One cannot “go founder mode” on relationships.
My view is that the generational barrier to AI safety isn’t just getting our elders to recognize the potential scale of calamity before us; but it is including them in the movement to avert it. Just now there are a lot of experienced diplomats available for hire, thanks to 2025 reductions in force. Diplomacy is not easy, and AI orgs that want to instrumentalize diplomatic processes should try harder to enlist trained hands to wield them.
My second point is somewhat more straightforward: the AI warning signs are not that easy for outsiders to parse. I can’t tell you how many times I’ve heard people struggle to correctly relay the significance of METR’s infamous graph of AI task length ability. The graph is fine, but we need to be thoughtful when explaining it in a presentation.
Even when findings are presented flawlessly, the conclusions are often more obvious to the presenter than to the audience needing convincing. It isn’t obvious to policy makers why models becoming amazing coders is a step toward our collective doom. We shouldn’t assume that policymakers have read Anthropic’s RSPs or that they will appreciate the importance of approximately 1.9x increase in model capabilities on well-specified tasks. If they understood both, they still might not grasp how a super intelligence could take over the physical world.
You can lead a horse to water but you can’t make it drink. If the US government is the horse, AI safety organizations are not leading it to water as much as they are describing the cellular process of dehydration and historical examples of how to monitor water for contamination.
My last point is perhaps the most important: we need to be realistic about what governments would and would not do. I speak with people who spend their professional lives agonizing over the coming wave of cyber- and biosecurity threats that AI will almost certainly usher in. They care deeply about ensuring that humanity can produce a vastly superior intelligence without becoming manipulated, overcome, or destroyed by it. To ward off such dangers, they want to see the United States and China develop a mutual agreement to not develop successively more powerful AI so quickly, at least until significant progress can render AI better aligned and more fully controlled.
Much thought has been put into verification regimes under which the US and China could hold one another accountable for compliance with a hypothetical controlled AI development agreement.12 These approaches presuppose a level of regard for ASI that does not exist yet.
Consider the specter of nuclear Armageddon under which the world lived throughout the Cold War. The first real nuclear arms control treaty, a partial testing ban, wasn’t signed until 1963, with the Treaty on the Non-Proliferation of Nuclear Weapons signed in 1968. It took the lived examples of Hiroshima and Nagasaki and the near-miss incident of the Cuban missile crisis to finally prompt binding international agreement. Of course, there are still non-signatories, nuclear aspiring countries, and the world continues onward at some persistent nuclear risk.
So, it may disappoint those passionate scholars to hear that this new wave of frightening possibilities will not, in the abstract, suffice to realize such a goal. I see no reason to believe that either China or the US is currently considering a pause unilaterally, much less that the only thing dissuading either from pausing is the perception that the other would not also pause.
Even if governments mandated a pause, how would they know when it was safe to resume AI development, or should they never resume? Would they have to pause again every time a critical mass of safety researchers declare an emergency? How many signatories justifies an intentional slow-down? Part of the problem with AI safety initiatives, in my view, is that they ask countries to commit to international agreements that exceed what they are willing to do at home. Asking any government to order the deceleration of research, or to submit itself to international scrutiny under penalty of sanctions, or worse, would be an extraordinary request that rightly requires extraordinary evidence to justify.
Perhaps there is value in the strategy of pushing for a unilateral or bilateral pause in AI development, despite the apparent futility of it. Given enough time, AI safety field building organizations could gradually staff government institutions with policy professionals more amenable to such drastic measures meant to avert ASI. By furnishing those institutions with professionals open to such options, there could be more preparation to implement them if and when a serious AI incident elevates ASI preparedness higher up the agenda of national governments. However, my interactions with officials does not give me the impression that they are particularly close to worrying about ASI in the way that they worry about conventional global security, the domestic economy, and the rest of “business as usual.” Since short term pivoting into a pause seems implausible, the long term aim of ‘safety-pilling’ the US government would be scant return on investment relative to pushing for realistic near-term objectives.
AI safety policy advocates need to think about this before they deliver briefings to officials or convene international dialogues involving influential actors. Decision makers have extremely limited time to review literature, sit through briefings, or to dedicate to multi day international dialogues. When they do, they have finite political capital to deploy in promoting a deal up the chain within their own government. The effort expended on convincing governments about the endgame risks of ASI are better spent on more realistic proposals than long shot bilateral agreements.
More feasible outcomes to pursue would be a redoubling of national and international commitment to those pertinent causes where agreement already exists. AI risks should be articulated precisely and in a national security framing. Specific dangerous applications of advanced AI are more tangible for governments than the nebulous concept of extreme capability. Embodied AI, military AI including weapons systems and decision support, offensive cyber agents, LLMs and bio models that reduce barriers to bioweapons design, and agentic AI in financial markets are prime candidates for targeted policy proposals.
The US and China have stated jointly3, as has a broader coalition of states in a political declaration4, that AI should never be in charge of the decision to use nuclear weapons. New international norms and standards of AI use in the military will make for more compelling and actionable proposals. These concrete examples of how an uncontrollable artificial intelligence can act in the physical world against other countries will be crucial to demonstrating the national security relevance of increasingly capable AI.
Virtually all governments oppose the creation of artificially developed bioweapons, and cooperation on responsible innovation, bioweapons banning, monitoring, and incident reporting are possible. The Biological Weapons Convention serves as evidence of the broad, lasting consensus on this issue, and might one day lead to a mechanism of action and verification. In the cases of military AI, or AI-enabled bioweapons, countries can conceptualize AI safety as tantamount to national security. If they develop their own national security policies pertinent to AI, then they will be more likely to engage in international treaties to normalize that globally. Countries can defend against some national security threats all by themselves, such as attacks by land, air, and sea. Other threats, like nuclear, biological, and yes, ASI, require states to depend upon on another. This is why I think - as do some others - that focusing on ASI in international dialogues will most likely be a lost cause.
A bright and exceptional field of AI policy researchers and scientists has sprung up to rigorously measure AI capabilities and model their potential risks. Their hearts are in the right place, they are not wrong to warn about ASI, and they can do great things. But although their movement sounds the alarm, it does not yet translate into a clear message. Crafting and delivering a clear message to all the necessary parties is not a task that should be taken over-ambitiously.