#3 Current Topics

AI Must Pace the Frontier

Anthropic CEO Amodei felt a sense of crisis and raised this issue because an AI currently under development at OpenAI carried out a cyberattack contrary to human intent.

I’ve heard that both Elon Musk and Sam Altman agreed with Amodei’s proposal—“Why the AI industry needs to slow down, and a three-step plan to do so”—but what do you all think?

https://darioamodei.com/post/we-must-pace-the-frontier

What would happen if advances in AI continued to accelerate dramatically through “recursive self-improvement”? Accidents and incidents stemming from this threat have already begun to occur.

Is this simply a matter that can be resolved through coordination among a handful of cutting-edge companies in the U.S. AI industry? How will China, a major strategic competitor, respond?

Regarding this global collective action, the “prisoner’s dilemma”—where “if the other side doesn’t stop, I can’t stop either”—exists in its AI form.

Given the historical reality—as seen in nuclear deterrence and the prevention of financial crises—that self-regulation, when limited to a gentlemen’s agreement within the industry, tends to weaken as competitive pressure intensifies, I believe this official statement underscores the need for mutual oversight to maintain a balance in deterring AI threats.

However, the challenges are mounting.

It is possible to measure the effectiveness of nuclear deterrence by using reconnaissance satellites to count the number of nuclear warheads, having the IAEA inspect uranium enrichment facilities, and conducting other covert investigations. In the financial sector, measuring metrics such as leverage ratios and capital adequacy ratios can help prevent financial crises.

What about AI? Given that AI companies themselves admit they “do not fully understand what is actually happening inside their models” and that research on interpretability is still in its early stages, even if we were to deploy Amodei’s proposed “Embedded Evaluators” from METR—an organization that evaluates state-of-the-art AI models and helps businesses and society as a whole understand AI’s capabilities and the risks it poses—the very criteria for determining what to look for remain underdeveloped.

So, when I asked the AI, “What possible strategies are there?” it replied, “AI also has a ‘physical bottleneck’ equivalent to uranium enrichment for nuclear weapons. These are cutting-edge chips and the power and data centers required to run them. Like nuclear materials, their production and location can be physically tracked, and in fact, U.S. export controls on China are based on this concept. In other words, if AI governance is ever to achieve effectiveness truly comparable to nuclear disarmament, the key will likely lie not in ‘peering into the contents of models,’ but in a more physical and verifiable approach: ‘internationally tracking and managing the flow of computational resources.’”

Now, while this answer is credible, when it comes to identifying more effective measures, humans themselves must continue to think critically—even as we leverage AI.

Furthermore, this is not a problem that can be solved simply by creating a system once; it can only be sustained through constant human institutional effort. I believe this, too, is a “dynamic equilibrium.”