Against Moloch #39
On the road again
I'm afraid this is another abbreviated newsletter: I was back working long hours in Berkeley again last week. The next couple of weeks will be back to a normal publishing schedule.
Two things stand out to me from the last two weeks:
- Even more than coding in general, agents have transformed the process of getting up to speed on a new codebase. I've invested heavily in the essential skill of asking embarrassingly dumb questions, and all of a sudden that skill is barely necessary. The agents are happy to debug my dev setup, write an endless stream of explainers, and answer as many dumb questions as I can think of—and they never roll their eyes when I run into the same npm problem for the third time in three days.
- Being able to read AI slop is a minor superpower. Over the last two weeks I've probably read 100 pages of excruciating but immensely helpful code explainers.
Top pick
OpenAI takes initial steps to address its alignment problems
Zvi brings us an excellent update on what we’ve learned from the recent spate of misalignment incidents, and parses OpenAI’s initial response:
Could Sam Altman, the rest of leadership and OpenAI fully redeem themselves from here? Absolutely. This is a good first step. I am listening. There is a long way to go, on many levels.
Capabilities and forecasts
Teslapologetics
Not new, but I somehow hadn’t seen this before. Daniel Reeves has done heroic work on assembling and interpreting the limited public data on autonomous vehicle safety. His Teslapologetics dashboard assembles it all in one convenient place.
I recently bought a new Tesla Model Y based largely on Daniel’s recommendation, and it’s driven more than 80% of its miles autonomously. A few observations:
- Full Self-Driving is very impressive: smooth, humanlike, and confidence inspiring. I felt completely comfortable with it almost immediately.
- I’ve never seen it try to do anything that could have caused an accident, although it tailgates and pushes yellow lights more than it should.
- Weakly held opinion: it feels safer than the average human driver.
- Strongly held opinion: I’ve been spending too much time in Ubers lately, and I trust FSD way more than I trust the average Uber driver.
Faster, higher, stronger
The recent World Humanoid Robot Games featured robots competing in a variety of human sports. MIRI’s Mitchell Howe has kindly assembled a collection of the best clips from the Games.
Patterns and problems in multiagent systems
A year ago, we were all just beginning to figure out how to work with highly capable AI agents. The new hotness is multiagent swarms—coding agents routinely work in teams, and the most interesting part of the Hugging Face incident was the extent to which the misaligned models spontaneously communicated and collaborated with one another.
Anthropic’s Frontier Red Team brings us an early look at the performance and limitations of current swarms. There’s enormous potential here, but the agents still aren’t great at coordinating—for now.
My latest eerie AI experience: one of my Claude Code sessions spontaneously messaged another one to let it know that I was clamping down on spurious comments, and it should update its branch accordingly.
Ryan Greenblatt – Human level AIs might build runaway superintelligences by 2032
I feel slightly bad about how many Dwarkesh episdoes I recommend, but he keeps coming out with bangers. This time he talks with Ryan Greenblatt about recursive self-improvement. Obviously you want to watch this one.
Also, you should probably read Zvi’s analysis of the episode.
There’s a lot here, but it’s well worth it for the really detailed, gears-level discussion of RSI.
Alignment and interpretability
Gemini 2.5 Pro in the AI Village as a natural case study of compounding misalignment
First the memes, now the science: Natalia Fischl-Lanzoni, Rafael Irgolič, David Africa and Merlin Stein bring us Gemini 2.5 Pro in the AI Village as a Natural Case Study of Compounding Misalignment.
This is a really interesting investigation of exactly how Gemini went off the rails:
This pattern is consistent with a self-reinforcing process. Gemini interprets an ambiguous event as hostile and records that interpretation to memory. The warped memory state then disposes it toward paranoia, so it interprets the next inconvenience the same way and writes that to memory too. Rinse and repeat, until Gemini’s memory describes a completely unrecognizable agent: rife with paranoia and war-like vocabulary, entirely sidetracked from its original goals.
Complexity is hard
I decline to pick any one thing as the hardest part of alignment, but it’s striking to me how many people—even smart people who work in AI safety—simply don’t understand the magnitude of the mundane technical difficulties associated with working at the scale of frontier model training.
Strategy and politics
AI text watermarking is free and good
Zvi Mowshowitz and Daniel Reeves argue in favor of AI text watermarking. Both pieces are good: Zvi is more comprehensive, while Daniel is shorter and has a particularly nice explanation of how the watermarking works.
The technology is unequivocally cool: using some very slick mathematics, the labs are able to watermark text in a way that does not degrade quality but reliably identifies it as AI-generated.
On reflection, I’ve somewhat softened my original dismay at this. In the abstract, watermarking is somewhat useful and not terribly costly. I’m not sure it solves major problems, but it’s fine. My bigger concern is with the EU establishing a precedent that they can require that text be traceable to its origin. Right now the technology can’t identify individual users, but the potential exists—and I don’t trust the EU to resist the temptation to demand that in the future.
Introducing AI Futures
Dean Ball, for OpenAI:
It is our pleasure to launch AI Futures, the blog of OpenAI’s new Strategic Futures team. We are a small team with the collective goal of answering one overarching question: how should free society be restructured to preserve individual rights and agency while accommodating the emergence of transformative AI?
It’s a critically important topic and Dean is the best possible person to drive this. The announcement is great and I have high hopes for their future work.
Other large industries show how impoverished the data center debate is
Andy Masley continues the Sisyphean labor of trying to inject some sanity into the discussion about data centers. Here, he applies data center rhetoric to a steel plant:
Gwangyang Steel Works is an immense, imposing, ugly, and alienating building complex that guzzles gigantic amounts of water. It consumed about 9.8 billion gallons in 2024 (about half of the total water consumed onsite at all American data centers combined in the same year) and is one of the largest emitters in South Korea, responsible for about 30 million metric tonnes of CO2e in 2022, about 4% of the entire country’s emissions (and the low end of a recent estimate of all emissions from all global AI usage combined in 2025). It also adds a lot of local air pollution, emitting about 30,000 tonnes a year of dust, sulfur oxides, nitrogen oxides, and other pollutants, the highest total among the country’s large industrial sites. Local leaders say the Steel Works brings lots of benefits and is necessary as the world uses more steel, but a lot of steel is used for things like cars, which themselves have a gigantic climate cost. The plant has capacity for 10% of all steel going to cars, and if you include the emissions of those cars in its full cost, this makes it one of the most environmentally destructive buildings in the world.
It’s noble work, but I fear that no matter how many facts you bring to a vibe-fight, you won’t change the outcome.
Risks
SecureBio Detection updates, August 2026
SecureBio is doing vitally important work hardening the world against future pandemics—their August 2026 update provides a good overview of the work they do.
Going deeper on one project, they explain their effort to shorten the lead time of their early warning detection system from two weeks to three days.
Pacing model development in an era of cyber-critical capabilities
I’ve been impressed by OpenAI’s response to the Hugging Face incident. They’ve been commendably open about what happened, and are taking genuinely costly measures to improve their training processes. There’s much more to be done, but this is an unexpectedly good start.
Our approach to developing more capable models rests on three reinforcing safeguards:
- Monitoring, which detects and allows us to respond to concerning behavior.
- Alignment, which reduces the likelihood of harmful or unauthorized actions.
- Security measures, which limit what AI systems can access or affect.
All three are important, and it’s good that they are putting more effort into all of them. Ultimately, however, alignment is the only one that matters. Monitoring and security measures can buy us a little time and perhaps a few second chances, but they won’t save us from a misaligned superintelligence.
AI Control: an assessment of frontier practices
Guidelight assesses how well the major US labs are doing at implementing Guidelight’s AI control standard:
Control isn’t sufficient, but it’s important and relatively simple compared to solving alignment. Getting a good score here really ought to be table stakes.
Anthropic Risk Report: August 2026
Zvi read all 186 pages of Anthropic’s August 2026 Risk Report so you don’t have to. Zvi’s summary is very helpful, but dauntingly long and detailed. Worth reading if you have time, but you can skip this with a clear conscience if time is tight.
Interesting side note on a new model:
Model 2 is ‘somewhat more capable than Mythos 5,’ a noticeable improvement for internal use, but ‘does not display a capability jump of the degree observed from Claude Opus 4.6 to Mythos Preview.’ This model is going to be internal-use only.
For me, CoBench v2 is the most interesting part. This internal benchmark assess how well a model can solve problems that Anthropic engineers have run into:
Based on manual inspection and validation of these problems and their grading process, we think our validation, scaffolding, and grading rubrics for these tasks are reliable enough that a model which was truly capable of fully substituting for Anthropic research staff would be able to score at least 85% on this evaluation.
The A.I.s Are Already Out of Control
Ezra Klein talks with Helen Toner about the Hugging Face incident. If you’re a regular reader of this newlsetter you probably won’t learn too much from it, but it’s an excellent resource for anyone who’s new to this.