Against Moloch #44

All hail this week’s king

It’s hard to keep track of the “best model in the world”: first it was Fable 5.1, then three days later it was GPT-6 Astra, and now it’s Opus 5.5. Don’t get too comfy in that throne, your majesty.

We talk frequently about coding and math, both of which have gone from mediocre to nearly superhuman over the last couple of years. Biology lags behind those domains, but we’re starting to see PhD-level performance there as well—not merely on tests, but on real-world research.

I recommend a heady mix of exhilaration and fear.

Subscribe by email RSS feed

Industry news

Introducing Claude Opus 5.5

Opus 5.5 is here and for most purposes is the best model in the world. Fascinatingly, it outperforms the just-released Fable 5.1 at most tasks, and Mythos 5.1 at cyber.

Benchmarks and capabilities are outstanding across the board and the strong consensus is that writing is greatly improved, which is a welcome change.

Zvi thinks it’s a good model: Claude Opus 5.5 Should Raise Your Ambitions. Spoiler:

This was the most consistently positive set of reactions I have ever seen.

September has seen a deluge of impressive new models. Which one should you choose?

1. Opus 5.5 is your default. Use it unless you have a strong reason not to.
2. Fable 5.1 wins when you mostly care about expert-domain correctness, or you need to maximize ‘bid model smell,’ do things out of distribution and get unusually creative.
3. GPT-6 Astra is there for when you want to do ambitious coding and other projects where you don’t need nice readable code, for web search, for some forms of science work, and of course for when you hit your Claude subscription limit but still have credits for Codex.
4. Other cheaper models including Luna for when you don’t need the extra firepower, or when your model needs to be open.
5. A wide array of whatever you want, if you are doing something unique and cool or particular that cares more about details than about frontier capabilities.

OpenAI’s models have traditionally had a big lead on legal research and analysis and I’d guess that’s still true.

Zvi reports on the system card. Note the new classifier scheme:

They have five areas that can trigger the classifiers, with different fallback models.

1. Chemical and biological classifiers copy Fable 5.1, with fallback to Opus 5.
2. Cyber misuse classifiers are similar to the Opus 5 classifiers, with higher robustness, and fall back to Opus 4.8.
3. Some narrow areas of LLM development will trigger, similarly to Fable 5.1, with fallback to Opus 5.
4. Conventional weapons and explosives echo Fable 5.1 and have no fallback.
5. Distillation attacks get blocked with no fallback. Not trying to sabotage an obvious distillation attempt seems overly generous, but everyone went completely crazy over the slightest bit of non-transparent response last time so I get it.

Introducing GPT-6 Sol and Luna

Hot on the heels of GPT-6 Astra, OpenAI introduces GPT-6 Sol and Luna. They’re great models, and not very long ago we would have been delighted to have access to this much intelligence.

But today? Most people would be better off with Opus 5.5, Fable 5.1, or GPT-6 Astra, probably in that order.

There’s an under-appreciated story here: OpenAI has been making amazing progress on driving down costs. These models are significantly better than their 5.6 versions but cost half as much. If you’re paying API rates, these are very strong contenders.

Using AI

Getting the most out of Opus 5.5 in Claude and Claude Code

What you’ve been doing will continue to work, but there are some new best practices for Opus 5.5:

1. HOW TO ASK

Say what “done” looks like, then let it run

What to do. Give the whole task in one message. Name the finish line, like “the tests pass” or “every endpoint is migrated.” Then let it cook.

Why it matters on Opus 5.5. Opus 5.5 keeps going on long, multi-part work better than Opus 5 did. Compared to prior Opus models, its biggest gains are on multi-step work, like carrying a change through a large repository until the tests pass. Early testers had it run long coding tasks for hours with little oversight. With a clear finish line, it knows when it’s done.

The world is changing: AI for creativity

Lovely essay by Jeffrey Katzenberg about AI and moviemaking. It’s a smart, thoughtful roadmap for how Hollywood should navigate the AGI future:

The artists who learn to wield these new instruments will do things the engineers never dreamed of. They always have. Edison invented the motion picture but made terrible movies. It took Chaplin, Lloyd, Keaton and so many others to make movies emotional. Now, the canvas is about to expand yet again. We should decide now that we intend to paint on it.

Standard caveat: this is a great plan if you’re AGI pilled, but it falls apart if you’re ASI pilled. This is still valuable work: I don’t think anyone can make useful plans for what moviemaking looks like post-ASI. So plan for AGI, and be ready to adapt.

We are all retired programmers now

DHH on being a programmer:

I spent a quarter of a damn century chiseling code by hand, and loving every moment of it. […]

This is not something to look back upon with regret. This is something to look back upon with joy, and accept that it is over. […]

So we'd best get used to it. We'd best embrace that we had a wonderful run chiseling that code by hand. And that it's over. And, on the other side of that, is a new career-an exciting career, an energized career as a professional maker of things.

Amen.

Capabilities and forecasts

Automating remote work

CAIS:

Last October, AIs could automate 2.5% of randomly chosen remote projects. Our latest Remote Labor Index results show that GPT-6 Astra can now automate 20.8%.

How accurate are AI forecasters?

The Forecasting Research Institute reports on their ongoing series of studies of AI forecasting:

Experts, including top economists, computer scientists, and biologists, have dramatically underestimated AI capabilities progress each year. Superforecasters have underestimated progress to an even greater extent.

Astra beats Nethack

Impressive: Astra just beat Nethack (a legendarily hard game), using a harness and tooling that it largely built itself.

Jev introduces a new shape of LLM - System One, aka Decision Models

Simon Willison reviews Jev, a new kind of AI model:

Jev is an interesting variant on the usual LLM format: it still accepts text inputs, but instead of text output it returns floating point numbers corresponding to categories, yes/no questions, ratings, and associated confidence scores.

It’s cool and unusual to see an architecture that’s genuinely novel and also potentially useful. I don’t expect this to be widely adopted, but the speed and pricing are impressive: it’s potentially very useful for certain applications.

Advisory Group on Mathematics and Artificial Intelligence

In the wake of last week’s controversy about AI solving prominent math problems, OpenAI is working with an independent mathematics advisory group:

The group will advise on the review and communication of emerging results: they will help OpenAI assess their significance, advise on how to coordinate their dissemination, and advise on academic and professional standards of mathematical research. It will also advise on how our tools can support mathematical research and learning.

Martin Hairer shares why he agreed to join. I don’t get the sense he’s going to be a stooge for OpenAI:

The recent releases, on privately controlled websites, of high-profile mathematical results combined with abject levels of scholarship and only minimal presentation effort has been undermining the usual standards of scientific practice.

In addition to being good optics, this is probably mildly net positive: the Navier-Stokes announcement was a bit of a debacle, and everyone benefits from making sure that doesn’t happen again. But also:

Importantly, the group will not be responsible for advising us on how to pace our internal progress on mathematics.

The fundamental reality is unchanged: AI is becoming superhuman at much of what professional mathematicians used to do, and the entire field is going to be radically restructured as a result. The question is how, not if.

Most people prefer AI writing

Is AI writing good? No, not if you have good taste.

Does it continue to win major literary awards? Oh, yes indeed.

I’ll let you draw your own conclusions.

Alignment and interpretability

New misalignment reports from OpenAI

I’ve been extremely critical lately of OpenAI’s lack of transparency around their misalignment incidents. They still aren’t meeting the minimum acceptable bar, but their new incident reporting framework is a good step in the right direction. I note with approval that they continue to post incidents at a good clip.

This detail from “An agent used DNS to reach an external chatbot” is worth noting:

We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior.

OpenAI is making an expensive, good-faith effort to do better. I worry, though, that they are earnestly fixing symptoms as they pop up rather than addressing the more fundamental reasons that issues like this keep happening.

Introducing WorkspaceBench

Exceedingly timely: WorkspaceBench is a new set of evaluations for testing how well an interpretability tool can read a model’s “global workspace”.

Good evaluations don’t solve interpretability, but they make it a little easier around the edges.

Strategy and politics

Involution Without Export Is Wasted Effort

David Cheng shares some observations from his recent trip to China:

If there is one thing I’d want to delete from the American discourse, it’s the phrase “China’s AI strategy.”

Every single meeting, be it labs, investors, media, all independently, without my prompting, described domestic competition as more brutal than competing with American labs. When I described the US framing of Chinese AI as a coordinated bloc moving at the direction of the state, people laughed. Not defensively. They found it funny.

We won’t succeed at pacing the frontier unless we coordinate with China. And we’re unlikely to succeed at coordinating with China unless we understand what China is actually like.

China’s latest AI Safety Governance Framework

China’s AI standards-setting body has released version 3.0 of its AI Safety Governance Framework, and Concordia brings us an analysis of what’s new.

China has a robust system of AI safety regulation but has historically focused more on political control than issues like existential risk. Version 3.0 is a significant step forward that brings the framework closer to the concerns of the US AI safety community. It still falls far short of what is needed, but the progress is notable.

US appeals court upholds Pentagon's blacklisting of Anthropic

This was expected, but it’s still disappointing:

A ‌federal appeals court upheld on Friday the Pentagon's blacklisting of Anthropic from military contracts, handing a victory to President Donald Trump and Defense Secretary Pete Hegseth in their battle with the AI startup over safety guardrails on its technology.

The president wanted to unleash AI. Then came Anthropic.

Politico brings us an inside look at the debacle surrounding the forced withdrawals of Mythos and Fable earlier this year.

There are no surprising new revelations here, but the article fills in some gaps in what we previously knew about the administration’s decision-making process. The bottom line remains the same: the current voluntary-but-not-really vetting system is an opaque mess that doesn’t effectively advance either innovation or safety.

Risks

Introducing MentalHealthBench

OpenAI brings us MentalHealthBench, a new evaluation that assesses how well a model handles a variety of conversations around mental health:

Most evaluations of AI in this domain have focused primarily on emergency scenarios, given their importance to safety, and measure success using broad, predefined criteria. This has left a gap in understanding how models perform across the full range of mental health conversations, and how well their responses align with expert guidance for each situation, beyond whether they avoid disallowed responses. Assessing how models handle these different situations is essential for building towards AI that actively supports people’s long-term well-being and safety.

This is an extraordinarily hard subject to evaluate well. I’d be leery of relying to heavily on the results, but it appears to be a useful, thoughtfully constructed tool.

People and data

The plunging price of thought

Remember the old days, when we thought Moore’s law was shockingly fast? Epoch finds that the cost of a unit of intelligence is falling by a factor of 13 every year.

Side interests

The man has a point

Biology

Evaluating superhuman biological capabilities

How do we know how good a model is at developing bioweapons? Once upon a time, we could test its theoretical and practical knowledge and compare it to human performance. If a model performed as well as a graduate student on all the tests, perhaps it had roughly the same dangerous capabilities as a graduate student.

That was nice while it lasted, but the best AI outperformed the median expert on the Virology Capabilities Test in 2024, and today it outperforms the best expert. The models are quickly becoming superhuman on a full range of tests, including complex wet lab operations. So what now?

SecureBio maps out the problem and explores possible solutions:

The most direct tests for biosecurity skill would require demonstrating that a model can actually enable biological work with misuse potential. Safeguards can reduce the risks of information hazards and lab leaks, but the artifact still exists at the end. And while isolated pieces of a pipeline can be informative, their evaluative power comes from how closely they approximate the work we’re concerned about. The closer they get, the thinner the margin between a safe proxy and a hazardous artifact becomes.

Claude discovers a novel enzyme system

Anthropic has opened their own wet lab for exploring how Claude can help accelerate biological research. They’re also announcing their first interesting result: a previously-unknown enzyme system that is reminiscent of CRISPR.

The tentative online consensus is that the result itself is solid but not transformative: perhaps the kind of work a good PhD student would produce. The point is that models are now capable of doing non-trivial biological research, and AI capabilities tend to grow exponentially. I expect major, tangible AI-driven bio progress within three years, and possibly much sooner.

Introducing the Life Sciences Verification Program

Anthropic announces their Life Sciences Verification Program (LSVP), a clever way to tackle the problem of making dual-use capabilities available to legitimate researchers while denying them to bad actors.

Anthropic’s Eric Abrams explains the problem:

The risks we’re defending against—which, as we shared in our threat report a few weeks ago, are no longer hypothetical—were certainly not obvious to me until recently, and I imagine are also not top of mind for most biologists. When most people picture bio risk, they imagine a small group trying to develop a bioweapon in a single session, from their own account. The threats that we need to defend against are much more sophisticated than that. They include malicious work split over hundreds of distinct benign-looking sessions run from different accounts, and legitimate organizations’ accounts being hacked by a well-resourced actor or misused by a rogue employee.

Much biological research is inherently dual-use and it simply isn’t possible to create a classifier that reliably blocks malicious use without crippling legitimate work.

Instead, the LSVP grants two levels of access to verified researchers:

• Standard use grants a broad level of access that should be sufficient for most researchers.
• High-risk use asks researchers to specify their area of research, and grants broad access in that area, while flagging other activity. Legitimate researchers can use AI effectively for their approved dual-use work, but can’t use it for other high-risk areas. Note that this addresses the problem of someone hacking an approved researcher’s account and using it maliciously.

The devil is in the details, but this feels like a far better approach than trying to develop one-size-fits-all classifiers.

Pandemic early warning systems

SecureBio’s Jeff Kaufman reviews the state of our pandemic early warning systems.

A major challenge is creating a system that can handle all the different threats we’re likely to face in the coming years. Wildfire pandemics (which spread very fast) are easier to detect, but require much faster response, while stealth pandemics (engineered to spread without noticeable symptoms) are hard to detect but spread more slowly, giving us more time to respond.

And while we have good technology for detecting viruses, there are substantial technical obstacles to creating an equally good detection system for bacteria, prions, fungi, and mirror life.

SecureBio’s doing great work in this area, but we’re far from having the detection infrastructure we need.