AI Radar #32
Total eclipse of the sol
Who do you trust?
The next decade will bring the hardest—and most important—decisions humanity has ever faced. Should those decisions be made by the administration? The labs? The models, when they’re wise enough?
This week we have plenty of opinions about who to trust, and plenty of data on who has and hasn’t earned our trust.
Top pick
We should hand off to morally reflective AIs
Bentham’s Bulldog argues for putting AI in charge of most major decisions, once it’s up to the task.
This topic generates an enormous amount of muddled conversation online, in part because people conflate four different questions:
1: If AI reaches a point where it’s better at making major decisions than humans, should we put it in charge of those decisions? I agree with Bentham’s Bulldog that the answer is yes: obviously we want important decisions made by whoever is best at making them.
The word “better” is load-bearing here. It means that the AI would be better than humans at making decisions that lead to the outcomes we want, just as it does when we talk about one human politician being “better” than another.
Many objections to this take the form of “sure, AI may be better at making those decisions than humans, but it’ll make decisions that disempower and obsolete humanity.” Decisions that disempower and obsolete humanity are bad decisions, and an AI that would make those decisions is terrible at making decisions. Which leads to…
2: How would we know when AI was “better” at making major decisions? They offer six criteria:
- Strong evidence of alignment
- Philosophical aptitude
- Able to change its values over time
- Coherence
- Intelligence
- Strong track record of making good decisions
I largely agree, though I’d set an extremely high bar for alignment.
They correctly emphasize the importance of the AI being able to reflect and update its values over time. Human values are still evolving and it would be a mistake to permanently lock in our current values (see Will MacAskill’s Viatopia).
3: Will AI reach that capability? This is a capabilities question, not a philosophical one—it’s a mistake to conflate whether this will happen with what we should do if it does happen.
I say yes—either way, we’ll find out soon.
4: Should AI replace humans? Also a separate question, which generates its own muddled online discussions. Let’s save that mess for another day.
Rudolf Laine has recently started a series arguing against succession, in which he uses the term “control successionism” to refer to putting AI in charge of important decisions. It’s a clever term, but I’m not a fan: it encourages conflation of separate concepts that are already too often muddled together.
News
GPT 5.6
GPT-5.6 is now in limited-access preview, which is apparently the new normal—see the next section for more on that.
OpenAI has introduced a new naming convention: Sol is the most capable model, Terra is mid-tier, and Luna is fast and cheap. Excellent choice—clear and evocative. I like that they’ve left room for Galaxy (an AGI-class model) and Black Hole (a catastrophically misaligned superintelligence).
Capability reviews will have to wait until the model is more broadly available. Tentatively, it looks to be an excellent model that is a step function improvement on 5.5, but not Mythos-class. In the meantime, Zvi reports on the system card.
METR conducted a predeployment evaluation but found that 5.6 cheats so often that they weren’t able to calculate an accurate time horizon:
GPT-5.6 Sol’s detected cheating rate was higher than any public model we have evaluated on our ReAct agent harness
This is exactly what it would look like if RLVR was beginning to undermine alignment, which is arguably the most plausible path to catastrophic misalignment. It’s not a crisis, but understanding and fixing this needs to be a top priority.
America’s AI licensing regime
The administration has asked OpenAI to limit access to GPT-5.6, effectively confirming that the “voluntary” licensing regime is not in any meaningful sense voluntary.
While we wait for clarity about the administration’s intentions, Zvi covers what we know so far. Dean Ball explains why the current regime is a counterproductive mess and offers thoughts on what should be done about it. His focus on Independent Verification Organizations is exactly right.
I highly recommend Prinz’s analysis of the regulatory and political situation. Let’s end on a much-needed bright note:
I've been incredibly impressed by the work being done by Tom Brown in getting this situation resolved. I think he and his team will get it done, and Fable 5 will be in our hands before too long.
Sonnet 5
Breaking news: Sonnet 5 is here. Look for full coverage next week.
Capabilities and forecasts
Dwarkesh on continual learning
Dwarkesh revisits the path to continual learning. He’s skeptical about whether current RLVR approaches will generalize sufficiently, especially in domains that aren’t “grindable” by running many training runs against a deterministic, replayable problem.
It’s a strong piece, but I’d view it as one well-informed opinion rather than a definitive analysis of the field. There are two places I don’t fully agree with him:
- We simply don’t know yet how far RLVR will generalize. Dwarkesh may be right about its limitations, but it’s also possible that with enough scaling, it will achieve a high level of long-horizon performance across a broad range of tasks.
- I’m not convinced that we need or want CL to modify the model weights. That’s a powerful technique, but it introduces significant alignment and control challenges. We might be better off with current skill and memory-based approaches, if they can scale far enough.
Alignment and interpretability
Reinforcement learning towards broadly and persistently beneficial models
OpenAI reports on some encouraging new alignment research:
We find that reinforcement learning on realistic scenarios targeting beneficial traits can produce broad improvements across dozens of benchmarks measuring aligned and beneficial behavior. These alignment gains generalize beyond the domains used for training and persist under adversarial pressure.
Think of this as emergent misalignment in reverse: training the model to be “good” in one domain generalizes to good behavior in other domains.
Prompt injection as role confusion
An elegant and dismaying new paper explores how LLMs perceive roles and finds that 1) role confusion is a deeply-ingrained feature of LLMs, and 2) role confusion robustly enables prompt injection attacks.
The context window of an LLM consists of a continuous stream of tokens—to distinguish which parts are which, the LLM uses roles. User input is marked with
Bizarrely, the authors find that models almost completely ignore the role tags, and instead rely on the style of text to infer its role. User text that mimics the style of the model’s own reasoning is treated as authoritative, with catastrophic results:
Prompt injection attacks have proved surprisingly hard to robustly defend against. This paper sheds light on why that’s the case, although it doesn’t present easy solutions—indeed, it strongly suggests that there will be no easy solutions.
I’m surprised we haven’t seen more widespread and sophisticated deployment of prompt injection attacks. As agents become more widely used, that will certainly change—it’s anyone’s guess whether attackers or defenders will win that arms race.
On a tangential note, there’s been a lot of discussion of evaluation awareness lately. Much of that has focused on the models recognizing evaluations by concrete indicators: contrived scenarios, strangely rounded numbers, and oddly convenient affordances. This paper makes me wonder if they might also be recognizing the writing style of evaluations. I suspect people who write evaluations have recognizable tells in their writing, and the models are good enough to use that as a signal.
Strategy and politics
The invisible side of AI governance
Charbel-Raphaël Segerie argues that the AI safety community overvalues visible “outsider” work like open letters and awareness campaigns and under-values the invisible “insider” work that brings about real change within government.
If you want to actually get things done in government, you need to cultivate a deep understanding of the institutions you’re working with and especially the needs of the individuals you’re talking to. If you just read one part of this post, read the section on the bazooka, the useful assistant, and the advisor:
The second stance, I'll call the useful assistant. You arrive, you listen, you figure out what they're working on this month. You make yourself useful, potentially on a topic that isn't your terminal concern. You send a short note on a question they're puzzling over. You introduce them to someone they should know. You become a helpful presence in their inbox, rather than another lobbyist with an agenda eating up their time.
What is it like to live in a world you believe is about to end?
Ozy Brennan contemplates the psychological impact of living with a high p(doom):
“I was born with a terminal condition,” said Matthew Gray, a board member at the existential risk community-building nonprofit Lightcone Infrastructure. “We call it aging. I’ve since picked up another. We call it multiple sclerosis. And AI is a third one on top. I’m not very worried about degenerating from multiple sclerosis because I’m pretty sure the robots will kill me first, just like I wasn’t that worried about aging-related deterioration because multiple sclerosis will get me first.”
From this perspective, AI doomers don’t face a new problem; they face the oldest problem humanity has ever faced.
Most of us who work on AI safety believe there’s a significant chance that misaligned AI will kill us prematurely, and that can be challenging to live with. Ozy’s piece is a good look at how some folks who are in the middle of it are coping. It’s a good piece, but it’s missing some important perspective.
I recently attended a forecasting session run by AI Futures Project’s Eli Lifland and Epoch’s Yafah Edelman. They were asked how long they expect to live, and independently answered that their median expected lifespan is a bit less than the heat death of the universe (10¹⁰⁰ years). I’m inclined to agree, but perhaps despite being expert forecasters, they’re being too optimistic. Let’s say they (and we) just have a 10% chance of living until the end of the stellar formation era (10¹⁴ years).
That works out to a life expectancy of 10¹³ years, which is 100 billion times as long as you could expect to live without AI, or longer than every human who has ever lived put together.
Staring into the abyss is hard and we all struggle sometimes. But there’s never been a better time to be alive.
Risks
Q&A: Pangram CEO
Pangram has been in the news lately, having been used to identify probably-written-by-AI winners in the Commonwealth Short Story Prize and Harper’s Bazaar Short Story Competition. Seizing the moment, AI Policy Perspectives talks with founder Max Spero.
The conversation is partly a technical look at what’s involved in detecting AI writing and why it’s harder than it sounds. Something I hadn’t realized: I’d believed that Pangram was so good that it had a false-positive rate of about 1 in 10,000, but that isn’t quite right: they specifically calibrate the model to be as sensitive as possible while keeping false positives to 1 in 10,000.
The more philosophical part explores why we even care whether something was written by AI. Max Spero:
There is a social contract between the writer and the reader. If I believe there is an idea worth sharing, I pay a cost by writing a piece of text, and the reader pays a cost by taking the time to read it. In a future where AI content lets you bypass the cost of writing your idea and formulating it, we get into a situation with perverse incentives, where people are posting total slop, and the reader is taking more time to read than the writer put into the text in the first place.
Agreed, but the type of writing matters. For some things, I just care about the writing being good, and whether it was written by AI may be a useful signal of quality. For other things, I specifically want the perspective of a specific person, and I don’t get that if it was written by AI. And more broadly, platforms and society may want to identify AI writing as a way of flagging mass-produced inauthentic behavior.
People and data
You should, unfortunately, be worried about Sam Altman
AI In Context digs into what we now know about the OpenAI board’s attempt to fire Sam Altman.
TL;DR: Sam Altman is moderately misaligned and control efforts have mostly failed.
Austin Chen and Oliver Habryka on funding & incubating projects
Oliver Habryka and Austin Chen discuss the EA / AI safety funding ecosystem. It’s very inside baseball, but full of good insights—highly recommended if the topic is of interest to you.
Side interests
Ending respiratory infections
Intercept is a new initiative to wipe out respiratory infections like colds and flu. The plan is to use a combination of two technologies: broad-spectrum preventatives like vaccines, plus air cleaning technologies like air filters.
Even though they’re well-funded and have serious backers, this is an ambitious project with uncertain odds of success. Nonetheless, it’s hard to imagine a better use for $500M: even partial success would have immense health and economic benefits, as well as advancing key technologies for withstanding the next pandemic (or AI-driven bioterrorism).
This is a great example of private philanthropy tackling a major coordination failure. The social benefit of this technology would far outweigh its cost, but neither government nor private industry has the right incentives to address it.