Against Moloch #43
Trouble with benchmarks
I’m afraid this is one of those weeks when there’s too much AI news for a single newsletter. Today will cover all the regular news, and over the next few days we’ll take a deeper look at the ongoing misalignment crisis, the shift in the Overton window, and progress toward recursive self-improvement.
Using AI
Claude Docs
Claude Docs are in beta: create documents, slides, and design projects to share with others.
OpenAI and Anthropic are busily experimenting with new features, and new ways of organizing and presenting existing features. I love all the new functionality, but the pace at which interfaces keep changing is more than a little disorienting.
Claude Projects
Another big beta release from Anthropic:
Projects are designed for long-running or agentic workflows: work that takes longer than one reply and has more than one part.
A year ago, agentic coding was just starting to take off. Agentic swarms are now reaching that point: expect very rapid evolution of capabilities and interfaces.
Astra for Law
OpenAI introduces Astra For Law, which is launching with an impressive set of plugins from established legal tech firms.
OpenAI’s models have long been particularly good at legal research, and this will only widen their lead. I'm very curious whether the plugin model is here to stay, or whether Astra will quickly incorporate all that functionality.
Google Home gets MCP
Very cool: Google Home is adding MCP support, so you’ll be able to do all kinds of wild agentic things with your smart house.
I really want this, but also…
Capabilities and forecasts
Epoch introduces Benchmark Reviews
A surprising number of important benchmarks have widespread problems that significantly impact their accuracy. Epoch is introducing Benchmark Reviews, which audit benchmarks for accuracy. A surprising fraction of their initial assessments found substantial issues:
How good are frontier models at physics?
Speaking of flawed benchmarks, a new paper finds that frontier models are much better at physics than previously realized. Fixing problems in common benchmarks leads to a dramatic increase in scores:
We find that GPT-5.6-Sol's measured mean@4 rises from 47.3% to 78.7% on HLE-Physics and from 61.0% to 87.2% on CMT-Benchmark […]
These findings suggest that current benchmarks substantially understate frontier models' ability to solve well-posed physics problems. Near-saturation on these closed-ended tasks highlights the need for more demanding, expert-validated evaluations.
Anthropic has set up a wet lab in San Francisco
Reuters reports that Anthropic is running its own wet lab, in addition to working with outside partners:
Now, Anthropic is bulking up its in-house lab efforts for speed and for firsthand experience in biology, though it outsources work when efficient, Kauderer-Abrams told Reuters.
“There’s some things that we can do much faster in our own hands,” he said, with the goal “to operate at the largest possible scale.”
Like coding, cyber, and math, AI-driven biotech won’t be a big deal until it suddenly is.
Alignment and interpretability
The case for reasoning transparency
GDM’s Rohin Shah and Anca Dragan have a great explanation of why CoT monitoring is a vital part of our alignment toolkit, why we’re in danger of losing it, and how to preserve it.
What if future reasoning models do give up on Chain of Thought for opaque methods? We might still be able to recover reasoning transparency with new and better tools for translating machine thoughts into human language. It’s a much tougher challenge than taking models that are already transparent, and making sure they stay that way, but it’s worth our efforts, and we’re already working on these challenges of interpretability. Minimally, we propose that if AI developers deviate from standard Chain of Thought, they should provide a robust and convincing demonstration that their new methods are comparably monitorable.
Detecting reward hacking with probes
New research from Goodfire finds that activation probes can detect reward hacking that CoT monitoring misses:
Our results suggest that models give us a useful signal for catching reward hacking: their own internal representations. Across models and environments, probes can identify reward hacking, generalize beyond the setting they were trained on, and surface a coherent internal concept associated with reward hacking. They can also catch warning signs that text-based monitors miss and make real-time monitoring feasible at scale.
This isn’t a silver bullet, but it sounds like a useful addition to the toolkit.
A philosopher’s guide to AI welfare
Writing for AI Frontiers, Jeff Sebo surveys what we know about AI welfare, and explains why studying it is hard:
a key challenge for this field: even if consciousness, sentience, and agency often occur together in humans and other animals, they might occur separately in AI systems. Therefore, we need to study each capacity individually. We also need to be ready for unexpected configurations, such as sentience existing in one part of a system and agency in another.
We barely understand consciousness, sentience, and agency in humans—barring unexpected breakthroughs, it’s hard to imagine that we’ll understand it before it arrives (if it does). Here, even more than in other parts of AI, I don’t really see a viable alternative to just trying to figure it out as we go.
Microsoft’s Humanist AI Code of Conduct
Perhaps someone should forward Jeff Sebo’s article to Microsoft AI? They just published a draft of their Humanist AI Code of Conduct, which is a master class in missing the point.
I’m far from the first to point out that the discussion of AI consciousness is an incoherent mess:
[AI] is not conscious and should not be designed to imitate consciousness. […] Whilst the science of AI consciousness is far from settled, we believe that training these systems to imitate consciousness-like states increases the challenge of containment, control, and alignment. We reject the pursuit of legal personhood, or the idea that models might deserve welfare, or be entitled to rights.
Obviously, if the science of AI consciousness is far from settled, it makes no sense to declare by fiat that AI is not conscious. And if AI might be conscious, categorically rejecting the idea that it might deserve welfare is abominable.
The document is equally confused about the fundamental nature of superintelligence:
At Microsoft AI, we’re working towards Humanist Superintelligence (HSI): incredibly advanced AI capabilities that always work for people and in the service of humanity more generally. We think of it as systems that are problem-oriented and tend towards the domain specific. Not an unbounded and unlimited entity with high degrees of autonomy
These are rookie mistakes. It is not the nature of superintelligence to be domain-specific, and superintelligence without agency is a mirage.
CheatBench
CAIS brings us CheatBench, which does exactly what you’d expect.
Strategy and politics
Quick notes from teaching technical profiles how to talk in public
AI safety is having a moment, which means many people are getting interview requests for the first time. If you’re one of them, Camille B. offers a crash course in improving your public speaking skills:
tl;dr : Deliberate practice is the bottleneck. Speak like you write, in fluid, uninterrupted sentences. Open with spoilers, be straight to the point. Make your voice go higher and lower than usual, have a high awareness of the social context, and focus on polishing the start and end of your intervention.
Be sure to check out Dave Kasten’s advice in the comments.
Risks
Hacking OpenAI
Ouch. A three person team from Hacktron used AI to chain together two vulnerabilities to break into OpenAI. They were able to access internal Slack, Github, and email—but probably not model weights.
This is concerning, but not terribly surprising: in the Mythos era, you should assume that every online system is vulnerable to a determined attacker. In particular, you should absolutely assume that China will be able to penetrate any of the frontier labs.
The bad guy with an AI named Claude
I briefly covered Anthropic’s report on Detecting and Countering Misuse of AI last week. If you want more detail, here’s Zvi with a detailed walkthrough. Sections 9 - 11 (covering distillation) are particularly interesting.
Deepseek, Moonshot, and Xiaomi all conduct distillation by secretly forwarding user queries to Claude and training on the results. An obvious-in-retrospect consequence of that strategy is that large amounts of confidential and/or classified Chinese data were sent to a US company, in violation of a number of Chinese laws. Oops.
The MIT report on AI and education
MIT’s report on AI and education has received considerable press in the last week, especially for popularizing the concept of “cognitive surrender”.
I don’t envy the authors their task: the college system was already badly out of sync with the modern world, and restructuring it for the AI era seems like an almost impossible task. But the report makes it painfully obvious that colleges are nowhere near ready for the disruptions that are already here, let alone those that are guaranteed to arrive over the next few years.
Side interests
Claude investigates the Voynich Manuscript
Delightful. Ethan Mollick asked Claude to investigate the Voynich Manuscript and make a movie about what it learned.
Math
AI use in math research
All of a sudden, it’s everywhere. Epoch finds that AI use in math papers is skyrocketing:
What does it mean to be a mathematician?
Last week brought us an open letter from 25 Fields Medalists warning about misalignment between AI companies and the mathematical community. This week, two prominent mathematicians respond.
Tim Gowers didn’t sign the letter even though he agreed with much of it, arguing it takes too narrow a view of why we do mathematics:
there is a spectrum of attitudes in mathematics to the relationship between problem-solving and conceptual understanding. At one end of the spectrum you have mathematicians who are primarily motivated by the wish to solve problems, who see conceptual understanding as a very important means to that end. At the other you have mathematicians who are primarily motivated by the wish to attain conceptual understanding, who see problem-solving as a very important means to that end.
I think that there are mathematicians all across the spectrum, and that that is a good thing
Daniel Litt explores what mathematics looks like in a world where “AI systems that are robustly superhuman at most or all aspects of mathematics will be here soon”. The way forward, he argues, must focus on what AI can’t do:
Broadly speaking I think we should focus on rewarding skill in the parts of our jobs that cannot be automated: the internal (e.g. understanding mathematics) and social-relational parts, and operationalizations that hew as closely to those aspects of the profession as possible. For example, talks and sustained mathematical discussion now demonstrate understanding much better than papers.
The math community is speedrunning some of the grief and disorientation that programmers have been navigating over the last year. What do you do when a machine is suddenly superhumanly good at the hard, prestigious profession you’ve dedicated your life to?
In an AGI world, Litt and Gowers are on the right track: focus on the high-level judgment that AI lacks, and on the social-relational parts. Perhaps more than ever, there’s a need for humans who really understand the profession.
But in an ASI world, the questions are much harder. I’m afraid that “the parts of our jobs that cannot be automated” will be the null set.
I appreciate the effort and thoughtfulness the math community is bringing to a challenge that we’ll all face before too long. I’m confident we can flourish and find purpose alongside superhuman AI, but that path is far from clear.