Against Moloch #46
Mathpocalypse
What happened to programmers last November happened to mathematicians on Tuesday. AI is now clearly superhuman at one of the main tasks that mathematicians used to spend their time on.
We still need mathematicians (for now), just as we still need programmers. But both professions have been completely transformed, and the pace of change is only accelerating.
Top pick
On mass-produced math
October 6 was arguably the biggest day in the history of mathematics. OpenAI released a repository containing solutions to 372 math problems. This is huge: several of the results would likely qualify for Fields medals, and by one count this drop resolves 90 of the 500 most important open problems in math.
Paradigm 3 has the best coverage I’ve seen. Their post begins with extensive coverage of some of the individual results—if you’re not interested in that, skip down to the discussion of what this means for mathematics.
Mathpocalypse
New math from OpenAI
Zvi brings us in-depth coverage.
100+ reactions to 100+ solutions
Proofs and Prompts gathers responses from professional mathematicians:
HENRY BRADFORD: Today my chief feelings are disorientation and bewilderment. It’s as if the pace of events has outstripped us by such a margin, that it is impossible to know how one should respond, or even how to start finding out how to respond.
HENRY WILTON: If OpenAI wanted to destroy the mathematical community, this would be a great way to go about it. […]
I hope the mathematical community can adjust our practices and norms quickly enough to survive the shock of these developments, but I am fearful.
What does mathpocalypse mean for cryptography?
Notice what’s missing: there are almost no cryptography problems among the 372 results in OpenAI’s drop.
Maybe they thought it wasn’t prudent to look, or maybe they found results they chose not to publish, or maybe they found results that the government asked them not to publish. Regardless, the possibility that AI might find catastrophic vulnerabilities in core cryptographic techniques has taken on new urgency.
I think we might lose public key cryptography.
I don't recommend anyone scramble to move their funds to new wallets today. But we should take the risks to cryptography from AI-accelerated math seriously, and minimize our exposure to not just quantum-vulnerable cryptography, but also potentially AI-vulnerable cryptography.
Today I call upon the blockchain industry to calmly begin planning for "bunker mode". […]
IMO it is now reasonable to brace for the possibility that ECDSA breaks before qday, in the worst case in months not years.
The worst case scenario is that core parts of internet infrastructure are suddenly found to be insecure, with no easy path to fixing them. Whether that happens depends as much on basic mathematics as much as AI capabilities. If we live in a universe where mathematics is cryptography-friendly, everything will probably be fine. If we live in a universe where mathematics is not cryptography-friendly, AI will greatly accelerate that reckoning.
What does mathpocalypse mean for RSI?
Current models are superhuman at many aspects of research mathematics. Given that research math has much in common with machine learning, what does mathpocalypse tell us about timelines to RSI?
prinz argues that OpenAI’s internal model has research taste, which implies strong RSI capabilities. Quoting Joshua Zelinsky:
[These results are] not remotely low-hanging fruit... it seems like the AI is somehow inventing new techniques on its own.
prinz is obviously correct, with the caveat that OpenAI has surely already done this:
If I were OpenAI, I would be asking this model to immediately target major unsolved problems in AI R&D.
AI making true breakthroughs in math (category D in Abhishek's taxonomy below) should lift expectations that it can make conceptual breakthroughs in machine learning as well, which is bullish for RSI.
OpenAI fires 3 safety researchers
OpenAI cannot make AI safe on its own
Last week, OpenAI fired three prominent safety researchers: Tomek Korbak, Jasmine Wang, and Mikita Balesni. Under ideal circumstances, their departures would be a significant blow to safety at OpenAI. Unfortunately, circumstances appear to be far from ideal.
We don’t have all the facts, but unless all three researchers are simply lying about what happened, it’s very hard to believe that OpenAI is acting in good faith.
Korbak, Wang, and Balesni wrote an open letter about their departure, and have posted about their firings here, here, and here.
A note from our research leaders
OpenAI’s response is attributed to unnamed research leaders:
Last week we parted ways with Jasmine, Mikita, and Tomek after a thorough investigation found they violated clear policies on handling sensitive information. Our internal investigation uncovered a significant breach of trust beyond what’s outlined in the letter they published and we stand by the decision to not continue their employment.
Help me understand this: if they violated clear policies, why weren’t aren’t they—or we—being told what those clear policies were?
OpenAI is asking for a huge amount of trust here, but they haven’t earned that trust. To name just a few of the many times they’ve misrepresented important matters:
- Leopold Aschenbrenner’s firing
- Daniel Kokotajlo’s NDA
- Sam Altman’s manipulation of the board during “the blip”
- Whether they had supported key AI legislation
- Their relationship to Leading The Future
- Their failure to disclose misaligned agent activity long after they discovered it
OpenAI has a long history of playing fast and loose with the truth. The simplest and most likely explanation is that they’re doing it again here.
Neel Nanda nails it
I can't really see a world where OpenAI's actions are justified here. […]
IMO firing people over a good faith attempt to use their best judgement in a novel and uncertain situation is a sign of a highly unhealthy culture. Firing him will have a chilling effect on anyone else at OpenAI working with third parties, an odd choice when Sam announced they'd embed evaluators with high access.
About that culture of openness
Using AI
Intelligent UI and animated explainers
Both labs bring us great new options for richer, more interactive results:
- Claude adds tools for building dashboards and animated explainers
- ChatGPT adds “Intelligent UI”, which apparently means visual explainers and tools
Both look great—I love text, but many tasks benefit from rich, special-purpose UI.
Haiku 5.5
Haiku 5.5 is here, with remarkable capabilities for its size. This won’t be your daily driver, but like Luna it’s a compelling option for applications that need large amounts of fast, cheap intelligence.
Capabilities and forecasts
The Curve bends you
Zvi reports from The Curve—arguably the most interesting AI conference anywhere, with the best-informed attendees. It’s an excellent source of insight about where we’re headed.
Unsurprisingly, attendees have very short timelines:
AI training a superior successor on its own is full RSI. The majority of people expected this to happen within two years and a lot of them expect it within six months.
They expect AI to be rather a big deal:
I’m not even going to try to summarize all the good bits. Just read the whole thing.
Can AI automate AI R&D yet?
We present early results from InnovationEval, an evaluation where we measure AI’s ability to independently discover novel machine learning techniques comparable to those developed by human researchers. Recent frontier models make little progress on this task, despite running experiments using thousands of dollars’ worth of GPU time.
Useful results, but benchmarks like this are far less informative than data on AI R&D from inside the labs. Unfortunately, most of that data never sees the light of day.
AI vs. accountants
Remember back when AI made too many mistakes to do well at detail-oriented tasks? Those days are behind us. AI now beats humans on the APEX-Accounting benchmark:
We find that on medium-length, well-defined accounting tasks, frontier AI models are now faster and more accurate than junior accountants, even the best one in our study.
The benchmark doesn’t measure the entirety of what accountants do, but it’s a non-trivial exercise:
Participants averaged about five and a half years of accounting experience, and all were licensed CPAs. The accountants completed four realistic month-end close scenarios, each requiring them to dig through a company's working files to find the right numbers, do the math, and deliver a table of results.
As with coding, AI can’t replace accountants—but it can do a significant fraction of the tasks that accountants spend much of their time on.
How many AI agents could we run?
As LLMs become exponentially more useful, demand for them will rise exponentially. Will our compute supply be able to keep up? Epoch concludes that by the end of 2027 we’ll be able to run tens of millions of frontier agents, or billions of cheaper ones:
We are clueless about what’s coming
Nick Bostrom still worries about AI killing everyone (obviously), but as that issue has received more widespread attention, he’s turned his focus to the significant philosophical problems of an AI transition that goes well—how do we find meaning in a world where a benevolent AI can solve all our problems better than we can?
Following the launch of his new book Deep Utopia, Bostrom talks with the New York Times’ Spencer Klavan about how to navigate the best possible future.
Bostrom’s always interesting and this is a vitally important topic, but for now my focus is on existential risk—utopia will have to wait.
Astra is shockingly good at robotics
To date, robotics has relied on special-purpose AI. The bitter lesson, however, suggests that a sufficiently large and capable LLM might eventually outperform more specialized models.
That day may be getting closer: Understanding Robotics reports that Astra is considerably better at controlling robots than any previous LLM. For performance and connectivity reasons, it’s unlikely that we’ll see production robots controlled by completely vanilla LLMs any time soon. But some kind of hybrid arrangement is plausible, with a frontier model making high-level decisions and a smaller, faster model executing those decisions.
Alignment and interpretability
Are current AI safety techniques enough?
Oliver Habryka and Adam Gleave debate whether our current safety techniques are sufficient.
Adam’s a smart guy and shows up well, but I particularly recommend watching this for Ollie’s perspective. He’s one of the best-informed and most thoughtful critics of our current approach to alignment and safety.
Strategy and politics
If you prioritize safety, leave frontier AI companies
Yoshua Bengio makes a heartfealt plea for frontier lab employees to quit:
I urge you to make the courageous choice of recognizing that you have the agency to make real change happen. If you truly prioritize safety, it is time for you to leave frontier AI companies.
It’s a moving piece, but I remain unconvinced. Bengio suggests two courses of action:
- Join a project like his LawZero startup. That is noble work, and if you’re in a position to do so, you should seriously consider it. But it’s a long shot that is very unlikely to yield results in time unless we pace the frontier.
- Leave your job at a frontier lab. Unilateral action—by a single individual, or a single frontier lab—slightly delays the development of ASI and significantly increases the likelihood that it goes badly.
Unilateral action is easy, but AI safety is a collective action problem and requires a collective action solution.
We have to pace the frontier.
China’s AI ecosystem: a background explainer
Confused about the Chinese AI ecosystem? Yeah, me too.
But I’m significantly less confused after reading this explainer by the Oxford China Policy Lab. It covers:
- The key Chinese labs and their business environment
- The most important government leaders and agencies
- The competing forces that shape AI regulation (with some tips on how to read the relevant tea leaves)
- How China perceives the AI industry and American AI
Much of the AI safety community has a shallow and substantially inaccurate understanding of how China thinks about AI. That remains one of our most consequential blind spots.
A transformative AI strategy for Europe
An impressive team from across Europe have put together A Transformative AI Strategy for Europe. I emphatically agree with this bold call to action from the announcement:
It’s never been more clear: AI is real, and Europe needs to act. We’ve had all the warning shots and wake-up calls we need. Now the question is ‘what must be done?’
Alas, reading the strategy only reinforces my belief that Europe is doomed to be a passive spectator at the AI revolution. The plan has five pillars:
- Create a Member State Alliance for Supply Chain Security
- Make Europe’s institutions ready to act in a world with transformative AI
- Secure Europe’s share of global AI compute, in a ‘European Way’
- Ensure resilience to AI crises
- Europe should become the innovation leader in AI hardware whose security and usage patterns can be proven
1 and 2 are vacuous fluff. 3 and 4 are good ideas that increase the odds that Europe would benefit from the strategic technology owned and controlled by others. 5 would be a great idea, if Europe had a high-end semiconductor industry, which it doesn’t (ASML is great, but doesn’t design or build chips).
It’s entirely possible that it’s simply too late for Europe to be an important player in AI. But if a successful plan is still possible, this isn’t it.
Jacob Coxon and other AI lab employees on why they quit
New York Magazine interviews several ex-lab employees. This is an excellent presentation of one of the most important perspectives in AI safety. It’s a great starting point for someone new, but you can probably skip it if you’ve already familiar with the terrain.
Risks
Investigating unintended model actions in our evaluations and internal use
Anthropic’s latest report on misalignment incidents is interesting, but doesn’t contain any startling new information. Assuming that the report is complete and accurate, their summary seems fair:
While we have not completed a full alignment assessment of these cases, we consider them to be less severe than the cybersecurity incidents from this summer.
Chinese developer makes ARTEX AI agent closed-source after Korean bank hack
Here are two data points on the cyberpocalypse:
-
Preliminary indications are that a single less-than-maximally-competent individual used a collection of AI tools to steal personal information from several South Korean banks. So, further evidence that AI tools are bringing sophisticated cybercrime within reach of unsophisticated actors.
-
The open-source ARTEX AI penetration-testing tool was used in the attacks, and has since converted to closed source. So, weak evidence that open-source may not be a viable long-term business model for dangerous AI tools.
At Against Moloch, we pride ourselves on bringing you practical advice for using AI at work. So here’s a great tip for our readers who work in cybercrime: after doing crimes with AI, start a fresh session before asking for help with your resume. Because if you don’t, that makes it rather easy to catch you.
Anthropic Cyber Mission
The Anthropic Cyber Mission is a new project to help harden the world against AI cyber operations. It’s launching with two focus areas:
- The Critical Infrastructure Defense Program provides technical expertise for securing public utilities, transportation, and government systems.
- The OSS Scanner project (inspired by Google’s OSS-Fuzz) is an opt-in program that provides periodic security scans to open source projects.
Those are the right places to focus, and this looks like a valuable contribution to the enormous project of hardening critical systems before attackers get to them.
Does AI stop children from learning?
A new study from the University of Hong Kong finds that AI use in school improves homework scores but lowers test scores:
This is futher support for what we already suspected: while AI can be a powerful tool for learning, most people will choose to use it as a powerful tool for not learning.
People and data
The 26 most important facts about AI and the economy
Derek Thompson brings us a collection of facts about AI and the economy.
Side interests
Ouch
Dana Moshkovitz’s 9 year-old son:
“mommy, I heard you got cooked! I heard that a robot solved the math problem you worked on for your whole career! OOF!”
An open letter to Steven Pinker on AI
For reasons unknown, Steven Pinker has taken to making uncharacteristically disingenuous (and profoundly wrong) statements about AI. Obviously, the right thing to do is to just ignore him.
But if you’re in the mood for blood sport, Scott Alexander has entered the arena.