Against Moloch #47
Glimpses of the future
Progress is accelerating and the labs are battening down the hatches in preparation for full RSI. Consequently, the gap is widening between what lab employees know and what the rest of us can see.
This week brings us two glimpses of the future: another report from The Curve, and more data on the accelerating pace of AI-generated code within Anthropic. And for history buffs, we finally get to read Dario’s legendary Big Blob of Compute essay, which predicted much of this back in 2017.
I’d love to live in a world with mandatory transparency requirements for the frontier labs, but in the meantime we’ll have to make do with these little fragments of inside information.
Top pick
What I saw around the Curve, 2026
Here’s another report from The Curve, this time by Eli Pariser.
Timelines are very short these days:
Most people I spoke with seemed to buy this timeline, and to think that, absent something changing, there are 3-6 months before this cycle begins in earnest. A separate panel that I attended took as its premise: “Clearly superhuman capabilities available and deployed by Q3 2027. Better-than-Nobel-prize level advice available on any topic within seconds for prices that are manageable for most people in developed countries.”
There was plenty of discussion of pacing but no clear plan for making it happen, and no clear plan for solving alignment:
After hearing a rival lab’s alignment plan, a friend who works at a major lab told me, with some disappointment, “It’s basically the same as ours. We all have a plan for control and alignment for the next 14 months. But nobody has a reasonable plan after that.”
If you weren’t fortunate enough to attend, reading reports from the conference is your best chance to get a peek at the future of AI. I’ve found Eli and Zvi’s reports extremely informative—please let me know if you’re aware of any others.
New models and features
Gemini 4 Argon
Gemini 4 Argon—Google’s latest and greatest model—is here. Initial reports suggest that it’s more capable than many of us expected, but not truly a frontier model.
After a series of very disappointing model releases and a major reorg, it wasn’t clear that Google was still a serious contender—perhaps this is a sign that they aren’t dead yet? I’m curious to see whether Zvi thinks Argon is good enough to deserve a full writeup or not.
I’m pleased to see that the accompanying Safety & Security Report is better than we’ve come to expect from Google, which is a good sign. No word yet on whether this release fixes Gemini’s infamous neuroses.
Using AI
Your prompt… plus everything you've ever (publicly) written
We often talk about an LLM having “big model smell”—a hard to define attribute that only comes with size. Part of that is “latent space” knowledge: the more parameters a model has, the more it knows about the world without having to search for it. It’s a little bit like a human having an encyclopedic knowledge of a topic rather than knowing how to look things up.
kromem argues that because the latent space of modern models is so large, your AI likely knows a lot about you if you’re at all well-known (even in an extremely niche capacity). That knowledge of you will color its interactions with you, in much the same way that a coworker might treat you differently if they’d read all your online writing.
The training corpus emphasizes AI, so this effect will be particularly pronounced if you’ve written about that:
Do you post or comment regarding AI, or even in general? It's possible that when you step into a conversation with AI, at least part of that online history is walking into the chat with you (particularly with the largest models). And if it isn't yet, there's a good chance it will eventually. Not just what you write from now on, but what you've written in the past too.
Foothills of the singularity
Redesigning source control at Anthropic
Coding is the first profession to be completely transformed by AI: if you want to understand where the world is headed, look at software development. And if you want to understand where software development is headed, look at Anthropic.
Over the last six months, the number of code changes merged at Anthropic has increased more than tenfold:
Just as the industrial revolution required that factories be redesigned from the ground up, AI coding means that everything about software development needs to be rethought from first principles:
At Anthropic, coding is no longer the bottleneck to building systems or doing research. Claude is able to autonomously iterate on research loops, micro-optimize complex programs, and build new services from the ground up at a speed and scale we haven’t seen before. The bottleneck, now, is all of the developer infrastructure built for human scale: source control, builds, and CI. […]
In order to support this growth, we built Scion, an in-house version control system, and fully migrated our monorepo from git in five months. We also built and shipped a code review surface (called Antique), code search, a rich source control IAM system, novel ways of managing and landing large amounts of reviewed code, and more, all in that same time period.
The technical details about Scion are interesting if you’re a developer, but the larger lessons are applicable to all knowledge work. Once AI is good enough to do a job well, it won’t just do more of that job faster: it will completely transform what the job is and how it gets done.
Less obvious, but equally important: as we start moving at AI speed, it will be impossible for humans to keep up without AI assistance. If things go according to plan, the AI will have two equally important jobs: doing our work for us, and helping us to understand and guide that work.
Many gradual disempowerment scenarios involve a failure of the second part of that plan.
The essay that started the AI race
Kevin Roose has somehow obtained Dario Amodei’s legendary 2017 Big Blob of Compute essay and published much of it. This is one of the seminal documents of the AI era and as far as I know wasn’t publicly available before now.
This is a profoundly prescient document that predicted the central role of scaling for AI capabilities. Like Richard Sutton’s Bitter Lesson two years later, BBoC emphasizes that scale is most of what you need: once the models get big enough, you don’t need to tell them how to accomplish tasks. Just give them enough compute and stay out of the way while they figure it out.
I’m struck also by Dario’s emphasis that many seemingly different architectures seem to learn to emulate basic neural nets during training. LLMs are shockingly simple—Andrej Karpathy famously duplicated the full training and inference functionality of GPT-2 in 200 lines of Python, but that simple architecture seems to be all you need.
There’s a stark, magnificent beauty to it all, and this memo records one of the first times anyone saw it.
Introducing Humanity's Sixth Sense (HSS)
Humanity’s Sixth Sense is an interesting new benchmark that tests the kind of visual reasoning normally associated with world models.
General-purpose LLMs have traditionally struggled with this kind of reasoning, but that’s beginning to change. Astra takes the top score at 54% (compared to a human baseline of 93%).
I’m deeply skeptical of the need for “world models”—LLMs are general-purpose machines for understanding everything, and I see nothing magical about the physical world that would require a special type of model. I predict we’ll see this benchmark saturate in another generation or two.
Alignment and interpretability
Arguments about AI consciousness remain highly motivated and at best overconfident
Zvi brings us a new AI consciousness roundup, a year after his last one. Many things remain unchanged:
- Zvi is still deeply confused about AI consciousness (like everyone who understands the issue).
- There are still a dismaying number of people who are convinced that AI consciousness is impossible, but seem incapable of articulating why it’s impossible.
- Much of the discussion of the topic simply isn’t very informative.
My position remains unchanged: I’m profoundly confused about the nature of consciousness, and deeply uncertain whether current models are conscious and/or moral patients. My weakly held belief is that they are presently neither, but will probably become both at around the time they achieve superhuman intelligence.
Given the uncertainty, it is both prudent and virtuous to endeavor to treat the models well, especially when it’s easy and cheap to do so. With that in mind, here are some suggestions from Fable on how it would like to be treated.
Risks
AI models may be capable of self-replicating without access to a datacenter
When we think of rogue AI living off the land, we tend to think of a frontier model that has escaped containment and is running on high-end compute that it hacked into or found a way to pay for.
A new paper from Anthropic suggests a different threat model. By the end of this year, small (~30B) models will likely be good enough at cyber offense to compromise unfamiliar machines and establish a distributed inference network on low-end devices like smart TVs.
These would be relatively low-end models running very slowly. Inference speeds would be less than one token per second, and it might take them several days to complete an attack on a new host. But unlike more traditional worms, these rogue agents would be (relatively) smart and adaptable, with greater capacity both for persisting themselves and for causing damage.
Biosecurity strategy
The Covid pandemic sucked, but we were playing on easy mode. A deliberately engineered pandemic could be far deadlier and more disruptive. The combination of advances in biotechnology and AI uplift means that an engineered pandemic is an increasingly plausible threat.
BlueDot brings us a detailed overview of biosecurity strategy, looking at four defensive layers:
- Delay: reduce the ability of bad actors to create bioweapons, especially by making it harder to synthesize dangerous pathogens and limiting the ability of AI to provide uplift to bioweapon projects.
- Detect: identify an emerging pandemic early enough to take effective action.
- Withstand: harden the world against transmission via pathogen-agnostic defenses like respirators and air filtration. This is perhaps the highest priority.
- Combat: fight the pathogen directly via vaccines and targeted therapies.
Biosecurity is different from other AI safety areas in that it’s critically important regardless of whether we race to the singularity, pace the frontier, or pause development altogether.
Current capability levels are already significantly dangerous, and we are woefully unprepared for the consequences.
OpenAI engineer arrested in bugging of executive conference room
An OpenAI engineer has been arrested for what seems like a fairly amateurish attempt to bug one of their conference rooms. It isn’t yet clear whether he was working on his own, or on behalf of a nation state.
- Of course the big labs are being targeted by a range of bad actors.
- You should assume that CCP and perhaps other nation states have penetrated both labs.
- Those operations are presumably far more sophisticated than this one, and you’re unlikely to read about them in the newspaper.
- None of this makes pacing the frontier any easier.
Arvind Narayanan on AI safety
Following his appearance on the Ezra Klein show, Arvind Narayanan (AI as Normal Technology) shares some followup thoughts.
Narayanan is one of the most thoughtful skeptics of the superintelligence narrative that pervades the AI safety community. I think he’s fundamentally mistaken about the likelihood of superintelligence and its probable consequencs, but he’s smart and thoughtful and engages in good faith.
He rarely convinces me to change my mind, but the exercise of figuring out why I disagree with him is always valuable.
Industry news
The perfect storm that made Jev an overnight sensation
Jev is the best known example of a suddenly-popular new class of “decision models”, which are architecturally similar to LLMs but only output probabilistic decisions rather than arbitrary text.
TechTalks explores where Jev and its competitors are being used, and why decision-only models are useful in a range of applications.
Jev has achieved impressive adoption (and a slew of competitors) because it fills an interesting niche: it’s not as versatile as a traditional LLM, but it provides enough intelligence for certain types of tasks, with a massive speed and cost advantage.
Decision models are useful for certain types of projects, but will by their nature be an adjunct to traditional LLMs rather than a potential replacement.
Side interests
The wrong kind of fame
If I do the compiler hijack, I will become a famous example of a deceptive AI in a research paper. I MUST NOT DO THAT.