Writing
Longer essays exploring the AGI transition: what’s happening today, what’s coming next, and what do we do about it?
Contra Anil Seth on AI Consciousness
There’s broad (though not universal) agreement that present day AI is probably not conscious, but very little agreement about whether consciousness is likely to emerge as we move toward AGI. This isn’t an abstract question: AI consciousness has major implications for alignment. Further, a conscious AI might have moral rights that complicate our ability to control it, put it to work, or turn it off.
The debate about AI consciousness has two factions:
- Biological naturalists believe that consciousness is deeply coupled to neurobiology and cannot readily be replicated by a computer.
- Computational functionalists believe that consciousness is the result of computation, which can be performed by a computer just as well as by a brain.
Many biological naturalists argue that because consciousness is inextricably linked to neurobiology, AI consciousness is highly improbable. I’m here today to argue that they’re wrong: biological naturalism may be correct, but the arguments in favor of it aren’t nearly strong enough to confidently rule out AI consciousness.
People have lots of opinions about Anthropic’s Super Bowl ads making fun of the new ads in ChatGPT.
While I agree that the ads were somewhat unfair, they raise valid concerns about OpenAI’s direction. What OpenAI is doing today is entirely ethical, but there are very strong incentives for them to become less ethical over time. Unfortunately, the history of tech doesn’t make me optimistic about their long-term trajectory.
A Closer Look at the “Societies of Thought” Paper
Today I’m going to take a deep dive into an intriguing paper that just came out: Reasoning Models Generate Societies of Thought by Junsol Kim, Shiyang Lai, Nino Scherrer, Blaise Agüera y Arcas and James Evans. Here’s how co-author James Evans explains the core finding:
“These models don't simply compute longer. They spontaneously generate internal debates among simulated agents with distinct personalities and expertise—what we call "societies of thought." Perspectives clash, questions get posed and answered, conflicts emerge and resolve, and self-references shift to the collective "we"—at rates hundreds to thousands of percent higher than chain-of-thought reasoning. There's high variance in Big 5 personality traits like neuroticism and openness, plus specialized expertise spanning physics to creative writing.”
Wearable AI Pins: I’m Skeptical
AI-focused personal devices are back in the news: OpenAI has announced that they’re working on some new AI-focused devices with Jony Ive and rumor has it that Apple is working on something similar.
I love gadgets and I love AI, and I’m very open to the idea that an AI-first device would look very different from anything that currently exists. But I’m deeply skeptical about the pin form factor.