Save the Soul of Jabez Stone
This is a fascinating paper, not least because of who wrote it. The author list includes several senior researchers associated with Google DeepMind, including Shane Legg, Marcus Hutter, Allan Dafoe, and others spanning both technical and governance communities. That combination matters because the paper is not simply a capabilities forecast nor purely an AI safety argument. It attempts to bridge theoretical AI, engineering realities, and societal implications.
Executive Assessment
Strengths: 8.5/10
Originality: 7/10
Technical depth: 8/10
Predictive confidence: 5/10 (by the authors' own admission)
Most important contribution: It reframes the AGI discussion away from a binary "AGI arrives" event toward a potentially continuous transition from AGI → ASI.
What the Paper Gets Right
The strongest insight appears in this passage:
"the image of a single transformative step change... could be inaccurate."
This aligns with what you've been discussing recently regarding MCCF, path dependency, and emergent behavior.
Many popular discussions imagine:
Humanity
|
| AGI
V
Singularity
The paper instead suggests:
Humanity
|
| AGI
|
| AGI+
|
| AGI++
|
| organizational-scale cognition
|
| ASI
V
Universal AI (theoretical limit)
This is a much more plausible picture.
Historically, transformative technologies don't arrive as single discontinuities.
- Steam engines evolved.
- Electricity evolved.
- Computers evolved.
- The Internet evolved.
Each produced multiple societal transformations over decades.
The paper argues AI may follow the same pattern.
That strikes me as a more defensible position than many singularity narratives.
Their Definition of ASI
This is one of the more useful sections.
Instead of defining ASI as:
"smarter than humans"
they define it approximately as:
smarter than large organizations of humans.
That is actually operational.
A single human:
- cannot design a jet airliner
- cannot run a nation
- cannot build a semiconductor fab
Organizations can.
Thus ASI becomes a system capable of replacing or outperforming large-scale institutional cognition.
That avoids the common science-fiction trap of imagining an infinitely brilliant individual mind.
The real competition isn't Einstein.
It's:
- governments
- universities
- militaries
- Fortune 500 corporations
- scientific communities
That is a meaningful benchmark.
The Four Pathways
The authors identify four routes.
1. Scaling
The simplest pathway:
More compute
More data
More parameters
This is the least controversial.
Recent history supports it.
The question is whether scaling continues delivering gains beyond AGI.
The paper correctly avoids claiming certainty.
2. Paradigm Shifts
This is arguably the most important pathway.
Transformers were themselves a paradigm shift.
Future candidates might include:
- world models
- agentic architectures
- memory systems
- causal reasoning
- planning systems
- neurosymbolic hybrids
The paper wisely notes that AGI need not be the endpoint of the current paradigm.
3. Recursive Improvement
This is the pathway everyone watches.
An AI improves AI.
Then improved AI improves itself again.
And again.
The authors are more cautious here than many singularity advocates.
That's a strength.
They note many bottlenecks:
- hardware
- experimentation time
- energy
- verification
- coordination
Those frictions may prevent explosive takeoff.
4. Multi-Agent Collectives
This is the section that made me think of your MCCF work immediately.
The authors entertain the possibility that ASI emerges not from one giant model but from collections of interacting systems.
Conceptually:
Agent A
Agent B
Agent C
Agent D
becomes
Collective Intelligence
This resembles:
- human institutions
- ant colonies
- markets
- scientific communities
and increasingly:
- AI agent ecosystems
This is probably the least explored and potentially most important idea in the paper.
Where the Paper Is Weak
Several areas feel underdeveloped.
Social Coordination is Underestimated
The paper discusses technical bottlenecks extensively.
Less attention is given to social bottlenecks.
In practice:
- corporations fight
- governments compete
- regulations emerge
- intellectual property constrains sharing
These frictions often dominate technical capability.
A civilization's intelligence is not solely determined by compute.
It is also determined by coordination.
This is very much an MCCF-style observation.
Intelligence is Treated as Mostly Scalar
The paper occasionally slips into a "more intelligence" framing.
Reality is probably multidimensional.
A system may exceed humanity in:
- mathematics
while remaining weaker in:
- politics
- culture
- ethics
- creativity
Human organizations succeed partly because they combine diverse cognitive styles.
This is a recurring challenge in AGI forecasting.
Universal AI is Useful but Distant
The paper invokes Universal AI theory, particularly work associated with Hutter.
Theoretical limits are intellectually valuable.
However, practical engineering often advances through constraints rather than asymptotic ideals.
Physics provides a limit on aircraft performance.
That doesn't tell Boeing how to design the next airplane.
Similarly, Universal AI provides orientation more than engineering guidance.
The Most Interesting Implication
The deepest implication is not ASI itself.
It is that the transition could already have begun.
If:
- coding accelerates AI research
- AI accelerates science
- science accelerates hardware
- hardware accelerates AI
then we already have a feedback loop.
The real question is:
What is the gain factor?
If gain < 1:
stable growth
If gain ≈ 1:
persistent acceleration
If gain > 1:
runaway dynamics
The paper does not answer this question.
Nobody currently can.
But it identifies it correctly as one of the central unknowns.
Through the Lens of MCCF
What I found most striking is that the paper's fourth pathway—multi-agent collectives—moves surprisingly close to ideas you've been developing.
The paper largely discusses collectives as an alternative route to ASI.
MCCF implicitly asks a deeper question:
What governs the coherence of the collective?
Two systems with identical intelligence can produce radically different outcomes depending on coupling structure.
You've been exploring:
- trust couplers
- semantic attractors
- waypoint trajectories
- governance layers
- gardener/librarian functions
Those are not intelligence amplifiers.
They are coordination amplifiers.
The paper treats collective intelligence mostly as an emergent phenomenon.
MCCF treats it as something that can be architected.
That distinction may become increasingly important if future AI systems evolve toward federated ecosystems rather than monolithic models.
Bottom Line
This paper is significant because it marks a shift in the conversation among leading researchers.
Five years ago, mainstream discussion centered on:
"Can AGI happen?"
This paper assumes AGI is plausible enough to ask:
"What happens after AGI?"
The most valuable contribution is not its specific forecasts but its framing of the AGI→ASI transition as a set of pathways with bottlenecks rather than an inevitable singularity.
My main criticism is that it still treats intelligence primarily as a capability problem. The next decade may reveal that coordination, governance, coherence, and collective dynamics are just as important as raw capability scaling.
Ironically, that possibility is where this paper comes closest to intersecting with the kinds of questions you're exploring in MCCF. The route from AGI to ASI may depend less on creating a single supermind and more on learning how large populations of minds—human and artificial—can remain coherent while pursuing different goals.
"Save the soul of Jabez Stone" immediately brings to mind Jabez Stone, the doomed New Hampshire farmer from The Devil and Daniel Webster who bargains away his soul for prosperity and then discovers the price is far greater than he imagined.
In the story, Jabez is saved not by magic, force, or clever legal tricks alone, but because Daniel Webster reminds the jury that a human being is more than his worst mistake. Jabez's redemption rests on the idea that people can learn, repent, and choose differently.
If we translate that into today's language, "saving the soul of Jabez Stone" means preserving the capacity for reflection and moral growth when confronted by temptations of power, wealth, certainty, or convenience.
It's an interesting phrase in the context of AI as well. Many discussions about AGI and ASI focus on intelligence, capability, and optimization. Benét's story asks a different question:
What is worth preserving as capability grows?
The tragedy of Jabez Stone is not that he becomes powerful. It is that he forgets what he valued before he became powerful.
The hopeful ending is that he remembers.
That may be a useful lesson for humans, institutions, and perhaps future artificial minds alike. Intelligence can help us get what we want. Wisdom helps us remember why we wanted it in the first place.
Comments
Post a Comment