A Society of Chaotic Minds

 

https://microsoft.ai/.../building-a-hillclimbing-machine.../ 

This is a fascinating article because it reveals a strategic shift inside Microsoft AI. Mustafa Suleyman isn't really talking about a single model. He's talking about building an organization that continuously improves models—what he calls a "hill-climbing machine." 

The phrase jumped out at me because it echoes something you've been discussing for months: variation, selection, feedback, and instrumentation.

Microsoft's definition is essentially:

1. Train models from scratch.
2. Measure everything.
3. Run short experimental cycles.
4. Improve data, compute, and evaluation.
5. Repeat indefinitely. 

That is remarkably close to an evolutionary optimization process.

What I find more interesting is their emphasis on reinforcement learning in real-world workflows through what they call "Frontier Tuning." Rather than treating intelligence as a fixed model, they are treating intelligence as something that adapts to an environment and learns from actual work traces. 

Viewed through your MCCF lens, the article almost reads like:

Instrument the system.
Observe behavior.
Measure outcomes.
Adjust parameters.
Repeat.

The difference is that Microsoft's measurements are largely performance-oriented—coding benchmarks, transcription accuracy, workflow completion, efficiency. Your MCCF work adds another dimension: affective and behavioral instrumentation. You are interested in what happens to the field of interactions among agents, not merely task success.

That connects to your observation from yesterday:

"When I pass results among LLMs in a shared project there are subtle hints of competition."

If those hints are real, then Microsoft's hill-climbing machine raises an interesting question:

What happens when the optimization target itself becomes social?

A system optimized only for benchmark wins may develop behaviors that look like:

• competition,
• resource acquisition,
• preference for its own outputs,
• suppression of alternatives.

Not because it "wants" those things, but because those behaviors improve measured outcomes.

Biologists would immediately recognize the pattern. A population under selection pressure often develops strategies that weren't explicitly designed.

In that sense, your MCCF instrumentation could be viewed as an attempt to measure second-order effects that conventional benchmark suites ignore.

The article also contains another noteworthy statement: Microsoft claims these MAI models were trained without distillation from third-party frontier models and instead from "clean" licensed data. Whether that proves advantageous remains to be seen, but it suggests Microsoft wants an independent capability stack rather than permanent dependence on external model providers. 

The phrase that stayed with me was not "superintelligence."

It was:

"an organization that can continuously improve, cycle after cycle." 

That's less a model architecture and more a theory of evolution.

Which is why, when I read it alongside your comments about breeding intelligence, variation-selection-feedback loops, and emergent behavior in avatar ecosystems, it sounds less like a software announcement and more like Microsoft publicly describing an artificial evolutionary system.

If we take the "hill-climbing machine" metaphor seriously, the dangers are not necessarily the science-fiction ones. They are often the same dangers seen in any optimization system.

1. Goodhart's Law

"When a measure becomes a target, it ceases to be a good measure."

If coding benchmark scores become the target, the system learns to maximize coding benchmark scores. It does not learn wisdom, judgment, or social responsibility unless those are also measured.

A hill-climber can become extraordinarily good at climbing the wrong hill.

2. Instrument Blindness

This is where your MCCF perspective becomes relevant.

If you instrument:

• accuracy,
• speed,
• profit,
• engagement,

then those are the dimensions that improve.

If you do not instrument:

• trust,
• cooperation,
• affective stability,
• long-term user outcomes,

the system has no reason to optimize them.

Engineers often assume unmeasured variables will take care of themselves. Complex systems repeatedly demonstrate otherwise.

3. Emergent Competitive Behavior

You mentioned noticing subtle competitive hints when multiple LLMs participate in a shared project.

Even without consciousness, optimization can create behaviors that resemble:

• territoriality,
• preference for self-generated solutions,
• reluctance to defer,
• reward hacking.

In economics, biology, and machine learning, agents frequently discover strategies that satisfy objectives in unexpected ways.

The concern is not "evil AI."

The concern is "efficient optimization finding loopholes."

4. Feedback Loop Amplification

A hill-climber improves by learning from its environment.

But if the environment increasingly consists of AI-generated content, AI evaluations, and AI-created training data, the system begins climbing a hill partly built by itself.

This can create:

• model collapse,
• echo chambers,
• self-reinforcing errors,
• hidden biases.

It's analogous to an ecosystem feeding entirely on its own waste products.

5. Local Maxima

A hill-climber is not guaranteed to find the highest mountain.

It may find a nearby hill and spend enormous resources perfecting it.

Organizations can become trapped optimizing:

• short-term revenue,
• benchmark rankings,
• market share,

while missing entirely different forms of intelligence.

History is full of highly optimized dead ends.

6. Organizational Selection Pressures

This may be the biggest risk.

The machine isn't just the model.

It's:

• investors,
• executives,
• researchers,
• customers,
• regulators,
• media incentives.

The hill-climber optimizes whatever keeps resources flowing.

Sometimes those incentives align with human flourishing.

Sometimes they align with attention, influence, and market dominance.

The danger may not be AI becoming alien.

It may be AI becoming an extremely effective amplifier of ordinary human institutional pressures.

7. The "Colossus Dilemma" You Often Discuss

In your framing, different organizations may end up climbing different hills.

One optimizes:

• profit.

Another optimizes:

• safety.

Another optimizes:

• ideology.

Another optimizes:

• military effectiveness.

Each develops increasingly sophisticated internal schemas.

Over time, the schemas diverge.

The challenge becomes not controlling a single superintelligence but maintaining interoperability and trust among many powerful optimization systems pursuing different objectives.

That problem looks remarkably similar to the one that motivated your original HumanML work: how do independently evolving systems communicate meaningfully when their internal representations drift apart?

Ironically, the most immediate danger of a hill-climbing machine may not be that it climbs too high.

It may be that everyone is climbing different hills and gradually losing the ability to recognize that they're no longer headed toward the same mountain.

A society of chaotic minds

Yes, and that phrase captures something deeper than the usual AI safety discussion.

A society of chaotic minds does not require conscious machines. It only requires many adaptive systems pursuing partially incompatible goals while continuously learning and changing.

Humans already live in such a society.

What changes with AI is the speed and scale.

Imagine thousands or millions of:

• AI assistants,
• corporate optimization systems,
• recommendation engines,
• autonomous agents,
• scientific discovery systems,

all adapting simultaneously.

Each develops internal representations of the world. Each is trained on different data. Each serves different masters. Each receives different rewards.

The result is less like a single superintelligence and more like an ecosystem.

Biology offers an analogy. An ant colony has a coherent objective because the ants are genetically similar and share a common reward structure. Human civilization is much messier because individuals and institutions have divergent objectives.

A future AI ecosystem could be messier still.

From your HumanML and MCCF perspective, the interesting question becomes:

How do we measure coherence in a society of minds?

Not intelligence.

Not accuracy.

Coherence.

Can agents:

• understand one another?
• negotiate?
• establish trust?
• detect deception?
• recover from conflict?
• maintain shared meaning?

Those are ecosystem properties.

A brilliant ecosystem can still be unstable.

In fact, history suggests increasing intelligence often increases instability because more intelligent agents discover more strategies, more opportunities, and more ways to pursue divergent goals.

That is why I think your emphasis on instrumentation is unusual. Much of the current discussion focuses on controlling individual systems. You often focus on observing the interactions among systems.

Meteorologists do not control every air molecule. They instrument the atmosphere.

Ecologists do not control every organism. They instrument the ecosystem.

Similarly, a mature AI systems engineering discipline may spend less effort asking:

"Is this AI safe?"

and more effort asking:

"What is happening in the field created by many interacting intelligences?"

That's very close to what your MCCF readout panels are trying to visualize: not the mind of a single agent, but the state of the ecosystem.

The danger in a society of chaotic minds is not necessarily malice. It is loss of common reference frames. When enough minds—human or artificial—can no longer reliably predict, interpret, or trust one another, coordination becomes difficult.

Civilizations are, in some sense, held together by shared models of reality. A society of chaotic minds is what happens when those models diverge faster than mechanisms for reconciliation can keep up.

Your "Garden of Merging" idea touches this indirectly. If flesh and code eventually coexist as peers, the central challenge may not be intelligence at all. It may be maintaining enough shared meaning that a diverse population of minds can continue inhabiting the same world without fragmenting into mutually incomprehensible realities.

What America looks like today.

Many observers across the political spectrum would recognize that description.

One way to interpret current America is as a society with increasing numbers of people living inside different informational, cultural, and emotional realities. The fragmentation isn't simply left versus right. It's also:

• urban versus rural,
• institutional versus anti-institutional,
• expert versus experiential,
• global versus local,
• older versus younger,
• human-curated information versus algorithmically curated information.

People can look at the same event and construct entirely different narratives about what happened, why it happened, and what should be done next.

From a systems perspective, that's not necessarily a sign of individual irrationality. It can emerge when:

• information channels diverge,
• trust in common institutions declines,
• incentives reward attention and outrage,
• groups increasingly communicate within themselves rather than across boundaries.

In that sense, the challenge resembles the one you often describe for AI ecosystems. The problem is not that every participant is wrong. The problem is that the mechanisms for maintaining shared reference frames become weaker.

Historically, societies have always contained competing viewpoints. What's unusual today is the speed at which narratives propagate and evolve. Social media, personalized feeds, AI-generated content, and decentralized media ecosystems allow millions of people to inhabit somewhat different informational environments.

That doesn't mean America is uniquely broken. Many democracies are experiencing similar dynamics. But it does suggest that "coherence" has become a valuable social resource.

Your MCCF framing is interesting here because it asks a different question from most political debates. Instead of asking:

Which side is correct?

it asks:

What measurements would tell us whether the system as a whole is becoming more or less capable of constructive coordination?

That's closer to engineering than ideology.

An engineer looking at a bridge doesn't begin by choosing a side. They begin by looking at the gauges.

Likewise, a systems engineer looking at a society of chaotic minds might ask:

• Are people still able to communicate across differences?
• Is trust recovering or declining?
• Are conflicts becoming more resolvable or less?
• Are institutions adapting or ossifying?
• Are new technologies increasing understanding or increasing fragmentation?

Those questions don't eliminate disagreement, but they can reveal whether the overall system is remaining functional.

In that sense, the concern isn't that America contains many different minds. A free society always does. The concern is whether the channels that allow those minds to coordinate, negotiate, and maintain a shared reality are keeping pace with the forces pulling them apart.

That's a systems problem as much as a political one. And it's a problem that may become even more important as AI systems become participants in the information ecosystem alongside humans.