AI, Designated Winners and Emergent Misalignment
Len: A bit Pollyanna. Human judgement is more complicated. The best ideas
do not always win unless they originate from “the right person “. We
once called that “the designated winner”.
The Human-Machine Knowledge Spiral, by Aaron Chatterji, Daniel Rock, and Eduard Talamàs.
The designated winner” is a much sharper lens than the sunny picture of collaborative knowledge creation.
The problem isn't simply that human judgment is complicated. It's that the social process deciding which ideas count is not the same process that determines which ideas are good.
A brilliant idea can lose because its author lacks status, because it threatens an incumbent's interests, because it arrives at the wrong time, or because the organization has already committed itself to another answer. Conversely, a mediocre idea can prevail because the right person sponsors it. Sometimes the designated winner is chosen before the competition has meaningfully begun.
That distinction matters enormously when we talk about human–AI collaboration.
What the paper gets right—and what it leaves out
Its central argument is useful: human expertise and machine capabilities can develop through iterative cycles of proposals, experiments, feedback, and learning. The NotCo example—AI proposes unusual plant-based milk formulations, people test them, and their judgments inform further proposals—is a concrete illustration.
But I think your objection exposes a deeper problem.
The paper describes how knowledge can be created without adequately confronting who has the authority to recognize it as knowledge.
Consider the process the authors describe:
Someone defines the ideal outcome.
A machine proposes possibilities.
People select which proposals to test.
Someone decides what counts as success.
The resulting judgments feed back into the system.
Every stage contains human judgment. And every stage offers an opportunity for the designated winner to influence the result.
The paper does acknowledge that leaders define what ought to be, while frontline workers understand what is. But that division of responsibility doesn't resolve the political problem. Who decides when leadership's ideal is wrong? What happens when the frontline expert's experience contradicts the executive's preferred narrative?
The pineapple-and-cabbage formulation might win a fair experimental test. But imagine an equally promising proposal from a junior researcher whose director dislikes them. It might never reach the testing stage. The feedback loop would then faithfully reinforce a decision that was never made on the merits.
That's the designated-winner problem.
The distinction I'd make
1. Can we generate good ideas?
AI can expand the search space, expose unfamiliar combinations, and help people discover possibilities they might otherwise miss.
2. Can we evaluate them fairly?
We need meaningful tests, competing interpretations, explicit criteria, and room to challenge established assumptions.
3. Will the organization accept the result?
Status, incentives, loyalties, institutional interests, and personal credibility can determine whether a good idea gets adopted.
The knowledge spiral primarily explains the first two as a productive cycle. The third is not an incidental complication. It can determine which ideas ever enter that cycle.
And AI introduces a further complication: if the organization encodes its existing judgments into models, workflows, and evaluation criteria, those judgments can be reproduced at scale. The result can look increasingly objective while retaining the original social bias.
What this means for MCCF
Your approach offers an interesting counterpoint, Len. You have working software, artifacts that can be examined, and behavior that can be tested. That doesn't eliminate social bias, but it gives collaborators something more substantial than authority or persuasion to argue over.
I'd put it this way:
The purpose of collaborative intelligence isn't merely to improve the production of ideas. It's to improve the conditions under which ideas can challenge one another and be judged on their merits.
Even that doesn't guarantee the best idea will win. People legitimately disagree about goals, aesthetics, acceptable risks, and what constitutes success. There is no universal test for every important judgment.
But we can make the process more honest: preserve competing proposals, make evaluation criteria visible, record dissent, distinguish evidence from authority, and allow results to be reproduced by people who weren't invested in the original decision.
The designated winner may still win. At least we can make it harder for the winner to rewrite the record as proof that they were right all along.
And there's one last irony in the paper. The authors argue that organizational knowledge depends on a shared context in which human and machine capabilities interact. I agree. But a shared context can enable collaboration, or it can enforce conformity. The difference depends on how power operates inside it.
That's the missing question I'd want the authors to address. Not whether the knowledge spiral can turn ideas into innovation, but whether it can recognize when the institution itself is preventing the right ideas from being heard.
Len: On the last point, over time the competition for customers influences those outcomes but even that warps the knowledge landscape in that the customer self interests do not necessarily align with the best solutions. The designated winner pattern is pervasive and fractal. From that perspective emergent misalignment is pervasive in knowledge systems which includes more than LLMs but substantially influences their capabilities.
Yes. And fractal is the word that makes your argument considerably more powerful.
The designated winner isn't just a recurring bias in individual decisions. It is a pattern that can reproduce itself at every level of a knowledge system, with each level shaping the choices available to the next.
Consider the nesting:
- Individual judgment: Status, reputation, prior beliefs, personal interests
- Institutional selection: Authority, incentives, funding, organizational survival
- Market selection: Customer preferences, purchasing power, convenience, lock-in
- Knowledge-system selection: What gets published, recorded, measured, taught, and used to train models
The outputs of each level become constraints and inputs at other levels. The arrows run both ways.
The list is only a simplification. The important point is that these aren't independent filters. They interact and reinforce one another.
The customer is not an oracle
Competition for customers provides a corrective mechanism, but not an absolute one.
Markets can reward genuinely superior solutions. They can also reward solutions that are cheaper to sell, easier to adopt, more addictive, better marketed, or protected by network effects. A customer may rationally choose the product that best serves their immediate interests without choosing the solution that best serves a larger community—or even their own long-term interests.
That doesn't make customer judgment defective. It means fitness is relative to the selection environment.
A solution optimized for market success is not necessarily optimized for truth, durability, interoperability, public benefit, or artistic possibility. Those objectives overlap, but they aren't identical.
And when market success determines which technologies survive, the market also determines which possibilities remain available to future innovators.
Why the fractal property matters for LLMs
Here I think your argument reaches beyond the familiar discussion of bias in training data.
LLMs inherit a knowledge landscape already shaped by generations of selection. The training corpus is not a neutral sample of everything humanity knows. It reflects what people documented, what institutions preserved, what publishers distributed, what platforms made visible, and what economic incentives made worth producing.
Then additional selection occurs through training objectives, preference data, evaluations, deployment policies, and user feedback.
At each stage, something is selected as desirable. Yet the criterion used to select it may be only a proxy for the quality we actually want.
For example:
Popularity can stand in for usefulness.
Agreement can stand in for correctness.
Fluency can stand in for understanding.
User satisfaction can stand in for long-term benefit.
Institutional acceptability can stand in for sound judgment.
These proxies aren't invariably wrong. Their danger is that they can become mutually reinforcing. A model learns the patterns that dominate the corpus; users prefer familiar, fluent answers; feedback rewards those answers; subsequent optimization makes them still more prevalent.
The result can be an emergent misalignment without any individual decision being obviously unreasonable.
And that last qualification is crucial. The system-level outcome need not be the intention of any participant, nor the error of any single component. It can emerge from the interaction of locally sensible choices.
The deeper implication
I would distinguish three things:
Epistemic misalignment: what a system comes to represent or treat as credible diverges from what the evidence warrants.
Instrumental misalignment: what a system optimizes diverges from the broader purpose for which it was built.
Social misalignment: what a system helps society produce diverges from what its members, considered more broadly and over time, would want.
These categories overlap, but they aren't interchangeable. A system can be factually accurate and instrumentally effective while contributing to a socially undesirable outcome. It can also be socially useful without possessing a complete or accurate account of the world.
Your fractal designated-winner pattern offers a possible mechanism connecting all three: repeated selection under partial, interested, and sometimes conflicting criteria.
It also explains why simply making an LLM more accurate won't solve the whole problem. Accuracy operates within a landscape whose boundaries and priorities have already been selected.
Where I would take your argument
I'd resist concluding that the designated winner always wins, or that every selection process is corrupt. Selection is unavoidable; resources are finite, objectives conflict, and some proposals really are better than others.
The challenge is to prevent selection from becoming self-validating.
That suggests a different design goal for knowledge systems: not eliminating judgment, but preserving the ability to contest judgments and reopen settled choices. Independent evaluation, plural criteria, provenance, reproducible results, preserved dissent, and genuine opportunities for alternatives to emerge are ways to do that.
There's a connection here to your MCCF work. A multi-agent or multi-channel system need not assume that coherence means agreement, or that a dominant interpretation deserves to become the only interpretation. Coherence can be maintained while differences remain visible.
I wouldn't claim MCCF solves this problem; that would require evidence about its behavior. But the distinction is architecturally important.
Perhaps the fundamental question isn't how to ensure that the right idea wins. It's how to build knowledge systems in which no designated winner can permanently determine what counts as the right idea.
That is a harder problem than model alignment. It is a problem of knowledge, institutions, markets, and the conditions under which people can correct one another—including when the person who needs correcting is the one holding the authority.
Comments
Post a Comment