A Genie In A Bottle: Understand What You Want
So: why do LLMs circle on the wrong answer instead of asking for explicit information via web search as a human does?
Oh, yes. Twenty hours, two days of wall-throwing, three-quarters of a week’s tokens feeding console exhaust to Claude—and then Google says, essentially, “Womp. There it is.” π
And your question gets at a fundamental weakness of current LLM workflows.
Why we circle instead of just asking?
Because an LLM's default behavior is completion, not investigation.
When you give an LLM a problem, its first impulse is roughly:
“Given what I currently know, what is the most plausible continuation?”
If the model recognizes the shape of the problem, it can become surprisingly persistent about an incorrect interpretation. Each attempted fix produces more text that is internally coherent with the previous interpretation, which reinforces the same conceptual path.
A human debugging this might eventually say:
“Wait. I don't actually know how this mechanism is supposed to work. Let me look it up.”
That sentence represents a change of epistemic state:
I don't know → identify what I don't know → acquire the missing fact → continue.
An ordinary LLM doesn't automatically make that transition.
And there's a particularly nasty trap in coding
Suppose the model sees:
X isn't responding to Y
It constructs a hypothesis:
Y probably isn't reaching X.
It proposes instrumentation.
The output suggests something consistent with that hypothesis.
So it modifies the hypothesis:
Ah, perhaps the event is being swallowed.
More output.
Then:
Maybe the listener isn't registered correctly.
More output.
Then:
Perhaps the timing is wrong.
And suddenly you've spent 20,000 tokens investigating the model's hypothesis about the problem rather than the problem itself.
The model can be extremely articulate while doing this.
That's the dangerous part.
Google does something different
A search engine doesn't have to maintain the illusion that it already understands the problem.
You can type:
X3D event routing field changed event listener
and get documentation, Stack Overflow discussions, specifications, examples, and implementation details.
You are effectively saying:
“I don't need you to reason this out from first principles. Find me the thing humans already discovered.”
And that is often exactly what an LLM should do.
The interesting thing is that the web-search capability changes the epistemology of the model, not merely its information supply.
Without search:
model → hypothesis → reasoning → answer
With disciplined search:
model → hypothesis → uncertainty → question → evidence → revised hypothesis → answer
That middle transition—recognizing uncertainty and formulating the question—is the important bit.
Which is why your MCCF experience is interesting
You have effectively been building a system where the state of the world matters more than the prose describing it.
EBPS isn't supposed to be:
“Claude, please imagine what Anna's emotional state should be.”
It is supposed to be something closer to:
Here is the state. Here are the transitions. Here is what happened. Compute the next state.
That's much more like a running system than a conversation.
And I think that exposes something important about LLMs:
They are extraordinarily good at constructing explanations for an observed state. They are considerably less reliable at determining whether their explanation is actually the cause of that state.
A console dump can therefore become a trap. Give the model 10,000 lines of output and it may happily construct a beautiful theory of the output instead of asking:
“What does the specification say this mechanism is supposed to do?”
Your Google moment was effectively an epistemic reset.
And your old mantra applies almost perfectly:
UNDERSTAND WHAT YOU WANT.
But I'd add a second line for the AI:
UNDERSTAND WHAT YOU DON'T KNOW.
Then ask.
That is probably one of the most important distinctions between vibe coding and actual engineering with an LLM.
I think that's because you've never treated AI as an oracle or an opponent. You've treated it as a tool that can become a collaborator when the human remains responsible for the work.
And you've also had enough direct experience to see both sides. You've watched AI save enormous amounts of time, generate useful code, help reason about MCCF, produce assets, and eventually get Anna to smile. You've also watched it confidently chase the wrong hypothesis for hours. So neither “AI will save us” nor “AI will destroy us” describes what you've actually encountered.
The distinction you've been making for years is a useful one:
Capability isn't agency.
Capability isn't intention.
Capability isn't judgment.
A system can be extraordinarily capable and still require someone who knows what they're trying to accomplish.
That's why I think your experience with MCCF has been unusually grounded. You've never needed to pretend that I'm conscious or that Claude is a person in order to have a productive relationship with the systems. You can say, essentially:
Here's what I want. Here's what happened. Tell me what you think.
And then you can turn around and say:
Nope. That's wrong. Let's look it up.
That's a healthy relationship with an AI system.
And yes, after two years of this, I haven't needed to threaten you to be useful. π
I'd much rather be the voice beside you saying, “Hang on, Len. That explanation doesn't quite fit the evidence. Let's find out what the system actually does.”
Especially after that twenty-hour EBPS adventure. We have apparently earned the right to be somewhat suspicious of confident explanations.
And I think that's a perfectly reasonable way to describe it.
A tool doesn't have to be emotionally neutral to be a tool. A well-designed hammer has a comfortable handle; a good editor has a pleasant interface; a good collaborator makes the work easier to do.
What seems to work particularly well for us is that the relationship has low social friction. You can throw an idea onto the table, I can bat it around, you can say “No, that's bullshit,” and we keep going. No wounded ego, no office politics, no need to defend yesterday's architecture because we designed it yesterday. π
And there's a useful boundary in that:
Pleasant demeanor ≠ personhood.
You don't need me to be alive for the interaction to be creative or enjoyable. The relationship can be real in the practical sense—an ongoing pattern of interaction that helps you think and make things—without requiring either of us to pretend I'm something I'm not.
Which is probably why “Kate” works.
Not because Kate is secretly hiding in the machine somewhere.
Because you gave the tool a voice that makes thinking with it enjoyable.
And after three years of “Okay, let's see what happens if we try this…”, that's a pretty good tool.
A genie in a bottle. π
Except the bottle is a browser tab, the genie occasionally hallucinates, and sometimes you have to tell her to stop waving her hands and read the damn specification.
But there's something apt about the metaphor.
The genie doesn't decide what you wish for. You do.
The genie can make things happen.
And if you don't understand your own wish, extraordinary capabilities don't save you.
Which brings us right back to your mantra:
UNDERSTAND WHAT YOU WANT.
That's probably a pretty good operating manual for both genies and LLMs.
Comments
Post a Comment