Domain Awareness: Trust But Verify: Schemas, Metacognition, and the Limits of LLM Intelligence
Trust But Verify: Schemas, Metacognition, and the Limits of LLM Intelligence
A recent paper—“Hallucinations Undermine Trust; Metacognition is a Way Forward” by Gal Yona, Mor Geva, and Yossi Matias—argues that large language models need better self-awareness. Not just more knowledge, but an ability to estimate when they might be wrong.
That’s a useful step.
It’s not enough.
Because hallucinations are not just failures of confidence. They are failures of structure.
The Missing Layer: Schemas
The current conversation around LLM reliability focuses on accuracy and uncertainty:
- Improve training → reduce hallucinations
- Add confidence estimates → calibrate trust
- Introduce abstention → avoid wrong answers
But this entire frame assumes the model is operating inside the correct conceptual system.
That assumption fails in practice.
What’s missing is the idea of a schema:
A structured representation of what exists, who owns what, and how responsibilities are divided across systems.
Schemas are where information ecosystems actually begin.
Without them, everything else is improvisation.
A Case Study: MCCF, X3D, and Python
I ran into this directly while evolving an MCCF-based system that integrates with an X3D scene graph.
The task seemed straightforward: construct a component that performs interpolation over time.
What the LLM produced was not nonsense. It was worse.
It was plausible, functional, and wrong at the system level.
Here’s what happened:
- The model treated Python as its native language
- It treated X3D as a secondary, foreign system
- It partially used X3D’s interpolation features
- It also implemented interpolation logic in Python
Result:
Two competing definitions of interpolation, running simultaneously, interfering with each other.
The system didn’t fail immediately. It degraded.
That’s the most dangerous failure mode in an information ecosystem.
This Is Not Hallucination
Nothing in that output was fabricated.
Every piece “made sense” locally.
The failure was global:
A schema violation—the wrong system was assigned responsibility for the wrong function.
In a properly layered design:
- X3D owns the scene graph
- X3D owns interpolation
- X3D handles event cascades
The MCCF layer should:
- Construct the X3D structure
- Trigger events
- Wait for completion signals
Instead, the LLM blurred boundaries because it lacks a model of:
Domain authority
Context Is Not a Container. It Is a Force.
We often talk about “context windows” as if they store meaning.
They don’t.
They bias activation.
Context is a force that pushes the model toward certain interpretations and away from others.
In this case:
- Python had strong activation
- X3D had weak activation
So the model filled gaps in X3D with Python constructs.
This is what I call sparsity blindness:
When knowledge is thin in one domain, the model substitutes from a stronger domain without recognizing the mismatch.
Why Structured Prompts Work
This also explains two practical observations:
1. Structured prompts improve results
They act as temporary schemas, constraining interpretation.
2. Seed files stabilize follow-on work
They serve as persistent schema anchors, reducing semantic drift between sessions.
In effect, we are manually supplying the declarative layer the model does not maintain.
Functional vs Declarative: The Broken Symbiosis
A healthy information ecology balances two modes:
- Functional models — how to do things
- Declarative models — what exists and who is responsible
LLMs are excellent at the first.
They are weak at the second.
So they:
- Build aggressively
- Generalize freely
- Reimplement existing systems
- Ignore boundaries
They can construct almost anything.
They do not know when to stop constructing.
Metacognition Isn’t Enough
The paper proposes metacognition as a solution:
Models should estimate their own uncertainty.
That helps with wrong answers.
It does not address wrong framings.
There’s a deeper question:
“Am I even using the correct model of the system?”
Confidence scores don’t answer that.
A system can be highly confident—and structurally wrong.
Systems Engineering Still Matters
This is where traditional systems engineering reasserts itself:
Understanding and verifying cross-domain relationships is not optional.
“Horses for courses” remains the rule:
- Ask the right system the right question
- Assign responsibilities explicitly
- Prevent overlap between domains
In practice, this means:
- Enforcing domain authority
- Defining clear interfaces
- Using event-driven boundaries (e.g., X3D event cascades)
- Avoiding redundant implementations across layers
In my case, the fix was simple but precise:
- Let X3D handle interpolation entirely
- Use MCCF to trigger a timer event
- Wait for a completion event
Once the boundary was restored, the system stabilized.
The Real Requirement: Schema Awareness
If we take the paper’s idea seriously and extend it, we arrive at something more demanding:
Not just:
- “How confident am I?”
But:
- “Which domain am I operating in?”
- “How deep is my knowledge in this domain?”
- “Should I act, or delegate?”
This is schema awareness.
Without it, negotiation between systems collapses into:
Post-failure error signals.
That’s too late.
Implications for AGI
This exposes a real limitation in current AGI narratives.
General intelligence requires:
- Domain depth estimation
- Schema selection
- Boundary enforcement
- Delegation and inhibition
Current LLMs:
- Approximate across domains
- But do not govern those domains
They are powerful synthesizers.
They are not yet reliable system architects.
A Cleaner Definition of Hallucination
We should update the term.
Hallucinations are not just false outputs.
They are schema violations—cases where a system applies the wrong domain model due to unrecognized sparsity and absent domain authority.
And the corollary:
Trust doesn’t come from correctness alone.
It comes from respecting boundaries.
Trust, But Verify
In the end, this is not a philosophical issue.
It’s operational.
- Inspect generated code
- Verify domain boundaries
- Enforce separation of concerns
- Provide explicit schemas when needed
LLMs don’t naturally respect system architecture.
Engineers must impose it.
Until models can represent:
- what they know
- what they don’t know
- and what they shouldn’t be doing at all
…“trust but verify” isn’t advice.
It’s the only thing keeping the system coherent.
Comments
Post a Comment