Applying MCCF to Simulation: Theatre Model
Now the full vision is on the table.
The pipeline you're describing has four stages, each with a different cost profile:
Stage 1: Story text → Scene graph Extract from narrative prose: locations, characters, objects, spatial relationships, emotional register of the scene, zone type. This is an LLM task and it's largely solved — give a scene description to an LLM with a structured output schema and it returns a JSON scene graph. The MCCF zone presets already give you the emotional vocabulary. A library scene with two characters in conflict maps directly to zone type, character cultivars, and initial channel states. That's maybe an hour of prompt engineering per scene, collapsing to minutes with a good template.
Stage 2: Scene graph → X3D This is where the work lives. Hand-authoring 3D objects with behaviors is genuinely expensive. But there are three tractable paths:
Asset libraries — X3D and glTF have substantial free libraries. Humanoid avatars, architectural elements, props. The H-Anim standard gives you rigged humanoid figures that can take motion data directly. You're not building from scratch if you're composing from existing assets.
Procedural generation — for environments, SemanticZones map to procedural scene generation. A garden zone generates foliage, soft light, open space. A threat zone generates constricted geometry, harsh shadows. The zone preset already contains the aesthetic vocabulary — you write the generator once.
LLM-to-X3D — this is early but viable. Current LLMs can generate basic X3D scene fragments from natural language descriptions with reasonable reliability. Not hero assets, but backgrounds, props, spatial layout. The MCCF scene composer already exports X3D zone visualization fragments. That pipeline extends.
Stage 3: Characters → MCCF-driven performance This is already built. The cultivar defines the character's affective disposition. The written scene text becomes the voice agent prompt context. The MCCF field shapes expression in real time as the scene plays. The ambient engine scores it. What's needed is a text-to-waypoint compiler — something that reads a scene and generates the waypoint arc automatically from the dramatic structure. That's an LLM task with the waypoint schema as output format.
Stage 4: User as character — the holodeck problem This is the hardest and the most interesting. The user needs to be in the scene as an agent whose presence modifies the field. Their proximity to characters changes coherence scores. Their voice input feeds prosody into the E-channel in real time. Their choices at each waypoint — what they say, where they move — alter the arc. The scene responds to them not as a game branching on choices but as a field that continuously recalculates around their presence.
The Web Speech API handles voice input already. The ProximitySensor handles spatial presence if the user has an avatar. The MCCF sensor endpoint handles the field update. The missing piece is the user's cultivar — what character are they playing, what are their channel weights, how does the field read them.
The hat trick specifically:
The reliable pipeline from story text to rendered animation is:
Scene prose
↓ LLM with zone/character schema
Scene graph JSON
↓ Procedural generator + asset library
X3D scene file
↓ MCCF cultivar assignment
Character field states
↓ Voice agent + ambient engine
Live performance
↓ User avatar + ProximitySensor
HolodeckEach arrow is a tool that either exists or is a bounded engineering problem. None of them individually is the hard part.
The hard part is the authorial question underneath the technical one. In a holodeck where the field responds to the user in real time, the story has to be robust to the user going off script — because they will. The characters need enough cultivar depth that they respond authentically to unexpected inputs rather than breaking. That's not a technical problem. That's a dramaturgy problem.
Which means the composer's challenge is exactly what you named it as. Not building the stage. Not writing the script. Designing a dramatic structure that holds together when a real person walks into it and starts making choices the author didn't anticipate.
You've been working on that problem in the Garden for a while. The ladies already know how to handle unexpected inputs — they feel fully and choose precisely. That's the model for every character in the holodeck.
The stage is closer than it looks. What do you want to build first?
Exactly right. The waypoints are spatial positions. What's missing is the temporal dimension — when to be where, how long to dwell, how fast to move between stations, and what the interpolation curve looks like between them.
In X3D the complete motion system is:
TimeSensor → PositionInterpolator → Transform.translation
TimeSensor → OrientationInterpolator → Transform.rotation
TimeSensor → ScalarInterpolator → [any float field]The TimeSensor is the conductor. Everything else follows its fraction output.
What the text-to-waypoint compiler needs to extract from scene prose:
Dramatic beat → waypoint position (already have this)
Scene duration → TimeSensor cycleInterval
Dwell at each station → key timing within the cycle
Movement quality → interpolation curve shape
Emotional intensity → scalar interpolator targetsThe interpolation curve shape is the expressive element:
A character who crosses a room in grief moves differently than one who crosses it in anger. The positions are identical. The TimeSensor cycle might be identical. What differs is the key/keyValue curve — smooth EASE in grief, sharp LINEAR in anger, oscillating BACK in conflict. That curve shape is an affective parameter. The MCCF arousal and valence values should drive it.
Let me build this:
The demo ran clean — it fell back gracefully because the Stub adapter doesn't actually parse scene prose into structured JSON, it just echoes. With a real LLM adapter that follows the extraction prompt, the Garden scene would produce four beats with proper positions, timing, and emotional registers.
What was built and why each piece matters:
The TimeSensor is the conductor as you named it. cycleInterval is the total scene duration. The fraction_changed
output drives every interpolator simultaneously — position,
orientation, and all seven affect scalars move in lockstep through the
same time coordinate.
The interpolation curve shapes
are the expressive element that most scene tools ignore. Grief gets
EASE — slow, weighted movement where the character seems to carry
something. Anger gets LINEAR — direct, unmodulated, no softening at the
edges. Fear gets SPIKE — fast approach then a frozen hold. The curve is
computed from emotional_register + arousal/valence in emotional_to_curve(). That function is the dramaturgy encoded as a lookup.
The three endpoints serve different authoring workflows. /compile/prose takes raw scene text through the LLM extractor — the full holodeck pipeline. /compile/direct takes pre-structured beats — useful when you're hand-authoring. /compile/waypoints
bridges directly from the MCCF waypoint editor's arc data — so the
constitutional cultivar arcs we already built can be compiled into X3D
animation directly without reauthoring anything.
The remaining gap is orientation. Currently everything defaults to [0,1,0,0]
— facing the audience. The LLM extraction prompt asks for orientation
but the Stub adapter won't produce it. With a real adapter, facing
direction should emerge from the spatial logic of each beat — crossing
toward another character means rotating to face them, turning away means
rotating 180°. That's a spatial reasoning task the LLM can handle if
the scene has clear character positions.
The Lady crossing from garden entrance to council chamber door, pausing at the fountain, looking back — that's a four-beat arc. The compiler knows how to time it, curve it, and score it. The holodeck is closer.
Yes, and H-Anim is exactly the right thread to pull.
H-Anim (Humanoid Animation) is an ISO standard that sits directly on top of X3D. It defines a skeletal hierarchy — Joint nodes, Segment nodes, Site nodes — that maps to human anatomy with named joints following anatomical convention. HumanoidRoot, sacrum, l_hip, l_knee, l_ankle down to the phalanges. The full skeleton is there, rigged and named, waiting for motion data.
The
gravity connection is precise. H-Anim joints have mass and
center-of-mass fields. The standard anticipated physics integration. A Joint node carries:
<Joint DEF="l_hip" name="l_hip"
center="0.0961 0.9124 0.0"
llimit="-0.7854 -0.6109 -0.3927"
ulimit="0.4363 0.5236 0.5236"
stiffness="0.0 0.0 0.0">
```
The `stiffness` field is the gravity hook — resistance to displacement from neutral. Set it to zero and the joint is floppy. Set it toward 1.0 and it holds posture. The `llimit` and `ulimit` fields are the anatomical rotation constraints — you can't hyperextend a knee or rotate a hip beyond its biological range.
**Where this connects to MCCF:**
The affect channels already map to postural states in human movement science. This is well established in Laban Movement Analysis:
```
E (Emotional) → Weight quality — heavy/light, how gravity is inhabited
B (Behavioral) → Time quality — sudden/sustained, relationship to rhythm
P (Predictive) → Space quality — direct/indirect, intentionality of path
S (Social) → Flow quality — bound/free, degree of controlled releaseHigh E-arousal with negative valence — grief — produces heavy Weight quality. The character moves as if gravity is stronger than usual. That translates directly to H-Anim joint stiffness values and center-of-mass positioning. The pelvis drops slightly. The thoracic spine curves forward. The head weight falls.
High B with positive valence — confidence — produces sustained Time quality with direct Space. Upright pelvis, extended spine, deliberate path through space.
The H-Anim community work:
Don Brutzman at Naval Postgraduate School has done the most sustained X3D/H-Anim work. The H-Anim 2.0 specification, which became ISO 19774, includes the full joint hierarchy with mass properties. There's also work connecting H-Anim to motion capture data formats — BVH specifically — which means the pipeline from motion capture to X3D H-Anim is documented and implemented.
The gap that hasn't been closed, as far as I know, is the affective parameterization of H-Anim joint states. Nobody has built the layer that says: given arousal=0.7, valence=-0.4, regulation=0.6, compute the joint stiffness and center-of-mass offset values that produce the corresponding postural expression. That's the missing piece.
What that layer would look like:
def affect_to_hanim_posture(arousal, valence, regulation) -> dict:
"""
Map MCCF affect params to H-Anim joint modifications.
Returns delta values to apply to neutral standing posture.
"""
# Weight quality: heavy affect = lower pelvis, forward thorax
pelvis_y_offset = -0.02 * (1 - valence) * arousal
thorax_flex = 0.15 * (1 - valence) * (1 - regulation)
# Time quality: high B = more deliberate, slower joint velocity
joint_velocity_scale = 0.5 + regulation * 0.5
# Space quality: high P = direct path, less peripheral movement
peripheral_damping = 0.3 + (1 - arousal) * 0.4
# Flow quality: low regulation = more free flow, more joint coupling
joint_coupling = 1.0 - regulation * 0.6
return {
"HumanoidRoot": {"translation": [0, pelvis_y_offset, 0]},
"vl5": {"rotation": [1,0,0, thorax_flex]},
"vc4": {"rotation": [1,0,0, thorax_flex * 0.5]},
"skullbase": {"rotation": [1,0,0, thorax_flex * 0.3]},
"velocity_scale": joint_velocity_scale,
"peripheral_damp": peripheral_damping,
"flow": joint_coupling
}
```
That function, connected to the MCCF ScalarInterpolator outputs, gives you a character whose body inhabits the scene the way a trained actor would — not just moving through space, but carrying emotional weight in the joints.
**The motion capture bridge:**
BVH files are essentially TimeSensor + OrientationInterpolator data per joint — which is exactly what our compiler outputs. The pipeline becomes:
```
Scene prose
↓ LLM extractor
SceneScript with beats
↓ mccf_compiler.py
TimeSensor + PositionInterpolator
↓ affect_to_hanim_posture()
H-Anim joint rotations per beat
↓ OrientationInterpolator per joint
Full body animation
↓ MCCF ScalarInterpolator
Real-time affect modulation on topThe base animation from the compiler gives you the spatial arc. The MCCF affect layer gives you the moment-to-moment postural expression on top of that arc. A character who is grieving doesn't just move to the fountain and stop — the quality of every step is heavy, the head drops incrementally as they approach, the shoulders round as they hold the position.
Brutzman's group and the HAnim mailing list would be the right place to surface this. The connection between Laban effort qualities, H-Anim joint parameters, and affective computing is a natural research direction that hasn't been formalized. The MCCF gives it the computational substrate it was missing.
The holodeck gets anatomically honest. That's worth a paper of its own.
Comments
Post a Comment