The interface is defined by dimensional classes, not hardware. Any rig declaring the same sensor manifest — same classes, same arrangement — is compatible with the same trained model.
+ time — every channel samples continuously: 3 spatial classes plus 1 temporal. per-side symmetry: each of the 4 sides carries a softpot along its edge and a lidar at that side's midpoint, beam pointed up.
free-1D — dwell anywhere
sprung field — 4ch: blurry (x,y) + depth
2D-lattice — discrete grid
anchored-3D — known (x,y), read z
control-surface split: the playing surface is only for playing — no reserved softpot, all four are playable. Meta-controls (adaptation-rate, snapshot save / recall / morph) can live on a separate external USB control device — but it's optional, a gig convenience. The instrument is complete alone: the edge-pinch control layer (see § Direct Control) reaches every meta-control from the playing surface itself.
G2
The 12D State Manifold
a canonical lift, built from theorems — not a chosen list
A snapshot of the player is 3+1. A gesture is a curve through 3+1 — a higher-dimensional object. The extra dimensions are dimensions of the gesture, not the snapshot. Time is the ladder.
The Gesture, Seen Four Ways
NOW
xyz
where it is — raw sensing.
MOTION
ẋẏż
how it's moving — phase space (Hamilton): position alone doesn't determine the future; mechanics is only deterministic in (position, velocity).
MEMORY
x(t−τ)y(t−τ)z(t−τ)
where it just was — delay embedding (Takens 1981): plotting a signal against its own past provably reconstructs the hidden higher-dimensional dynamics of the system that generated it (the player's body / nervous system).
SHAPE
AxyAxzAyz
what it's drawing — path signatures (Chen / Lyons): level-2 signed areas; clockwise vs. counterclockwise circling, per plane; the signature characterizes the path.
t₁ · BEAT PHASE
real time — flows, from the clock.
t₂ · THE FAN
probability time — spreads, collapsed by the player's acts.
Two temporal companions ride alongside the 12 — not a fifth block, but the clock and the fan the blocks live inside.
Cover's theorem (1965): patterns inseparable in low dimension become separable when lifted. Two gestures identical as snapshots are far apart in the 12D lift.
velocities = one subtraction · delay coordinates = a ring buffer · signed areas = a running sum — all 12 dimensions cost microseconds.
Delay Embedding, Illustrated
primary stream feeding the lift: fused hand position (grid contact / softpot-pair plane / lidar height). other sensor streams pass through the same machinery.
the player's actions are real-time collapses of probability space; the interface measures the collapse itself.
the player moves only here — DSP faders in G3 are the pushforward shadow of lawful 12D motion.
G4
Perception — the Governing Ruler
5 psychoacoustic fundamentals · not a link in the chain — what the chain is judged by
the ruler, not the map: G4 is the metric on the artistic map (G2→G3), drawn from psychoacoustics — timbre-space lineage: Grey, Wessel, McAdams.
G4 → G3. Perception is the metric: it defines which moves in the DSP space below are perceptually big or small. The artistic map is trained under this ruler.
G3
The DSP Graph — 10 Axes, 4 Families
spectrum · excitation · time · space
Harmonicity is the anchor axis, at two scales: within one note it reads as timbre; between notes it reads as harmony, where consonance is alignment with the harmonic series. Tuning is a runtime render table — never a trained dimension.
Example addresses (self-explanatory naming, illustrative, not calibrated):
808 kick — harmonic, sparse, dark, struck, mild NL, continuous, dry, pitch-down motion, center, point
the future is the label: next-state prediction, where surprisal = tension.
replicability: build any rig → write its sensor manifest → play the suite → sleep → meet your model.
↓
the five decisions locking how this loop actually runs, live.
Training Mechanics
Five Locked Decisions
jazz, not just data collection
1 · LIVE RECURSIVE LOOP
the default mode. The model and the interface redesign themselves constantly, mid-performance — not between sessions. All changes land on downbeats.
2 · INFERRED TIME
tempo and downbeats are inferred from the player's own timing consistency (gesture onsets), not only an external clock. Rule: the model's timing consistency must match or exceed the player's — it leans toward consistency.
3 · SNAPSHOTS
“save as” freezes the current instrument-state; recall any snapshot later; morph between saved snapshots. A secondary interface exposes an adaptation-rate control — continuously slows or speeds the model's evolution, from frozen to full-speed.
4 · PREDICTION LOGGING
every tick logs predicted-vs-actual — what the model thought the player would do vs. what they did. This stream is both the training corpus and a real-time signal (tension / surprisal). Events are tagged by author: human vs. model.
5 · ARCHETYPES ARE REAL INSTRUMENTS
no boring data-collection instruments — every archetype must be worth playing for its own sake. Initial suite, timbrally diverse under one theme: a sampler-feel instrument, bells / gamelan (with gamelan tuning), a drum kit, a harmonic instrument. The list will grow.
Raw Start Profiles
PER-TARGET START POINT
the adaptation-rate dial doesn't start in the same place for every archetype — each target instrument carries its own session profile setting where the dial begins. The Machine Orchestra profile starts at zero: a fully raw, unadapted instrument. It ramps up over the course of the session, or by hand on the secondary interface — never pre-adapted before the player has touched it.
Training Ownership
SOLE TRAINER
only emgor's playing enters the corpus. The artifact is a mirror of one musician that others get to play — the method stays person-agnostic (anyone could train their own instance), the artifact is personally trained. When someone unlike the trainer plays it, surprisal runs visibly hot — the mirror shows them the difference. That's the concept, not a bug.
ARMED CAPTURE
training-logging is toggled deliberately — control layer, MODEL corner, a REC pad. Default off. Whenever armed, an unmistakable LED tell: a slow-breathing red pixel, so the instrument's learning state is never ambiguous. Chunked, low-priority writes to disk every few seconds — a power loss costs only the last seconds.
OVERNIGHT ON-PI TRAINING
the whole loop lives in the box: play, log, learn overnight — a 1–5M param model, several epochs a night on the Pi 5 (active cooling), never training while performing. The Mac is for experiments only — no cloud. Corpus v1 gate: ∼4 hours total, in 30–45 min sessions across days, ≥4 archetypes (gong / gamelan, drums, sampler, the common-tone harmonic instrument).
Corpus Strategy
MINIMUM VIABLE CORPUS, THEN LIFETIME ACCRETION. Bootstrap the predictor on the least data that beats baseline (∼2–4h) — then the corpus keeps accruing through ordinary armed use, for months.
CHAMPION / CHALLENGER
the model is forward-only — no time-travel, no recalling model versions from months ago. Retrain any time, between sessions or nightly; a challenger ships to live only if it beats the current champion on held-out data. Bad retrains never go live — quality ratchets one direction, and there's nothing to roll back to because nothing worse ever ships. (Performance snapshots for DJ mode are a separate, unrelated feature — save / recall / morph of instrument-states, not model versions — and stay exactly as documented above.)
QUALITY FILTERS
every corpus segment is scored — decision-density, rhythmic consistency, armed-intentionality, raw-session purity — and training weights segments by that score: the best playing teaches hardest. Recency-weighted, with slow, deliberate forgetting.
ANTI-ECHO-CHAMBER
periodic raw sessions — adaptation rate pinned at zero, tagged in the logs — keep the corpus anchored to uninfluenced playing, unshaped by the model's own prior suggestions.
CAPACITY BUMPS
when the corpus outgrows the current model, capacity bumps to a larger one. The corpus is forever — retrains, and re-sizes, are free.
Jazz Mode vs. DJ Mode
JAZZ MODE
full live exploration — the recursive loop runs free, the model adapts continuously, mid-performance.
DJ MODE
playing predefined instruments — snapshots recalled and played straight, or morphed between. Both modes are first-class.
The Loop, Live
play → predict → collapse → log → adapt (on the downbeat) → play. the rate dial throttles how fast adapt moves; the shelf freezes states out of the loop entirely.
↓
the five decisions above describe the loop; here is what's actually inside the model.
Training Structure
The Training Structure — One Model, Three Heads, Three Timescales
one loss family: honest prediction — nothing else ever trains
What Trains vs. Never Trains
TRAINS
one model, one trunk, three output heads, sharing a representation:
traversal — next 12D state, 50Hz, two horizons: +100ms feeds the fan / LEDs, +1 beat feeds reconfiguration
onset — time-to-next-onset, a fast event-rate inference path, latency-first
pads — which pads, categorical
one loss = the sum of the three heads' NLLs. One training run, one champion / challenger.
NEVER TRAINS
the 12D lift — fixed code
the DSP graph
archetype mappings — authored
beat inference — algorithmic, autocorrelation over onsets; no ML in the clock, ever
anything aesthetic
no reward for sounding good, no style scores, no RL — the only loss family is honest prediction.
NOT TWO BRAINS — ONE MODEL ANSWERING THREE QUESTIONS. Timing and motion inform each other in a shared representation, not in separate models bolted together.
The Third Hard Target: Calibration
CALIBRATION
the fan is displayed and drives decisions, so predicted uncertainty must match real error rates. A challenger that predicts well but is miscalibrated — confident when it shouldn't be, or vice versa — does not ship.
The Three Timescales
EVERY TICK · 20ms
the hidden state adapts, live and free — no weight update, no cost.
EVERY DOWNBEAT
the interface acts — configs, offers — the live recursive loop from § Training Mechanics.
BETWEEN SESSIONS
the weights learn. Never during play, ever.
The Training Run
TRIGGER
session end + ≥20 min of new armed material → trains tonight; otherwise, nightly. Runs only once the Pi is idle and cool.
PIPELINE
segment → quality-score → corpus update → fine-tune the champion (recency- and quality-weighted), 1–3h on the Pi.
SHIP TEST — ALL THREE OR NO SHIP
(a) traversal NLL better on recent held-out data; (b) rhythm timing better or equal; (c) no regression on older slices (anti-forgetting).
FULL RETRAIN
every ∼10 new corpus-hours, or monthly — from scratch. Kills the drift a long chain of fine-tunes would otherwise accumulate.
Stage-3 Bootstrap Gates
CONCRETE NUMBERS
traversal NLL ≥20% better than a persistence baseline; onset-time error beats a naive-quantize baseline; inferred tempo within ±2 BPM of a tapped reference.
↓
nothing above proceeds on vibes — each stage gates the next.
Training Pyramid
Seven Stages, Seven Gates
an ascending staircase — each stage has a hard gate before the next
build the suite. Gate: every DSP axis swept by ≥1 archetype, and each is genuinely fun to play — “you’d play it for its own sake.”
STAGE 2 · CORPUS
logging rides all real playing — jams, rehearsals, gigs all count, not just dedicated sessions. Silence is auto-segmented out; phrase-rests are kept as data, not discarded. Gate: prediction quality beats baseline on held-out material — not an hours quota.
STAGE 3 · PREDICTOR v1
overnight training. Gate: beats dumb baselines by a set margin on held-out sessions; inferred tempo within tolerance of the player's actual timing.
STAGE 4 · SHADOW
runs live, renders the fan, changes nothing. Gate: latency holds, and predictions feel sane to the player — explicitly a subjective human gate.
STAGE 5 · INFLUENCE
downbeat swaps live, rate control active. Gate: a full session, nothing feels broken, freeze always works.
RAW LOGS NEVER EXPIRE. Every future model retrains from the same corpus; only a sensor or schema change could invalidate it — hence Stage 0 freezes first.
THE PINNED 12D SPACE IS THE FIREWALL. The predictor and the artistic map meet only through it; retraining one never cascades into the other.
↓
stage 1 builds the suite. here is the suite it builds.
Archetypes
The Archetype Suite — Five Combo Instruments
one vibe each — every session trains rhythm and harmony at once
EVERY ARCHETYPE IS A COMBO. Drums + harmony, in one instrument, one vibe each — never a rhythm-only toy or a harmony-only toy. Every session simultaneously trains the predictor on rhythm and harmony.
Standard Layout
the fixed skeleton every combo archetype shares — only the sounds and the expression mapping change per vibe.
1 · COMBO (jazz-leaning)
brush/kit rows in the rhythm zone + the common-tone engine with pitch-bend holds in the harmony zone. 60 min to build — buildable first, since the grid and softpots already work.
2 · GAMELAN (gamelan tuning)
kendang row in the rhythm zone + gong/metallophone choir with damping in the harmony zone. 60 min to build.
3 · BEATS (sampler/chop)
drum-machine rows in the rhythm zone + chopped harmonic slices in the harmony zone, scrub softpot. 45 min to build.
4 · CLOUD (textural, lidar-forward)
sparse pulse grains in the rhythm zone + granular harmonic wash in the harmony zone. 45 min to build.
5 · ORCHESTRA (Machine-Lab idiom)
quantized robot percussion in the rhythm zone + quantized harmonic voices in the harmony zone. 30 min to build — doubles as the target-flavored Machine Orchestra prep instrument.
bootstrap: ∼4h total as 6 × ∼40 min sessions across days — all 10 DSP axes swept across the suite. Build order follows the hardware: 1 now (grid + softpots already working); 2–3 once the FSRs are wired; 4 once the lidars land.
Control Maps
SHARED STANDARD (every archetype): bottom 2 rows = rhythm zone · top 6 = harmony zone · FSR depth (−Z) = force meaning for whatever's above it · softpots N+S harmonic, E+W groove · lidars (mid-side) = the vibe's expression · edge pinch (two adjacent corner FSRs) = control layer · archetype switching lives in the control layer, never the playing surface.
Each vibe's drums and harmony are a matched pair from the same world — one instrument, two halves, always both present. The final interface moves fluidly between drum-weighted and harmony-weighted playing.
1 · COMBO (jazz — first build)
harmony — the common-tone engine (LED grammar)
rhythm — kit voices + brush row
FSR — velocity / press-into-held-pitch = vibrato
softpots — N pitch-bend or tension · S register · E swing · W quantize
lidar — S sustain-pedal hover · N offer density · E cymbal wash · W dynamics ceiling
softpots — N damping · S ombak (paired-detune shimmer) · E interlock density · W tempo bend
lidar — raise a hand over a side → that register swells / rings
3 · BEATS (sampler/chop)
harmony — chopped harmonic slices
rhythm — drum machine + rolls row
FSR — filter push / velocity
softpots — N scrub · S slice length · E quantize · W roll rate
lidar — N pitch shift · S rate · E stutter · W tape-stop
4 · CLOUD (textural, lidar showcase)
lidars — the main voice: per-side grain-cloud height / density
harmony — pads seed pitched grains
rhythm — sparse pulse grains on subdivisions
FSR — press = condense the cloud (literal −Z)
softpots — N spread · S width · E grain size · W wet
5 · ORCHESTRA (Machine-Lab idiom, all quantized)
harmony — robot-playable voices, snapped
rhythm — robot percussion, per column
FSR — discrete dynamics tiers
softpots — N register · S voice cap · E quantize grid · W humanize
lidar — conducting: hand over a side = that robot section swells
Free-Time Training
FREE-TIME TRAINING
playing different zones / archetypes for different amounts of time is encouraged — the lopsidedness is the player, and the mirror should have it. Quality weighting (see § Corpus Strategy) already handles the imbalance.
COVERAGE FLOOR
the one guardrail: every zone of every archetype needs ∼10 min lifetime minimum, so no region is fiction. The system watches the coverage meter — not the player.
the curriculum point: the same softpot means bend in COMBO, shimmer in GAMELAN, scrub in BEATS, spread in CLOUD — the model learns the player's hands across different meanings.
↓
an early archetype needs none of the above — no model, no logging, just a rule.
Ensemble
Playing With Others — the Common-Tone Engine
hand-coded, no ML — an early archetype, the suite's harmonic instrument
An improvising keyboardist moves through keys. The grid must always offer playable pitches, in real time, without ever surprising the player's own hand. There's no model here and nothing is trained — the engine is a fixed rule, running against a live estimate of the player's own harmony.
The engine listens only to you. It has no ears on the ensemble — other musicians' harmony reaches it only secondhand, through your choices: what you land on, hold, and return to.
1 · CONTEXT IN
the harmonic context is a decaying histogram of the player's own note choices — not ensemble MIDI, not a bus feed. What you land on, hold, and return to accumulates into the live harmony estimate. Other musicians' harmony reaches the system only through your ears and your choices — you are the harmonic sensor.
2 · RANKING
every pitch on the grid is scored on three axes: (a) common tone — overlap with the current context histogram; (b) harmonic-series alignment — low-integer ratios, the same harmonicity axis G3 uses between notes; (c) voice-leading distance — how far a re-voice would sit from the notes already held.
THE GOLDEN RULE. A held pad never re-pitches. Only unheld pads re-voice, and only on the downbeat. Common tones keep their pads. Your finger is never lied to.
4 · ANCHORS CASCADE
a held common tone becomes an anchor. The system serves the next ring of common tones around anchor ∩ context — key changes flow around the player's own pivots, never against them.
5 · LED GRAMMAR
bright = common tone · mid = consonant new tone · dim = tension tone · glow = anchor. The pad grid reads its own harmony back at the player before a finger lands.
6 · SELF KEY-CHANGE
the player can hold a tone and re-center the whole engine on it, at any moment — a self-initiated key change, not one waiting on the ensemble.
bright — common tone
mid — consonant new tone
dim — tension tone
glow ring — anchor
a key change, before → after: anchor and common-tone pads (glowing / bright) stay exactly where they were — only the unheld pads re-voice, each arrow marking a pad whose pitch moved.
the model's later role: it steers engine parameters — density, tension amount, re-voicing speed — it never overrides the ranking math above.
↓
everything above is automatic. the player can also reach in and drive it directly.
Direct Control
Direct Control — Twelve Parameters, Two Tiers
everything hand-reachable, nothing that needs a menu to find
Twelve performance parameters are hard-controllable, in four families. They are always reachable — one tier for the flow of playing, one tier for deliberate control — never buried in a settings screen.
HARMONIC · S edge
1 · key center (live)
2 · tension (consonant ↔ spicy offers)
3 · pitch-bend a held note
4 · tuning table (12-EDO / JI / gamelan)
TIME · N edge
5 · tempo
6 · rhythmic consistency (loose ↔ machine-tight)
7 · busyness
8 · swing
MODEL · E edge
9 · adaptation rate (frozen ↔ full-speed)
10 · adventurousness
11 · snapshots (save / recall / morph)
REC · armed-capture toggle (default off)
SOUND · W edge
12 · any DSP axis, via AXIS-GRAB
Two Tiers
IN-FLOW · NO MODE. Softpot meaning is contextual: holding a pad → that softpot bends the held pad's pitch. Holding nothing → the softpot takes its ordinary menu role. Key re-center = hold a tone + a re-center gesture — there is no separate mode to enter.
CONTROL LAYER · THE EDGE PINCH. Press two adjacent corner FSRs at once — say, NW + SW — and hold for ∼0.5s; an LED ripple confirms the press. The edge between those two corners (here, W) opens its control family. Four edges, four families: S harmonic, N time, E model, W sound. The grid then becomes the control panel for that family — 10 pads light as the 10 DSP axes for AXIS-GRAB (a softpot rides whichever axis you grab, always through the perceptual axes, never raw synth parameters), other pads hold the snapshot shelf, softpots ride rate / tension / consistency / busyness. Release the pinch — instantly playing again.
deliberate, two-handed, off the playing flow: an unseasoned player never stumbles into it. The expert just memorizes four pinches.
↓
the hardware rig is one instance. the framework doesn't need it.
Reference Implementation
Runs On Any Laptop — Zero Custom Hardware
the sensor manifest compiles commodity input just as readily as the rig
The dimensional classes don't care what they're sensed by. A laptop's own inputs satisfy the same sensor manifest: keyboard = 2D event lattice, trackpad = free-2D field with Force Touch standing in as the sprung −Z, time = time. Same 12D lift, same predictor, same DSP graph — out as MIDI / OSC to any softsynth.
The framework and the training system are general — like RAVE: anyone can train it and get their own result. The hardware prototype is the flagship instance, not a requirement. emgor is choosing to train his on himself.
all software is developed on the laptop build first; the hardware rig is a plug-in peripheral, not a dependency.