rocky_mini: A Robot's Brain, Built and Tested Without the Robot
An embodied AI character (Rocky, the Eridian engineer from Andy Weir's Project Hail Mary) built sim-first on a fully local, zero-cost stack. Every external service sits behind a Protocol with a Fake, so the whole four-latency-domain brain runs and passes 164 tests with no robot, no GPU, and no local LLM in the loop. Built and proven in sim first; the physical robot has since arrived and runs.
rocky_mini is a personal homage project: Rocky, the Eridian engineer from Andy Weir's Project Hail Mary, role-played on a Reachy Mini Wireless robot from Pollen Robotics. He is framed as a curious alien child on Earth who learns and remembers across sessions and answers in chords and a ring-modulated voice. Rocky and the Eridians are Weir's creations, not mine, and the Reachy Mini is Pollen's hardware; I am naming both up front.
The reason this is a portfolio piece is not the fun premise. It is that I built and tested the entire software brain in simulation, without the robot, without the GPU, and without the local language model in the loop. The artifact worth pointing at is the structure that makes that possible, and the discipline that keeps it honest about what has actually run.
The seam that lets one codebase live in two worlds
Every external dependency (the language model, speech to text, text to speech, and the robot's own motion and audio surfaces) sits behind a Protocol. The sim and test paths use Fakes. A rule-based SimResponder plays Rocky well enough to drive a real conversation turn and the settings UI; the real path swaps in OllamaLLM behind the same interface. The imports for the real clients are lazy and guarded, so the sim genuinely never touches Ollama, faster-whisper, Piper, or the Reachy SDK.
The distinction I care about is Protocol-with-Fake versus mock. A mock lives in a test file, is built by the test, and knows what the test expects, which lets the production code keep an implicit dependency on a concrete client because nothing forces the interface to be written down. A Protocol is the opposite arrangement: the interface is the production artifact, both implementations are shipped code, and the runtime picks one from config.
That buys two things. The Fake gets exercised by the app rather than only by the suite, so when SimResponder breaks it breaks in the settings UI while I am using it. And the awkward parts of the interface surface early. Streaming is the obvious one: if the LLM Protocol returns a string, the Fake is easy and the real path is a rewrite, because the whole latency story depends on chunking tokens as they arrive. Writing the Fake as an async generator that yields clause-sized pieces meant the chunker, the tic linter, and the pipelined TTS stage were designed against streaming from the first line of code. The same argument made the Windows sounddevice audio path a first-class tested surface instead of a workaround bolted on beside the robot path.
There is also exactly one place in the codebase where sim becomes real, so "what has actually run" is a question with a checkable answer instead of a vibe.
The left column is code that ran and passed tests before any hardware existed. The right column was written against the same Protocols and documented for the machine with the GPU and the robot, and it stayed unexecuted until the robot arrived. Keeping those two columns separate for the whole sim-first build is the point, not a disclaimer: when bring-up day came, "what has actually run" was a checkable answer, and the swap was the one seam line the design promised.
Four latency domains, one process
A talking robot is a real-time problem. If the acknowledgement is late, or the mouth and the voice drift, or two threads fight over a joint, the character breaks. Each of those failures lives in a different time budget, so each gets its own owner: one process runs four threads plus one asyncio loop, communicating over queues, a locked MotionState, and a global generation counter.
MotionThread, 100 Hz. The single owner of set_target, one call per tick. A primary layer (emote trajectories, internal minimum-jerk moves, breathing idle) is summed with additive secondary offsets (speech wobble, direction-of-arrival orientation, a listening freeze), blended, then clamped before anything reaches a joint: pitch and roll bounded, head-versus-body yaw bounded, antennas resting a few degrees off zero rather than at zero. The clamp is the last statement in the tick, so no upstream behavior can route around it.
Mixer, frame-paced. The single owner of push_audio_sample. A generation-tagged VoiceBus carrying text-to-speech PCM and a ChordBus carrying Eridian stingers are summed, ring-modulated, and soft-clipped in one place, then pushed as fixed-size frames. It also owns the playout clock that drives the speech wobble, so the head moves with the audio the speaker is emitting rather than with the audio the loop generated.
AudioIn. Microphone samples into a ring buffer, then voice activity detection with a hangover window before a speech event is published. On hardware it also polls direction of arrival, the only spatial input Rocky gets, because he echolocates and deliberately has no face tracking.
ConversationLoop, asyncio. A state machine over idle, listening, transcribing, thinking, and speaking. It fires the acknowledgement chord and thinking pose first, then runs speech to text, the streamed model, the chunker that splits on clause boundaries and inserts the verbal tics and emote tags, and a pipelined text-to-speech stage that stays a couple of chunks ahead.
The load-bearing rules are the ownership ones: exactly one thread writes motion, exactly one writes audio, everything else asks. That is what turns three concurrent intentions (breathe, turn toward the speaker, speak a chord) into one composed movement instead of a fight over the neck. The generation counter is the same rule in the time dimension. Barge-in increments it and cancels the in-flight task, and the Mixer discards any queued chunk whose tag is stale; draining buffers from the interrupting thread instead would have put a second writer on the audio surface at exactly the moment timing matters most. Half-duplex stays the default on the development machine, because there is no acoustic echo cancellation on a PC and motor noise into an open microphone is a hardware-tuning problem.
What the 164 tests actually cover
The suite is green at 164 tests. That count was recently corrected downward after I found the number I had been quoting was stale, which is the sort of correction that matters when the whole pitch of the project is that the ledger is honest.
The shape of the suite follows the risk. Pure functions get property-style tests: the DSP tests assert ring modulation produces the expected sideband structure under an FFT and that carrier phase is continuous across chunk boundaries, because a phase discontinuity is an audible click, the kind of bug ears rationalize and an FFT does not. The chunker tests pin the tic insertion table and the emote and sound-effect tag extraction. The persona builder gets a byte-stability test, since the prompt prefix has to be byte-identical for cache reuse on the serving side and for fine-tune evaluations to be reproducible.
Concurrency gets tested at the seam rather than by sleeping and hoping: the Mixer tests drive the generation counter directly and assert stale chunks are dropped, and the turn tests run the whole state machine against Fakes, so a full conversation turn is deterministic and takes milliseconds.
Above that sit the integration layers: the FastAPI settings surface, a sim end-to-end pass driving a typed turn from UI input to pushed audio samples, and one real-browser Playwright pass over the chat round trip, the fact table, and the metrics panel. The naivety suite is deliberately not a pass/fail gate. It is a 20-probe red-team set scored as a thresholded regression metric, because a zero-leak gate on a language model fails for reasons unrelated to the change under review and gets muted within a week.
What the suite cannot cover is the right column of the diagram. There is no test here that proves a servo moves.
Local and zero-cost, by constraint
There are no API keys anywhere, and that constraint drove the architecture more than any aesthetic choice did. The brain is designed as Ollama-served Qwen2.5-7B-Instruct on an RTX 4080, quantized so it fits in VRAM with headroom for the speech model beside it, and held resident so the first token on the real path does not pay a load. Speech to text is faster-whisper, text to speech is Piper, and Piper's slightly synthetic timbre is a feature here: it is already halfway to Rocky before the ring modulator touches it.
The split falls out of the hardware. The Reachy Mini is a Raspberry Pi CM4 with 4 GB of memory, so a 7B model does not run there in any useful sense: the PC becomes a LAN brain server and the robot becomes a thin client doing voice activity detection, motion, and audio mixing and nothing else. That is why the detector is an ONNX runtime model rather than a Torch one, and why the audio and motion paths are written to a small, steady CPU and memory profile. It also sets the honest failure mode, which is that the robot is inert when the PC is asleep. The answer to that is not a smaller model on the Pi, it is a canned in-character fallback and a chords-only mode, and those are drills that need hardware to fail against.
Latency is treated as a budget with a mask rather than a number to chase. The acknowledgement chord and thinking pose fire within milliseconds of the speech event, which buys the model round trip its cover; the end-to-end budget is measured from the user's last voiced frame, hangover included, and asserted in the sim integration test rather than quoted from a good run.
The LoRA is the piece easiest to overstate, so: it is a pipeline, not an artifact. A seed dataset, a QLoRA training scaffold, an export path, and a ship-gate evaluation that refuses to promote a fine-tune unless it beats the stock model plus a good system prompt on naivety, tic metrics, and tool-call validity. The gate runs. The thing it exists to gate does not exist yet, because training needs a GPU and a WSL2 environment that were not part of this build.
Honest status
The sim story stands: the settings UI works (sim chat, a fact table, a latency meter against a 2.5-second budget, memory export), the suite is green at 164 tests including a sim end-to-end pass and a real-browser Playwright pass, and motion, audio DSP, the brain loop, the naivety gate, memory, and choreography were all built and tested with no robot present.
And then the robot arrived. The Reachy Mini Wireless is assembled and on my desk, and Rocky runs on it: the sim-first brain driving real joints instead of Fakes.
Still open, and stated plainly: the soak-and-failure drills that need hardware to fail against over long sessions, latency measured on the real path rather than asserted against the sim budget, and the Rocky LoRA, which remains a pipeline with a runnable ship gate rather than a trained artifact.
I wrote up the full methodology, with the concurrency walk-through and both diagrams, as a companion post: