Raising an Alien in Simulation: Building a Robot's Brain Without the Robot

I am building Rocky, the Eridian engineer from Andy Weir's Project Hail Mary, on a Reachy Mini Wireless robot. The framing is a curious alien child dropped on Earth: I teach him about the world, he remembers what he learns across sessions, and he answers in chords and a ring-modulated voice that is not quite human. The whole brain is built to run on my own machine at zero paid-API cost. This is a personal homage to Weir's novel and to Pollen Robotics' little robot, not an original character, and I want to name both sources before anything else.

Here is the catch, and it is also why this post exists. I built and tested Rocky's entire software brain in simulation, without the robot, without the GPU, and without the local language model in the loop. Nothing below describes a physical robot that has been driven or a local LLM that has answered a real prompt in this build. The interesting engineering is exactly that: how you write and test a robot's whole nervous system when you do not have the robot, the model, or the hardware to run any of it, and how you keep that codebase honest about which half has actually run.

The seam is the whole idea

The reason to work sim-first is boring and correct: hardware and a local model stack are slow and awkward to iterate against. A servo you can burn out, a 5.5 GB model you have to keep warm in VRAM, an audio backend that fights the operating system. None of that belongs in the inner loop where you are changing a persona rule for the fortieth time in an afternoon.

So every external service in rocky_mini sits behind a Protocol, and the sim and test paths use Fakes. The language model is a Protocol. Speech to text is a Protocol. Text to speech is a Protocol. The robot's own motion and audio surfaces are Protocols. In the sim path a rule-based SimResponder plays Rocky in character, well enough to drive the settings UI and run a full conversation turn end to end. The real path swaps in OllamaLLM behind the same interface. The imports for the real clients are lazy and guarded, so the sim never touches Ollama, faster-whisper, Piper, or the Reachy SDK. That is not a mock in a test file. It is the seam the runtime code is built around.

The left side of that diagram is real code that runs and passes tests today. The right side is real code too, written against the same Protocols and documented for the machine that has the GPU and the robot, but it has not executed in this build. The easy mistake in a sim-first build is to let a green test suite quietly imply that the robot works. It does not. The robot has never moved. What works is everything up to the seam.

Four latency domains, one process

The part of this project I would actually defend in a systems interview is the concurrency model, because a talking robot is a real-time problem wearing a toy costume. If Rocky takes a beat too long to acknowledge you, the illusion breaks. If his mouth and his voice drift apart, the illusion breaks. If a motion update and an audio sample fight over the same resource, you get a jerk or a click, and the illusion breaks. Each of those failures lives in a different time budget, so I gave each one its own owner.

One process runs four threads plus one asyncio loop, and each owns exactly one real-time surface.

Follow one turn across the lanes. AudioIn takes the microphone through voice activity detection and emits a speech event. That event wakes the ConversationLoop, an asyncio loop that does the language work. Its first move is to fire an acknowledgement chord, a short Eridian stinger that plays within milliseconds so Rocky feels like he heard you before he has thought of anything to say. Then the streamed LLM produces tokens, a chunker splits them on clause boundaries and inserts Rocky's verbal tics and emote tags, and each chunk goes to text to speech as soon as it is ready rather than waiting for the full reply.

The audio all lands on the Mixer, the single owner of push_audio_sample. The ack chord arrives on a ChordBus, the spoken chunks arrive on a generation-tagged VoiceBus, and the Mixer is the one place that ring-modulates and soft-clips them into a single output stream. Meanwhile the MotionThread runs at 100 Hz as the single owner of set_target, layering idle breathing, emotes, an additive wobble, and a direction-of-arrival gaze that turns Rocky toward whoever spoke. Every frame it computes gets clamped to safe position, velocity, and acceleration limits before it goes anywhere near a joint.

The two load-bearing rules are the ownership ones. There is exactly one set_target owner and exactly one push_audio_sample owner. Every other thread that wants to move a joint or make a sound asks those owners; nothing else writes. That is the difference between a robot that composes breathing, a turn of the head, and a spoken chord into one smooth motion, and a robot that stutters because three threads all decided to be in charge of the neck at once. The generation tag on the VoiceBus is the same idea in the audio domain: when you interrupt Rocky mid-sentence, the barge-in bumps the generation and the Mixer drops the stale chunks instead of talking over itself. On the development machine that barge-in is half-duplex by default, because there is no acoustic echo cancellation on a PC and open-mic barge-in is something to tune at hardware bring-up, not to pretend it works now.

A local, zero-cost stack

There are no API keys anywhere in this project, and that was a hard constraint, not a nice-to-have. The brain is Qwen2.5-7B-Instruct, Apache-2.0 licensed, served by Ollama on my own PC with an RTX 4080, held warm in VRAM (keep_alive set to never expire) so that on the real path the first token comes back fast. There is an optional Rocky LoRA on top of it. Speech to text is faster-whisper; text to speech is Piper with the en_US-lessac-medium voice, ring-modulated afterward into Rocky's register. The PC does all of the thinking and acts as a LAN brain server. The Reachy Mini itself, a Raspberry Pi CM4 with 4 GB of memory, is a thin client: it does voice activity detection, motion, and audio mixing, and runs no model at all.

I want to be precise about the LoRA, because it is the easiest part of this stack to overstate. There is no trained Rocky LoRA. What exists is a fine-tuning pipeline: a small seed dataset, a training scaffold for WSL2 with Unsloth, and a ship-gate evaluation you can actually run. The eval is the honest part. It refuses to promote a fine-tune that does not beat the stock model plus a good system prompt on a naivety probe. Training that LoRA needs a GPU and a WSL2 setup that were not part of this build, so the pipeline is written and documented and the gate is runnable against the sim brain, but the artifact it is meant to gate does not exist yet. Saying otherwise would be the exact inflation this whole post is trying to avoid.

Character and learning, kept in their lane

Rocky is supposed to feel like a curious child, and most of that is small systems rather than one big model prompt. He does not track faces; he holds a blind, slightly-off gaze on purpose. He has verbal tics and chord stingers. He has a sleep-watch ritual and a set of idle behaviors. Under that is the part I find more interesting: a naivety auditor that enforces what Rocky is allowed to know. An alien child who has been on Earth for a week should not casually explain compound interest or quote a pop song, so the auditor runs a 20-probe red-team suite and treats naivety as a thresholded regression metric rather than a hard zero-leak gate. A brittle zero-leak gate would block every change; a tracked metric lets me see the persona drift and decide.

Memory is a JSONL store with a confidence lifecycle: new facts start provisional, get confirmed or deleted through the settings UI, and age through a digest and reflection pass so Rocky's sense of the world consolidates instead of just growing. It lives outside the app directory so it survives a reinstall, and it exports to a zip, which is how a Rocky raised in sim would eventually move onto the robot. I am describing the design here rather than quoting the novel, both out of respect for the source and because the engineering is the transferable part.

What actually runs, and what only compiles

The honest ledger. The sim runs end to end. The settings UI is a real, running artifact: a sim chat that drives a genuine conversation turn, a fact table you can confirm and delete against, emote buttons, live server-sent metrics with a latency meter measured against a 2.5-second budget, a model toggle, and memory export. The test suite is green at 126 tests, spanning unit tests, the FastAPI surface, a sim end-to-end integration pass, and one real-browser Playwright pass over that settings UI. Motion, the Eridian audio DSP, the brain loop, the naivety gate, memory, and the choreography rituals are all built and tested in sim.

What is seamed but not exercised is the entire right column of that first diagram. Voice-in has its VAD, STT, and barge-in seams and Fakes in place, but real speech to text runs through the brain server, which needs the models. Hardware bring-up is documented with guarded code paths and has never met a robot. The soak-and-failure drills exist as a canned Eridian-only fallback mode but the drills themselves need the hardware to fail against. In milestone terms, the software milestones through the sim brain are done and tested, hardware bring-up and the soak drills wait on the physical robot, and the LoRA waits on a GPU. I would rather write that plainly than round a "seamed" up to a "working."

What this was worth

None of this proves Rocky works, and I have tried not to imply that it does. What it shows is that you can build and test the full software nervous system of an embodied character without any of the embodiment present, if you are willing to put every external dependency behind a Protocol and live with Fakes in the inner loop. The transferable pieces are the ones I would carry into any real-time or edge-AI job: a concurrency design where each latency domain has a single owner, a local and offline LLM serving path with a language model kept warm behind a clean interface, and a seams-and-Fakes testing discipline that lets one codebase run today in sim and drive hardware the day the hardware arrives.

The next real test is hardware bring-up, where the clamps meet actual joints and the half-duplex assumption meets a real room. That is the point where the right column starts to run, and where I will find out how much of this survives contact with a body. Until then, the claim is bounded exactly: the brain is built and green in simulation, and it has not yet moved a robot.

The code, including the same honest scope this post leads with, is here: github.com/TirtheshJani/rocky_mini.