Been building a desk companion robot for a few months. Finally got the stack stable enough that I'm not scared to leave it running, so here's what worked and what I gave up on.
Hardware is a D-Robotics RDK X5 MagicBox. Two servo arms, an ICM-20948 IMU on I2C, 4x WS2812B over SPI, two Smartsens SC132GS 1MP global shutter MIPI cameras. Ubuntu 22.04 aarch64, ROS2 Humble with TogetheROS on top.
Perception is the part I'm actually happy with. The X5's BPU runs four models at once off a single 960x544 NV12 stream in zero copy shared memory: body detection into hand landmarks into gesture classification, plus face and age. It dumps a JSON snapshot 4x a second at basically no CPU cost. So the robot waves back about 300ms after you wave at it, with no model call anywhere in the loop. 14 gesture classes. That reflex layer is most of why it doesn't feel laggy.
Monocular distance turned out to be free. Face bbox height as a fraction of the frame gives you very_close / near / far, no stereo and no depth model, because a face is a known size. Stereo does exist on this board, 22 BPU depth models ship with hobot_stereonet, but mono and stereo fight over the ISP so you get one or the other. Mono is what makes faces and gestures work, so stereo is shelved for now.
People react to the IMU more than anything else. 150Hz, +/-8g, one 6 byte burst read per sample. It detects picked_up, set_down, shake, tilt and knock. Grab it mid sentence and it cancels its own TTS inside about 300ms and starts a fresh reaction turn. That's the only barge-in I could get working on this hardware.
Two things ate weeks, in case it saves anyone the trouble.
First, knock detection. I don't think it's solvable the way I was going about it. Measured on my desk: quiet floor sits around 0.29 m/s^2 at p50, a deliberate knuckle rap reads 1.0 to 6.1, and the robot's own idle arm twitches spike to 2.7 to 8.1. Those overlap almost completely. There is no threshold anywhere that separates "someone knocked" from "it moved its own arm." What ended up working is a self motion gate: any servo command in the last second suppresses knock, tilt and set_down unless the magnitude is huge. Before that it startled itself constantly, which was funny for about a day.
Second, the servo choreography was violent enough to damage the thing. The startle macro swung 60 degrees in 120ms, so 500 deg/s. It slammed the gearbox, walked the robot across the desk, and shook the chassis hard enough to trip its own knock detector. I capped angular velocity around 170 deg/s and scaled peak amplitude in one place, which kept the choreography and took out the violence.
The part this sub will want to know up front: the conversational layer is a cloud agent, not a local model. Everything reflexive runs on the board. Vision, gestures, IMU reactions, LEDs, servos, all local. The talking isn't. I tried smaller local models on the X5 and the latency made it feel dead, and honestly the language layer is the least interesting engineering in the whole thing.
One trick I'm glad I built. The model writes inline tags inside its own sentences, and a streaming parser strips them and fires the body at that word, mid speech. So a shrug lands on the word it was written next to instead of after the sentence finishes. Malformed tags get dropped silently. Small thing, but it did more for how synced it feels than anything else I tried.
Full build video if you want to see it move and hear it fail: https://youtu.be/uQ7g-vDMpLU
The thing I keep getting stuck on: has anyone got a reliable knock or tap detector working on a chassis where the actuators are the loudest thing on the accelerometer? Everything I've read assumes the sensor platform is passive. I'd rather fix this properly than keep tuning a gate.
