I rearranged my kitchen to fool my robot. It found the truck anyway
30 bumps of a phone became 147,731 training labels, a 385-parameter model, and an honest experiment.
The idea: expensive sensors teach, cheap sensors ship. During training, an iPhone's LiDAR, ARKit pose tracking, and accelerometer do all the labeling work. The model that comes out consumes none of them — RGB images are its only runtime input. The first truck prototype still carries the whole phone (it provides the camera and the compute); what's being tested is whether plain camera images are the only perception signal the finished model actually requires.
I've shipped this pattern before. I spent twenty months building a running-gait system where an expensive camera rig generated the labels that trained a watch model — and the watch shipped without the camera. This project is that pattern, made visible.
How the labels happen
I walk the room for a few minutes. When I'm at arm's length from something the truck shouldn't hit, I bump the phone body with my free hand — the phone never touches the obstacle. Walking never exceeds about 0.6 g of accelerometer deviation (g as in units of Earth's gravity); bumps land at 1.6–4.7 g, so detection is a threshold. The accelerometer identifies when a bump occurred, LiDAR identifies the nearby obstacle in 3-D, and ARKit projects that location back into every earlier frame that saw it on approach — those patches are labeled "no-go." Floor I walk over seconds later labels itself "go." Thirty bumps became 147,731 labeled image patches. No manual image labeling — nobody draws a box.
The model is deliberately tiny: frozen DINOv2 image features feed a logistic-regression head with 385 trained parameters. It retrains locally in about two minutes. Model size doesn't make an evaluation trustworthy by itself, so the experiments use held-out time blocks and pass/fail gates defined before training.
The gate, then the experiment
Before training, I wrote down a pass/fail gate with two separate conditions: at least 0.85 AUC against LiDAR-assisted labels on held-out time blocks, AND heatmap overlays that visibly outline the obstacles. A good score doesn't prove a good picture. The first session — a 3.5-minute walk, 21 bumps — passed both: 0.932, and you could see the obstacles in the heat.
Same-session held-out is the easiest honest test there is. So when the kitchen got rearranged, I treated it as a free experiment — and evaluated the old model on the new room before retraining anything. What changed: the parked RC truck moved to a new spot; a battery and charger were removed; a 3D-printed plastic hand appeared (made for last Halloween; never seen, never bumped); a stool and a drone case stayed put.
Results, old model on the new room: the heatmap outlined the truck at its new location — it found the truck, not the truck's old spot — and showed no comparable response in the empty space where the battery and charger had been. Then I re-ran the whole loop in the new room: 30 bumps, ~2 minutes of training → AUC 0.9403 on that recording's held-out labels. The old model on those same held-out labels: 0.9224 — a 1.8-point gap on identical test data, for a model that had never seen this arrangement. (0.932 and 0.9224 come from different recordings, so I'm not attributing their difference to the rearrangement alone.)
Why test like this at all? An earlier wrist-IMU model from my running project dropped 21 percentage points in a matched test on another day — different task, different metric, so the numbers aren't comparable; together, though, those two experiments are why I now test every system under changed conditions instead of trusting a single held-out score.
The useful failure
The 3D-printed hand is the best part. Under the old model it produced a strong response nearby but only a faint one from across the room: predicted no-go probability 0.19, in a frame whose median was 0.00. After retraining with bumps that included the hand, the same distant viewpoint — measured on a separate no-bump walk — reads 1.00, in a frame whose median is 0.16. The retrained model runs hotter overall, so the margin over the median is the result — and it's one object at one viewpoint, not a generalization claim. But that's the loop: new dangers need new bumps — and the bumps work.
Honest limits
Not tested yet, and therefore not claimed: the truck's ~15 cm camera viewpoint (all data is hand-held so far), another building, another operator, dynamic obstacles, closed-loop control. One person, one home, real measurements. Goal for November: phone mounted on the truck (the mount is real — it's in the video's first seconds), the danger overlay on-device, and driver assistance — a camera-only model flags danger while a human drives. If testing supports it, the next step is shared autonomy, with the human keeping immediate override.
Lineage: BADGR (UC Berkeley) and Wild Visual Navigation (ETH Zürich), shrunk to consumer hardware. Every number above traces to a dated findings ledger.
Next log: the truck goes outside.
Video: youtu.be/4TfdhlGQR4A · Full build log with receipts: coming to Hackaday.io — link will land here.