Vague, but exciting…

My robot drove itself down a trail. Then it met a corner.

· Colin Rooney

Straights: solved. Tested five times on a straight section, 94–100 % of the time on the trail. Corners: it runs wide every time, and two field days of tuning couldn't fix it. Here's everything that happened since the kitchen — and why the next step is a racetrack on my kitchen floor.

Last time, this was a phone in my hand, walking a kitchen. This time the phone is bolted to an RC truck (the same one that was an obstacle in the kitchen video), and the truck drives itself. I call it the Robo Bump Truck.

Nothing about the method changed. That's the point of this log.

Figure 1. The truck on the trail, from three cameras, plus what the robot sees and what the model draws.
Figure 1. Left to right: the tripod referee on day 1; on the straight and from the bank on day 2; the robot's own view; and the steering model's output on that view: yellow patches score high for "trail", the magenta line is the fitted centre, the dot is the goal point.

1 · What changed on the truck

It now carries a Jetson Orin Nano in a printed cradle, a Pixhawk flight controller running ArduRover, a RadioMaster MT12 transmitter talking to an ExpressLRS receiver, and an iPhone on a wedge mount looking down the trail. The phone's video camera is the only thing the model sees, and it's all the model uses to drive.

2 · Training: two models, no hand-drawn labels

The whole project rests on one idea: expensive sensors teach, cheap sensors ship. The phone carries LiDAR, motion tracking and an accelerometer. All of it is used to label the training images automatically. None of it is used to drive. The field calls this self-supervised learning, in a specific sense: the robot's own other sensors write the labels, and no human does. The training step itself is ordinary supervised learning against that answer key — the word "self" refers to who wrote the labels, not to how the model learns. The point of it, beyond the shipping argument, is time: every hour the sensors spend labelling is an hour a human doesn't. There are two models, and each has a different teacher. If you want to dig deeper, the obstacle model is in the lineage of Berkeley's BADGR, which labels images from the robot's own collision and bumpiness sensors; the steering model is in the lineage of ETH's Wild Visual Navigation, which labels images from the path the robot actually traversed.

2.1 · The obstacle model (DangerNet) — taught by bumps

This is Log #1's method. I walk up to each obstacle, and at arm's length I bump the phone body with my free hand — the phone never touches the obstacle. The accelerometer marks the moment; the LiDAR marks where the obstacle is; and that spot is projected back into every frame of the approach as "no-go." The floor I walk over in the next few seconds is projected back into those same frames as "go" (Figure 2). Thirty bumps became 147,731 labelled image patches.

Figure 2. One bump, two kinds of label.
Figure 2. One bump, two kinds of label. The obstacle's location is projected back into the 12 seconds of approach frames as no-go; the floor the camera walks over in the next 1–8 seconds is projected back as go. Where they overlap, no-go wins.

2.2 · The steering model — taught by where the truck went

The model that drove the truck has a different teacher: the truck's own path. I drive a lap by hand while the phone records video and its motion-tracking pose. Afterwards, the path the truck actually took is projected back into every frame, and the image patches that path passes through are labelled "trail" (Figure 3). No LiDAR in this chain at all, and no human drew a line to teach it.

Figure 3. The steering label is the path the truck drove.
Figure 3. Motion tracking records where the truck went. That path, projected back into each earlier frame, is the label: "the trail was here." The model learns to find it from pixels alone.

I did paint trail centrelines by hand on 53 frames — but to check the result, never to train it. On held-out data the model's centre landed about twice as close to my hand-painted centre as the pose labels it learned from.

2.3 · One backbone, two heads

Both models are the same shape (Figure 4): a frozen DINOv2 vision backbone that turns each image into a grid of 850 patch features, and on top of it a tiny logistic-regression head, one per model, that scores each patch. The obstacle head answers "must I not hit this?"; the steering head answers "is this trail?". Each head is 385 numbers and trains in a couple of minutes on a laptop. At runtime they see RGB pixels, nothing else.

Figure 4. One frozen backbone, two tiny heads.
Figure 4. The backbone is pretrained and never changes; it is where the general visual knowledge lives. Each head is a 385-parameter classifier over the backbone's patch features. The expensive part, the backbone pass, is shared, so a second head costs one extra matrix multiply. The driving tool has a flag to run both on the same frame. On these trips the obstacle head was logged and drawn on the display only; it never steered.

Each head answers one yes/no question for every patch, independently, and the answer is a score from 0 to 1 (Figure 5). A heat map is just those 850 scores painted over the frame.

Figure 5. Per-patch scores from each head.
Figure 5. Left: DangerNet's no-go score per patch on a kitchen frame from Log #1, red high. Right: the steering head's trail score per patch on the trail, yellow high, with the line fitted through the row centres. Neither head knows what an object is; each only scores patches.

On this trip the obstacle model ran display-only. It was trained handheld at walking height, and at truck height on a trail it's out of its depth — literally. It gets retrained from the truck next. The steering model is what drove. Which expensive sensor does the teaching depends on which model you mean (Figure 6), and neither one keeps it.

Figure 6. Expensive sensors teach, cheap sensors ship.
Figure 6. Which expensive sensor teaches depends on the model. Both ship seeing only camera pixels.

3 · Live inference: getting the picture to the robot, fast

3.1 · The robot's eye is a phone, and it talks over one cable

The video gets from the phone to the Orin over a single USB cable. No capture card, no wifi, no radio link for the video. That sounds unremarkable until you try it. The published failure reports from people trying this describe the phone never appearing at all: usbmuxd #281, filed on the same phone and iOS version I'm using, and record3d #120. It works here because the Orin's usbmuxd is started with preflight disabled: the device enumerates and a tunnel binds before the pairing handshake completes, which is the whole bypass.

Once it worked, I wanted to know whether it kept working, because the neighbouring reports describe setups that connect and then die: usbmuxd #215, "it works fine for a random amount of minutes, until it dies", and record3d #72, a robot streaming from an iPhone over USB with "variable lag in the video stream (1-3 seconds)", open since 2023. So, a soak test. Thirty minutes continuous:
delivered rate 15.02 frames/s sustained
frames 27,000
gaps over 1 s 0
disconnects 0

That was indoors, on a bench, in a cool room. Outside is a hotter place, and other people's phones have overheated doing this. Hold that thought — the field had opinions about it.

3.2 · How long does the robot take to see something?

This matters because it sets the floor under everything downstream. The loop has three stages: perception (what am I looking at?), control (what should I do about it?) and actuation (move the steering servo). This delay sits in front of all three. Control engineers call it dead time: however good the rest of the loop is, it is acting on a world that has already moved on. At 1 m/s, 60 ms is 6 cm of travel before the robot sees anything at all. Dead time is also what turns a confident controller into an oscillating one, so I needed the number.

The obvious approach doesn't work. You can't ask the phone "when did you see this?" The phone's clock and the robot's clock are different clocks, and the question spans both.

Here's the intuition, and it's a yodel. You're alone on one side of a valley and you want to know how long your voice takes to get across it. There's nobody over there to shout back. So you rig a billboard on the far side that flashes the instant it hears you, you yodel "Yo!", and you run a stopwatch until you see the flash. No partner required: you made something happen at a moment you knew, and you timed the consequence coming back.

(The yodel is for human intuition only. Nothing in the real rig makes a sound: light crosses a valley far too fast to time with a stopwatch, which is exactly why the real measurement is all visual, and why the slow thing being measured is the camera pipeline rather than any distance.)

So the robot rigs the same trick with light, on a billboard it controls (Figure 7). It serves a web page that flashes black to white on a schedule set by its own clock; I put that page on a monitor and pointed the phone at it. However long it takes for that flash to turn up in a frame the robot has actually received — that's how long the robot takes to see the world.

Figure 7. The billboard rig.
Figure 7. The robot flashes a page on its own clock and watches, through the phone, for its own flash to arrive.

About 60 milliseconds. Eleven clean flashes, mean 60.5 ms, standard deviation 4.6 ms.

Two things I have to say about that number. It's an upper bound: it includes the monitor's own display lag, which this method can't separate out, so the camera path alone is faster. And "eleven clean" means I threw four out: they came back at 322 to 990 ms, and in each of those the camera was still recovering its exposure after the screen went dark, so it never saw a flash at all — excluded on a brightness rule I'd set before seeing the numbers. I posted the 60 ms figure on a forum before writing that down, which was the wrong order.

3.3 · How a frame becomes a steering command

Every frame, the steering model scores each patch for "trail." In each row, the probability-weighted centre of the patches it calls trail is that row's trail centre, rows where that mass is spread out are dropped, and a line is fitted through the rest. That is the drawn trail centre. A steering law called pure pursuit picks a goal point on that line a fixed distance ahead, the lookahead, and computes the curvature (in 1/m: how tightly the truck must turn) that would carry it through that point; a gain scales how hard it steers for that curvature. The Orin converts the result to a servo pulse width (in microseconds) and hands it to the flight controller, which drives the steering servo (Figure 8). Every stage (the fitted line, the curvature, the command, and what the radio and servo actually did) is logged per frame, so every pass can be replayed afterwards. That log is how every fault in this post was found.

Figure 8. Signal path.
Figure 8. Camera to steering model to pure pursuit to servo pulse to flight controller to servo, with a log tap on every stage.

4 · Day one: it drove itself

First real trail, and it worked.

Five passes down the straight at the settings I'd ended up on (lookahead 2.0 m, gain about 1.8), judged by a tripod I set up on the bank — a fixed camera watching the truck, with a tracker reading the ground colour underneath it. 94 to 100 % on the trail on every pass, one excursion longer than two seconds in 227 seconds of driving.

I want to be exact about what that number covers: the tripod's view was fixed down the straight, and the corner was out of frame. So it's a measurement of the straight. It is also not the robot grading its own homework, which is the part I care about — the robot's own confidence had no vote.

Then the corner. It saw the turn and started it, and then it went one of two ways. At a low gain it turned too gently: the curvature it was commanding climbed from 0.33 to 0.74 1/m, it asked for about 55 % of steering lock, and it arced wide into the grass. With the gain turned up it hunted everywhere, straight or corner: 152 sign changes of the steering command in 319 seconds, weaving back and forth across the trail.

The fix for the zigzag wasn't the gain. Standing there watching it, my read was "we need to look farther up the trail," which turns out to be pure pursuit's damping knob. Pushing the lookahead from 1.5 m to 2.0 m cut the weaving by more than half (0.45 to 0.19 direction changes per second). Nothing at that lookahead fixed the running wide.

5 · Day two: the app broke, then the corner didn't yield

Day two was the first field use of Robot Eyes, the iPhone app I'd built overnight. I'd been recording training data with a scanner app and streaming live video with a separate one, and no app did both from the same camera session. I wanted to gather training and validation data while the model drove on the same frames. It's the tool that later lets you replay exactly what the model saw and did. So I wrote one. It failed in the field in a way I'm glad happened early.

The phone got hot. ARKit throttled the camera to 20 frames/s, which is documented behaviour. The bug was mine: the app decimated frames by a fixed divider, so when capture dropped to a third, what reached the robot dropped to a third with it: 5 frames/s. The steering loop ended up running at 5 Hz. I fixed it sitting in the car between the afternoon and evening sessions, so I could keep recording training data and driving live off the same frames that same day; the evening ran at the full rate; the divider is now recomputed every second from the measured capture rate, so the app targets a frame rate instead of a ratio. The proper bench acceptance that night found two more defects, which is the argument for benches.

Heat is still the open problem, and I'd rather say what I'm doing about it than pretend it's solved. What's actually in: the app now reports its own thermal state every second and halves the video bitrate when iOS says the phone is struggling, so the robot at least knows, and I can read it back afterwards from the session file. What I want to try, none of it tested yet: airplane mode, to stop the phone spending power on radios it doesn't need; a small travel router on the truck so the phone isn't running wifi at all; and settling whether running it on the charger helps or hurts, because a charging battery is its own heat source. The phone is doing camera capture, depth, motion tracking and video encoding at once, in the sun, which is a lot to ask of something designed to live in a pocket.

The second fault I found with my hands rather than any log. Driving by hand on the way to the trail, the truck tracked along the left edge, in the grass, exactly parallel to the trail. Strange. I trimmed it straight on the transmitter. Then, under the model, it did exactly the same thing again. The transmitter's trim and the model's idea of "straight" turn out to be two different numbers that can't see each other: a steering servo is commanded with a pulse width, in microseconds, and the flight controller holds a centre value it calls straight (1500 µs). The truck's actual straight was 1463 µs (Figure 9). So every time the model asked for "no steering," it got a little left: a constant bias, in control-theory terms a steady-state error. Setting the flight controller's centre to 1463 fixed it. Pose data later confirmed the value independently: fitting the truck's recorded path curvature against the servo pulse, the zero crossing lands at 1461.2 µs — 1.8 µs from the number I set by feel.

Figure 9. Steering neutral.
Figure 9. The flight controller's centre pulse against the truck's real straight. 37 µs is about 9 % of one side's steering travel — enough to drift.

With both fixed, the straight held: 99–100 % on the trail across three passes, measured on the straight. But that number is graded differently: the tripod wasn't up, so it comes from a second small classifier on the robot that reports whether the camera is looking at trail, the robot's own trail head (Figure 10). It's the robot marking its own homework, which is exactly what day one's number wasn't, so it isn't comparable and I'm not claiming it as an improvement.

Figure 10. Two graders.
Figure 10. Day one's number came from an external referee; day two's from a classifier on the robot. They are not the same measurement.

And then I spent the rest of the session on the corner, systematically: one knob at a time. I kept a matrix table of the settings, and I moved the tripod to the corner and used it to count how many times the truck left the trail, because I wanted empirical validation and it was too much to hold in my head. Every lookahead from 1.0 to 3.0 metres at one gain, plus one pass at a higher gain: seven passes, one per cell, and the log of each one pulled and graded before the next.

It didn't work. Not one of them held the turn. Every pass left the trail in the bend or just after it, and rejoined further on. Short lookahead, long lookahead, more gain: the truck ran wide every single time.

And the sweep couldn't even tell me which way to go next. A 2.5 m lookahead graded worst and 2.8 m graded best, which isn't physics — it's two single passes at different speeds on different entry lines. Across two field days I'd tried every lookahead from one to three metres and gains from 1.0 to 2.1. None of them made the corner, and the differences between them were noise.

6 · The trail can't measure this problem

That's the actual finding, and it took me embarrassingly long to see it as one.

I was trying to tune the steering law with a test that couldn't resolve the thing being tuned. The throttle was my trigger finger, so no two passes entered the bend at the same speed or on the same line, and with one pass per setting I couldn't rank two lookaheads half a metre apart under that spread. Speed and entry line were uncontrolled variables, and they were bigger than the variable I was changing.

So the answer isn't a better gain. It's a controlled course, where speed is held and the same lap repeats, and a grader that needs neither a tripod nor the robot's opinion.

I built a racetrack out of blue tape on my kitchen floor.

That's the next log.

What I'm not claiming

Glossary

Full build notes, including every wall I hit and how I got past it, are in the troubleshooting guide. Video: Log #1's film is here; Log #2's six short clips land on the same channel, one a week. Previous log: I rearranged my kitchen to fool my robot. It found the truck anyway.