73% of a slow video render was cv2.VideoCapture seeking — a frame cache cut it 443s→69s

August 28, 2026 · Colin Rooney

A composed "highlights" video in my gait-analysis pipeline — eight segments plus section cards, built from one source clip — took 443 seconds to render on a cloud GPU worker. A 4K clip took 63 minutes. The app timed out; the render pool starved. The obvious suspects were the drawing code (lots of cv2/numpy overlay work) and the software encode.

Both innocent. cProfile said 73% of wall-clock was cv2.VideoCapture.set(CAP_PROP_POS_FRAMES) — seeking the source video.

Why seeking dominated

The composer builds ~8 segments, each revisiting different, overlapping frame ranges out of order. Frame access went through a small LRU cache (60 frames); on a miss it did:

cap.set(cv2.CAP_PROP_POS_FRAMES, idx)
ok, img = cap.read()

The trap: VideoCapture.set(CAP_PROP_POS_FRAMES) is not O(1). Video is inter-frame compressed, so OpenCV decodes forward from the nearest keyframe on every seek. For a 600-frame clip accessed out of order across 8 segments, the 60-frame cache thrashed — nearly every access re-seeked. Measured locally: ~1,251 seeks × ~61 ms ≈ 76 s of a 104 s render, pure seeking. Drawing was ~10%; encoding ~11%. The render was never the bottleneck — frame I/O was.

The fix: decode once, sequentially, into memory

Read the whole clip front-to-back one time into a dict keyed by frame index; frame(idx) becomes a memory lookup. Guard against huge clips so a long or 4K source can't OOM the container:

def _predecode(self):
    self._decoded = True
    nframes = int(self.cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
    if nframes <= 0 or nframes * self.W * self.H * 3 > 6 * 1024**3:
        return  # too big — keep the per-access seek fallback
    self.cap.set(cv2.CAP_PROP_POS_FRAMES, 0)
    i = 0
    while True:
        ok, img = self.cap.read()
        if not ok:
            break
        if img.shape[1] != self.W:
            img = cv2.resize(img, (self.W, self.H))
        self._cache[i] = img
        i += 1

Sequential decode is what video codecs are optimized for; you pay the decode cost exactly once.

Results (measured)

Byte-identical output (same md5) — a pure speedup, which also made it safe to ship without re-validating the video content.

The check most optimizations skip: does the rest of the pipeline have this bug?

No — and knowing why is the useful part. The main renderer reads frames sequentially (cap.read() in a loop, no seeking), so it was already fine. One other POS_FRAMES use grabs ~3 frames per event — bounded, ~1 s. The thrash was unique to the one component with an out-of-order, multi-segment access pattern. The fix stayed scoped to that file, and nothing else needed touching.

Limits

• Cost still scales with source pixels: the renderer draws on full-res frames even though the output canvas is small. A 4K source is ~63 minutes and should be downscaled before rendering (that change touches pose-coordinate scaling, so it carries correctness risk and wasn't done). Our capture path emits 1080p, and the memory guard degrades gracefully, so production is safe as-is.
• The 6 GiB guard is sized for a 16 GiB container; scale to yours.

The transferable rule

For any renderer that revisits frames out of order: decode once into a cache; never random-seek per frame. And before optimizing a slow multi-pass renderer, profile — python -m cProfile -o out.prof … then sort by cumulative — because the plausible bottleneck (drawing, encoding) and the actual one (I/O) are routinely different, and one run of the profiler settles it.