Corso Mazzini
Pedestrian tracking on a city square, projected through a homography onto an orthophoto — the video and the ground plane driven by one timeline, so a person and their trajectory are the same click.
Introduction
Real-time pedestrian tracking with homography-based spatial projection. Two minutes of a fixed camera over Corso Mazzini, every person the detector found, and the same people plotted on an orthophoto of the square they are standing in.
The idea came out of a conversation with my friend Raffaella Arena.
The two panels below are the same instant in two coordinate systems. Click anyone in either one and the other follows: their box, the ground point under their feet, the projected dot, and the path they have walked so far.
Context
A detector's output is a stack of boxes in image space. Nobody can read footfall, dwell or desire lines off boxes — the question a square actually poses is where people walk, and that lives on the ground, not on the sensor.
Detect and track once, offline; take each box's ground point; push it through a homography onto an orthophoto of the square; ship the recorded result and replay it against the source clip, so both coordinate systems are visible at the same instant.
Methodology
The detector runs once, offline, over the clip. Each detection carries a track id from the tracking step, and a ground point — the bottom-centre of its box, which is where the person meets the pavement. That single point is what the homography acts on: a 3×3 matrix fitted from four correspondences between the camera image and an orthophoto of the square maps it from sensor pixels to map pixels.
Only the ground point can be transformed this way. A homography maps one plane to another, and the only plane a standing person shares with the pavement is the one under their feet — which is why the box's centre, or its top, would land somewhere the person is not.
- frame / timestamp_seconds
- Position in the clip
- object_id
- The tracker's identity, stable across frames
- bbox_xyxy
- Corner coordinates in 1920×1080 source pixels
- bbox_xywh
- The same box as centre, width and height
- x1_x2_mean_min_y
- The ground point: box centre-x, box bottom-y
- confidence
- The detector's score for that box
- map_coords
- The ground point after the homography, in reference-image pixels
Interactive
Results
- 2,160 frames at 18 fps — 120 seconds — carrying 31,568 detections across 324 track identities, between 8 and 22 people in frame at any moment.
- 70% of projected points land inside the orthophoto crop; the rest are the far end of the street, which is genuinely outside the reference image rather than a projection failure.
- The tracker holds one identity for the full two minutes and loses most others within a second: 171 of 324 ids survive 18 frames, only 69 survive five seconds.
- The whole demo is a video file plus a 1 MB export — the 8.3 MB pipeline output rewritten as columns — with no backend and no inference in the browser.
Technical explanation
There is one clock: the video element's currentTime, read once per animation frame and turned into a frame number by binary search over the exported timestamps. Both panels are handed that same integer. Nothing caches a position, so playing, pausing, scrubbing, stepping a frame and jumping five seconds are the same operation — and the map cannot drift out of step with the overlay, because there is no second thing to drift.
Coordinates are stored in their source spaces — boxes in the video's 1920×1080, projected points in the reference image's 510×748 — and each panel applies one contain fit at draw time. A box exported at pixel (287, 342) is drawn at pixel (287, 342) whether the panel is 320 or 1200 pixels wide, and a click maps back through the same transform to hit-test against the stored data.
The pipeline's own export is 8.3 MB of nested objects. A build step rewrites it as columns — one flat array per field, plus an offset table so frame N's detections are a slice — which the browser turns into typed arrays once, at load. Playback then allocates nothing: drawing a frame is two integer lookups and a loop over a slice. A second index, built in the same pass, gives each object_id its points in frame order, so a trajectory is found by binary search rather than by filtering.
Neither panel is React. Both are canvases with a render(frame) handle called from the clock; the playhead and the clock readout are written straight to the DOM. Steady playback costs zero re-renders, which is what keeps twenty two-thousand-point trails smooth.
Conclusion
- The homography is the honest part and the fragile part at once: it is exact on the pavement and meaningless off it, so a detection that clips a balcony or a shop sign projects to a confident, wrong place on the map. Those points are drawn where the pipeline put them rather than filtered out.
- Roughly 30% of projected points fall outside the orthophoto — the far end of the street, which the reference image simply does not cover. The map fades at that edge instead of pretending the trail ended there.
- Tracking is the weak link, not detection. One identity survives all 120 seconds; the median track lasts twenty frames. Following a single person across the square is possible, but the ids underneath are re-issued far more often than the picture suggests.
- Hashing a track id straight to a colour looked fine and was not: with fifteen people in frame, the birthday paradox puts two indistinguishable markers on screen most of the time. Colours are assigned instead by colouring the co-visibility graph, so two people who are ever on screen together never share one — 24 slots cover all 324 tracks.
- Identity is still encoded three ways — hue, marker shape and the printed id — because 22 people in frame is past the number of colours anyone can tell apart, and colour alone excludes readers who cannot separate hues.
- Smoothing on the trails is a rendering choice, applied to a separate copy of the path. The markers, the readouts and the hit-testing all use the pipeline's own coordinates.