Context
Industrial sites are often covered by fixed cameras whose footage is only reviewed after an incident and yields no usable data. The aim was to turn existing cameras, and occasional drone footage, into a measuring instrument: for every detected object, a latitude/longitude position and metric distances rather than a pixel box.
This is hard for three reasons. A single camera gives no depth, so scene scale must be inferred. The usual georeferencing method, a ground-plane homography, assumes a flat floor and fails silently for raised objects or uneven terrain. And high-resolution video is expensive, while the common shortcut of downscaling to 640 px discards the geometric detail ground positions depend on.
Approach
Methods
- Detection. A YOLO-family detector, at several model sizes, in four modes (detect, segment, pose, oriented boxes). Inference runs at native resolution with no forced
imgsz. Segmentation is the default because masks give cleaner depth samples than boxes. PyTorch, ONNX, TensorRT and CoreML weights are accepted (CUDA and Apple MPS). - Monocular depth. A metric monocular depth model predicts a depth map, which is resampled to the frame size. Each object’s depth is a robust estimate over its mask pixels, after outlier rejection. A validation step compares predicted depth with reference zones whose distance is known, to detect calibration drift.
- Geolocation by ray casting. A ray is cast from the camera centre through the object’s pixel and stopped at the object’s metric depth. The resulting point is converted from the camera frame to a local North-East-Down frame, then to WGS84. Unlike a homography, this does not assume a flat ground. An optional DEM supplies ground elevation. Positions can be overlaid on a georeferenced orthophoto for visual checks.
- Tracking. A custom multi-object tracker with a motion model, confidence-aware association and a track-fusion step that links fragments using gap, heading and speed limits. When depth is available, each track stores a 3D trajectory, from which the distance travelled and the speed in m/s are derived.
- Inter-object distances. Later consolidated into a reusable vision library. Two segmented objects are back-projected into 3D, and the minimum point-to-point distance is reported. Overlapping masks are treated as contact. A Monte Carlo simulation perturbs depth, focal length and pixel positions to give a standard deviation and a 95 % interval. Zone membership uses the share of the mask inside the zone, with hysteresis, rather than a centre-point test.
- Engineering. The configuration supports a two-GPU split (detection on one device, depth estimation on the other), RTSP discovery and recording, and processing of several cameras in parallel.
Results
Measured on a single NVIDIA RTX 5090. The detection benchmark covers detection and tracking only, with depth disabled.
| Setting | Video | Throughput | Output |
|---|---|---|---|
| Largest detector, batch 1 | 3840×2160, 50 fps, 3,301 frames | 5.1 fps | 320,217 detections, 1,941 tracks |
| Smallest detector, batch 1 | 3840×2160, 50 fps | 13.0 fps | 158,989 detections, 908 tracks |
| Largest detector, batch 1 | 2304×1296, 15 fps, 5,000 frames | 17.9 fps | 73,502 detections, 2,706 tracks |
| Smallest detector, batch 1 | 2304×1296, 15 fps | 39.0 fps | 51,890 detections, 1,499 tracks |
| Full chain (detection + depth + tracking + storage) | 2304×1296, 300 frames | 2.02 fps | 12.3 detections per frame |
In the full-chain run, depth estimation took 117.6 s of the 148.3 s total (about 80 %), and detection took 20.9 s. Depth estimation, not the detector, is the bottleneck. On 4K footage the largest detector runs about ten times slower than real time. That suits the intended use, which is analysis of recorded archives, not live alerting.
Limitations & next steps
- No geolocation accuracy has been measured against surveyed ground truth yet. The ray-casting stage was added after the timing runs above, so its cost is not included in them.
- The metric scale depends on the depth model and on the camera pose and intrinsics. Errors in tilt or focal length propagate directly into ground positions.
- The depth model predicts at its own internal resolution, so fine depth detail is limited even when detection runs in 4K.
- Next steps: an accuracy study with GNSS-surveyed targets, profiling the two-GPU split, and measuring how accurate Monte Carlo distance intervals are on objects at known separations.
Stack
Python · PyTorch · OpenCV · NumPy · SciPy · pyproj · rasterio · SQLite · CUDA · Apple MPS