3D Vision on Boris Dupoux

Handheld Video to Photogrammetry and Gaussian Splatting

Thu, 01 Oct 2026 00:00:00 +0000

Context

Indoor sites are often documented with a handheld gimbal camera rather than a survey-grade scanner. The goal was to turn such video into usable 3D deliverables (dense cloud, textured mesh, plan and elevation orthoviews, a Gaussian Splatting scene and lightweight web exports) with no manual steps between capture and output. The pipeline runs natively on Windows on an RTX 5090 (Blackwell, sm_120) that is shared with other long-running GPU workloads, so it must never pre-empt them.

Approach

flowchart TB
 A["inbox/project/videos + GO file"] --> B["Keyframe extraction
optical-flow overlap + sharpness"]
 B --> C["COLMAP 4.2.1 SfM
SIFT, global mapper"]
 C --> D{"QC: registration, reprojection,
trajectory continuity"}
 D -- fail --> C2["Fallback: incremental mapper"] --> D
 D -- pass --> E["OpenMVS 2.4 (sm_120 build)
densify, mesh, refine, texture"]
 E --> F["Ortho alignment
gravity, floor RANSAC, Manhattan yaw"]
 F --> G["Blender renders
orthos, axonometries, orbit"]
 E --> H["gsplat training
MCMC or default strategy"]
 G --> I["Web exports
Potree, GLB, SOG"]
 H --> I
 I --> J["outbox: deliverables, report.json, STATUS"]

Methods

Overlap-driven keyframe selection. KLT tracking with a RANSAC similarity transform estimates overlap with the last keyframe. A frame is emitted when overlap drops below 0.80, tracking is lost or a time limit passes. The sharpest candidate (variance of the Laplacian) is the one kept.
SfM with explicit QC. COLMAP’s global mapper is fast but non-deterministic. In testing it produced reconstructions that met registration and reprojection criteria but were geometrically incoherent. A trajectory-continuity check (jumps above 25× the median camera step) triggers an automatic fallback to the incremental mapper. Below 20 points per image the step fails.
MVS hardening. The official OpenMVS Windows binary has only sm_89 kernels, so it was rebuilt for sm_120. The object ROI is off for rooms because it cropped the cloud. Refinement is dropped when it diverges, and seam leveling (which blackened faces) is off.
Metric-ready orthoviews. The up axis comes from camera orientations. A RANSAC floor plane sets z = 0 and a histogram of wall-normal azimuths (mod 90°) sets Manhattan yaw. An optional two-point reference fixes scale. The 4×4 transform is stored so it can be applied again to the mesh and the splat.
Gaussian Splatting. gsplat 1.6.0 was compiled for sm_120 with CUDA 13 under MSVC, which required local patches. Training uses the MCMC strategy with a Gaussian budget and falls back to the default 3DGS strategy when MCMC diverges.
Operations. Projects start in watch-folder mode when a GO file appears. Stages can be resumed or redone. A GPU guard waits until enough VRAM has been free for a stable period, serialises GPU stages through a lock file and records peak VRAM per stage in report.json. Degraded QC produces a DONE_WITH_ALERTS status instead of a silent success.

Results

First real deployment: five locations inside a commercial showroom, one 4K handheld video per location, processed unattended in about 30 minutes per location.

Location	Images	SfM registered / points / reproj.	Dense cloud	Splat PSNR / SSIM
1	116	116/116 · 21,458 · 1.49 px	1.40 M	23.2 / 0.86
2	103	103/103 · 18,972 · 1.75 px	1.23 M	20.6 / 0.81
3	47	47/47 · 13,167 · 1.17 px	1.26 M	23.1 / 0.86
4	121	121/121 · 11,970 · 1.34 px	1.99 M	21.9 / 0.85 (default strategy)
5	64	64/64 · 12,106 · 1.48 px	0.91 M	21.8 / 0.83

All frames were registered in every location. Reprojection error was 1.17–1.75 px.
Measured VRAM peaks: SIFT +1.3 GB, matching +1.6 GB, texturing +0.5 GB, Blender +5.4 GB, and splatting with 3 M Gaussians at 1920 px +25.2 GB.
The sm_120 OpenMVS build densified the synthetic smoke test in 17 s. The CPU fallback took about 40 min.
Metadata analysis showed shutter speeds of 1/50–1/100 s (about 14 px of motion blur at 4K) and automatic ISO and white balance. Exposure compensation in gsplat (ppisp) gave no gain (PSNR 20.5 vs 20.6). The quality limit is in the capture, not the solver.

Limitations & next steps

PSNR in the low 20s reflects motion blur and auto-exposure. The capture protocol now requires a 1/250–1/500 s shutter, locked exposure and white balance, and a dedicated floor pass.
Merging videos from different locations caused false matches and an incoherent model, so each location is processed separately.
Dark glossy tiled floors reconstruct poorly. Manhattan alignment is not yet reliable with furniture and glazing.
CPU stages are the bottleneck on a 4-vCPU virtual machine.
Planned: comparison with a commercial photogrammetry suite on the same frames, and retesting mesh refinement with the sm_120 build.

Stack

Python 3.11 · OpenCV · Open3D · NumPy · COLMAP 4.2.1 · OpenMVS 2.4 · gsplat 1.6.0 · PyTorch 2.14 (CUDA 13) · Blender 5.2 · FFmpeg · PotreeConverter · glTF-Transform · splat-transform

Geolocated Video Analysis and Inter-Object Distance Measurement

Thu, 01 Jan 2026 00:00:00 +0000

Context

Industrial sites are often covered by fixed cameras whose footage is only reviewed after an incident and yields no usable data. The aim was to turn existing cameras, and occasional drone footage, into a measuring instrument: for every detected object, a latitude/longitude position and metric distances rather than a pixel box.

This is hard for three reasons. A single camera gives no depth, so scene scale must be inferred. The usual georeferencing method, a ground-plane homography, assumes a flat floor and fails silently for raised objects or uneven terrain. And high-resolution video is expensive, while the common shortcut of downscaling to 640 px discards the geometric detail ground positions depend on.

Approach

flowchart TB
 A["Video source
file · RTSP · drone clip (up to 3840×2160)"] --> B["Object detector at native resolution
detect · segment · pose · OBB"]
 A --> C["Metric monocular depth model
depth map"]
 B --> D["Multi-object tracker
motion model + association"]
 B --> E["Per-object depth
mask pixels, outlier rejection"]
 C --> E
 E --> F["Ray casting
pixel ray + metric depth + camera pose"]
 F --> G["WGS84 position
optional DEM for ground elevation"]
 E --> H["3D distances
between masks · along trajectories"]
 D --> H
 G --> I["SQLite store
detections · tracks · GPS trajectories"]
 H --> I
 I --> J["Analytics
zones · flows · orthophoto overlay"]

Methods

Detection. A YOLO-family detector, at several model sizes, in four modes (detect, segment, pose, oriented boxes). Inference runs at native resolution with no forced imgsz. Segmentation is the default because masks give cleaner depth samples than boxes. PyTorch, ONNX, TensorRT and CoreML weights are accepted (CUDA and Apple MPS).
Monocular depth. A metric monocular depth model predicts a depth map, which is resampled to the frame size. Each object’s depth is a robust estimate over its mask pixels, after outlier rejection. A validation step compares predicted depth with reference zones whose distance is known, to detect calibration drift.
Geolocation by ray casting. A ray is cast from the camera centre through the object’s pixel and stopped at the object’s metric depth. The resulting point is converted from the camera frame to a local North-East-Down frame, then to WGS84. Unlike a homography, this does not assume a flat ground. An optional DEM supplies ground elevation. Positions can be overlaid on a georeferenced orthophoto for visual checks.
Tracking. A custom multi-object tracker with a motion model, confidence-aware association and a track-fusion step that links fragments using gap, heading and speed limits. When depth is available, each track stores a 3D trajectory, from which the distance travelled and the speed in m/s are derived.
Inter-object distances. Later consolidated into a reusable vision library. Two segmented objects are back-projected into 3D, and the minimum point-to-point distance is reported. Overlapping masks are treated as contact. A Monte Carlo simulation perturbs depth, focal length and pixel positions to give a standard deviation and a 95 % interval. Zone membership uses the share of the mask inside the zone, with hysteresis, rather than a centre-point test.
Engineering. The configuration supports a two-GPU split (detection on one device, depth estimation on the other), RTSP discovery and recording, and processing of several cameras in parallel.

Results

Measured on a single NVIDIA RTX 5090. The detection benchmark covers detection and tracking only, with depth disabled.

Setting	Video	Throughput	Output
Largest detector, batch 1	3840×2160, 50 fps, 3,301 frames	5.1 fps	320,217 detections, 1,941 tracks
Smallest detector, batch 1	3840×2160, 50 fps	13.0 fps	158,989 detections, 908 tracks
Largest detector, batch 1	2304×1296, 15 fps, 5,000 frames	17.9 fps	73,502 detections, 2,706 tracks
Smallest detector, batch 1	2304×1296, 15 fps	39.0 fps	51,890 detections, 1,499 tracks
Full chain (detection + depth + tracking + storage)	2304×1296, 300 frames	2.02 fps	12.3 detections per frame

In the full-chain run, depth estimation took 117.6 s of the 148.3 s total (about 80 %), and detection took 20.9 s. Depth estimation, not the detector, is the bottleneck. On 4K footage the largest detector runs about ten times slower than real time. That suits the intended use, which is analysis of recorded archives, not live alerting.

Limitations & next steps

No geolocation accuracy has been measured against surveyed ground truth yet. The ray-casting stage was added after the timing runs above, so its cost is not included in them.
The metric scale depends on the depth model and on the camera pose and intrinsics. Errors in tilt or focal length propagate directly into ground positions.
The depth model predicts at its own internal resolution, so fine depth detail is limited even when detection runs in 4K.
Next steps: an accuracy study with GNSS-surveyed targets, profiling the two-GPU split, and measuring how accurate Monte Carlo distance intervals are on objects at known separations.

Stack

Python · PyTorch · OpenCV · NumPy · SciPy · pyproj · rasterio · SQLite · CUDA · Apple MPS