Handheld Video to Photogrammetry and Gaussian Splatting

Thu, 01 Oct 2026 00:00:00 +0000

Context

Indoor sites are often documented with a handheld gimbal camera rather than a survey-grade scanner. The goal was to turn such video into usable 3D deliverables (dense cloud, textured mesh, plan and elevation orthoviews, a Gaussian Splatting scene and lightweight web exports) with no manual steps between capture and output. The pipeline runs natively on Windows on an RTX 5090 (Blackwell, sm_120) that is shared with other long-running GPU workloads, so it must never pre-empt them.

Approach

flowchart TB
 A["inbox/project/videos + GO file"] --> B["Keyframe extraction
optical-flow overlap + sharpness"]
 B --> C["COLMAP 4.2.1 SfM
SIFT, global mapper"]
 C --> D{"QC: registration, reprojection,
trajectory continuity"}
 D -- fail --> C2["Fallback: incremental mapper"] --> D
 D -- pass --> E["OpenMVS 2.4 (sm_120 build)
densify, mesh, refine, texture"]
 E --> F["Ortho alignment
gravity, floor RANSAC, Manhattan yaw"]
 F --> G["Blender renders
orthos, axonometries, orbit"]
 E --> H["gsplat training
MCMC or default strategy"]
 G --> I["Web exports
Potree, GLB, SOG"]
 H --> I
 I --> J["outbox: deliverables, report.json, STATUS"]

Methods

Overlap-driven keyframe selection. KLT tracking with a RANSAC similarity transform estimates overlap with the last keyframe. A frame is emitted when overlap drops below 0.80, tracking is lost or a time limit passes. The sharpest candidate (variance of the Laplacian) is the one kept.
SfM with explicit QC. COLMAP’s global mapper is fast but non-deterministic. In testing it produced reconstructions that met registration and reprojection criteria but were geometrically incoherent. A trajectory-continuity check (jumps above 25× the median camera step) triggers an automatic fallback to the incremental mapper. Below 20 points per image the step fails.
MVS hardening. The official OpenMVS Windows binary has only sm_89 kernels, so it was rebuilt for sm_120. The object ROI is off for rooms because it cropped the cloud. Refinement is dropped when it diverges, and seam leveling (which blackened faces) is off.
Metric-ready orthoviews. The up axis comes from camera orientations. A RANSAC floor plane sets z = 0 and a histogram of wall-normal azimuths (mod 90°) sets Manhattan yaw. An optional two-point reference fixes scale. The 4×4 transform is stored so it can be applied again to the mesh and the splat.
Gaussian Splatting. gsplat 1.6.0 was compiled for sm_120 with CUDA 13 under MSVC, which required local patches. Training uses the MCMC strategy with a Gaussian budget and falls back to the default 3DGS strategy when MCMC diverges.
Operations. Projects start in watch-folder mode when a GO file appears. Stages can be resumed or redone. A GPU guard waits until enough VRAM has been free for a stable period, serialises GPU stages through a lock file and records peak VRAM per stage in report.json. Degraded QC produces a DONE_WITH_ALERTS status instead of a silent success.

Results

First real deployment: five locations inside a commercial showroom, one 4K handheld video per location, processed unattended in about 30 minutes per location.

Location	Images	SfM registered / points / reproj.	Dense cloud	Splat PSNR / SSIM
1	116	116/116 · 21,458 · 1.49 px	1.40 M	23.2 / 0.86
2	103	103/103 · 18,972 · 1.75 px	1.23 M	20.6 / 0.81
3	47	47/47 · 13,167 · 1.17 px	1.26 M	23.1 / 0.86
4	121	121/121 · 11,970 · 1.34 px	1.99 M	21.9 / 0.85 (default strategy)
5	64	64/64 · 12,106 · 1.48 px	0.91 M	21.8 / 0.83

All frames were registered in every location. Reprojection error was 1.17–1.75 px.
Measured VRAM peaks: SIFT +1.3 GB, matching +1.6 GB, texturing +0.5 GB, Blender +5.4 GB, and splatting with 3 M Gaussians at 1920 px +25.2 GB.
The sm_120 OpenMVS build densified the synthetic smoke test in 17 s. The CPU fallback took about 40 min.
Metadata analysis showed shutter speeds of 1/50–1/100 s (about 14 px of motion blur at 4K) and automatic ISO and white balance. Exposure compensation in gsplat (ppisp) gave no gain (PSNR 20.5 vs 20.6). The quality limit is in the capture, not the solver.

Limitations & next steps

PSNR in the low 20s reflects motion blur and auto-exposure. The capture protocol now requires a 1/250–1/500 s shutter, locked exposure and white balance, and a dedicated floor pass.
Merging videos from different locations caused false matches and an incoherent model, so each location is processed separately.
Dark glossy tiled floors reconstruct poorly. Manhattan alignment is not yet reliable with furniture and glazing.
CPU stages are the bottleneck on a 4-vCPU virtual machine.
Planned: comparison with a commercial photogrammetry suite on the same frames, and retesting mesh refinement with the sm_120 build.

Stack

Python 3.11 · OpenCV · Open3D · NumPy · COLMAP 4.2.1 · OpenMVS 2.4 · gsplat 1.6.0 · PyTorch 2.14 (CUDA 13) · Blender 5.2 · FFmpeg · PotreeConverter · glTF-Transform · splat-transform