Handheld Video to Photogrammetry and Gaussian Splatting

Thu, 01 Oct 2026 00:00:00 +0000

What’s original

Keyframes chosen by overlap, not by clock. A new frame is kept when the view has moved enough, and the sharpest candidate wins, so the reconstruction gets fewer, better images.
Quality checks that catch “valid but wrong” models. A fast reconstruction can pass the usual registration and reprojection criteria while being geometrically incoherent; a trajectory-continuity check detects this and switches to a slower, safer method.
Built to share a GPU. GPU stages wait for free memory and run one at a time, so long jobs already on the card are never pre-empted; degraded quality is flagged rather than reported as a silent success.

How it works

flowchart LR
 A["Handheld 4K video"] --> B["Keyframe selection"]
 B --> C["Structure from motion"]
 C --> D{"Quality checks"}
 D -- fail --> C
 D -- pass --> E["Dense cloud + textured mesh"]
 E --> F["Gravity + Manhattan alignment"]
 F --> G["Orthoviews + renders"]
 E --> H["Gaussian Splatting"]
 G --> I["Web exports + report"]
 H --> I

COLMAP recovers camera poses, OpenMVS builds the dense cloud and mesh, and gsplat trains the splat. The scene is then aligned to gravity, the floor and the dominant wall directions, so plan and elevation views come out square; the same transform is applied to the mesh and the splat. Every stage can be resumed or redone.

Results

At five locations of an indoor commercial site, one 4K video each, every selected keyframe was registered in every location.
The geometry check worked as intended: it raised an alert on every run rather than reporting a silent success.
Capture metadata showed slow shutter speeds with automatic ISO and white balance; exposure compensation during splat training gave no gain on one location (single run).
Merging videos from different locations produced false matches, so each location is processed on its own.

Evidence & limits

Reprojection error is above the pipeline’s own 1 px target, and splat PSNR sits in the low 20s on held-out views.
One run per location, with a small held-out set; the figures come from the project README, as run logs were not archived.
That quality is limited by capture (motion blur, auto-exposure) is a hypothesis, not yet tested by recapturing with locked settings.
Dark glossy floors reconstruct poorly, and Manhattan alignment is not yet reliable with furniture and glazing. A comparison with a commercial photogrammetry suite is planned.

Technical details

Five locations at one indoor commercial site, one handheld 4K video per location, each processed once, unattended, on a single high-end GPU. PSNR/SSIM on held-out keyframes.

Location	Keyframes	SfM registered · points · mean reproj.	Dense cloud	Splat PSNR / SSIM
1	116	116/116 · 21,458 · 1.49 px	1.40 M	23.2 / 0.86
2	103	103/103 · 18,972 · 1.75 px	1.23 M	20.6 / 0.81
3	47	47/47 · 13,167 · 1.17 px	1.26 M	23.1 / 0.86
4	121	121/121 · 11,970 · 1.34 px	1.99 M	21.9 / 0.85
5	64	64/64 · 12,106 · 1.48 px	0.91 M	21.8 / 0.83

Stack