Context
Safety recall campaigns depend on reaching every affected vehicle, but letters and phone calls leave a residual population of owners who never respond. In island and overseas territories the problem is sharper: network coverage is uneven, field logistics span several territories, and the registry of who drives what is incomplete. The practical question becomes a fleet-management one: which vehicles of the circulating fleet are still affected, where are they seen, and how far has each one progressed towards repair?
Approach
Methods
Data ingestion and versioning. The recall listing is pulled automatically on a schedule or imported by hand. Each import is diffed against the previous one with stable identifiers, versioned by hash, and only the delta is sent to the field units. Manufacturer type codes are translated into readable model names on import.
Matching under OCR noise. Plate detection and character recognition run on the field laptop with ONNX models, with no network dependency. Two safeguards reduce false matches: (i) a positional correction step uses the known structure of national plate formats (current and legacy) to resolve classic OCR confusions (O/0, I/1, S/5, B/8, Z/2, G/6), rejecting any ambiguous reading; (ii) a double-read confirmation only counts a plate when it is read twice within a short time window. A smartphone web page offers a manual fallback (photo or typed plate, plus VIN check-digit validation).
Offline-first synchronisation. Matches are queued locally and pushed to the central server on the next connection, so detections survive multi-day outages. Units report a heartbeat (software version, recall-list version, reading counters).
Field operations. Supervisors plan tours as ordered, geolocated checkpoints on a map, assign them to teams and units, reuse templates, and follow progress. Agents report incidents with a severity level. A contact workflow records the outcome of each owner approach, and the dashboard derives a conversion funnel (identified → approached → engaged → repaired) with weekly trends, the median delay to repair and a breakdown by territory.
Privacy by design. Plates that do not match the recall list are never stored or transmitted; retained records have fixed retention periods with automatic purge; actions are written to an audit log; dashboard access is role-based, including read-only accounts; data at rest is encrypted. A processing register was written alongside the code.
Deployment & operations (DevOps)
The system runs on two very different kinds of machine: Windows laptops in agents’ vehicles, which may stay offline for days, and one server per instance. After the first weeks in the field, most of the engineering work went into making both sides easy to deploy, observe and recover, by a single person, without breaking field work.
Field laptops: packaging, installation and updates
Packaging. The field application (Python backend, React interface) is
frozen with PyInstaller into a one-folder Windows executable that embeds the
ONNX models and the OCR configuration, so a unit needs no Python install and
no network to read plates. Built-in --diag and --selftest modes check the
bundled runtime and read a test image without starting the interface, which
made start-up failures diagnosable on the field machine itself.
One-click installer. The build is delivered as an archive with a
double-click INSTALLER.bat. It elevates itself and runs a PowerShell script
that removes the “downloaded from the Internet” mark from the files (otherwise
the bundled runtime fails to load), disables USB power saving so the GPS
dongle is not suspended, and installs the unit’s configuration file where the
executable reads it. The script refuses to guess: if several different
configuration files are present, it lists them and asks which one is the new
one. An optional reset clears the local database when a unit moves to another
server; it shows the target server before asking for confirmation and warns
that unsynchronised detections would be lost. The unit’s access token is
never displayed. An update follows the same path: new build, re-run the
installer.
Provisioning. A Python script that uses only the standard library, so it runs on a bare Windows machine, creates the units from a CSV file through the server’s admin API and generates, for each unit, its configuration file (server address, unit code, display name, token) and a README to hand over with the laptop. The admin key is read from an environment variable or a masked prompt, never from a command-line argument.
Versioning: software and recall list
Software version. Each unit reports its application version in its heartbeat. The dashboard compares it with the expected release and flags the units that still need updating.
Recall-list version. The recall list is versioned by hash. On each cycle, a unit compares its local hash with the server’s: identical, nothing to do; no local copy, full download; otherwise, the server returns only the rows added and removed since that hash (and the unit falls back to a full download if the server no longer knows its version). The change is applied in a single transaction, and the local hash only moves forward if everything was applied.
That last rule came from a field incident. A database constraint made the diff fail on one unit, the error was swallowed by the sync loop, and the unit kept sending heartbeats while its recall list stayed frozen for more than two weeks, so it alerted on vehicles that had already been repaired. Release 1.4.1 fixed the cause, added an explicit rollback, and shipped a schema migration that keeps existing data. It was tested with a 62-test run that included reproducing the bug and migrating a database with the old schema. Monitoring now raises a “recall sync stuck” alert when a unit has not received a published list after a grace period.
Incremental, resilient synchronisation
Every five minutes, or on demand, a unit runs a sync cycle made of independent steps: heartbeat, push of detections (in batches), GPS track, incidents, checkpoint updates, then pull of the recall list and the tours. Pushes come before pulls, so field data leaves first. A failing step no longer blocks the others: in an earlier version the cycle was monolithic, and a failed heartbeat once delayed detections by 22 minutes. Requests use short timeouts, downloads longer ones. The heartbeat carries the software version, the recall-list version, the number of pending detections and the reading counters, which is what the dashboard and the monitoring rely on.
Server deployment
Instance kit. Each deployment is an independent instance described by a short identifier, on an Ubuntu LTS virtual server. A bootstrap script, which is interactive and can be safely re-run, installs the system packages, Docker and Caddy from their official repositories. It enables unattended security updates without automatic reboot, restricts SSH to key authentication, enables a default-deny firewall and brute-force protection, and checks that the DNS record points to the server before applying the TLS configuration. Secrets live in root-only configuration files on the server, never in the repository.
Container and reverse proxy. A multi-stage Dockerfile builds the React dashboard with Node, then runs the FastAPI server on a slim Python image as an unprivileged user. Its health check allows a three-minute start period because the plate and OCR models are loaded before the server listens. The container is published on the loopback interface only. Caddy is the only public entry point: automatic Let’s Encrypt certificate, explicit HSTS, and no access log, because scan URLs contain plates. The application’s own access log is disabled for the same reason.
Deploy and rollback. Code leaves the admin workstation only as a
git archive of a committed tree: uncommitted changes are listed and not
sent, and line endings are forced to LF. On the server, the new tree is
unpacked next to the current one and swapped in, the previous one is kept,
and the deployed commit is recorded. The deploy script then detaches from the
SSH session, so a dropped connection interrupts neither the deployment nor a
rollback. A lock prevents concurrent deployments. The script builds the image
(labelled with its commit), keeps the previous image as rollback, recreates
the container and waits until it reports healthy. It gives up after five
minutes, or as soon as the container has restarted twice. On failure it rolls
back automatically to the previous image and source tree, sets the failed
code aside and sends an alert. Going back to any earlier version means
re-deploying that commit. An image rollback does not undo a schema
migration, so the documented procedure is to take a backup before any
deployment that changes the schema.
Encrypted volume and safe restarts
Server data (database, contact photos, recall archives) lives on a LUKS2 file container mounted with restrictive options. Its passphrase is never stored on the machine, so after a reboot a person unlocks the volume. The container restarts on its own, but the server refuses to start unless a sentinel file on the encrypted volume is present. Without it, a reboot would have left an empty directory in place of the volume, and the application would have created a fresh, unencrypted database there and accepted field data into it. The guard was checked both ways on a throwaway container. A separate script grows the volume while it is mounted.
Backups and recovery
A daily systemd timer takes a consistent SQLite snapshot through the
database’s backup API, archives it with the photos and recall archives, and
encrypts the result with age to a public key. The private key is shown
once, at creation, and kept off the server. Backups are kept locally and,
optionally, copied to S3-compatible object storage with rclone, configured
through environment variables only. A restore test mode decrypts a backup
into a temporary directory on the encrypted volume, runs an integrity check,
counts rows per table, then deletes everything. The apply mode moves the
current state aside instead of deleting it. The documentation includes a
step-by-step rebuild after total loss of the server. The continuity plan
targets a recovery time of 4 hours and a maximum data loss of 24 hours.
Monitoring
A watchdog runs every 15 minutes and only posts to a chat webhook when a state changes (problem, then “resolved”). A fuller digest is sent twice a day, timed to the field shifts. It checks that the encrypted volume is mounted, the API health endpoint (with retries, because the daily import can block it for about a minute), the container health status, that the database is readable, disk usage on both the system disk and the encrypted volume, the size of the SQLite write-ahead log, TLS certificate expiry, backup age and size, units silent for more than 48 hours (not 24, since units are switched off at weekends) and units whose recall list is stuck. A separate watcher alerts when no recall import has happened for more than 30 hours. Several of these checks were added after real incidents: the encrypted volume once reached 90 % without any alert because only the system disk was watched. A weekly timer cleans up Docker build caches and keeps only the current and rollback images. A dedicated script truncates the SQLite log with a short lock wait, so field writes are never blocked for long.
Offline resilience
The field side is designed to keep working through a server outage: plate reading, matching against the local recall copy, alerts, tours already downloaded and the local statistics tab all work without a network, and the queue empties at the next connection. Only the dashboard (and the server-hosted smartphone page) is unavailable while the server is down.
Testing
There is no CI pipeline in the repository. Releases are tested before they ship, and the commits record those runs: 12/12 and 6/6 for the plate correction and confirmation stages, 62 tests for release 1.4.1. Behaviours that cannot be unit-tested, such as the volume guard or a live deployment, were checked on throwaway containers and against the production health endpoint.
Results
- In early field operation, agents flagged 32 % of detections as misreadings over roughly one week. This measurement motivated the positional-correction and double-read confirmation stages, and the dashboard statistics now exclude detections marked as false positives.
- The new stages shipped with unit tests: 12/12 for plate correction and 6/6 for confirmation, settings and reading log.
- After the model-name mapping was extended, only one row of the production recall listing remained without a decoded model name.
No end-to-end recognition rate has been measured yet.
Limitations & next steps
- Recognition accuracy has not been measured against a ground-truth benchmark. The specification targets (read rate, latency) are design goals, not results.
- No legacy-format plate reading was observed at first. A dedicated counter now tracks these readings end to end, but the fix still needs a field check.
- A false match against a large listing still costs agent time. Calibrating the confidence threshold and confirmation window on labelled field video is the next step.
Stack
Python · FastAPI · SQLite · ONNX Runtime · fast-alpr (YOLOv9 plate detector + OCR) · OpenCV · PaddleOCR · React · TypeScript · Vite · Leaflet · Docker · Caddy · LUKS · systemd · age · rclone · PyInstaller · PowerShell