<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Data Synchronisation on Boris Dupoux</title><link>https://borisdupoux.com/tags/data-synchronisation/</link><description>Recent content in Data Synchronisation on Boris Dupoux</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><lastBuildDate>Mon, 01 Jun 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://borisdupoux.com/tags/data-synchronisation/index.xml" rel="self" type="application/rss+xml"/><item><title>Field Identification of Recall-Affected Vehicles</title><link>https://borisdupoux.com/research/fleet-management/</link><pubDate>Mon, 01 Jun 2026 00:00:00 +0000</pubDate><guid>https://borisdupoux.com/research/fleet-management/</guid><description>&lt;h2 id="context"&gt;&lt;a href="#context" class="header-anchor"&gt;&lt;/a&gt;Context
&lt;/h2&gt;&lt;p&gt;Safety recall campaigns depend on reaching every affected vehicle, but
letters and phone calls leave a residual population of owners who never
respond. In island and overseas territories the problem is sharper: network
coverage is uneven, field logistics span several territories, and the
registry of who drives what is incomplete. The practical question becomes a
fleet-management one: &lt;em&gt;which vehicles of the circulating fleet are still
affected, where are they seen, and how far has each one progressed towards
repair?&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="approach"&gt;&lt;a href="#approach" class="header-anchor"&gt;&lt;/a&gt;Approach
&lt;/h2&gt;&lt;pre class="mermaid" style="visibility:hidden"&gt;flowchart TB
 A["Recall listing&lt;br/&gt;(scheduled file pull or manual CSV)"] --&gt; B["Central server&lt;br/&gt;import · versioning · model-name decoding"]
 B --&gt;|"incremental diff"| C["Field units (laptops in agent vehicles)&lt;br/&gt;local recall copy"]
 D["Camera stream"] --&gt; E["Plate detection + OCR&lt;br/&gt;(on-device, offline)"]
 E --&gt; F["Format-aware correction&lt;br/&gt;+ double-read confirmation"]
 F --&gt; G{"Match against&lt;br/&gt;local recall list?"}
 G --&gt;|"no"| H["Discarded — nothing stored"]
 G --&gt;|"yes"| I["Alert to agent&lt;br/&gt;+ local queue"]
 I --&gt;|"sync when network returns"| B
 B --&gt; J["Dashboard&lt;br/&gt;units · tours · detections · incidents"]
 J --&gt; K["Owner contact workflow&lt;br/&gt;identified → contacted → repaired"]&lt;/pre&gt;&lt;h2 id="methods"&gt;&lt;a href="#methods" class="header-anchor"&gt;&lt;/a&gt;Methods
&lt;/h2&gt;&lt;p&gt;&lt;strong&gt;Data ingestion and versioning.&lt;/strong&gt; The recall listing is pulled
automatically on a schedule or imported by hand. Each import is diffed
against the previous one with stable identifiers, versioned by hash, and
only the delta is sent to the field units. Manufacturer type codes are
translated into readable model names on import.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Matching under OCR noise.&lt;/strong&gt; Plate detection and character recognition run
on the field laptop with ONNX models, with no network dependency. Two
safeguards reduce false matches: (i) a &lt;em&gt;positional correction&lt;/em&gt; step uses the
known structure of national plate formats (current and legacy) to resolve
classic OCR confusions (O/0, I/1, S/5, B/8, Z/2, G/6), rejecting any
ambiguous reading; (ii) a &lt;em&gt;double-read confirmation&lt;/em&gt; only counts a plate when
it is read twice within a short time window. A smartphone web page offers a
manual fallback (photo or typed plate, plus VIN check-digit validation).&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Offline-first synchronisation.&lt;/strong&gt; Matches are queued locally and pushed
to the central server on the next connection, so detections survive
multi-day outages. Units report a heartbeat (software version, recall-list
version, reading counters).&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Field operations.&lt;/strong&gt; Supervisors plan tours as ordered, geolocated
checkpoints on a map, assign them to teams and units, reuse templates, and
follow progress. Agents report incidents with a severity level. A contact
workflow records the outcome of each owner approach, and the dashboard
derives a conversion funnel (identified → approached → engaged → repaired)
with weekly trends, the median delay to repair and a breakdown by territory.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Privacy by design.&lt;/strong&gt; Plates that do not match the recall list are never
stored or transmitted; retained records have fixed retention periods with
automatic purge; actions are written to an audit log; dashboard access is
role-based, including read-only accounts; data at rest is encrypted. A
processing register was written alongside the code.&lt;/p&gt;
&lt;h2 id="deployment--operations-devops"&gt;&lt;a href="#deployment--operations-devops" class="header-anchor"&gt;&lt;/a&gt;Deployment &amp;amp; operations (DevOps)
&lt;/h2&gt;&lt;p&gt;The system runs on two very different kinds of machine: Windows laptops in
agents&amp;rsquo; vehicles, which may stay offline for days, and one server per
instance. After the first weeks in the field, most of the engineering work
went into making both sides easy to deploy, observe and recover, by a single
person, without breaking field work.&lt;/p&gt;
&lt;pre class="mermaid" style="visibility:hidden"&gt;flowchart LR
 subgraph ADM["Admin workstation"]
 GIT["Git repository&lt;br/&gt;committed tree only"]
 PROV["Provisioning script&lt;br/&gt;one config file per unit"]
 BLD["Field build&lt;br/&gt;PyInstaller bundle + ONNX models"]
 end
 subgraph SRV["Server (Ubuntu LTS VPS)"]
 CAD["Caddy&lt;br/&gt;TLS · HSTS · no access log"]
 APP["Docker container&lt;br/&gt;FastAPI API + dashboard&lt;br/&gt;loopback port only"]
 VOL[("LUKS2 volume&lt;br/&gt;SQLite · photos · archives")]
 TIM["systemd timers&lt;br/&gt;monitoring · backup · cleanup"]
 end
 FLD["Field laptops&lt;br/&gt;local SQLite · sync loop"]
 OBJ[("Off-site object storage&lt;br/&gt;age-encrypted backups")]
 CHAT["Chat alerts&lt;br/&gt;state changes + daily digests"]
 GIT --&gt;|"git archive over SSH → deploy script"| APP
 PROV --&gt;|"admin API"| APP
 PROV --&gt;|"config file + README"| FLD
 BLD --&gt;|"archive + one-click installer"| FLD
 FLD &lt;--&gt;|"HTTPS: heartbeat · pushes · recall diff"| CAD
 CAD --&gt; APP
 APP --- VOL
 TIM --&gt; VOL
 TIM --&gt;|"encrypted copy"| OBJ
 TIM --&gt; CHAT&lt;/pre&gt;&lt;h3 id="field-laptops-packaging-installation-and-updates"&gt;&lt;a href="#field-laptops-packaging-installation-and-updates" class="header-anchor"&gt;&lt;/a&gt;Field laptops: packaging, installation and updates
&lt;/h3&gt;&lt;p&gt;&lt;strong&gt;Packaging.&lt;/strong&gt; The field application (Python backend, React interface) is
frozen with PyInstaller into a one-folder Windows executable that embeds the
ONNX models and the OCR configuration, so a unit needs no Python install and
no network to read plates. Built-in &lt;code&gt;--diag&lt;/code&gt; and &lt;code&gt;--selftest&lt;/code&gt; modes check the
bundled runtime and read a test image without starting the interface, which
made start-up failures diagnosable on the field machine itself.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;One-click installer.&lt;/strong&gt; The build is delivered as an archive with a
double-click &lt;code&gt;INSTALLER.bat&lt;/code&gt;. It elevates itself and runs a PowerShell script
that removes the &amp;ldquo;downloaded from the Internet&amp;rdquo; mark from the files (otherwise
the bundled runtime fails to load), disables USB power saving so the GPS
dongle is not suspended, and installs the unit&amp;rsquo;s configuration file where the
executable reads it. The script refuses to guess: if several &lt;em&gt;different&lt;/em&gt;
configuration files are present, it lists them and asks which one is the new
one. An optional reset clears the local database when a unit moves to another
server; it shows the target server before asking for confirmation and warns
that unsynchronised detections would be lost. The unit&amp;rsquo;s access token is
never displayed. An update follows the same path: new build, re-run the
installer.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Provisioning.&lt;/strong&gt; A Python script that uses only the standard library, so it
runs on a bare Windows machine, creates the units from a CSV file through the
server&amp;rsquo;s admin API and generates, for each unit, its configuration file
(server address, unit code, display name, token) and a README to hand over
with the laptop. The admin key is read from an environment variable or a
masked prompt, never from a command-line argument.&lt;/p&gt;
&lt;h3 id="versioning-software-and-recall-list"&gt;&lt;a href="#versioning-software-and-recall-list" class="header-anchor"&gt;&lt;/a&gt;Versioning: software and recall list
&lt;/h3&gt;&lt;p&gt;&lt;strong&gt;Software version.&lt;/strong&gt; Each unit reports its application version in its
heartbeat. The dashboard compares it with the expected release and flags the
units that still need updating.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Recall-list version.&lt;/strong&gt; The recall list is versioned by hash. On each cycle,
a unit compares its local hash with the server&amp;rsquo;s: identical, nothing to do; no
local copy, full download; otherwise, the server returns only the rows added
and removed since that hash (and the unit falls back to a full download if the
server no longer knows its version). The change is applied in a single
transaction, and the local hash only moves forward if everything was applied.&lt;/p&gt;
&lt;p&gt;That last rule came from a field incident. A database constraint made the
diff fail on one unit, the error was swallowed by the sync loop, and the unit
kept sending heartbeats while its recall list stayed frozen for more than two
weeks, so it alerted on vehicles that had already been repaired. Release 1.4.1
fixed the cause, added an explicit rollback, and shipped a schema migration
that keeps existing data. It was tested with a 62-test run that included
reproducing the bug and migrating a database with the old schema. Monitoring
now raises a &amp;ldquo;recall sync stuck&amp;rdquo; alert when a unit has not received a
published list after a grace period.&lt;/p&gt;
&lt;h3 id="incremental-resilient-synchronisation"&gt;&lt;a href="#incremental-resilient-synchronisation" class="header-anchor"&gt;&lt;/a&gt;Incremental, resilient synchronisation
&lt;/h3&gt;&lt;p&gt;Every five minutes, or on demand, a unit runs a sync cycle made of
independent steps: heartbeat, push of detections (in batches), GPS track,
incidents, checkpoint updates, then pull of the recall list and the tours.
Pushes come before pulls, so field data leaves first. A failing step no
longer blocks the others: in an earlier version the cycle was monolithic,
and a failed heartbeat once delayed detections by 22 minutes. Requests use
short timeouts, downloads longer ones. The heartbeat carries the software
version, the recall-list version, the number of pending detections and the
reading counters, which is what the dashboard and the monitoring rely on.&lt;/p&gt;
&lt;h3 id="server-deployment"&gt;&lt;a href="#server-deployment" class="header-anchor"&gt;&lt;/a&gt;Server deployment
&lt;/h3&gt;&lt;p&gt;&lt;strong&gt;Instance kit.&lt;/strong&gt; Each deployment is an independent instance described by a
short identifier, on an Ubuntu LTS virtual server. A bootstrap script, which is
interactive and can be safely re-run, installs the system packages, Docker
and Caddy from their official repositories. It enables unattended security
updates without automatic reboot, restricts SSH to key authentication,
enables a default-deny firewall and brute-force protection, and checks that
the DNS record points to the server before applying the TLS configuration.
Secrets live in root-only configuration files on the server, never in the
repository.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Container and reverse proxy.&lt;/strong&gt; A multi-stage Dockerfile builds the React
dashboard with Node, then runs the FastAPI server on a slim Python image as an
unprivileged user. Its health check allows a three-minute start period
because the plate and OCR models are loaded before the server listens. The
container is published on the loopback interface only. Caddy is the only
public entry point: automatic Let&amp;rsquo;s Encrypt certificate, explicit HSTS, and
no access log, because scan URLs contain plates. The application&amp;rsquo;s own
access log is disabled for the same reason.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Deploy and rollback.&lt;/strong&gt; Code leaves the admin workstation only as a
&lt;code&gt;git archive&lt;/code&gt; of a committed tree: uncommitted changes are listed and not
sent, and line endings are forced to LF. On the server, the new tree is
unpacked next to the current one and swapped in, the previous one is kept,
and the deployed commit is recorded. The deploy script then detaches from the
SSH session, so a dropped connection interrupts neither the deployment nor a
rollback. A lock prevents concurrent deployments. The script builds the image
(labelled with its commit), keeps the previous image as &lt;code&gt;rollback&lt;/code&gt;, recreates
the container and waits until it reports healthy. It gives up after five
minutes, or as soon as the container has restarted twice. On failure it rolls
back automatically to the previous image and source tree, sets the failed
code aside and sends an alert. Going back to any earlier version means
re-deploying that commit. An image rollback does not undo a schema
migration, so the documented procedure is to take a backup before any
deployment that changes the schema.&lt;/p&gt;
&lt;h3 id="encrypted-volume-and-safe-restarts"&gt;&lt;a href="#encrypted-volume-and-safe-restarts" class="header-anchor"&gt;&lt;/a&gt;Encrypted volume and safe restarts
&lt;/h3&gt;&lt;p&gt;Server data (database, contact photos, recall archives) lives on a LUKS2
file container mounted with restrictive options. Its passphrase is never
stored on the machine, so after a reboot a person unlocks the volume. The
container restarts on its own, but the server refuses to start unless a
sentinel file &lt;em&gt;on the encrypted volume&lt;/em&gt; is present. Without it, a reboot
would have left an empty directory in place of the volume, and the
application would have created a fresh, unencrypted database there and
accepted field data into it. The guard was checked both ways on a throwaway
container. A separate script grows the volume while it is mounted.&lt;/p&gt;
&lt;h3 id="backups-and-recovery"&gt;&lt;a href="#backups-and-recovery" class="header-anchor"&gt;&lt;/a&gt;Backups and recovery
&lt;/h3&gt;&lt;p&gt;A daily systemd timer takes a consistent SQLite snapshot through the
database&amp;rsquo;s backup API, archives it with the photos and recall archives, and
encrypts the result with &lt;code&gt;age&lt;/code&gt; to a public key. The private key is shown
once, at creation, and kept off the server. Backups are kept locally and,
optionally, copied to S3-compatible object storage with &lt;code&gt;rclone&lt;/code&gt;, configured
through environment variables only. A restore &lt;em&gt;test&lt;/em&gt; mode decrypts a backup
into a temporary directory on the encrypted volume, runs an integrity check,
counts rows per table, then deletes everything. The &lt;em&gt;apply&lt;/em&gt; mode moves the
current state aside instead of deleting it. The documentation includes a
step-by-step rebuild after total loss of the server. The continuity plan
targets a recovery time of 4 hours and a maximum data loss of 24 hours.&lt;/p&gt;
&lt;h3 id="monitoring"&gt;&lt;a href="#monitoring" class="header-anchor"&gt;&lt;/a&gt;Monitoring
&lt;/h3&gt;&lt;p&gt;A watchdog runs every 15 minutes and only posts to a chat webhook when a
state changes (problem, then &amp;ldquo;resolved&amp;rdquo;). A fuller digest is sent twice a day,
timed to the field shifts. It checks that the encrypted volume is mounted, the
API health endpoint (with retries, because the daily import can block it for
about a minute), the container health status, that the database is readable,
disk usage on both the system disk and the encrypted volume, the size of the
SQLite write-ahead log, TLS certificate expiry, backup age and size, units
silent for more than 48 hours (not 24, since units are switched off at
weekends) and units whose recall list is stuck. A separate watcher alerts
when no recall import has happened for more than 30 hours. Several of these
checks were added after real incidents: the encrypted volume once reached
90 % without any alert because only the system disk was watched. A weekly
timer cleans up Docker build caches and keeps only the current and rollback
images. A dedicated script truncates the SQLite log with a short lock wait,
so field writes are never blocked for long.&lt;/p&gt;
&lt;h3 id="offline-resilience"&gt;&lt;a href="#offline-resilience" class="header-anchor"&gt;&lt;/a&gt;Offline resilience
&lt;/h3&gt;&lt;p&gt;The field side is designed to keep working through a server outage: plate
reading, matching against the local recall copy, alerts, tours already
downloaded and the local statistics tab all work without a network, and the
queue empties at the next connection. Only the dashboard (and the
server-hosted smartphone page) is unavailable while the server is down.&lt;/p&gt;
&lt;h3 id="testing"&gt;&lt;a href="#testing" class="header-anchor"&gt;&lt;/a&gt;Testing
&lt;/h3&gt;&lt;p&gt;There is no CI pipeline in the repository. Releases are tested before they
ship, and the commits record those runs: 12/12 and 6/6 for the plate
correction and confirmation stages, 62 tests for release 1.4.1. Behaviours
that cannot be unit-tested, such as the volume guard or a live deployment,
were checked on throwaway containers and against the production health
endpoint.&lt;/p&gt;
&lt;h2 id="results"&gt;&lt;a href="#results" class="header-anchor"&gt;&lt;/a&gt;Results
&lt;/h2&gt;&lt;ul&gt;
&lt;li&gt;In early field operation, agents flagged &lt;strong&gt;32 %&lt;/strong&gt; of detections as
misreadings over roughly one week. This measurement motivated the
positional-correction and double-read confirmation stages, and the
dashboard statistics now exclude detections marked as false positives.&lt;/li&gt;
&lt;li&gt;The new stages shipped with unit tests: &lt;strong&gt;12/12&lt;/strong&gt; for plate correction
and &lt;strong&gt;6/6&lt;/strong&gt; for confirmation, settings and reading log.&lt;/li&gt;
&lt;li&gt;After the model-name mapping was extended, only &lt;strong&gt;one&lt;/strong&gt; row of the
production recall listing remained without a decoded model name.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;No end-to-end recognition rate has been measured yet.&lt;/p&gt;
&lt;h2 id="limitations--next-steps"&gt;&lt;a href="#limitations--next-steps" class="header-anchor"&gt;&lt;/a&gt;Limitations &amp;amp; next steps
&lt;/h2&gt;&lt;ul&gt;
&lt;li&gt;Recognition accuracy has not been measured against a ground-truth
benchmark. The specification targets (read rate, latency) are design goals,
not results.&lt;/li&gt;
&lt;li&gt;No legacy-format plate reading was observed at first. A dedicated counter
now tracks these readings end to end, but the fix still needs a field check.&lt;/li&gt;
&lt;li&gt;A false match against a large listing still costs agent time. Calibrating
the confidence threshold and confirmation window on labelled field video
is the next step.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="stack"&gt;&lt;a href="#stack" class="header-anchor"&gt;&lt;/a&gt;Stack
&lt;/h2&gt;&lt;p&gt;&lt;code&gt;Python&lt;/code&gt; · &lt;code&gt;FastAPI&lt;/code&gt; · &lt;code&gt;SQLite&lt;/code&gt; · &lt;code&gt;ONNX Runtime&lt;/code&gt; · &lt;code&gt;fast-alpr&lt;/code&gt; (YOLOv9 plate detector + OCR) · &lt;code&gt;OpenCV&lt;/code&gt; · &lt;code&gt;PaddleOCR&lt;/code&gt; · &lt;code&gt;React&lt;/code&gt; · &lt;code&gt;TypeScript&lt;/code&gt; · &lt;code&gt;Vite&lt;/code&gt; · &lt;code&gt;Leaflet&lt;/code&gt; · &lt;code&gt;Docker&lt;/code&gt; · &lt;code&gt;Caddy&lt;/code&gt; · &lt;code&gt;LUKS&lt;/code&gt; · &lt;code&gt;systemd&lt;/code&gt; · &lt;code&gt;age&lt;/code&gt; · &lt;code&gt;rclone&lt;/code&gt; · &lt;code&gt;PyInstaller&lt;/code&gt; · &lt;code&gt;PowerShell&lt;/code&gt;&lt;/p&gt;</description></item></channel></rss>