Data Engineering on Boris Dupoux

Points of Interest and Urban Flows in Fort-de-France City Centre from Open Data

Thu, 01 Oct 2026 00:00:00 +0000

Context

Fort-de-France (Martinique) has almost no local open data. The city’s own datasets were last updated in 2019, and there are no public pedestrian counters, open transit ridership figures or open road counts at city-centre scale. The project asks how far national open data and continuously archived feeds can go in describing the city centre as a system of points of interest (shops, administrations, schools, health, culture, entry gates) and flows (residents, commuters, pupils, visitors, cruise passengers, vehicles, buses) over a ten-year horizon (2016–2026).

No single source answers that question. The business register knows where establishments are but not who visits them. The census knows where people sleep but not where they spend the day. Transit feeds show the buses but not their passengers, and cruise schedules show the ships but not the people ashore. The approach is therefore to cross many heterogeneous sources on a common spatial and temporal frame, so that each one fills a blind spot of the others, and to check the result against the few measured signals that do exist.

Approach

The method comes from an earlier knowledge-base project of mine. Raw data is immutable and addressed by content. Every database is disposable and rebuilt from it. Every value keeps its provenance, and a set of invariants is checked on each rebuild.

flowchart TB
 A[Source catalogue
sources.yaml] --> B[Immutable raw store
sha256 + JSONL manifest]
 C[Scheduled collectors
Meta every 14 days · GTFS-RT 60 s] --> B
 B --> D[Versioned extractors
normalised Parquet]
 D --> E[DuckDB spatial warehouse
rebuilt, never edited]
 E --> F[QA invariants I1–I6]
 E --> G[Statistics
requetes/*.sql]
 E --> H[Desktop application
map + time selector]
 G --> I[Calibration against
measured signals]

Methods

Acquisition. Each download is hashed (SHA-256) and logged in a JSONL manifest, and acquisition is idempotent. National files larger than 1 GB (Sirene business register) are read as remote HTTP-range extracts filtered to the municipality. The filtering query is recorded as part of the provenance.
Ephemeral sources. Meta Movement Distribution keeps only 90 days online, so a scheduled task archives it every 14 days. A bus collector polls GTFS-RT vehicle positions every 60 s and predictions every 10 min, and zips each day with a SHA-256 checksum.
Spatial frame. Data is stored in WGS84 and computed in EPSG:5490 on the INSEE 200 m grid. Points of interest are classified by one typology (NAF codes, OSM tags, stops), with family, weight and an hourly rhythm over seven time slots.
Models. Hourly presence per grid cell combines residents (Filosofi 2021, distributed dasymetrically to buildings), employees, pupils and tourists. A Huff model estimates catchment. Cruise passengers onshore are modelled from ship schedules and capacities. An island-wide gravity model with BPR assignment covers visitors and through-traffic.
Quality. Six invariants are checked, from provenance to the absence of personal data. Unconfirmed parameters go in a decision log.

Data sources

Crossing the sources is the core of the project. Business registers (Sirene, BODACC, URSSAF) give the stock and turnover of activities, but say nothing about the people around them. The census and tax data (RP, Filosofi) place residents, their income and housing vacancy, but only at night. Commuting files and the national travel survey (MOBPRO, MOBSCO, EMP) say who travels, why and when. Transit, accident and air-quality feeds (GTFS-RT, BAAC, Madininair) are the only measured traces of movement. Cruise schedules and ship capacities bring a population that no census counts. The geographic frames (IGN, OpenStreetMap, the INSEE 200 m grid) are what make all of these comparable cell by cell.

The catalogue lists 120 sources: 51 integrated, 32 to integrate, 7 to verify, 13 available on request only, 5 paid, 1 by partnership and 11 not retained. The tables below list every integrated source and summarise the others by theme.

Businesses, jobs & amenities

Source	What it contributes	Period / frequency	Status
Sirene business register — INSEE	Establishments, activity code (NAF), creation and closure dates, headcount bands	Full history; stock rebuilt at any date 2016–2026	Integrated
BODACC — DILA	Business turnover and failures (sales of goodwill, insolvency proceedings)	2016–2026, daily	Integrated
Permanent amenities database (BPE), geolocated — INSEE	Shops, services, health, administrations, education, sport, tourism	2024 and 2025 editions, annual	Integrated
Establishments and salaried headcount by municipality × activity — URSSAF / Acoss	Private salaried employment by activity	2006–2025, annual	Integrated
Education directory and pupil numbers — MENJ / DEPP	Schools, location, pupils per school year, staff	Pupil numbers 2016–2025	Integrated
FINESS — DREES / ARS	Health and social facilities, pharmacies	Extract of May 2026, bimonthly	Integrated
Overture Maps places — Overture Maps Foundation	Points of interest with a confidence score (comparison layer)	Frozen extract of release 2026-09-23, monthly	Integrated
OpenStreetMap (Overpass) — OpenStreetMap contributors	Points of interest, shops, streets, pedestrian paths	Current state	Integrated
Historic monuments (Mérimée / POP) — Ministry of Culture	Heritage attractors (cathedral, Schœlcher library, Fort Saint-Louis)	Current	Integrated
23 more — e.g. Flores localised employment (INSEE), Foursquare Open Source Places, LOCOMVAC vacant commercial premises (DGFiP), Google Places			Catalogued

Population, income & housing

Source	What it contributes	Period / frequency	Status
Census, IRIS-level files (RP) — INSEE	Residents, age, occupations, dwellings (vacancy, car ownership), workforce	RP 2016 → RP 2022, annual	Integrated
Filosofi by IRIS — INSEE	Disposable and declared income, poverty	2016–2021, annual	Integrated
Filosofi 200 m grid files — INSEE	Tax households, individuals, poverty, social housing and age per 200 m cell	2015, 2017, 2019, 2021	Integrated
Filosofi 200 m grid, Géoplateforme WFS layer — INSEE	Grid cell geometries	2015 layer	Integrated
IRCOM income tax by municipality — DGFiP	Tax households, declared income	2016–2024, annual	Integrated
Cafdata — CNAF	Recipients of minimum income, housing and family benefits	2020–2024 (2022 missing), annual	Integrated
Priority neighbourhoods (QPV) — ANCT / INSEE	Priority urban policy areas, which include the city centre	2024 boundaries	Integrated
LOVAC — Cerema / DGALN	Short-term and structural (≥ 2 years) vacancy of private housing	2020–2026 editions, annual	Integrated
RPLS social housing — SDES	Social dwellings (address, surface, rent, vacancy)	2023 and 2025	Integrated
Sitadel building permits — SDES	Dwellings and premises authorised, started, demolished	2013–2026, monthly	Integrated
Geolocated DVF — DGFiP / Etalab	Property transactions (price, surface, commercial premises)	2021–2025, rolling 5 years	Integrated
8 more — e.g. census detail files and legal populations (INSEE), DV3F and land files (Cerema / DGFiP), WorldPop / GHSL gridded population			Catalogued

Mobility & transport

Source	What it contributes	Period / frequency	Status
Commuting flows (MOBPRO) — INSEE	Home → work flows between municipalities	2016–2022, annual	Integrated
School commuting (MOBSCO) — INSEE	Home → study flows between municipalities	2016–2022, annual	Integrated
National travel survey 2018–2019 (EMP) — SDES	Trip purposes, departure and return times, activity durations, distances	2018–2019	Integrated
GTFS, urban network Centre zone (Mozaïk, BRT lines) — Martinique Transport	Scheduled transit supply	Archives since June 2023	Integrated
GTFS-RT, continuous collection — Martinique Transport	Observed bus speeds, buses in service, declared occupancy	Since 7 October 2026; 60 s (positions), 10 min (predictions)	Integrated
GTFS, ferry shuttles — Martinique Transport	Shuttles landing on the waterfront	Versions since August 2024	Integrated
Road injury accidents (BAAC) — ONISR	Geolocated accidents (pedestrians, vehicles), with time	2016–2024 loaded	Integrated
Movement Distribution — Meta Data for Good / HDX	Share of users at home, within 10 km, 10–100 km, beyond 100 km	Daily; 90-day window, archived every 14 days	Integrated
Movement Range Maps — Meta Data for Good / HDX	Change in movement, share of users staying put	March 2020 → May 2022, daily (discontinued)	Integrated
BD TOPO main road network — IGN	Island road graph for traffic assignment	Current	Integrated
BD TOPO road sections — IGN	Carriageway width, lanes and direction, used to measure kerbside parking	Current	Integrated
OpenStreetMap car parks — OpenStreetMap contributors	Car parks, type, declared capacity	Current	Integrated
18 more — e.g. ridership and ticket validations (Martinique Transport, on request), road counts (on request), Orange Flux Vision mobile data (paid), Waze for Cities / Strava Metro (partnership), a pedestrian counting campaign (to be created)			Catalogued

Tourism & cruise

Source	What it contributes	Period / frequency	Status
Cruise call schedule, port of Fort-de-France — CruiseMapper	Ship-by-ship calls with arrival and departure times	March 2026 → March 2029; archived monthly, as past months disappear	Integrated
Ship pages — CruiseMapper	Passenger capacity, crew, tonnage of ships in port	Current	Integrated
Cruise season reviews — Martinique Tourism Authority (CMT), via the press	Passengers and calls per season, cruise terminals	Seasons 2024–25 and 2025–26	Integrated
PortWatch — IMF / HDX	Daily port calls from AIS	2019–2026, daily	Integrated
Monthly commercial air traffic — DGAC	Arrivals at Martinique Aimé Césaire airport (tourist seasonality)	1990–2025, monthly	Integrated
Classified tourist accommodation — Atout France	Classified hotels and residences, rooms and capacity	Current stock	Integrated
Tourist accommodation capacity — INSEE	Hotels by rating, campsites, residences, holiday villages	2015–2026, annual	Integrated
7 more — e.g. DATAtourisme (ADN Tourisme), INSEE hotel occupancy survey, Inside Airbnb / AirDNA, port authority call data			Catalogued

Context: weather, air, calendar, safety

Source	What it contributes	Period / frequency	Status
Climatological data — Météo-France	Rain, temperature, wind (Fort-de-France Desaix and Le Lamentin stations)	2010–2026, hourly / daily	Integrated
Air quality — Madininair	NO₂ as a traffic proxy, plus PM10, PM2.5, O₃ and H₂S from sargassum	Hourly; monthly NO₂ 2016–2026	Integrated
Recorded crime by municipality — SSMSI	Recorded offences	2016–2025, annual	Integrated
Public holidays in Martinique — DINUM	Calendar, including 22 May	2006–2031	Integrated
School calendar — MENJ	School holidays, including Carnival	2017–18 → 2026–27	Integrated
Pageviews — Wikimedia Foundation	Online interest in Fort-de-France, Carnival and heritage	2016–2026, monthly	Integrated
4 more — e.g. NASA Black Marble night lights, Google Trends / GDELT, an events agenda (to be compiled)			Catalogued

Geography & base maps

Source	What it contributes	Period / frequency	Status
Study perimeters — project	City centre and extended centre (provisional, see D1)	Version 0.1	Integrated
Administrative boundaries API (geo.api.gouv.fr) — DINUM / IGN / INSEE	Municipal boundary and legal population	Current	Integrated
IRIS boundaries — IGN / INSEE	Sub-municipal census units	Current layer	Integrated
BD TOPO — IGN	Buildings (height, use), roads, facilities	Quarterly editions	Integrated
BD TOPO residential buildings, extended centre — IGN	Use, height and footprint used to place households in buildings	Current edition	Integrated
OpenStreetMap vector base map — OpenStreetMap contributors	Streets, water, parks, coastline drawn in the application	Current	Integrated
9 more — e.g. Etalab cadastre, National Address Base, BD ORTHO aerial imagery (IGN), local urban plan (Géoportail de l’Urbanisme), natural risk maps (Géorisques)			Catalogued

Results

The project started in October 2026 and is at an early, uncalibrated stage (application v0.6.0). All city-centre figures depend on a provisional perimeter (see below). Each result below states which sources it crosses: none of them could be produced from a single dataset.

Catalogue. 120 sources are catalogued, of which 51 are integrated.
Warehouse (first build). 117,161 entities, about 25,000 measures and 1,278 indicators, with all 6 invariants passing.
Retail decline vs residents (measured, registers × census × tax data). Active personal-goods establishments (Sirene) fell from 498 (2016) to 363 (2026), a drop of 27 %, and the city centre’s share of the municipality in this sector fell from 75.5 % to 64.7 %. Over the same period, population in the city-centre IRIS (census) fell from 13,444 (2016) to 12,898 (2022), and housing vacancy rose from 24.9 % to 27.6 %. 47 % of city-centre households are below the poverty line, against 30 % for the municipality (Filosofi 2021). Read together, the three sources show a centre losing shops, residents and purchasing power at the same time.
Day/night rhythm (modelled, six sources crossed). About 3,700 people are present at night and 11,200 in the morning, a ratio of 3.0. Night-time residents come from the Filosofi 200 m grid placed in BD TOPO buildings; daytime workers from Sirene headcount bands; pupils from the education directory; tourists from classified accommodation capacity. The morning slot is adjusted on the hourly NO₂ profile of traffic stations.
Observed bus speed vs model (GTFS-RT × road graph × travel surveys). Over one full day, buses ran at 12.7 km/h at 05:00, falling to 7–8 km/h during the day and 10.6 km/h at 19:00. The assignment model, built from BD TOPO roads, MOBPRO commuting, EMP trip profiles and BPE amenities, predicts a 20 % slowdown at peak hours, whereas the buses show 45 %. The measured feed exposes what the model misses.
Cruise check (schedules × ship capacities × reported counts). Onshore passengers are modelled from the CruiseMapper call schedule and ship capacities. On the one documented cruise day (24 April 2026), the estimates came within about 1–2 % of the passenger counts reported in the press (3,075 vs 3,112, and 1,995 vs 1,969). The fill rate used in the model was set on that same day, so this is a consistency check, not an independent validation.

Limitations & next steps

D1: the city-centre perimeter is provisional. It is a rectangle around the historic core plus a 1 km extended zone, used until an official perimeter is provided. Every city-centre figure depends on it.
No pedestrian, ridership or road-count data is open. Hourly rhythms, attractiveness weights and parking assumptions are still uncalibrated. The measured controls so far are hourly NO₂ at traffic stations and observed bus speeds.
The 2016–2026 business stock is rebuilt from creation and closure dates only.
Next steps: build up several months of GTFS-RT data for measured congestion, request survey and counting data, and calibrate presence on field counts.

Stack

Python · DuckDB (spatial, httpfs) · Parquet · SQL · GTFS / GTFS-RT · pytest · ruff · Electron · React · TypeScript · ECharts · Leaflet · GitHub Actions

Codex — High-Density Physical Archiving of Media on Paper

Tue, 01 Sep 2026 00:00:00 +0000

Context

Consumer digital media decay on the scale of years. Unpowered SSDs lose charge, optical discs delaminate, formats go out of use and accounts get closed. Paper can last for centuries, needs no electricity and can be checked by eye. The question is how much information an ordinary A4 sheet can hold if it must still be read back reliably with a standard flatbed scanner and no special hardware.

The problem is hard because the print-and-scan channel is noisy in several ways at once. Toner spreads and clumps, density varies across the page, and scanners add rotation, anisotropic scaling, uneven paper feed, blur, vignetting and gamma. Paper also stretches non-linearly: at the target density, 0.2 % of stretch shifts the grid by about three cells. Common 2D codes such as QR and DataMatrix are designed for robustness rather than density. A long-term archive also has to remain decodable when today’s software no longer exists.

Approach

flowchart TB
 A["Source media: photo, video, voice, text"] --> B["Format-conservative reduction
JPEG baseline · MJPEG/PCM AVI · PCM WAV · UTF-8"]
 B --> C["Archive: tar + zlib + SHA-256"]
 C --> D["Inter-page erasure code
Cauchy MDS over GF(256): any K of N pages"]
 D --> E["Per-block Reed–Solomon + CRC-32
page-wide interleaving, triplicated header"]
 E --> F["Multi-level cell rendering
1-bit A4 PDF at 1200 dpi"]
 F --> G["Print → scan
(simulated channel)"]
 G --> H["Fiducials → homography → local alignment markers"]
 H --> I["Grey-level calibration from the page itself"]
 I --> J["RS/CRC decode → erasure decode → SHA-256 check"]
 J --> K["Byte-exact files, or partial recovery"]
 F -.-> L["Printed appendix: format spec + decoder source in OCR-B with per-line CRC-16"]
 L -.-> J

Left: page 1 of 10 of the photo archive, as sent to the printer (A4, 1200 dpi, a grid of 1,145 × 1,576 cells), downscaled. Right: the boxed top-left corner at printer resolution (about 10 × 9 mm, each printer dot drawn as 2 × 2 pixels): the solid corner fiducial, the start of the header band, and the first data cells. Each cell is 8 × 8 dots (0.169 mm) and its number of inked dots encodes one of 4 levels. The square at the corner of the grid is an alignment marker.

A page of the printed appendix (page 11 of 30): the start of cdx_read.py (360 lines), set in OCR-B. Each line carries its number and its CRC-16, so a line misread by OCR can be found and checked by hand.

Methods

Multi-level cells instead of binary modules. Each cell carries 2 bits. Cells are drawn as clustered dots at the printer’s native resolution, so the driver does no halftoning and produces no moiré.
Geometry recovery in stages. Four corner fiducials, one of them hollow to fix orientation, give a global homography. Alignment markers spread across the page then absorb local, non-linear paper stretch.
Self-calibrating reader. The decoder estimates the printed grey levels and spatial shading from calibration structures on each page, so it does not depend on the printer’s dot settings.
Two-level error control, a product code. Reed–Solomon and CRC-32 inside each page correct random and burst errors such as stains and scratches. A Cauchy MDS erasure code across pages, decoded block by block, lets any K of N pages rebuild an archive.
Graceful failure. If the SHA-256 check fails, the decoder still extracts every complete file that comes before the damaged region.
Self-describing archive. A printed appendix contains the format specification, a reading procedure and the full decoder source in OCR-B, with a line number and CRC-16 on each line. The decoder needs only NumPy, Pillow and reedsolo. Text uses DEFLATE (zlib), which can be reimplemented in a few hundred lines, rather than the stronger LZMA.
Earlier prototype (codex_proto/). This version explored several physical media: paper, laser-marked stainless steel, intaglio plates and microfilm frames. It used 2- to 16-level cells, fountain-style parity pages, MLP-based grey-level readers, a printed optics calculator for reading microfilm, and lossy media reduction (parametric voice coding, wavelet-residual photo coding).

Results

All figures below come from simulation only. The software models laser defects (spot blur, toner grain, dot gain, banding, non-uniform density, textured paper), scanner defects (±1.2° rotation, ±0.3 % x/y scale, feed wobble, blur, vignetting, noise, gamma 2.2), coffee stains, scratches, and one deliberately lost page per archive. No physical print-and-scan test has been run yet.

Net payload: 383,469 bytes per A4 page, about 59 bits/mm².
End-to-end: 46 of 50 pages were scanned and all 4 of 4 archives were restored byte-identical (photo, video, voice, text). No Reed–Solomon codeword failed, and no inter-page block was lost.
Error rates: clean pages needed about 0.007–0.008 % of bytes corrected. Pages with stains and/or scratches needed at most 0.90 %.
Archive sizes: photo 2.86 MB → 10 pages (8 data + 2 parity); video (800×600, 24 s, MJPEG + 16 kHz PCM) 8.30 MB → 25 pages; voice 0.91 MB → 5 pages; text 5.58 MB → 10 pages.
Text: the public-domain Mémoires d’outre-tombe (5.43 M characters) would take 1,357 typeset A4 pages. Encoded, it takes 6 data pages, a ratio of about 226. zlib compresses this prose by 2.62 and LZMA by 3.58.
Partial recovery: an archive truncated to a quarter of its length still returned 1.42 MB of the 5.58 MB of text.
Decoder robustness: the decoder also worked on scans rotated 90° and 180°, on 600 dpi scans and on JPEG (q92) scans.
Decoder size: 731 lines of Python for the printed decoder modules.

The same region of the photo page (380 × 300 dots, about 8 × 6.4 mm), enlarged. Left: as encoded. Right: after the simulated print-and-scan channel, resampled onto the page grid through the fiducial homography. Toner spread, blur and noise soften the cells, and the edge of a simulated coffee stain darkens the left part. The reader calibrates the grey levels locally, so it still separates the four levels. In the simulation run reported above, stained or scratched pages needed at most 0.90 % of bytes corrected.

The video archive: an underwater clip (school of fish over a reef) used as test footage. Top: the original 3840 × 2880 HEVC frames at 3.2 s, 12.8 s and 16.0 s, downscaled. Bottom: the same instants decoded from the simulated scans of the 25 pages (one deliberately lost). The paper channel itself loses nothing: the restored AVI is byte-identical to the archived one. What differs from the original is the reduction made before printing (800 × 600 MJPEG, 12.5 frames/s, 16 kHz mono sound) so that 24 s of video fit in 8.30 MB.

Close-up of the 12.8 s frame. Left: a 1152 × 864-pixel area of the original, downscaled. Right: the matching 240 × 180-pixel area of the restored frame, enlarged with no smoothing. The fish and the reef stay recognisable. Fine texture and colour detail are lost, and MJPEG block artefacts appear. This is the price of fitting the clip into 25 pages.

Limitations & next steps

No physical test yet. The cell dot patterns were tuned in simulation. The next step is a real 1200 dpi print with a 1200 dpi scan to set them and measure the actual error margin.
Small test set. The simulation covers one run over four archives. A sweep across defect severity, scan resolution and paper stock is still needed.
Unmeasured physical properties. Long-term ageing of paper and toner, and how practical it is to OCR the decoder back from paper, have not been measured.
Font dependency. The appendix body text relies on a system font.

Stack

Python · NumPy · SciPy · Pillow · reedsolo · ReportLab · soundfile · ffmpeg · Tesseract (OCR-B)

Explainable Processing Atoms for Documents & Data

Tue, 01 Sep 2026 00:00:00 +0000

Context

Production systems for document reconciliation and OCR, built for clients such as a wholesale distributor and an accounting firm, left several codebases that each solved the same sub-problems (page routing, OCR tuning, LLM extraction, field comparison) with their own bugs. The atoms library consolidates them into one package. Its central question is the one behind the original systems: can a document check be automated so that a human can audit and contest every verdict? Extracting a field is not enough. The system must state whether two documents agree, why, and from what evidence.

Approach

Everything is built from atoms. An atom has one responsibility, typed inputs and outputs, no global state, and injected dependencies (LLM client, OCR engine, storage). A pipeline is a composition of atoms, either written in code or declared as a YAML recipe that is validated when it loads, before anything runs.

flowchart TB
 A["Input documents (PDF, scans, CSV, mail)"] --> B{"Page routing: native text or OCR?"}
 B -- "native" --> C["Positioned native text"]
 B -- "scan" --> D["PaddleOCR (production settings)"]
 C --> E["Segmentation & classification"]
 D --> E
 E --> F["LLM JSON extraction + verbatim grounding check"]
 F --> G["Cross-document reconciliation (invoice / delivery note / order)"]
 G --> H["Tri-state verdict + named reason + trace"]
 H --> I["Report / human review"]

The catalogue

The 459 atoms are grouped into 32 families, one Python package each, ordered along the processing chain: capture the world, understand it, reconcile and decide, deliver, then operate in production. Two further packages assemble them: steps (77 named, parameterised steps) and recipes (the YAML loader and validator).

Stage	Family	Atoms	Role
Base	`core`	25	pivot types, pipeline, configuration, resilience (retry, rate limits), files, concurrency
Capture	`connect`	40	resilient HTTP, mailboxes (Graph, IMAP), SFTP, OneDrive, read-only ERP source, open data, audit trail
	`google`	24	Gmail, Sheets, Drive, Apps Script, with resilient execution
	`web`	17	driver-agnostic web robots (Selenium, Playwright): waits, resumable skeleton, forms
	`pdf`	9	positioned native text, OCR-or-not decision, rendering, page handling, duplicates
	`ocr`	9	PaddleOCR with production settings, EasyOCR
	`layout`	3	page geometry: lines, columns, deskew, monospace rendering
	`audio`	16	chunked transcription, segment post-processing, microphone ↔ screen alignment
	`vision`	32	video / RTSP streams, detection, tracking, depth, segmentation, calibration
	`vlm`	10	screen reading with vision-language models: change detection, label location
	`geo`	11	geodesic measures, combined GPS fixes, line-distance filtering, spatial SQL
	`calendar`	4	public holidays, working days and opening hours
Understand	`text`	32	numbers and amounts, dates, normalisation, pattern-based fields, identifiers
	`segment`	7	pages → logical documents, typed document numbers, page nature
	`classify`	12	file name, sender, weighted patterns, TF-IDF, visual and LLM classifiers, cascades
	`llm`	24	provider-agnostic contract, providers, pool, prompts, structured extraction
	`anchor`	5	fields and tables located relative to printed labels, native text or OCR
	`kb`	8	traced knowledge base: validated event log folded into a SQLite index
Reconcile	`match`	25	EAN/GTIN, indexes, multi-criteria scores, optimal assignment, entity resolution
	`compare`	9	tri-state, explainable comparators for fields, records and sets
	`check`	9	generic tri-state checks: totals, VAT, balance, completeness, plausibility
	`reconcile`	12	set pairing, order → delivery → invoice chains, N-way, ledger and bank matching
	`validate`	14	rules engine, identifiers, amount consistency, corrected LLM verdicts, final decision
	`rules`	8	safe condition language, explainable first-match tables, escalation, state machines
	`events`	11	timestamped events: windows, expected-event watch, sessions, status transitions
	`analytics`	18	supply-chain analytics without pandas: demand, stock-outs, service level, safety stock
Deliver	`export`	8	CSV for French Excel, shared Excel workbooks, HTML / Markdown / PDF reports
	`report`	6	persistable discrepancies, summaries, review queue
	`render`	12	HTML deliverables: pre-rendered Mermaid, ECharts charts, GeoJSON maps
	`diagram`	6	process diagrams: draw.io / BPMN JSON in, Mermaid and BPMN XML out
Operate	`ops`	26	high availability (heartbeats, leader election), scheduled daemon, monitoring, pre-flight checks
	`qa`	7	editorial non-regression (golden files), simulation replay, inter-annotator agreement

Named recipes

Each of the 17 recipes is a YAML list of steps, named <family>.<atom>. Three examples:

invoice_multi_doc: a batch of multi-document PDFs becomes checked logical documents. io.collect → pdf.read → segment.documents → classify.documents → llm.extract → validate.ids → validate.amounts → validate.decide → export.csv → google.sheets.upsert → connect.notify. Each document ends as valid, to check, or invalid.
doc_reconciliation: a file of invoices, credit notes and delivery notes, checked against orders. Guarded LLM extraction (llm.extract_docs) is cross-checked with pattern extraction (extract.regex, compare.extractions, reconcile.fields), then check.document and check.completeness, lines against the order reference (compare.reference), the order → delivery → invoice chain (reconcile.nway), and a discrepancy file plus review queue (report.discrepancies, report.review). 17 steps.
meeting_transcript: a meeting recording becomes a timestamped transcript with speakers. audio.decode → audio.vad → audio.transcribe → audio.diarize → audio.speakers → audio.segments → export.csv → export.transcript → connect.notify.

The others include delivery_vs_order, ledger_reconciliation, invoice_dedup, article_base_cleaning, stock_policy, mail_to_sheet, draft_mails, pdf_text_extract, rubric_eval, screen_session, camera_flow and process_from_transcript.

The Atlas

The Atlas is a single offline HTML page generated from the code itself (syntax trees and YAML), with no hand-written data. It draws the catalogue as a periodic table: each family is an element with a symbol, each module an atom, each recipe a molecule. It also flags atoms that lack documentation, tests or a usage example. The interface is in French.

The “Capture” series of the periodic table. Each box is a family, with its symbol, number of modules and size in thousands of lines; each tile below it is an atom.

The “Reconcile” series: the families that pair, compare, check and decide.

An atom’s page (compare.fields): what it does, taken from its docstring, then each public name with its full signature.

The invoice_multi_doc recipe as a molecule: 11 steps, the context keys that link them, and the services they need (dashed when optional).

Methods

Tri-state results. Every comparison returns ok ∈ {True, False, None} (conforms / discrepancy / inconclusive) with a named reason code and a trace. A missing or unreadable value counts as inconclusive, never as an anomaly. Inconclusive results are still counted, not silently dropped.
Hybrid native-text/OCR routing. Each PDF page is classified as native, OCR or empty, based on how much native text it holds and how much of it is covered by images. OCR runs only where it is needed.
Grounded LLM extraction. Structured JSON extraction is validated against Pydantic schemas. Each extracted value is then checked verbatim against the source text, so altered or invented values are flagged.
Cross-document reconciliation. Invoices, delivery notes and orders are compared line by line using named tolerances and dominance rules. For example, a matching line amount can override a price difference.
Provenance. Every ported atom carries a Provenance: header and an entry tracing it to repo@commit:path:lines in the original production code. Bugs found while porting were fixed and each fix is covered by a non-regression test.
Enforced architecture. Layering is checked by 30 import contracts: an atom imports only the core package. Strict typing, a ≥ 80 % coverage gate and a check that no client names appear in source code are also enforced.
Device abstraction. Vision and audio atoms target cuda, mps or cpu via one setting.
End-to-end evaluation. Tests run with real engines (PaddleOCR, Ollama qwen3:8b and qwen3-vl:8b) on data with known ground truth. Each metric is recorded against a threshold in a JSON/HTML report. Metrics that miss their threshold are kept and shown as failing.

Results

Codebase (current tree): 459 atom modules in 32 families, 77 parameterised steps, 17 YAML recipes, 30 import contracts. The suite collects 2,959 unit and recipe tests, plus 109 end-to-end tests that run separately. All 17 recipes have at least one end-to-end test, and 34 of the 35 Atlas elements are exercised end to end.

Selected end-to-end scores (synthetic documents with known ground truth, local 8B models):

Task	Engine	Measured
Invoice + delivery note vs. order reconciliation	`qwen3:8b`	document types and fields 14/14; expected discrepancies 5/5; 0 false discrepancies on the compliant case
LLM JSON extraction (4 native invoices)	`qwen3:8b`	valid JSON 4/4; exact fields 28/28; all deliberately altered values caught by grounding
OCR on a photographed label	PaddleOCR	word recall 0.94 on the raw photo; a contrast-enhancement defect (0.65–0.71) was found and fixed (0.88–0.94)
Table reconstruction from OCR	PaddleOCR	100 % exact cells (proportional font), 95 % (monospace)

Limitations & next steps

Verbatim grounding cannot catch a value that exists in the document but was put in the wrong field: an absent field was wrongly filled in 1 of 8 cases.
The 8B model judged a +50 % quantity gap a “minor” discrepancy, against the instruction. LLM verdicts are therefore normalised and kept downstream of deterministic rules.
Scores come mostly from synthetic data. A real-document test set is being built outside the repository, and only cases that have been reviewed are scored.
Next steps: measure the automatic-reconciliation rate and the false-positive rate on production data.

Stack

Python · Pydantic · PyYAML · PaddleOCR · Ollama (qwen3:8b, qwen3-vl:8b) · Mistral / Anthropic / OpenAI APIs · pytest · mypy · import-linter · ruff

Field Identification of Recall-Affected Vehicles

Mon, 01 Jun 2026 00:00:00 +0000

Context

Safety recall campaigns depend on reaching every affected vehicle, but letters and phone calls leave a residual population of owners who never respond. In island and overseas territories the problem is sharper: network coverage is uneven, field logistics span several territories, and the registry of who drives what is incomplete. The practical question becomes a fleet-management one: which vehicles of the circulating fleet are still affected, where are they seen, and how far has each one progressed towards repair?

Approach

flowchart TB
 A["Recall listing
(scheduled file pull or manual CSV)"] --> B["Central server
import · versioning · model-name decoding"]
 B -->|"incremental diff"| C["Field units (laptops in agent vehicles)
local recall copy"]
 D["Camera stream"] --> E["Plate detection + OCR
(on-device, offline)"]
 E --> F["Format-aware correction
+ double-read confirmation"]
 F --> G{"Match against
local recall list?"}
 G -->|"no"| H["Discarded — nothing stored"]
 G -->|"yes"| I["Alert to agent
+ local queue"]
 I -->|"sync when network returns"| B
 B --> J["Dashboard
units · tours · detections · incidents"]
 J --> K["Owner contact workflow
identified → contacted → repaired"]

Methods

Data ingestion and versioning. The recall listing is pulled automatically on a schedule or imported by hand. Each import is diffed against the previous one with stable identifiers, versioned by hash, and only the delta is sent to the field units. Manufacturer type codes are translated into readable model names on import.

Matching under OCR noise. Plate detection and character recognition run on the field laptop with ONNX models, with no network dependency. Two safeguards reduce false matches: (i) a positional correction step uses the known structure of national plate formats (current and legacy) to resolve classic OCR confusions (O/0, I/1, S/5, B/8, Z/2, G/6), rejecting any ambiguous reading; (ii) a double-read confirmation only counts a plate when it is read twice within a short time window. A smartphone web page offers a manual fallback (photo or typed plate, plus VIN check-digit validation).

Offline-first synchronisation. Matches are queued locally and pushed to the central server on the next connection, so detections survive multi-day outages. Units report a heartbeat (software version, recall-list version, reading counters).

Field operations. Supervisors plan tours as ordered, geolocated checkpoints on a map, assign them to teams and units, reuse templates, and follow progress. Agents report incidents with a severity level. A contact workflow records the outcome of each owner approach, and the dashboard derives a conversion funnel (identified → approached → engaged → repaired) with weekly trends, the median delay to repair and a breakdown by territory.

Privacy by design. Plates that do not match the recall list are never stored or transmitted; retained records have fixed retention periods with automatic purge; actions are written to an audit log; dashboard access is role-based, including read-only accounts; data at rest is encrypted. A processing register was written alongside the code.

Deployment & operations (DevOps)

The system runs on two very different kinds of machine: Windows laptops in agents’ vehicles, which may stay offline for days, and one server per instance. After the first weeks in the field, most of the engineering work went into making both sides easy to deploy, observe and recover, by a single person, without breaking field work.

flowchart LR
 subgraph ADM["Admin workstation"]
 GIT["Git repository
committed tree only"]
 PROV["Provisioning script
one config file per unit"]
 BLD["Field build
PyInstaller bundle + ONNX models"]
 end
 subgraph SRV["Server (Ubuntu LTS VPS)"]
 CAD["Caddy
TLS · HSTS · no access log"]
 APP["Docker container
FastAPI API + dashboard
loopback port only"]
 VOL[("LUKS2 volume
SQLite · photos · archives")]
 TIM["systemd timers
monitoring · backup · cleanup"]
 end
 FLD["Field laptops
local SQLite · sync loop"]
 OBJ[("Off-site object storage
age-encrypted backups")]
 CHAT["Chat alerts
state changes + daily digests"]
 GIT -->|"git archive over SSH → deploy script"| APP
 PROV -->|"admin API"| APP
 PROV -->|"config file + README"| FLD
 BLD -->|"archive + one-click installer"| FLD
 FLD <-->|"HTTPS: heartbeat · pushes · recall diff"| CAD
 CAD --> APP
 APP --- VOL
 TIM --> VOL
 TIM -->|"encrypted copy"| OBJ
 TIM --> CHAT

Field laptops: packaging, installation and updates

Packaging. The field application (Python backend, React interface) is frozen with PyInstaller into a one-folder Windows executable that embeds the ONNX models and the OCR configuration, so a unit needs no Python install and no network to read plates. Built-in --diag and --selftest modes check the bundled runtime and read a test image without starting the interface, which made start-up failures diagnosable on the field machine itself.

One-click installer. The build is delivered as an archive with a double-click INSTALLER.bat. It elevates itself and runs a PowerShell script that removes the “downloaded from the Internet” mark from the files (otherwise the bundled runtime fails to load), disables USB power saving so the GPS dongle is not suspended, and installs the unit’s configuration file where the executable reads it. The script refuses to guess: if several different configuration files are present, it lists them and asks which one is the new one. An optional reset clears the local database when a unit moves to another server; it shows the target server before asking for confirmation and warns that unsynchronised detections would be lost. The unit’s access token is never displayed. An update follows the same path: new build, re-run the installer.

Provisioning. A Python script that uses only the standard library, so it runs on a bare Windows machine, creates the units from a CSV file through the server’s admin API and generates, for each unit, its configuration file (server address, unit code, display name, token) and a README to hand over with the laptop. The admin key is read from an environment variable or a masked prompt, never from a command-line argument.

Versioning: software and recall list

Software version. Each unit reports its application version in its heartbeat. The dashboard compares it with the expected release and flags the units that still need updating.

Recall-list version. The recall list is versioned by hash. On each cycle, a unit compares its local hash with the server’s: identical, nothing to do; no local copy, full download; otherwise, the server returns only the rows added and removed since that hash (and the unit falls back to a full download if the server no longer knows its version). The change is applied in a single transaction, and the local hash only moves forward if everything was applied.

That last rule came from a field incident. A database constraint made the diff fail on one unit, the error was swallowed by the sync loop, and the unit kept sending heartbeats while its recall list stayed frozen for more than two weeks, so it alerted on vehicles that had already been repaired. Release 1.4.1 fixed the cause, added an explicit rollback, and shipped a schema migration that keeps existing data. It was tested with a 62-test run that included reproducing the bug and migrating a database with the old schema. Monitoring now raises a “recall sync stuck” alert when a unit has not received a published list after a grace period.

Incremental, resilient synchronisation

Every five minutes, or on demand, a unit runs a sync cycle made of independent steps: heartbeat, push of detections (in batches), GPS track, incidents, checkpoint updates, then pull of the recall list and the tours. Pushes come before pulls, so field data leaves first. A failing step no longer blocks the others: in an earlier version the cycle was monolithic, and a failed heartbeat once delayed detections by 22 minutes. Requests use short timeouts, downloads longer ones. The heartbeat carries the software version, the recall-list version, the number of pending detections and the reading counters, which is what the dashboard and the monitoring rely on.

Server deployment

Instance kit. Each deployment is an independent instance described by a short identifier, on an Ubuntu LTS virtual server. A bootstrap script, which is interactive and can be safely re-run, installs the system packages, Docker and Caddy from their official repositories. It enables unattended security updates without automatic reboot, restricts SSH to key authentication, enables a default-deny firewall and brute-force protection, and checks that the DNS record points to the server before applying the TLS configuration. Secrets live in root-only configuration files on the server, never in the repository.

Container and reverse proxy. A multi-stage Dockerfile builds the React dashboard with Node, then runs the FastAPI server on a slim Python image as an unprivileged user. Its health check allows a three-minute start period because the plate and OCR models are loaded before the server listens. The container is published on the loopback interface only. Caddy is the only public entry point: automatic Let’s Encrypt certificate, explicit HSTS, and no access log, because scan URLs contain plates. The application’s own access log is disabled for the same reason.

Deploy and rollback. Code leaves the admin workstation only as a git archive of a committed tree: uncommitted changes are listed and not sent, and line endings are forced to LF. On the server, the new tree is unpacked next to the current one and swapped in, the previous one is kept, and the deployed commit is recorded. The deploy script then detaches from the SSH session, so a dropped connection interrupts neither the deployment nor a rollback. A lock prevents concurrent deployments. The script builds the image (labelled with its commit), keeps the previous image as rollback, recreates the container and waits until it reports healthy. It gives up after five minutes, or as soon as the container has restarted twice. On failure it rolls back automatically to the previous image and source tree, sets the failed code aside and sends an alert. Going back to any earlier version means re-deploying that commit. An image rollback does not undo a schema migration, so the documented procedure is to take a backup before any deployment that changes the schema.

Encrypted volume and safe restarts

Server data (database, contact photos, recall archives) lives on a LUKS2 file container mounted with restrictive options. Its passphrase is never stored on the machine, so after a reboot a person unlocks the volume. The container restarts on its own, but the server refuses to start unless a sentinel file on the encrypted volume is present. Without it, a reboot would have left an empty directory in place of the volume, and the application would have created a fresh, unencrypted database there and accepted field data into it. The guard was checked both ways on a throwaway container. A separate script grows the volume while it is mounted.

Backups and recovery

A daily systemd timer takes a consistent SQLite snapshot through the database’s backup API, archives it with the photos and recall archives, and encrypts the result with age to a public key. The private key is shown once, at creation, and kept off the server. Backups are kept locally and, optionally, copied to S3-compatible object storage with rclone, configured through environment variables only. A restore test mode decrypts a backup into a temporary directory on the encrypted volume, runs an integrity check, counts rows per table, then deletes everything. The apply mode moves the current state aside instead of deleting it. The documentation includes a step-by-step rebuild after total loss of the server. The continuity plan targets a recovery time of 4 hours and a maximum data loss of 24 hours.

Monitoring

A watchdog runs every 15 minutes and only posts to a chat webhook when a state changes (problem, then “resolved”). A fuller digest is sent twice a day, timed to the field shifts. It checks that the encrypted volume is mounted, the API health endpoint (with retries, because the daily import can block it for about a minute), the container health status, that the database is readable, disk usage on both the system disk and the encrypted volume, the size of the SQLite write-ahead log, TLS certificate expiry, backup age and size, units silent for more than 48 hours (not 24, since units are switched off at weekends) and units whose recall list is stuck. A separate watcher alerts when no recall import has happened for more than 30 hours. Several of these checks were added after real incidents: the encrypted volume once reached 90 % without any alert because only the system disk was watched. A weekly timer cleans up Docker build caches and keeps only the current and rollback images. A dedicated script truncates the SQLite log with a short lock wait, so field writes are never blocked for long.

Offline resilience

The field side is designed to keep working through a server outage: plate reading, matching against the local recall copy, alerts, tours already downloaded and the local statistics tab all work without a network, and the queue empties at the next connection. Only the dashboard (and the server-hosted smartphone page) is unavailable while the server is down.

Testing

There is no CI pipeline in the repository. Releases are tested before they ship, and the commits record those runs: 12/12 and 6/6 for the plate correction and confirmation stages, 62 tests for release 1.4.1. Behaviours that cannot be unit-tested, such as the volume guard or a live deployment, were checked on throwaway containers and against the production health endpoint.

Results

In early field operation, agents flagged 32 % of detections as misreadings over roughly one week. This measurement motivated the positional-correction and double-read confirmation stages, and the dashboard statistics now exclude detections marked as false positives.
The new stages shipped with unit tests: 12/12 for plate correction and 6/6 for confirmation, settings and reading log.
After the model-name mapping was extended, only one row of the production recall listing remained without a decoded model name.

No end-to-end recognition rate has been measured yet.

Limitations & next steps

Recognition accuracy has not been measured against a ground-truth benchmark. The specification targets (read rate, latency) are design goals, not results.
No legacy-format plate reading was observed at first. A dedicated counter now tracks these readings end to end, but the fix still needs a field check.
A false match against a large listing still costs agent time. Calibrating the confidence threshold and confirmation window on labelled field video is the next step.

Stack

Python · FastAPI · SQLite · ONNX Runtime · fast-alpr (YOLOv9 plate detector + OCR) · OpenCV · PaddleOCR · React · TypeScript · Vite · Leaflet · Docker · Caddy · LUKS · systemd · age · rclone · PyInstaller · PowerShell