Home on Boris Dupoux

A Fully Local AI Agent with a Meaning Graph and Multimodal Emotion Measurement

Thu, 01 Oct 2026 00:00:00 +0000

Context

Personal digital traces (messaging, mail, photos, music listening, notes, calendar, health exports, code repositories) are among the most sensitive data a person owns, and also among the richest for studying behaviour — and feeling — over time. Sending them to a cloud LLM is not acceptable, so everything here runs on a single Apple-silicon desktop (M4, 32 GB unified memory), and no data leaves the machine.

The aim is an agent that answers questions about one’s own life while keeping a strict line between what is observed, what is computed, what is hypothesised by a model, and what is accepted as true by its owner.

What is original

Local agents over personal files are not new. Two aspects of this project are.

1. A meaning graph on top of a measurement layer. Knowledge is held at two levels. An observatory stores facts and signals in long format (item, dimension, value, confidence, model, version). Above it, a meaning graph stores what those signals mean: relations (reified, so that a relationship can evolve and be measured), affective episodes (who felt what, when, how intensely, through which modality), themes, chapters of a life, and truths — statements with supporting and contradicting evidence that only the owner can validate. Every node and edge carries its provenance down to the source element, and an origin ranked by trust (human > declared > structural > rule/computed > vendor > model > inferred).

2. Emotion measured across modalities, along unusual dimensions. The same fixed vocabulary of 53 emotions (Plutchik-based, each with valence and arousal coordinates) is applied to messages (local LLM), voice (five acoustic and multimodal layers fused with the transcript, with an explicit text/voice mismatch flag) and photos (deterministic colour and light measures, SigLIP embeddings, face signals treated as weak evidence). These sit in a catalogue of 929 dimensions in 88 groups that goes well beyond the usual metadata. A few examples of its less common entries:

modality	dimension	how it is defined
photo	colour temperature	mean red–blue balance of the image (warm vs cool)
photo	colourfulness	Hasler & Süsstrunk (2003) opponent-colour metric
photo	golden hour / blue hour	sun elevation computed offline from date and position
photo	face size	share of the frame taken by a face (a proxy for proximity)
voice	jitter, shimmer, harmonics-to-noise ratio	openSMILE eGeMAPS prosody
voice	laughter, crying, sighs	sound-event detection (AudioSet classes)
voice	text/voice mismatch	voice valence and transcript valence differ by ≥ 0.6
message	valence, arousal, tone	per participant, constrained to the fixed grid
message	expressive punctuation, emoji palette	deterministic text measures
relation	reply-latency asymmetry	log ratio of the two parties’ median reply delays
relation	initiatives	conversations opened after 3 h of silence, per side
relation	trend	slope of relationship intensity over 90 days (cooling < 0)
period	musical nostalgia	mean age of the tracks listened to
period	creation / consumption	time in creative tools ÷ time consuming

From signals to meaning

flowchart LR
 PH["Photo"] --> PM["Measures
light · contrast · colour temperature
colourfulness · palette · sharpness"]
 PH --> PS["SigLIP embedding
(image ↔ text similarity)"]
 PH --> PF["Vendor face signals
smile · expression · gaze · face size"]
 VN["Voice note"] --> VT["Whisper transcript"]
 VN --> VL["Acoustic layers
valence/arousal/dominance · categories
sound events · prosody · audio-LLM"]
 VT --> VX["Text emotion of transcript"]
 VL --> VF["Weighted fusion
confidence = agreement
text/voice mismatch flag"]
 VX --> VF
 MS["Messages
(pseudonymised: Me, P1, P2)"] --> ML["Local LLM, fixed grid
emotion · intensity · valence
arousal · tone · topics"]
 PM --> OB[("Observatory
929 dimensions")]
 PS --> OB
 PF --> OB
 VF --> OB
 ML --> OB
 OB -- "emotion with enough confidence" --> AFF["Affective episode (AFF)
who · emotion · when · intensity · modality"]
 PF -. "weak cue, confidence ≤ 0.4" .-> AFF
 ML -- "topics" --> THM["Themes"]
 OB -- "aggregates" --> AG["Day / period indicators
valence · light · palette"]
 AFF --> G["Meaning graph
relations · episodes · themes · chapters · truths
(all proposed)"]
 THM --> G
 AG --> G
 G --> HV(("Owner
validate · reject · correct"))

Each affective episode is a node linked to the person who felt it, to the emotion in the fixed vocabulary, and — when it exists — to the conversation or moment it occurred in. Inference rules then carry affect upwards (for instance, the affect of a chapter or of a routine), and strong aggregate correlations can become proposed truths with their measures as evidence.

Emotion measurement in detail

Vocabulary. The grid is fixed and versioned: Plutchik’s 8 primary emotions at 3 intensities, his 8 primary dyads, plus social and reflective emotions useful for a life (gratitude, tenderness, pride, shame, loneliness, nostalgia, wonder, relief, exhaustion…), 53 in total. Each carries starting coordinates in Russell’s circumplex — valence in [−1, 1], arousal in [0, 1] — meant to be calibrated later on the owner’s own validations. Plutchik oppositions are encoded, and health-app mood labels are mapped onto the grid.

Messages. A local LLM (Qwen3-30B-A3B) reads each conversation with participants replaced by “Me”, “P1”, “P2”, messages framed as data it must not obey, and media reduced to placeholders such as “[voice note 23 s]”. Output is constrained by a JSON schema: for each participant, the dominant emotion from the grid (or none), intensity, valence, arousal, a tone from a closed list of 19, and a confidence; plus 1–3 topics that become themes. Long conversations are reduced to their beginning and end; short ones are batched ten per call. An affective episode is proposed only when confidence is at least 0.3.

Voice. Each recording is converted to 16 kHz mono and passes through:

Whisper large-v3-turbo (MLX) for the transcript;
a wav2vec2 dimensional model giving valence, arousal and dominance on 15 s windows, weighted by window length, plus arousal variability;
emotion2vec+ large for categorical emotion, mapped onto the grid (kept at ≥ 0.4);
an AST AudioSet classifier for laughter, crying, sighs, shouting, whispering, music and ambience;
openSMILE eGeMAPS prosody: pitch (semitones) and its variability, loudness, jitter, shimmer, harmonics-to-noise ratio, voiced segments per second, mean pause length;
Qwen2.5-Omni-7B (llama.cpp, Metal), which listens to the audio and returns an emotion from the grid, a valence and short acoustic cues;
the text-emotion model above, applied to the transcript.

The categorical opinions (voice categories, audio-LLM, transcript) are combined by a weighted vote (weights 1.0, 1.0 and 0.8); valence is a weighted mean of the dimensional model, the audio-LLM and the transcript. Confidence is agreement between layers: one agreeing source gives 0.45, two give 0.65, three give 0.85, minus 0.1 for each source that disagrees. When the voice and the words disagree by 0.6 or more in valence (an “I’m fine” said sadly), a text/voice mismatch is flagged rather than averaged away. Laughter and crying scores are kept alongside.

Photos. On 256 px thumbnails, without any model: luminance, contrast, saturation, colour temperature, colourfulness, sharpness (variance of the Laplacian) and a 5-colour palette with named dominant hue. SigLIP so400m embeddings make the library searchable by text. Vendor face attributes (smile, smile type, expression, gaze, eye state, face size) are read as signals, but the schema states that a photographed smile is not a felt joy: the face modality is a weak cue whose confidence is capped at 0.4. Music mood is handled the same way: the mood of a track is never, on its own, evidence of the listener’s mood. At period level, valence is fused with a fixed precedence: declared > voice > text > face.

Human validation. Every episode, theme, chapter or truth is emitted as proposed. The PWA shows a validation queue — affective episodes ordered by confidence, with the evidence (the conversation or recording it came from) — and the owner validates, rejects or corrects. Only that human decision event turns a proposal into validated, an equivalence checked automatically. The planned next step is a small personal classifier on emotion2vec features, trained on these validations.

Architecture

flowchart TB
 S["Sources (read-only snapshots)
messaging · mail · photos · music · notes · calendar · exports"]
 C["Connectors
dedicated ingestion process, sole holder of disk access"]
 V["Vault (owner-only, 0700)
content-addressed raw store"]
 O["Observatory
facts + signals (SQLite)"]
 K["Meaning graph
append-only JSONL log → SQLite fold + FTS"]
 N["Nightly / hourly pipelines
thumbnails · measures · SigLIP · emotions
(GPU lock, one model at a time)"]
 M["MCP server (stdio, no port)
20 read-only / propose-only tools"]
 H["Hermes Agent
local LLM via Ollama"]
 P["PWA (HTTPS + TOTP)
chat · validation queue"]
 U(("Owner"))
 S --> C --> V --> O --> K
 N --> O
 O --> M
 K --> M
 M --> H --> P
 P -- "validate / reject / correct" --> K
 U -- "NetBird VPN" --> P

Raw data is snapshotted (SQLite opened in mode=ro&immutable=1) and never written to; everything derived lives in a separate owner-only vault, and the code repository holds no data. The meaning graph is built on the atoms library’s knowledge-base module: an append-only, schema-validated JSONL event log folded into a disposable SQLite index.

The schema itself is data, verified mechanically: 27 node types and 50 predicates, 9 provenance origins, 31 locator kinds tying every claim to its source element, the 53-emotion vocabulary, 17 inference rules, and 42 sources behind the 929 dimensions (610 native, 126 vendor-computed, 86 aggregates, 74 deterministic measures, 29 model interpretations).

Security model

Source content is assumed adversarial (prompt injection in a message or mail), and the model may hallucinate or misuse tools. Defences are structural:

No ambient capabilities. The agent framework (Hermes Agent) has 20 built-in toolsets disabled (terminal, files, code execution, web, browser, delegation, memory, cloud speech…); no cloud provider, update check or external login; the only model endpoint is a loopback Ollama server.
One narrow interface. Data is reachable only through a local MCP server over stdio, with closed schemas, validated arguments, explicit truncation and a content-free access log. Tools can propose, never decide.
Privilege separation and pseudonymisation. Only ingestion holds full-disk access; analysis has no network. Third parties are opaque codes, and a real name in an identifier is rejected by the schema.
Sensitivity levels (0–3) per element with a lower ceiling for remote sessions; remote access only through a NetBird VPN, TOTP and strict session cookies.
Ethical exclusions: vendor-inferred ethnicity, skin tone and gender are never read.

Models

All inference is local; one large model is loaded at a time under a shared GPU lock.

task	model	runtime
agent (tool use, ≥ 64k context)	gpt-oss:20b, qwen3:8b fallback	Ollama
text emotions, voice-transcript emotions	Qwen3-30B-A3B-Instruct-2507 (Q4)	Ollama
text / image embeddings	bge-m3 · SigLIP so400m	Ollama · PyTorch MPS
image reading	qwen3-vl:8b	Ollama
speech-to-text	Whisper large-v3-turbo	MLX
voice emotion (5 layers)	wav2vec2 dimensional · emotion2vec+ · AST · openSMILE eGeMAPS · Qwen2.5-Omni-7B	PyTorch MPS · CPU · llama.cpp (Metal)

Each model is smoke-tested on synthetic data only (a fictitious French sentence synthesised by a local TTS, then passed through the whole audio chain).

Evaluation & results

Agent answer quality. A 16-question benchmark (2 trap questions), with ground truth computed by calling the tools directly, scores tool choice, numerical fidelity (every number in the answer must appear in that turn’s tool outputs), correctness, trap handling and partial-period flagging: 73 % → 100 % after tool-description, argument-validation and configuration fixes.
Server-side error correction. With a reduced 8-tool interface, simple questions went from 76 % to 90 % and cross-source questions from 73 % to 90 %; 20 deliberately unusual questions scored 89 % and exposed silent-zero failures, since fixed.
Context and latency. Tool descriptions cut from 23.5 k to 9.5 k characters; a heavy aggregate query from 172 s to 0.1 s; all heavy benchmark queries ≤ 0.3 s.
Cross-exploration benchmark. 60 placeholder-only scenarios with oracle code; oracles pass 61/61, with no leak at remote sensitivity ceilings 1 and 2.
Verification. 56 named schema checks green; 631 native/vendor columns confirmed against real database structures; 1,235 Python unit tests, 15 JavaScript test files and 110 quality tests on real data that report counts only.

Limitations & next steps

Evaluation sets are small and authored by the system’s designer; they measure fidelity and regressions rather than general capability. Emotion outputs are hypotheses: the valence/arousal coordinates of the vocabulary are starting values, and personal calibration of voice emotion from validated examples is still pending, as is the larger target agent model. Agent answers take tens of seconds on a 20B model.

Stack

Python · SQLite / FTS5 · Ollama · llama.cpp (Metal) · MLX · PyTorch MPS · Hermes Agent · MCP · SigLIP · Whisper · emotion2vec · openSMILE · NetBird · PWA · launchd

Handheld Video to Photogrammetry and Gaussian Splatting

Thu, 01 Oct 2026 00:00:00 +0000

Context

Indoor sites are often documented with a handheld gimbal camera rather than a survey-grade scanner. The goal was to turn such video into usable 3D deliverables (dense cloud, textured mesh, plan and elevation orthoviews, a Gaussian Splatting scene and lightweight web exports) with no manual steps between capture and output. The pipeline runs natively on Windows on an RTX 5090 (Blackwell, sm_120) that is shared with other long-running GPU workloads, so it must never pre-empt them.

Approach

flowchart TB
 A["inbox/project/videos + GO file"] --> B["Keyframe extraction
optical-flow overlap + sharpness"]
 B --> C["COLMAP 4.2.1 SfM
SIFT, global mapper"]
 C --> D{"QC: registration, reprojection,
trajectory continuity"}
 D -- fail --> C2["Fallback: incremental mapper"] --> D
 D -- pass --> E["OpenMVS 2.4 (sm_120 build)
densify, mesh, refine, texture"]
 E --> F["Ortho alignment
gravity, floor RANSAC, Manhattan yaw"]
 F --> G["Blender renders
orthos, axonometries, orbit"]
 E --> H["gsplat training
MCMC or default strategy"]
 G --> I["Web exports
Potree, GLB, SOG"]
 H --> I
 I --> J["outbox: deliverables, report.json, STATUS"]

Methods

Overlap-driven keyframe selection. KLT tracking with a RANSAC similarity transform estimates overlap with the last keyframe. A frame is emitted when overlap drops below 0.80, tracking is lost or a time limit passes. The sharpest candidate (variance of the Laplacian) is the one kept.
SfM with explicit QC. COLMAP’s global mapper is fast but non-deterministic. In testing it produced reconstructions that met registration and reprojection criteria but were geometrically incoherent. A trajectory-continuity check (jumps above 25× the median camera step) triggers an automatic fallback to the incremental mapper. Below 20 points per image the step fails.
MVS hardening. The official OpenMVS Windows binary has only sm_89 kernels, so it was rebuilt for sm_120. The object ROI is off for rooms because it cropped the cloud. Refinement is dropped when it diverges, and seam leveling (which blackened faces) is off.
Metric-ready orthoviews. The up axis comes from camera orientations. A RANSAC floor plane sets z = 0 and a histogram of wall-normal azimuths (mod 90°) sets Manhattan yaw. An optional two-point reference fixes scale. The 4×4 transform is stored so it can be applied again to the mesh and the splat.
Gaussian Splatting. gsplat 1.6.0 was compiled for sm_120 with CUDA 13 under MSVC, which required local patches. Training uses the MCMC strategy with a Gaussian budget and falls back to the default 3DGS strategy when MCMC diverges.
Operations. Projects start in watch-folder mode when a GO file appears. Stages can be resumed or redone. A GPU guard waits until enough VRAM has been free for a stable period, serialises GPU stages through a lock file and records peak VRAM per stage in report.json. Degraded QC produces a DONE_WITH_ALERTS status instead of a silent success.

Results

First real deployment: five locations inside a commercial showroom, one 4K handheld video per location, processed unattended in about 30 minutes per location.

Location	Images	SfM registered / points / reproj.	Dense cloud	Splat PSNR / SSIM
1	116	116/116 · 21,458 · 1.49 px	1.40 M	23.2 / 0.86
2	103	103/103 · 18,972 · 1.75 px	1.23 M	20.6 / 0.81
3	47	47/47 · 13,167 · 1.17 px	1.26 M	23.1 / 0.86
4	121	121/121 · 11,970 · 1.34 px	1.99 M	21.9 / 0.85 (default strategy)
5	64	64/64 · 12,106 · 1.48 px	0.91 M	21.8 / 0.83

All frames were registered in every location. Reprojection error was 1.17–1.75 px.
Measured VRAM peaks: SIFT +1.3 GB, matching +1.6 GB, texturing +0.5 GB, Blender +5.4 GB, and splatting with 3 M Gaussians at 1920 px +25.2 GB.
The sm_120 OpenMVS build densified the synthetic smoke test in 17 s. The CPU fallback took about 40 min.
Metadata analysis showed shutter speeds of 1/50–1/100 s (about 14 px of motion blur at 4K) and automatic ISO and white balance. Exposure compensation in gsplat (ppisp) gave no gain (PSNR 20.5 vs 20.6). The quality limit is in the capture, not the solver.

Limitations & next steps

PSNR in the low 20s reflects motion blur and auto-exposure. The capture protocol now requires a 1/250–1/500 s shutter, locked exposure and white balance, and a dedicated floor pass.
Merging videos from different locations caused false matches and an incoherent model, so each location is processed separately.
Dark glossy tiled floors reconstruct poorly. Manhattan alignment is not yet reliable with furniture and glazing.
CPU stages are the bottleneck on a 4-vCPU virtual machine.
Planned: comparison with a commercial photogrammetry suite on the same frames, and retesting mesh refinement with the sm_120 build.

Stack

Python 3.11 · OpenCV · Open3D · NumPy · COLMAP 4.2.1 · OpenMVS 2.4 · gsplat 1.6.0 · PyTorch 2.14 (CUDA 13) · Blender 5.2 · FFmpeg · PotreeConverter · glTF-Transform · splat-transform

Points of Interest and Urban Flows in Fort-de-France City Centre from Open Data

Thu, 01 Oct 2026 00:00:00 +0000

Context

Fort-de-France (Martinique) has almost no local open data. The city’s own datasets were last updated in 2019, and there are no public pedestrian counters, open transit ridership figures or open road counts at city-centre scale. The project asks how far national open data and continuously archived feeds can go in describing the city centre as a system of points of interest (shops, administrations, schools, health, culture, entry gates) and flows (residents, commuters, pupils, visitors, cruise passengers, vehicles, buses) over a ten-year horizon (2016–2026).

No single source answers that question. The business register knows where establishments are but not who visits them. The census knows where people sleep but not where they spend the day. Transit feeds show the buses but not their passengers, and cruise schedules show the ships but not the people ashore. The approach is therefore to cross many heterogeneous sources on a common spatial and temporal frame, so that each one fills a blind spot of the others, and to check the result against the few measured signals that do exist.

Approach

The method comes from an earlier knowledge-base project of mine. Raw data is immutable and addressed by content. Every database is disposable and rebuilt from it. Every value keeps its provenance, and a set of invariants is checked on each rebuild.

flowchart TB
 A[Source catalogue
sources.yaml] --> B[Immutable raw store
sha256 + JSONL manifest]
 C[Scheduled collectors
Meta every 14 days · GTFS-RT 60 s] --> B
 B --> D[Versioned extractors
normalised Parquet]
 D --> E[DuckDB spatial warehouse
rebuilt, never edited]
 E --> F[QA invariants I1–I6]
 E --> G[Statistics
requetes/*.sql]
 E --> H[Desktop application
map + time selector]
 G --> I[Calibration against
measured signals]

Methods

Acquisition. Each download is hashed (SHA-256) and logged in a JSONL manifest, and acquisition is idempotent. National files larger than 1 GB (Sirene business register) are read as remote HTTP-range extracts filtered to the municipality. The filtering query is recorded as part of the provenance.
Ephemeral sources. Meta Movement Distribution keeps only 90 days online, so a scheduled task archives it every 14 days. A bus collector polls GTFS-RT vehicle positions every 60 s and predictions every 10 min, and zips each day with a SHA-256 checksum.
Spatial frame. Data is stored in WGS84 and computed in EPSG:5490 on the INSEE 200 m grid. Points of interest are classified by one typology (NAF codes, OSM tags, stops), with family, weight and an hourly rhythm over seven time slots.
Models. Hourly presence per grid cell combines residents (Filosofi 2021, distributed dasymetrically to buildings), employees, pupils and tourists. A Huff model estimates catchment. Cruise passengers onshore are modelled from ship schedules and capacities. An island-wide gravity model with BPR assignment covers visitors and through-traffic.
Quality. Six invariants are checked, from provenance to the absence of personal data. Unconfirmed parameters go in a decision log.

Data sources

Crossing the sources is the core of the project. Business registers (Sirene, BODACC, URSSAF) give the stock and turnover of activities, but say nothing about the people around them. The census and tax data (RP, Filosofi) place residents, their income and housing vacancy, but only at night. Commuting files and the national travel survey (MOBPRO, MOBSCO, EMP) say who travels, why and when. Transit, accident and air-quality feeds (GTFS-RT, BAAC, Madininair) are the only measured traces of movement. Cruise schedules and ship capacities bring a population that no census counts. The geographic frames (IGN, OpenStreetMap, the INSEE 200 m grid) are what make all of these comparable cell by cell.

The catalogue lists 120 sources: 51 integrated, 32 to integrate, 7 to verify, 13 available on request only, 5 paid, 1 by partnership and 11 not retained. The tables below list every integrated source and summarise the others by theme.

Businesses, jobs & amenities

Source	What it contributes	Period / frequency	Status
Sirene business register — INSEE	Establishments, activity code (NAF), creation and closure dates, headcount bands	Full history; stock rebuilt at any date 2016–2026	Integrated
BODACC — DILA	Business turnover and failures (sales of goodwill, insolvency proceedings)	2016–2026, daily	Integrated
Permanent amenities database (BPE), geolocated — INSEE	Shops, services, health, administrations, education, sport, tourism	2024 and 2025 editions, annual	Integrated
Establishments and salaried headcount by municipality × activity — URSSAF / Acoss	Private salaried employment by activity	2006–2025, annual	Integrated
Education directory and pupil numbers — MENJ / DEPP	Schools, location, pupils per school year, staff	Pupil numbers 2016–2025	Integrated
FINESS — DREES / ARS	Health and social facilities, pharmacies	Extract of May 2026, bimonthly	Integrated
Overture Maps places — Overture Maps Foundation	Points of interest with a confidence score (comparison layer)	Frozen extract of release 2026-09-23, monthly	Integrated
OpenStreetMap (Overpass) — OpenStreetMap contributors	Points of interest, shops, streets, pedestrian paths	Current state	Integrated
Historic monuments (Mérimée / POP) — Ministry of Culture	Heritage attractors (cathedral, Schœlcher library, Fort Saint-Louis)	Current	Integrated
23 more — e.g. Flores localised employment (INSEE), Foursquare Open Source Places, LOCOMVAC vacant commercial premises (DGFiP), Google Places			Catalogued

Population, income & housing

Source	What it contributes	Period / frequency	Status
Census, IRIS-level files (RP) — INSEE	Residents, age, occupations, dwellings (vacancy, car ownership), workforce	RP 2016 → RP 2022, annual	Integrated
Filosofi by IRIS — INSEE	Disposable and declared income, poverty	2016–2021, annual	Integrated
Filosofi 200 m grid files — INSEE	Tax households, individuals, poverty, social housing and age per 200 m cell	2015, 2017, 2019, 2021	Integrated
Filosofi 200 m grid, Géoplateforme WFS layer — INSEE	Grid cell geometries	2015 layer	Integrated
IRCOM income tax by municipality — DGFiP	Tax households, declared income	2016–2024, annual	Integrated
Cafdata — CNAF	Recipients of minimum income, housing and family benefits	2020–2024 (2022 missing), annual	Integrated
Priority neighbourhoods (QPV) — ANCT / INSEE	Priority urban policy areas, which include the city centre	2024 boundaries	Integrated
LOVAC — Cerema / DGALN	Short-term and structural (≥ 2 years) vacancy of private housing	2020–2026 editions, annual	Integrated
RPLS social housing — SDES	Social dwellings (address, surface, rent, vacancy)	2023 and 2025	Integrated
Sitadel building permits — SDES	Dwellings and premises authorised, started, demolished	2013–2026, monthly	Integrated
Geolocated DVF — DGFiP / Etalab	Property transactions (price, surface, commercial premises)	2021–2025, rolling 5 years	Integrated
8 more — e.g. census detail files and legal populations (INSEE), DV3F and land files (Cerema / DGFiP), WorldPop / GHSL gridded population			Catalogued

Mobility & transport

Source	What it contributes	Period / frequency	Status
Commuting flows (MOBPRO) — INSEE	Home → work flows between municipalities	2016–2022, annual	Integrated
School commuting (MOBSCO) — INSEE	Home → study flows between municipalities	2016–2022, annual	Integrated
National travel survey 2018–2019 (EMP) — SDES	Trip purposes, departure and return times, activity durations, distances	2018–2019	Integrated
GTFS, urban network Centre zone (Mozaïk, BRT lines) — Martinique Transport	Scheduled transit supply	Archives since June 2023	Integrated
GTFS-RT, continuous collection — Martinique Transport	Observed bus speeds, buses in service, declared occupancy	Since 7 October 2026; 60 s (positions), 10 min (predictions)	Integrated
GTFS, ferry shuttles — Martinique Transport	Shuttles landing on the waterfront	Versions since August 2024	Integrated
Road injury accidents (BAAC) — ONISR	Geolocated accidents (pedestrians, vehicles), with time	2016–2024 loaded	Integrated
Movement Distribution — Meta Data for Good / HDX	Share of users at home, within 10 km, 10–100 km, beyond 100 km	Daily; 90-day window, archived every 14 days	Integrated
Movement Range Maps — Meta Data for Good / HDX	Change in movement, share of users staying put	March 2020 → May 2022, daily (discontinued)	Integrated
BD TOPO main road network — IGN	Island road graph for traffic assignment	Current	Integrated
BD TOPO road sections — IGN	Carriageway width, lanes and direction, used to measure kerbside parking	Current	Integrated
OpenStreetMap car parks — OpenStreetMap contributors	Car parks, type, declared capacity	Current	Integrated
18 more — e.g. ridership and ticket validations (Martinique Transport, on request), road counts (on request), Orange Flux Vision mobile data (paid), Waze for Cities / Strava Metro (partnership), a pedestrian counting campaign (to be created)			Catalogued

Tourism & cruise

Source	What it contributes	Period / frequency	Status
Cruise call schedule, port of Fort-de-France — CruiseMapper	Ship-by-ship calls with arrival and departure times	March 2026 → March 2029; archived monthly, as past months disappear	Integrated
Ship pages — CruiseMapper	Passenger capacity, crew, tonnage of ships in port	Current	Integrated
Cruise season reviews — Martinique Tourism Authority (CMT), via the press	Passengers and calls per season, cruise terminals	Seasons 2024–25 and 2025–26	Integrated
PortWatch — IMF / HDX	Daily port calls from AIS	2019–2026, daily	Integrated
Monthly commercial air traffic — DGAC	Arrivals at Martinique Aimé Césaire airport (tourist seasonality)	1990–2025, monthly	Integrated
Classified tourist accommodation — Atout France	Classified hotels and residences, rooms and capacity	Current stock	Integrated
Tourist accommodation capacity — INSEE	Hotels by rating, campsites, residences, holiday villages	2015–2026, annual	Integrated
7 more — e.g. DATAtourisme (ADN Tourisme), INSEE hotel occupancy survey, Inside Airbnb / AirDNA, port authority call data			Catalogued

Context: weather, air, calendar, safety

Source	What it contributes	Period / frequency	Status
Climatological data — Météo-France	Rain, temperature, wind (Fort-de-France Desaix and Le Lamentin stations)	2010–2026, hourly / daily	Integrated
Air quality — Madininair	NO₂ as a traffic proxy, plus PM10, PM2.5, O₃ and H₂S from sargassum	Hourly; monthly NO₂ 2016–2026	Integrated
Recorded crime by municipality — SSMSI	Recorded offences	2016–2025, annual	Integrated
Public holidays in Martinique — DINUM	Calendar, including 22 May	2006–2031	Integrated
School calendar — MENJ	School holidays, including Carnival	2017–18 → 2026–27	Integrated
Pageviews — Wikimedia Foundation	Online interest in Fort-de-France, Carnival and heritage	2016–2026, monthly	Integrated
4 more — e.g. NASA Black Marble night lights, Google Trends / GDELT, an events agenda (to be compiled)			Catalogued

Geography & base maps

Source	What it contributes	Period / frequency	Status
Study perimeters — project	City centre and extended centre (provisional, see D1)	Version 0.1	Integrated
Administrative boundaries API (geo.api.gouv.fr) — DINUM / IGN / INSEE	Municipal boundary and legal population	Current	Integrated
IRIS boundaries — IGN / INSEE	Sub-municipal census units	Current layer	Integrated
BD TOPO — IGN	Buildings (height, use), roads, facilities	Quarterly editions	Integrated
BD TOPO residential buildings, extended centre — IGN	Use, height and footprint used to place households in buildings	Current edition	Integrated
OpenStreetMap vector base map — OpenStreetMap contributors	Streets, water, parks, coastline drawn in the application	Current	Integrated
9 more — e.g. Etalab cadastre, National Address Base, BD ORTHO aerial imagery (IGN), local urban plan (Géoportail de l’Urbanisme), natural risk maps (Géorisques)			Catalogued

Results

The project started in October 2026 and is at an early, uncalibrated stage (application v0.6.0). All city-centre figures depend on a provisional perimeter (see below). Each result below states which sources it crosses: none of them could be produced from a single dataset.

Catalogue. 120 sources are catalogued, of which 51 are integrated.
Warehouse (first build). 117,161 entities, about 25,000 measures and 1,278 indicators, with all 6 invariants passing.
Retail decline vs residents (measured, registers × census × tax data). Active personal-goods establishments (Sirene) fell from 498 (2016) to 363 (2026), a drop of 27 %, and the city centre’s share of the municipality in this sector fell from 75.5 % to 64.7 %. Over the same period, population in the city-centre IRIS (census) fell from 13,444 (2016) to 12,898 (2022), and housing vacancy rose from 24.9 % to 27.6 %. 47 % of city-centre households are below the poverty line, against 30 % for the municipality (Filosofi 2021). Read together, the three sources show a centre losing shops, residents and purchasing power at the same time.
Day/night rhythm (modelled, six sources crossed). About 3,700 people are present at night and 11,200 in the morning, a ratio of 3.0. Night-time residents come from the Filosofi 200 m grid placed in BD TOPO buildings; daytime workers from Sirene headcount bands; pupils from the education directory; tourists from classified accommodation capacity. The morning slot is adjusted on the hourly NO₂ profile of traffic stations.
Observed bus speed vs model (GTFS-RT × road graph × travel surveys). Over one full day, buses ran at 12.7 km/h at 05:00, falling to 7–8 km/h during the day and 10.6 km/h at 19:00. The assignment model, built from BD TOPO roads, MOBPRO commuting, EMP trip profiles and BPE amenities, predicts a 20 % slowdown at peak hours, whereas the buses show 45 %. The measured feed exposes what the model misses.
Cruise check (schedules × ship capacities × reported counts). Onshore passengers are modelled from the CruiseMapper call schedule and ship capacities. On the one documented cruise day (24 April 2026), the estimates came within about 1–2 % of the passenger counts reported in the press (3,075 vs 3,112, and 1,995 vs 1,969). The fill rate used in the model was set on that same day, so this is a consistency check, not an independent validation.

Limitations & next steps

D1: the city-centre perimeter is provisional. It is a rectangle around the historic core plus a 1 km extended zone, used until an official perimeter is provided. Every city-centre figure depends on it.
No pedestrian, ridership or road-count data is open. Hourly rhythms, attractiveness weights and parking assumptions are still uncalibrated. The measured controls so far are hourly NO₂ at traffic stations and observed bus speeds.
The 2016–2026 business stock is rebuilt from creation and closure dates only.
Next steps: build up several months of GTFS-RT data for measured congestion, request survey and counting data, and calibrate presence on field counts.

Stack

Python · DuckDB (spatial, httpfs) · Parquet · SQL · GTFS / GTFS-RT · pytest · ruff · Electron · React · TypeScript · ECharts · Leaflet · GitHub Actions

Codex — High-Density Physical Archiving of Media on Paper

Tue, 01 Sep 2026 00:00:00 +0000

Context

Consumer digital media decay on the scale of years. Unpowered SSDs lose charge, optical discs delaminate, formats go out of use and accounts get closed. Paper can last for centuries, needs no electricity and can be checked by eye. The question is how much information an ordinary A4 sheet can hold if it must still be read back reliably with a standard flatbed scanner and no special hardware.

The problem is hard because the print-and-scan channel is noisy in several ways at once. Toner spreads and clumps, density varies across the page, and scanners add rotation, anisotropic scaling, uneven paper feed, blur, vignetting and gamma. Paper also stretches non-linearly: at the target density, 0.2 % of stretch shifts the grid by about three cells. Common 2D codes such as QR and DataMatrix are designed for robustness rather than density. A long-term archive also has to remain decodable when today’s software no longer exists.

Approach

flowchart TB
 A["Source media: photo, video, voice, text"] --> B["Format-conservative reduction
JPEG baseline · MJPEG/PCM AVI · PCM WAV · UTF-8"]
 B --> C["Archive: tar + zlib + SHA-256"]
 C --> D["Inter-page erasure code
Cauchy MDS over GF(256): any K of N pages"]
 D --> E["Per-block Reed–Solomon + CRC-32
page-wide interleaving, triplicated header"]
 E --> F["Multi-level cell rendering
1-bit A4 PDF at 1200 dpi"]
 F --> G["Print → scan
(simulated channel)"]
 G --> H["Fiducials → homography → local alignment markers"]
 H --> I["Grey-level calibration from the page itself"]
 I --> J["RS/CRC decode → erasure decode → SHA-256 check"]
 J --> K["Byte-exact files, or partial recovery"]
 F -.-> L["Printed appendix: format spec + decoder source in OCR-B with per-line CRC-16"]
 L -.-> J

Left: page 1 of 10 of the photo archive, as sent to the printer (A4, 1200 dpi, a grid of 1,145 × 1,576 cells), downscaled. Right: the boxed top-left corner at printer resolution (about 10 × 9 mm, each printer dot drawn as 2 × 2 pixels): the solid corner fiducial, the start of the header band, and the first data cells. Each cell is 8 × 8 dots (0.169 mm) and its number of inked dots encodes one of 4 levels. The square at the corner of the grid is an alignment marker.

A page of the printed appendix (page 11 of 30): the start of cdx_read.py (360 lines), set in OCR-B. Each line carries its number and its CRC-16, so a line misread by OCR can be found and checked by hand.

Methods

Multi-level cells instead of binary modules. Each cell carries 2 bits. Cells are drawn as clustered dots at the printer’s native resolution, so the driver does no halftoning and produces no moiré.
Geometry recovery in stages. Four corner fiducials, one of them hollow to fix orientation, give a global homography. Alignment markers spread across the page then absorb local, non-linear paper stretch.
Self-calibrating reader. The decoder estimates the printed grey levels and spatial shading from calibration structures on each page, so it does not depend on the printer’s dot settings.
Two-level error control, a product code. Reed–Solomon and CRC-32 inside each page correct random and burst errors such as stains and scratches. A Cauchy MDS erasure code across pages, decoded block by block, lets any K of N pages rebuild an archive.
Graceful failure. If the SHA-256 check fails, the decoder still extracts every complete file that comes before the damaged region.
Self-describing archive. A printed appendix contains the format specification, a reading procedure and the full decoder source in OCR-B, with a line number and CRC-16 on each line. The decoder needs only NumPy, Pillow and reedsolo. Text uses DEFLATE (zlib), which can be reimplemented in a few hundred lines, rather than the stronger LZMA.
Earlier prototype (codex_proto/). This version explored several physical media: paper, laser-marked stainless steel, intaglio plates and microfilm frames. It used 2- to 16-level cells, fountain-style parity pages, MLP-based grey-level readers, a printed optics calculator for reading microfilm, and lossy media reduction (parametric voice coding, wavelet-residual photo coding).

Results

All figures below come from simulation only. The software models laser defects (spot blur, toner grain, dot gain, banding, non-uniform density, textured paper), scanner defects (±1.2° rotation, ±0.3 % x/y scale, feed wobble, blur, vignetting, noise, gamma 2.2), coffee stains, scratches, and one deliberately lost page per archive. No physical print-and-scan test has been run yet.

Net payload: 383,469 bytes per A4 page, about 59 bits/mm².
End-to-end: 46 of 50 pages were scanned and all 4 of 4 archives were restored byte-identical (photo, video, voice, text). No Reed–Solomon codeword failed, and no inter-page block was lost.
Error rates: clean pages needed about 0.007–0.008 % of bytes corrected. Pages with stains and/or scratches needed at most 0.90 %.
Archive sizes: photo 2.86 MB → 10 pages (8 data + 2 parity); video (800×600, 24 s, MJPEG + 16 kHz PCM) 8.30 MB → 25 pages; voice 0.91 MB → 5 pages; text 5.58 MB → 10 pages.
Text: the public-domain Mémoires d’outre-tombe (5.43 M characters) would take 1,357 typeset A4 pages. Encoded, it takes 6 data pages, a ratio of about 226. zlib compresses this prose by 2.62 and LZMA by 3.58.
Partial recovery: an archive truncated to a quarter of its length still returned 1.42 MB of the 5.58 MB of text.
Decoder robustness: the decoder also worked on scans rotated 90° and 180°, on 600 dpi scans and on JPEG (q92) scans.
Decoder size: 731 lines of Python for the printed decoder modules.

The same region of the photo page (380 × 300 dots, about 8 × 6.4 mm), enlarged. Left: as encoded. Right: after the simulated print-and-scan channel, resampled onto the page grid through the fiducial homography. Toner spread, blur and noise soften the cells, and the edge of a simulated coffee stain darkens the left part. The reader calibrates the grey levels locally, so it still separates the four levels. In the simulation run reported above, stained or scratched pages needed at most 0.90 % of bytes corrected.

The video archive: an underwater clip (school of fish over a reef) used as test footage. Top: the original 3840 × 2880 HEVC frames at 3.2 s, 12.8 s and 16.0 s, downscaled. Bottom: the same instants decoded from the simulated scans of the 25 pages (one deliberately lost). The paper channel itself loses nothing: the restored AVI is byte-identical to the archived one. What differs from the original is the reduction made before printing (800 × 600 MJPEG, 12.5 frames/s, 16 kHz mono sound) so that 24 s of video fit in 8.30 MB.

Close-up of the 12.8 s frame. Left: a 1152 × 864-pixel area of the original, downscaled. Right: the matching 240 × 180-pixel area of the restored frame, enlarged with no smoothing. The fish and the reef stay recognisable. Fine texture and colour detail are lost, and MJPEG block artefacts appear. This is the price of fitting the clip into 25 pages.

Limitations & next steps

No physical test yet. The cell dot patterns were tuned in simulation. The next step is a real 1200 dpi print with a 1200 dpi scan to set them and measure the actual error margin.
Small test set. The simulation covers one run over four archives. A sweep across defect severity, scan resolution and paper stock is still needed.
Unmeasured physical properties. Long-term ageing of paper and toner, and how practical it is to OCR the decoder back from paper, have not been measured.
Font dependency. The appendix body text relies on a system font.

Stack

Python · NumPy · SciPy · Pillow · reedsolo · ReportLab · soundfile · ffmpeg · Tesseract (OCR-B)

Explainable Processing Atoms for Documents & Data

Tue, 01 Sep 2026 00:00:00 +0000

Context

Production systems for document reconciliation and OCR, built for clients such as a wholesale distributor and an accounting firm, left several codebases that each solved the same sub-problems (page routing, OCR tuning, LLM extraction, field comparison) with their own bugs. The atoms library consolidates them into one package. Its central question is the one behind the original systems: can a document check be automated so that a human can audit and contest every verdict? Extracting a field is not enough. The system must state whether two documents agree, why, and from what evidence.

Approach

Everything is built from atoms. An atom has one responsibility, typed inputs and outputs, no global state, and injected dependencies (LLM client, OCR engine, storage). A pipeline is a composition of atoms, either written in code or declared as a YAML recipe that is validated when it loads, before anything runs.

flowchart TB
 A["Input documents (PDF, scans, CSV, mail)"] --> B{"Page routing: native text or OCR?"}
 B -- "native" --> C["Positioned native text"]
 B -- "scan" --> D["PaddleOCR (production settings)"]
 C --> E["Segmentation & classification"]
 D --> E
 E --> F["LLM JSON extraction + verbatim grounding check"]
 F --> G["Cross-document reconciliation (invoice / delivery note / order)"]
 G --> H["Tri-state verdict + named reason + trace"]
 H --> I["Report / human review"]

The catalogue

The 459 atoms are grouped into 32 families, one Python package each, ordered along the processing chain: capture the world, understand it, reconcile and decide, deliver, then operate in production. Two further packages assemble them: steps (77 named, parameterised steps) and recipes (the YAML loader and validator).

Stage	Family	Atoms	Role
Base	`core`	25	pivot types, pipeline, configuration, resilience (retry, rate limits), files, concurrency
Capture	`connect`	40	resilient HTTP, mailboxes (Graph, IMAP), SFTP, OneDrive, read-only ERP source, open data, audit trail
	`google`	24	Gmail, Sheets, Drive, Apps Script, with resilient execution
	`web`	17	driver-agnostic web robots (Selenium, Playwright): waits, resumable skeleton, forms
	`pdf`	9	positioned native text, OCR-or-not decision, rendering, page handling, duplicates
	`ocr`	9	PaddleOCR with production settings, EasyOCR
	`layout`	3	page geometry: lines, columns, deskew, monospace rendering
	`audio`	16	chunked transcription, segment post-processing, microphone ↔ screen alignment
	`vision`	32	video / RTSP streams, detection, tracking, depth, segmentation, calibration
	`vlm`	10	screen reading with vision-language models: change detection, label location
	`geo`	11	geodesic measures, combined GPS fixes, line-distance filtering, spatial SQL
	`calendar`	4	public holidays, working days and opening hours
Understand	`text`	32	numbers and amounts, dates, normalisation, pattern-based fields, identifiers
	`segment`	7	pages → logical documents, typed document numbers, page nature
	`classify`	12	file name, sender, weighted patterns, TF-IDF, visual and LLM classifiers, cascades
	`llm`	24	provider-agnostic contract, providers, pool, prompts, structured extraction
	`anchor`	5	fields and tables located relative to printed labels, native text or OCR
	`kb`	8	traced knowledge base: validated event log folded into a SQLite index
Reconcile	`match`	25	EAN/GTIN, indexes, multi-criteria scores, optimal assignment, entity resolution
	`compare`	9	tri-state, explainable comparators for fields, records and sets
	`check`	9	generic tri-state checks: totals, VAT, balance, completeness, plausibility
	`reconcile`	12	set pairing, order → delivery → invoice chains, N-way, ledger and bank matching
	`validate`	14	rules engine, identifiers, amount consistency, corrected LLM verdicts, final decision
	`rules`	8	safe condition language, explainable first-match tables, escalation, state machines
	`events`	11	timestamped events: windows, expected-event watch, sessions, status transitions
	`analytics`	18	supply-chain analytics without pandas: demand, stock-outs, service level, safety stock
Deliver	`export`	8	CSV for French Excel, shared Excel workbooks, HTML / Markdown / PDF reports
	`report`	6	persistable discrepancies, summaries, review queue
	`render`	12	HTML deliverables: pre-rendered Mermaid, ECharts charts, GeoJSON maps
	`diagram`	6	process diagrams: draw.io / BPMN JSON in, Mermaid and BPMN XML out
Operate	`ops`	26	high availability (heartbeats, leader election), scheduled daemon, monitoring, pre-flight checks
	`qa`	7	editorial non-regression (golden files), simulation replay, inter-annotator agreement

Named recipes

Each of the 17 recipes is a YAML list of steps, named <family>.<atom>. Three examples:

invoice_multi_doc: a batch of multi-document PDFs becomes checked logical documents. io.collect → pdf.read → segment.documents → classify.documents → llm.extract → validate.ids → validate.amounts → validate.decide → export.csv → google.sheets.upsert → connect.notify. Each document ends as valid, to check, or invalid.
doc_reconciliation: a file of invoices, credit notes and delivery notes, checked against orders. Guarded LLM extraction (llm.extract_docs) is cross-checked with pattern extraction (extract.regex, compare.extractions, reconcile.fields), then check.document and check.completeness, lines against the order reference (compare.reference), the order → delivery → invoice chain (reconcile.nway), and a discrepancy file plus review queue (report.discrepancies, report.review). 17 steps.
meeting_transcript: a meeting recording becomes a timestamped transcript with speakers. audio.decode → audio.vad → audio.transcribe → audio.diarize → audio.speakers → audio.segments → export.csv → export.transcript → connect.notify.

The others include delivery_vs_order, ledger_reconciliation, invoice_dedup, article_base_cleaning, stock_policy, mail_to_sheet, draft_mails, pdf_text_extract, rubric_eval, screen_session, camera_flow and process_from_transcript.

The Atlas

The Atlas is a single offline HTML page generated from the code itself (syntax trees and YAML), with no hand-written data. It draws the catalogue as a periodic table: each family is an element with a symbol, each module an atom, each recipe a molecule. It also flags atoms that lack documentation, tests or a usage example. The interface is in French.

The “Capture” series of the periodic table. Each box is a family, with its symbol, number of modules and size in thousands of lines; each tile below it is an atom.

The “Reconcile” series: the families that pair, compare, check and decide.

An atom’s page (compare.fields): what it does, taken from its docstring, then each public name with its full signature.

The invoice_multi_doc recipe as a molecule: 11 steps, the context keys that link them, and the services they need (dashed when optional).

Methods

Tri-state results. Every comparison returns ok ∈ {True, False, None} (conforms / discrepancy / inconclusive) with a named reason code and a trace. A missing or unreadable value counts as inconclusive, never as an anomaly. Inconclusive results are still counted, not silently dropped.
Hybrid native-text/OCR routing. Each PDF page is classified as native, OCR or empty, based on how much native text it holds and how much of it is covered by images. OCR runs only where it is needed.
Grounded LLM extraction. Structured JSON extraction is validated against Pydantic schemas. Each extracted value is then checked verbatim against the source text, so altered or invented values are flagged.
Cross-document reconciliation. Invoices, delivery notes and orders are compared line by line using named tolerances and dominance rules. For example, a matching line amount can override a price difference.
Provenance. Every ported atom carries a Provenance: header and an entry tracing it to repo@commit:path:lines in the original production code. Bugs found while porting were fixed and each fix is covered by a non-regression test.
Enforced architecture. Layering is checked by 30 import contracts: an atom imports only the core package. Strict typing, a ≥ 80 % coverage gate and a check that no client names appear in source code are also enforced.
Device abstraction. Vision and audio atoms target cuda, mps or cpu via one setting.
End-to-end evaluation. Tests run with real engines (PaddleOCR, Ollama qwen3:8b and qwen3-vl:8b) on data with known ground truth. Each metric is recorded against a threshold in a JSON/HTML report. Metrics that miss their threshold are kept and shown as failing.

Results

Codebase (current tree): 459 atom modules in 32 families, 77 parameterised steps, 17 YAML recipes, 30 import contracts. The suite collects 2,959 unit and recipe tests, plus 109 end-to-end tests that run separately. All 17 recipes have at least one end-to-end test, and 34 of the 35 Atlas elements are exercised end to end.

Selected end-to-end scores (synthetic documents with known ground truth, local 8B models):

Task	Engine	Measured
Invoice + delivery note vs. order reconciliation	`qwen3:8b`	document types and fields 14/14; expected discrepancies 5/5; 0 false discrepancies on the compliant case
LLM JSON extraction (4 native invoices)	`qwen3:8b`	valid JSON 4/4; exact fields 28/28; all deliberately altered values caught by grounding
OCR on a photographed label	PaddleOCR	word recall 0.94 on the raw photo; a contrast-enhancement defect (0.65–0.71) was found and fixed (0.88–0.94)
Table reconstruction from OCR	PaddleOCR	100 % exact cells (proportional font), 95 % (monospace)

Limitations & next steps

Verbatim grounding cannot catch a value that exists in the document but was put in the wrong field: an absent field was wrongly filled in 1 of 8 cases.
The 8B model judged a +50 % quantity gap a “minor” discrepancy, against the instruction. LLM verdicts are therefore normalised and kept downstream of deterministic rules.
Scores come mostly from synthetic data. A real-document test set is being built outside the repository, and only cases that have been reviewed are scored.
Next steps: measure the automatic-reconciliation rate and the false-positive rate on production data.

Stack

Python · Pydantic · PyYAML · PaddleOCR · Ollama (qwen3:8b, qwen3-vl:8b) · Mistral / Anthropic / OpenAI APIs · pytest · mypy · import-linter · ruff

Mission Knowledge Graphs: From Consulting Evidence to an Ontology Queried Through MCP

Tue, 01 Sep 2026 00:00:00 +0000

Context

Process-observation consulting produces flowcharts, interview notes, screen recordings and recommendations. Once a mission is delivered, that material is effectively dead: the next mission starts from a blank page, even when the answer was already found elsewhere. The research question is how to turn deliverables from documents into typed, sourced, queryable data, so that an assistant can answer “where have we seen this before, and what did we conclude?” while citing verifiable evidence.

Two requirements structure the work: traceability (every assertion points to its proof) and a strict separation between evidence and knowledge (what was observed is not what was generalised). Retrieval-augmented generation is fast to build but cites passages, not checkable assertions; a formal ontology is traceable but expensive. This project explores a practical path between the two.

Approach

flowchart TB
 A["`Raw mission sources
immutable, hashed`"] --> B["`Append-only event log
source of truth`"]
 B --> C["`Rebuildable SQLite base
disposable`"]
 C --> D[Deterministic extraction]
 C --> E["`LLM extraction
proposer only`"]
 D --> F["`Proposed nodes and edges
with provenance`"]
 E --> F
 F --> G{"`Human review
outside MCP`"}
 G -->|validated| H[Knowledge graph]
 H --> I[MCP: evidence]
 H --> J[MCP: knowledge]
 H --> K[MCP: ontology]
 I --> L[AI assistant]
 J --> L
 K --> L

The core design choice is to split the interface into three MCP servers by epistemic status: evidence (attested, sourced observations), knowledge (generalisations), and ontology (the meta-model itself). The split prevents an assistant from presenting a generalisation as an observation.

flowchart TB
 subgraph SCHEMA["SCHEMA · naming"]
 ON["`ontology server
How do we name it?`"]
 end
 subgraph PROVEN["PROVEN · observed"]
 EV["`evidence server
What did we capture, and where?`"]
 end
 subgraph GENERAL["GENERALISED · concluded"]
 KN["`knowledge server
What do we conclude?
proposes, never validates`"]
 end
 ON -. "types" .-> EV
 EV == "grounds every conclusion" ==> KN
 EV --> AI["`AI assistant
answers labelled by status`"]
 KN --> AI
 ON --> AI

The graph is single; only the façades are split. The boundary that matters runs between evidence and knowledge: the evidence server only returns what was observed and where, the knowledge server returns conclusions that must chain back to that evidence, and no MCP tool can mark anything as validated.

Methods

Meta-model. A fixed core plus mission profiles and a versioned ontology (v0). Each node carries an origin (structural, deterministic, LLM-extracted) and a status (proposed → validated).
Invariants (I1–I6). Provenance is mandatory on every derived node and edge; a recommendation must chain back to a sourced finding; LLM-extracted quotes must be literal substrings of their fragment; “validated” requires a recorded human decision. Raw sources are immutable; the database is disposable and rebuilt from the log.
Competency questions before implementation. A v0 set of 33 questions, typed as factual, aggregate, traceability, contradiction, temporal, cross-mission and negative (requests the system must refuse, e.g. cross-client leakage), in an executable format with expected answers and references.
Evaluation protocol. At least 30 natural-language tasks, scored automatically or by a judge. Gate thresholds were fixed up front: accuracy ≥ 0.8, valid citations ≥ 0.98, grounding ≥ 0.95, leakage = 0, median ≤ 6 tool calls per task; each question solvable in ≤ 3 calls; deterministic extraction precision ≥ 0.95 per type; inter-annotator κ ≥ 0.7.
Phases P0–P6 with human gates. Implementation is delegated to an agent session that must stop at each gate and log open decisions.
Black-box software mapping. The same method was applied to a commercial business-management software, without reading its source code. The target is a triplet of user, software and database truths linked by proven edges. Eleven heuristics (H1–H11) cover value matching, inverse and direct perturbation, calculated fields, conditional visibility, scenario diffs, invariant mining, input fuzzing, semantic priors (never treated as proof), documentation-to-test and symptom-to-rule. Inputs: the menu tree via AT-SPI, a screen survey, column profiling, and an event typology from the application audit log. A hypothesis is accepted only with independent counter-evidence; vendor-produced maps are held in quarantine as a scoring oracle.

An illustrative excerpt of the typed graph (labels are fictitious): the recommendation is only admissible because the chain down to a literal fragment of a source is complete.

flowchart TB
 R["`Recommendation
single entry point`"] -- responds to --> C["`Finding
double data entry at goods receipt`"]
 O["`Observation
“stock is re-keyed by hand”`"] -- attests --> C
 O -- extracted from --> F["`Fragment
interview notes, lines 40–52`"]
 F -- cites --> P["`Source file
hashed, immutable`"]
 C -- concerns --> A["`Activity
goods receipt`"]
 A -- uses --> S["`Screen
receipt form`"]
 S -- module of --> Y["`System
business software`"]

Status & results

This is mainly a design with partial implementation, not a system in service.

Pilot (observation corpus, deterministic pass). 6,195 files (6.92 GB) acquired idempotently; 13,272 nodes and 6,022 edges extracted in 2.1 s at zero LLM cost; all six invariants green over 57,969 checks.
Gold-set measurement. 145 annotation units plus 100 entity-resolution pairs. Human annotation was partial and the second annotator was an LLM. Human × LLM agreement: κ = 0.72 (software), 0.41 (screen), 0.31 (category), 0.52 (segment boundaries). Extractor precision against human reference: 0.74, 0.43, 0.04, 0.63. Gate G2 (≥ 0.95) not passed; the category result reflects a definitional mismatch, not a computation error.
Software mapping. Schema of 427 tables with 0 declared foreign keys, triggers or views (all logic is applicative); 1,048 menu entries (899 leaves) mapped; 154 of 899 screens surveyed (17 %); 2,845 rule hypotheses reduced to 1,519 by a local LLM filter, with 1 decision error in a 50-item sample reviewed by the supervising agent. A read-only MCP prototype exists; no rule has yet been proven in the software. Work paused after the host VM was overloaded.

Gold-set measurement against the gates fixed in advance: bars are the measured values per field, the orange line is the gate. Only one bar clears its gate.

---
config:
 xyChart:
 width: 540
 height: 320
 themeVariables:
 xyChart:
 backgroundColor: "transparent"
 plotColorPalette: "#3F86B8, #C46A2B"
---
xychart-beta
 title "Human × LLM agreement κ (gate 0.7)"
 x-axis ["software", "screen", "category", "segments"]
 y-axis "κ" 0 --> 1
 bar [0.72, 0.41, 0.31, 0.52]
 line [0.7, 0.7, 0.7, 0.7]

---
config:
 xyChart:
 width: 540
 height: 320
 themeVariables:
 xyChart:
 backgroundColor: "transparent"
 plotColorPalette: "#3F86B8, #C46A2B"
---
xychart-beta
 title "Extractor precision vs human reference (gate 0.95)"
 x-axis ["software", "screen", "category", "segments"]
 y-axis "Precision" 0 --> 1
 bar [0.74, 0.43, 0.04, 0.63]
 line [0.95, 0.95, 0.95, 0.95]

The software mapping, as a funnel: the menu tree is fully mapped, but screen coverage and proof lag far behind, and the hypothesis pipeline has not yet produced a single proven rule.

---
config:
 xyChart:
 width: 540
 height: 360
 themeVariables:
 xyChart:
 backgroundColor: "transparent"
 plotColorPalette: "#3F86B8"
---
xychart-beta
 title "Black-box mapping: from inventory to proof"
 x-axis ["Menu entries", "Leaves", "Screens", "Hypotheses", "Filtered", "Proven"]
 y-axis "Count" 0 --> 3000
 bar [1048, 899, 154, 2845, 1519, 0]

The MCP servers and end-to-end evaluation (P4–P5) are not implemented.

Limitations & next steps

Pilot results are below threshold and rest on partial human annotation. Next steps: redefine the category label, complete a fully human reference, re-measure G1/G2, then build the three servers and run the pre-registered evaluation. For the software mapping, the write-perturbation heuristics await written authorisation and a dedicated VM.

Stack

Python 3.11 · JSONL event log · SQLite · YAML ontology · MCP · AT-SPI · Ollama · pytest

Deterministic Process Automation Under Strong Security Constraints

Mon, 01 Jun 2026 00:00:00 +0000

Context

A client operating under strong security constraints manages several hundred sites and runs four recurring back-office processes every day: maintaining field equipment, handling recurring spurious equipment events, writing intervention reports for its customers, and checking that each site’s periodic health check has arrived. The inputs are scattered across three systems with very different access paths: PDF reports delivered by e-mail, a business web portal with no public API, and Excel workbooks maintained by hand over many years.

The goal: automate these processes on an office workstation while keeping a human operator as the final authority on every customer message.

Design constraints

No language model at run time. Every decision comes from explicit rules and every sentence from a template. This makes the robot’s output reproducible: the same inputs always produce the same drafts, so a new version can be compared word for word against production, and any change traces back to a rule or a data source. No customer data goes to an external model service. A vision-language model was used once, offline, during design, to map recorded operator screen sessions; it is not part of the deployed system.
Drafts, not sends. The robot only creates Outlook drafts; the operator reviews and sends them. A global kill switch forces draft mode regardless of per-process settings.
Local execution, no new infrastructure. Python on a Windows workstation, scheduled tasks, a shared NAS folder for state and configuration.
No plaintext secrets. Credentials and tokens live in the Windows Credential Manager (DPAPI); configuration is YAML the operator can edit without a rebuild.

Architecture

flowchart TB
 SCHED["Scheduled passes (morning / afternoon)"] --> ING
 MAIL["Mailbox (Microsoft Graph, read)"] --> ING["Ingestion PDF reports grouped by day"]
 ING --> PARSE["PDF parser (pypdf + regular expressions)"]
 PORTAL["Business web portal Playwright login → session token → JSON endpoints replayed"] --> RULES
 XLS["Reference & history workbooks (Excel)"] --> RULES
 PARSE --> RULES["Rule engine 4 processes, ordered rules & exclusions"]
 RULES --> TPL["Template composer French agreement, names, dates"]
 TPL --> DRAFTS["Outlook drafts (Graph, never sent)"]
 RULES --> TRACK["Tracking workbooks (atomic append)"]
 DRAFTS --> OP(["Operator reviews & sends"])
 OP -. "sent items: corrected recipients" .-> RULES
 NAS[("NAS: locks, heartbeats, journal, idempotence ledger")] --- RULES
 RULES --> HEALTH["Health alerts webhook + dead-man check"]

The four processes

Equipment maintenance. Fault lines parsed from the daily PDF reports are matched against each device’s battery and repair history. An ordered decision tree (first match wins) chooses between ignoring, waiting, battery replacement, device replacement or radio-range testing, and drafts a technician work order; each device on a multi-device order keeps its own decision.

Recurring spurious events. Event counts per site and per device over 24 hours are compared with thresholds to detect recurring spurious events. Portal data (an incident confirmed as real, the system being switched off minutes after a burst) overrides the counts; the outcome is an advisory e-mail to the customer, a work order, or nothing.

Intervention reports. For each on-site intervention, the robot rebuilds a single timeline from the portal’s event log, call log and field-staff records, handles events that straddle midnight, and drafts a customer report with the media files attached.

Periodic health checks. Sites whose periodic health check is missing are identified, the status of each connectivity channel is checked on the portal, and a customer e-mail, a technician work order or an internal note is drafted.

Composing an e-mail: from parameters to a draft

Each draft is the end of a long chain of explicit decisions. Counted in the source code (October 2026), the four rule engines have 53 terminal outcomes (each place where a rule returns a decision), mapped onto 37 decision types and 13 draft builders.

The customer advisory e-mail of the spurious-events process shows the scale. A site reaches it only after up to 14 ordered tests in the rule engine (16 terminal outcomes for this process alone). The composer then reads 21 inputs — 17 fields of the site’s decision record and customer record, the recipient list, the attachments, and two learned resources (zone names from the portal and a spelling dictionary learned from the operator’s own history workbook). Between the decision and the final text there are 108 conditional branch points (if statements, conditional expressions and boolean operands, counted on the syntax tree of the composer and the ten helpers it calls, before the HTML pass).

flowchart TB
 IN(["1 · Rule engine — 14 ordered tests, first match wins. Input: daily event reports · portal journal · customer record · learned site profile"]) --> T0

 T0["Blocking flag on the site record?"] -- yes --> X0["No e-mail"]
 T0 -- no --> T1["R6 · incident confirmed as real today?"]
 T1 -- yes --> X1["No e-mail"]
 T1 -- no --> T2["R17 · media events < 30 s apart across zones?"]
 T2 -- yes --> X2["Manual review"]
 T2 -- no --> T3["R12 · external cause confirmed on media?"]
 T3 -- no --> T4["R8 · ≥ 8 integrity events?"]
 T4 -- yes --> T4b["persisting ≥ 3 days?"]
 T4b -- yes --> X3["Technician work order"]
 T4b -- no --> X4["Keep watching"]
 T4 -- no --> T5["≥ 2 integrity events, ≥ half of the burst?"]
 T5 -- yes --> X5["No e-mail"]
 T5 -- no --> T6["R16 · ≥ 140 events, advisory ignored ≥ 4 weeks? (off by default)"]
 T6 -- yes --> X6["Billing notice"]
 T6 -- no --> T7["R15 · shared-building flag?"]
 T7 -- yes --> X7["Coordination e-mail"]
 T7 -- no --> T8["R5 · weekend + chronic-device flag?"]
 T8 -- yes --> X8["Temporary masking request"]
 T8 -- no --> T9["R1 · same device ≥ 7 events?"]
 T9 -- no --> T10["R2 · ≥ 5 devices and ≥ 8 events?"]
 T10 -- no --> T11["R4 · ≥ 8 events, ≥ 50 % 05:00–07:00?"]
 T11 -- no --> T12["R3 · ≥ 8 events, ≥ 60 % 12:00–18:00, ≥ 70 % in 4 h?"]
 T12 -- no --> T13["R1bis · site total ≥ 7 events?"]
 T13 -- no --> X9["No e-mail"]

 T3 & T9 & T10 & T11 & T12 & T13 -- yes --> ADV

 ADV(["2 · Composer — advisory e-mail, rule id carried along. Customer name: legal form removed, re-cased, acronyms kept"])
 ADV --> SUBJ["≥ 30 events?"]
 SUBJ -- yes --> S1["Subject with urgent prefix"]
 SUBJ -- no --> S2["Plain subject"]
 S1 & S2 --> GR["Contact cell"]
 GR -- "several people, or a role / organisation word" --> G1["Generic greeting"]
 GR -- "title + surname" --> G2["Greeting + normalised title (7 abbreviations) + SURNAME"]
 GR -- "surname only" --> G3["Greeting + SURNAME"]
 G1 & G2 & G3 --> DEV["Devices involved"]
 DEV -- one --> D1["Portal zone name if known, else cleaned label + jargon map + learned accents"]
 DEV -- several --> D2["Generic phrase + list of ≤ 6 by count, each with its device code if known"]
 D1 & D2 --> HND["System switched on/off within minutes of the burst? (R1–R4 only)"]
 HND -- "off just after" --> C1["Handling-error wording A"]
 HND -- "on just before" --> C2["Handling-error wording B"]
 HND -- no --> ACT["Site in activity? — all of: R1–R4 or R1bis · ≥ 2 devices · ≥ 10 events · no intervention in the window · professional site · ≥ 90 % in learned hours · media sensors only: ≥ 90 % while on"]
 ACT -- yes --> C3["Site-in-activity wording"]
 ACT -- no --> RUL["Rule that fired"]
 RUL -- "R1, R2" --> C4["Generic environmental cause"]
 RUL -- "R3, R4" --> C5["Staff-passage cause"]
 RUL -- R12 --> C6["External cause confirmed on media"]
 RUL -- R1bis --> C7["Cause to investigate on site"]
 C4 & C5 --> ATT["Media actually attached?"]
 ATT -- yes --> A1["+ reference to the attachments"]
 ATT -- no --> FLG
 A1 & C1 & C2 & C3 & C6 & C7 --> FLG["Site flag: frequent contractor works?"]
 FLG -- yes --> F1["Request: switch off before works"]
 FLG -- no --> F2["Request: treat the cause"]
 F1 & F2 --> REC["Addresses in the customer record?"]
 REC -- yes --> RC1["All addresses found"]
 REC -- no --> RC2["Internal fallback mailbox"]
 RC1 & RC2 --> DRAFT(["Plain text → Outlook HTML (unfolding, lists, fixed typography, signature) → drafts folder, never sent"])
 classDef q stroke-dasharray:4 3,stroke-width:2px;
 class T0,T1,T2,T3,T4,T4b,T5,T6,T7,T8,T9,T10,T11,T12,T13,SUBJ,GR,DEV,HND,ACT,RUL,ATT,FLG,REC q;

The wording space of this single e-mail: 2 subject forms, 3 greeting forms, 2 opening-sentence forms, 7 reachable cause wordings (two of them with or without a reference to attached media), 2 requested actions, and an optional device list of 2 to 6 lines. Every threshold above lives in the YAML configuration, not in the code.

Parameter	Source	Effect on the e-mail
Event count over 24 h	daily reports	figure in the opening sentence; ≥ 30 adds an urgent prefix to the subject
Rule that fired	rule engine	selects the cause wording; gates the handling-error and site-in-activity checks
Devices involved (cleaned, de-duplicated)	daily reports	one named device, or a generic phrase plus a list of up to 6 lines ordered by count
Device code per report line	daily reports	appended in parentheses to each listed device, only when known
Zone name of the main device	portal	replaces the raw report label when available
Learned spellings	operator’s history workbook	restores accents and casing of device names
Title and contact-name cell	customer record	one of 3 greeting forms; several people, a role or an organisation name fall back to the generic form
Company name	customer record	legal form removed, re-cased with acronyms kept, used in subject and body
Site type, learned opening hours, interventions in the window	tracking workbook, portal journal	enable or suppress the site-in-activity wording
System switched on/off around the burst	portal journal	handling-error wording (2 variants), overriding every other cause
Site flags	customer record	choice between the 2 requested actions
Attachments present	portal media	reference to the attachments, or nothing
Recipient addresses	customer record	every address found; otherwise an internal fallback mailbox

The other drafts draw on the same shared French-language helpers: the preposition before a site name agrees with its first noun (6 forms, from chez to à l’ and aux); report titles agree in gender and number; dates are written in full French form; a “Contact” heading becomes plural when several contacts are listed; and an 8-entry map turns technical equipment jargon into customer vocabulary.

Engineering for reliability

Idempotence. A daily ledger keyed on site, message type and subject hash prevents duplicate drafts; a cross-day register prevents re-issuing a work order before the previous intervention has happened. Missed days are caught up automatically.
Concurrency on a shared NAS. Exclusive lock files (O_CREAT|O_EXCL) with a time-to-live for orphans; two workstations publish heartbeats, and the secondary takes over only when the primary has been silent for two minutes.
Safe spreadsheet writes. Writes go through a temporary file and an atomic replace, with retries when the workbook is open by a user; inserted rows shift footer formulas so totals stay correct.
Network robustness. Graph calls handle token refresh, HTTP 429 Retry-After and 5xx back-off.
Explicit failure. A crash wrapper with no dependency on the main package sends a redacted alert and a dead-man ping even when imports fail; a sentinel raises an alert when the portal returns no data at all instead of reporting success.
Shadow benchmarking. A sandbox build replays a day in read-only mode and diffs its drafts against production. The acceptance metric is the share of drafts the operator sends without changing a single word.

Results

1,766 automated tests in the repository (October 2026), built on mock mail and portal adapters, with live-system tests marked separately.
An audit of the corpus of received reports showed that 9.5 % of event lines were being silently discarded, because event codes and labels are concatenated in the extracted PDF text; the parser was rewritten to recognise the code forms actually emitted.
Caching the fault-history workbook (1.49 MB, 15,533 rows) cut the equipment maintenance process on a replay of real reports from 161.1 s to 23.3 s; workbook opens fell from 26 to 3.
First clean shadow comparison against production: 14 drafts on each side, 14 in common, 0 missed, 0 added.

Stack

Python 3.11 · Microsoft Graph · MSAL · httpx · Playwright · pypdf · openpyxl · pydantic · keyring · pytest · PyInstaller

Field Identification of Recall-Affected Vehicles

Mon, 01 Jun 2026 00:00:00 +0000

Context

Safety recall campaigns depend on reaching every affected vehicle, but letters and phone calls leave a residual population of owners who never respond. In island and overseas territories the problem is sharper: network coverage is uneven, field logistics span several territories, and the registry of who drives what is incomplete. The practical question becomes a fleet-management one: which vehicles of the circulating fleet are still affected, where are they seen, and how far has each one progressed towards repair?

Approach

flowchart TB
 A["Recall listing
(scheduled file pull or manual CSV)"] --> B["Central server
import · versioning · model-name decoding"]
 B -->|"incremental diff"| C["Field units (laptops in agent vehicles)
local recall copy"]
 D["Camera stream"] --> E["Plate detection + OCR
(on-device, offline)"]
 E --> F["Format-aware correction
+ double-read confirmation"]
 F --> G{"Match against
local recall list?"}
 G -->|"no"| H["Discarded — nothing stored"]
 G -->|"yes"| I["Alert to agent
+ local queue"]
 I -->|"sync when network returns"| B
 B --> J["Dashboard
units · tours · detections · incidents"]
 J --> K["Owner contact workflow
identified → contacted → repaired"]

Methods

Data ingestion and versioning. The recall listing is pulled automatically on a schedule or imported by hand. Each import is diffed against the previous one with stable identifiers, versioned by hash, and only the delta is sent to the field units. Manufacturer type codes are translated into readable model names on import.

Matching under OCR noise. Plate detection and character recognition run on the field laptop with ONNX models, with no network dependency. Two safeguards reduce false matches: (i) a positional correction step uses the known structure of national plate formats (current and legacy) to resolve classic OCR confusions (O/0, I/1, S/5, B/8, Z/2, G/6), rejecting any ambiguous reading; (ii) a double-read confirmation only counts a plate when it is read twice within a short time window. A smartphone web page offers a manual fallback (photo or typed plate, plus VIN check-digit validation).

Offline-first synchronisation. Matches are queued locally and pushed to the central server on the next connection, so detections survive multi-day outages. Units report a heartbeat (software version, recall-list version, reading counters).

Field operations. Supervisors plan tours as ordered, geolocated checkpoints on a map, assign them to teams and units, reuse templates, and follow progress. Agents report incidents with a severity level. A contact workflow records the outcome of each owner approach, and the dashboard derives a conversion funnel (identified → approached → engaged → repaired) with weekly trends, the median delay to repair and a breakdown by territory.

Privacy by design. Plates that do not match the recall list are never stored or transmitted; retained records have fixed retention periods with automatic purge; actions are written to an audit log; dashboard access is role-based, including read-only accounts; data at rest is encrypted. A processing register was written alongside the code.

Deployment & operations (DevOps)

The system runs on two very different kinds of machine: Windows laptops in agents’ vehicles, which may stay offline for days, and one server per instance. After the first weeks in the field, most of the engineering work went into making both sides easy to deploy, observe and recover, by a single person, without breaking field work.

flowchart LR
 subgraph ADM["Admin workstation"]
 GIT["Git repository
committed tree only"]
 PROV["Provisioning script
one config file per unit"]
 BLD["Field build
PyInstaller bundle + ONNX models"]
 end
 subgraph SRV["Server (Ubuntu LTS VPS)"]
 CAD["Caddy
TLS · HSTS · no access log"]
 APP["Docker container
FastAPI API + dashboard
loopback port only"]
 VOL[("LUKS2 volume
SQLite · photos · archives")]
 TIM["systemd timers
monitoring · backup · cleanup"]
 end
 FLD["Field laptops
local SQLite · sync loop"]
 OBJ[("Off-site object storage
age-encrypted backups")]
 CHAT["Chat alerts
state changes + daily digests"]
 GIT -->|"git archive over SSH → deploy script"| APP
 PROV -->|"admin API"| APP
 PROV -->|"config file + README"| FLD
 BLD -->|"archive + one-click installer"| FLD
 FLD <-->|"HTTPS: heartbeat · pushes · recall diff"| CAD
 CAD --> APP
 APP --- VOL
 TIM --> VOL
 TIM -->|"encrypted copy"| OBJ
 TIM --> CHAT

Field laptops: packaging, installation and updates

Packaging. The field application (Python backend, React interface) is frozen with PyInstaller into a one-folder Windows executable that embeds the ONNX models and the OCR configuration, so a unit needs no Python install and no network to read plates. Built-in --diag and --selftest modes check the bundled runtime and read a test image without starting the interface, which made start-up failures diagnosable on the field machine itself.

One-click installer. The build is delivered as an archive with a double-click INSTALLER.bat. It elevates itself and runs a PowerShell script that removes the “downloaded from the Internet” mark from the files (otherwise the bundled runtime fails to load), disables USB power saving so the GPS dongle is not suspended, and installs the unit’s configuration file where the executable reads it. The script refuses to guess: if several different configuration files are present, it lists them and asks which one is the new one. An optional reset clears the local database when a unit moves to another server; it shows the target server before asking for confirmation and warns that unsynchronised detections would be lost. The unit’s access token is never displayed. An update follows the same path: new build, re-run the installer.

Provisioning. A Python script that uses only the standard library, so it runs on a bare Windows machine, creates the units from a CSV file through the server’s admin API and generates, for each unit, its configuration file (server address, unit code, display name, token) and a README to hand over with the laptop. The admin key is read from an environment variable or a masked prompt, never from a command-line argument.

Versioning: software and recall list

Software version. Each unit reports its application version in its heartbeat. The dashboard compares it with the expected release and flags the units that still need updating.

Recall-list version. The recall list is versioned by hash. On each cycle, a unit compares its local hash with the server’s: identical, nothing to do; no local copy, full download; otherwise, the server returns only the rows added and removed since that hash (and the unit falls back to a full download if the server no longer knows its version). The change is applied in a single transaction, and the local hash only moves forward if everything was applied.

That last rule came from a field incident. A database constraint made the diff fail on one unit, the error was swallowed by the sync loop, and the unit kept sending heartbeats while its recall list stayed frozen for more than two weeks, so it alerted on vehicles that had already been repaired. Release 1.4.1 fixed the cause, added an explicit rollback, and shipped a schema migration that keeps existing data. It was tested with a 62-test run that included reproducing the bug and migrating a database with the old schema. Monitoring now raises a “recall sync stuck” alert when a unit has not received a published list after a grace period.

Incremental, resilient synchronisation

Every five minutes, or on demand, a unit runs a sync cycle made of independent steps: heartbeat, push of detections (in batches), GPS track, incidents, checkpoint updates, then pull of the recall list and the tours. Pushes come before pulls, so field data leaves first. A failing step no longer blocks the others: in an earlier version the cycle was monolithic, and a failed heartbeat once delayed detections by 22 minutes. Requests use short timeouts, downloads longer ones. The heartbeat carries the software version, the recall-list version, the number of pending detections and the reading counters, which is what the dashboard and the monitoring rely on.

Server deployment

Instance kit. Each deployment is an independent instance described by a short identifier, on an Ubuntu LTS virtual server. A bootstrap script, which is interactive and can be safely re-run, installs the system packages, Docker and Caddy from their official repositories. It enables unattended security updates without automatic reboot, restricts SSH to key authentication, enables a default-deny firewall and brute-force protection, and checks that the DNS record points to the server before applying the TLS configuration. Secrets live in root-only configuration files on the server, never in the repository.

Container and reverse proxy. A multi-stage Dockerfile builds the React dashboard with Node, then runs the FastAPI server on a slim Python image as an unprivileged user. Its health check allows a three-minute start period because the plate and OCR models are loaded before the server listens. The container is published on the loopback interface only. Caddy is the only public entry point: automatic Let’s Encrypt certificate, explicit HSTS, and no access log, because scan URLs contain plates. The application’s own access log is disabled for the same reason.

Deploy and rollback. Code leaves the admin workstation only as a git archive of a committed tree: uncommitted changes are listed and not sent, and line endings are forced to LF. On the server, the new tree is unpacked next to the current one and swapped in, the previous one is kept, and the deployed commit is recorded. The deploy script then detaches from the SSH session, so a dropped connection interrupts neither the deployment nor a rollback. A lock prevents concurrent deployments. The script builds the image (labelled with its commit), keeps the previous image as rollback, recreates the container and waits until it reports healthy. It gives up after five minutes, or as soon as the container has restarted twice. On failure it rolls back automatically to the previous image and source tree, sets the failed code aside and sends an alert. Going back to any earlier version means re-deploying that commit. An image rollback does not undo a schema migration, so the documented procedure is to take a backup before any deployment that changes the schema.

Encrypted volume and safe restarts

Server data (database, contact photos, recall archives) lives on a LUKS2 file container mounted with restrictive options. Its passphrase is never stored on the machine, so after a reboot a person unlocks the volume. The container restarts on its own, but the server refuses to start unless a sentinel file on the encrypted volume is present. Without it, a reboot would have left an empty directory in place of the volume, and the application would have created a fresh, unencrypted database there and accepted field data into it. The guard was checked both ways on a throwaway container. A separate script grows the volume while it is mounted.

Backups and recovery

A daily systemd timer takes a consistent SQLite snapshot through the database’s backup API, archives it with the photos and recall archives, and encrypts the result with age to a public key. The private key is shown once, at creation, and kept off the server. Backups are kept locally and, optionally, copied to S3-compatible object storage with rclone, configured through environment variables only. A restore test mode decrypts a backup into a temporary directory on the encrypted volume, runs an integrity check, counts rows per table, then deletes everything. The apply mode moves the current state aside instead of deleting it. The documentation includes a step-by-step rebuild after total loss of the server. The continuity plan targets a recovery time of 4 hours and a maximum data loss of 24 hours.

Monitoring

A watchdog runs every 15 minutes and only posts to a chat webhook when a state changes (problem, then “resolved”). A fuller digest is sent twice a day, timed to the field shifts. It checks that the encrypted volume is mounted, the API health endpoint (with retries, because the daily import can block it for about a minute), the container health status, that the database is readable, disk usage on both the system disk and the encrypted volume, the size of the SQLite write-ahead log, TLS certificate expiry, backup age and size, units silent for more than 48 hours (not 24, since units are switched off at weekends) and units whose recall list is stuck. A separate watcher alerts when no recall import has happened for more than 30 hours. Several of these checks were added after real incidents: the encrypted volume once reached 90 % without any alert because only the system disk was watched. A weekly timer cleans up Docker build caches and keeps only the current and rollback images. A dedicated script truncates the SQLite log with a short lock wait, so field writes are never blocked for long.

Offline resilience

The field side is designed to keep working through a server outage: plate reading, matching against the local recall copy, alerts, tours already downloaded and the local statistics tab all work without a network, and the queue empties at the next connection. Only the dashboard (and the server-hosted smartphone page) is unavailable while the server is down.

Testing

There is no CI pipeline in the repository. Releases are tested before they ship, and the commits record those runs: 12/12 and 6/6 for the plate correction and confirmation stages, 62 tests for release 1.4.1. Behaviours that cannot be unit-tested, such as the volume guard or a live deployment, were checked on throwaway containers and against the production health endpoint.

Results

In early field operation, agents flagged 32 % of detections as misreadings over roughly one week. This measurement motivated the positional-correction and double-read confirmation stages, and the dashboard statistics now exclude detections marked as false positives.
The new stages shipped with unit tests: 12/12 for plate correction and 6/6 for confirmation, settings and reading log.
After the model-name mapping was extended, only one row of the production recall listing remained without a decoded model name.

No end-to-end recognition rate has been measured yet.

Limitations & next steps

Recognition accuracy has not been measured against a ground-truth benchmark. The specification targets (read rate, latency) are design goals, not results.
No legacy-format plate reading was observed at first. A dedicated counter now tracks these readings end to end, but the fix still needs a field check.
A false match against a large listing still costs agent time. Calibrating the confidence threshold and confirmation window on labelled field video is the next step.

Stack

Python · FastAPI · SQLite · ONNX Runtime · fast-alpr (YOLOv9 plate detector + OCR) · OpenCV · PaddleOCR · React · TypeScript · Vite · Leaflet · Docker · Caddy · LUKS · systemd · age · rclone · PyInstaller · PowerShell

From Multitrack Recordings to an Editable Timeline: Transcription, Diarisation and AAF Export

Mon, 01 Jun 2026 00:00:00 +0000

Context

A conference recorded on several microphones (speakers, room, music) produces hours of raw multitrack audio. Most editing time goes into locating material — who speaks when, on which track, where the silences are — before any cutting.

Open-source speech tooling (Whisper and its derivatives, pyannote, Silero VAD) solves transcription and “who spoke when” well, but stops at text. The question addressed here is whether that analysis can be delivered as an editing object: a multitrack timeline in AAF, the interchange format that audio workstations open natively, with tracks qualified and dead time removed.

Before → after

One real job processed end to end: a conference recorded on 16 tracks. Everything on the right-hand side was produced automatically, with no manual cut; figures are read from the job’s analysis database and from the delivered AAF.

	Input (raw recording)	Generated output (AAF timeline)
Tracks	16 × 1 h 32 min 54 s (≈ 24 h 46 min of audio to review)	16 tracks kept in the AAF, 6 carrying clips
Track roles	4 speech · 2 music (one auto-flagged as ambience) · 10 silent	10 silent tracks left empty, no manual triage
Material	204 speech regions and 5 music passages detected	41 clips (37 speech, 4 music)
Cuts	—	40 hard cuts between sources, of which 33 microphone switches
Transitions	—	0 crossfades: straight cuts, each speech clip extended by a 0.3 s tail
Dead time	—	3 gaps ≥ 4.5 s rippled out (36.9 s); timeline 1 h 32 min 17 s

Top: the 16 input tracks over the full recording, coloured by the detected role and activity; the four speech microphones are active almost all the time because each one also picks up its neighbours. Bottom: the generated edit, one lane per track as in the AAF — a single source is kept at any moment, ticks mark the 40 source switches, hatched areas in the top panel are the removed dead time.

Speech clips are few and long (median 79 s, longest 10 min 11 s): the montage follows the active speaker rather than chopping on every breath, and micro-switches shorter than 0.8 s are absorbed by the anti-flicker smoothing. Ten clips are still shorter than 6 s — short interjections, the first place to look when checking the edit.

Approach

flowchart TB
 A["Multitrack WAV recording"] --> B["Decode to 16 kHz mono + RMS energy per 100 ms"]
 B --> C["Voice activity detection (Silero)"]
 C --> D["Transcription (Whisper large-v3, chunked)"]
 D --> E["Forced word alignment (WhisperX / wav2vec2)"]
 C --> F["Speaker diarisation (pyannote)"]
 E --> G["Speaker assignment + segment post-processing"]
 F --> G
 B --> H["Track role: speech / music / silent"]
 G --> H
 H --> I["Montage: active microphone per 100 ms frame"]
 I --> J["AAF export (linked to original WAV names)"]
 I --> K["Mono mixdown WAV"]

Methods

Transcription. Whisper large-v3 via faster-whisper (CTranslate2), run on fixed-length chunks inside an isolated, persistent worker process: the model is loaded once, and a native crash only loses one chunk, which is reported rather than silently dropped. Decoding uses beam search with Silero VAD filtering and condition_on_previous_text disabled to limit repetition loops.

Word timing and speakers. Word-level timestamps come from WhisperX forced alignment (wav2vec2). Speaker turns come from pyannote diarisation and are assigned to words by temporal overlap.

Post-processing. Pure filters remove known Whisper hallucinations (stock subtitle credits, “thanks for watching”-type phrases, music-only segments), non-Latin output on French audio and duplicates, then merge consecutive same-speaker segments.

Track qualification and montage. Each track is classified as speech, music or silent by combining frame energy, VAD coverage and transcript statistics. The montage selects, for every 100 ms frame, the microphone with the highest signal-to-noise ratio above its own noise floor (reducing cross-talk “bleed”), applies anti-flicker smoothing and ripples out global dead time.

Synchronisation. Separate recorders are aligned by matching distinctive words across transcripts (robust median of offsets), after a coarse clock-based placement corrected for clock drift.

Export. The edit list is written as a multitrack AAF with pyaaf2, either embedding the audio or linking to it.

Deployment. A web portal accepts jobs and a separate compute worker polls it over HTTPS. Clients upload only compressed analysis copies together with a manifest of their original files; the delivered AAF references the original WAV names, so the editor relinks full-quality sources locally and the originals never leave the client’s machine.

Results

Benchmark (bench_stt.py) on real conference audio: one-minute speech windows selected by VAD on four speech tracks, Whisper large-v3 throughout, Apple Silicon laptop, warm timings excluding model loading. No ground-truth transcript was available, so quality figures are divergence from a pseudo-reference (WhisperX output), not absolute error.

Pipeline (CPU)	Real-time factor
openai-whisper	0.43
WhisperX	0.51
WhisperX + forced alignment	0.54
openai-whisper + word timestamps	0.57
faster-whisper (beam 5 + VAD)	0.66
faster-whisper + word timestamps	0.70

Word timing: native word timestamps deviate from forced alignment by a median of ~205 ms (faster-whisper) and ~210 ms (openai-whisper); forced alignment adds about 1.2 s per minute of audio.
Transcript divergence: 7–11 % word-level divergence between pipelines, mostly punctuation, segmentation and normalisation on manual inspection.
Repetitions: 1–2 % repeated content; no hallucinated passages observed on these clips.
Track detection: the three music/silence tracks of the seven-track recording were identified by VAD and skipped automatically.

Limitations & next steps

Quality is measured against a pseudo-reference; a manually transcribed test set is needed to report true word error rate and diarisation error rate.
Benchmark clips are short (one minute per track) and all single-speaker windows, so diarisation accuracy on multi-speaker overlap is not yet quantified.
CTranslate2 has no Metal backend: on Apple Silicon, faster-whisper and WhisperX run on CPU, while CUDA is used where available. Device choice per stage was benchmarked rather than assumed; a CUDA comparison remains to be measured.
AAF export targets Logic Pro import; other editors have not been systematically tested.

Stack

Python · faster-whisper · CTranslate2 · WhisperX · pyannote.audio · Silero VAD · PyTorch (MPS / CUDA) · ffmpeg · pyaaf2 · SQLite · PHP · MySQL

Geolocated Video Analysis and Inter-Object Distance Measurement

Thu, 01 Jan 2026 00:00:00 +0000

Context

Industrial sites are often covered by fixed cameras whose footage is only reviewed after an incident and yields no usable data. The aim was to turn existing cameras, and occasional drone footage, into a measuring instrument: for every detected object, a latitude/longitude position and metric distances rather than a pixel box.

This is hard for three reasons. A single camera gives no depth, so scene scale must be inferred. The usual georeferencing method, a ground-plane homography, assumes a flat floor and fails silently for raised objects or uneven terrain. And high-resolution video is expensive, while the common shortcut of downscaling to 640 px discards the geometric detail ground positions depend on.

Approach

flowchart TB
 A["Video source
file · RTSP · drone clip (up to 3840×2160)"] --> B["Object detector at native resolution
detect · segment · pose · OBB"]
 A --> C["Metric monocular depth model
depth map"]
 B --> D["Multi-object tracker
motion model + association"]
 B --> E["Per-object depth
mask pixels, outlier rejection"]
 C --> E
 E --> F["Ray casting
pixel ray + metric depth + camera pose"]
 F --> G["WGS84 position
optional DEM for ground elevation"]
 E --> H["3D distances
between masks · along trajectories"]
 D --> H
 G --> I["SQLite store
detections · tracks · GPS trajectories"]
 H --> I
 I --> J["Analytics
zones · flows · orthophoto overlay"]

Methods

Detection. A YOLO-family detector, at several model sizes, in four modes (detect, segment, pose, oriented boxes). Inference runs at native resolution with no forced imgsz. Segmentation is the default because masks give cleaner depth samples than boxes. PyTorch, ONNX, TensorRT and CoreML weights are accepted (CUDA and Apple MPS).
Monocular depth. A metric monocular depth model predicts a depth map, which is resampled to the frame size. Each object’s depth is a robust estimate over its mask pixels, after outlier rejection. A validation step compares predicted depth with reference zones whose distance is known, to detect calibration drift.
Geolocation by ray casting. A ray is cast from the camera centre through the object’s pixel and stopped at the object’s metric depth. The resulting point is converted from the camera frame to a local North-East-Down frame, then to WGS84. Unlike a homography, this does not assume a flat ground. An optional DEM supplies ground elevation. Positions can be overlaid on a georeferenced orthophoto for visual checks.
Tracking. A custom multi-object tracker with a motion model, confidence-aware association and a track-fusion step that links fragments using gap, heading and speed limits. When depth is available, each track stores a 3D trajectory, from which the distance travelled and the speed in m/s are derived.
Inter-object distances. Later consolidated into a reusable vision library. Two segmented objects are back-projected into 3D, and the minimum point-to-point distance is reported. Overlapping masks are treated as contact. A Monte Carlo simulation perturbs depth, focal length and pixel positions to give a standard deviation and a 95 % interval. Zone membership uses the share of the mask inside the zone, with hysteresis, rather than a centre-point test.
Engineering. The configuration supports a two-GPU split (detection on one device, depth estimation on the other), RTSP discovery and recording, and processing of several cameras in parallel.

Results

Measured on a single NVIDIA RTX 5090. The detection benchmark covers detection and tracking only, with depth disabled.

Setting	Video	Throughput	Output
Largest detector, batch 1	3840×2160, 50 fps, 3,301 frames	5.1 fps	320,217 detections, 1,941 tracks
Smallest detector, batch 1	3840×2160, 50 fps	13.0 fps	158,989 detections, 908 tracks
Largest detector, batch 1	2304×1296, 15 fps, 5,000 frames	17.9 fps	73,502 detections, 2,706 tracks
Smallest detector, batch 1	2304×1296, 15 fps	39.0 fps	51,890 detections, 1,499 tracks
Full chain (detection + depth + tracking + storage)	2304×1296, 300 frames	2.02 fps	12.3 detections per frame

In the full-chain run, depth estimation took 117.6 s of the 148.3 s total (about 80 %), and detection took 20.9 s. Depth estimation, not the detector, is the bottleneck. On 4K footage the largest detector runs about ten times slower than real time. That suits the intended use, which is analysis of recorded archives, not live alerting.

Limitations & next steps

No geolocation accuracy has been measured against surveyed ground truth yet. The ray-casting stage was added after the timing runs above, so its cost is not included in them.
The metric scale depends on the depth model and on the camera pose and intrinsics. Errors in tilt or focal length propagate directly into ground positions.
The depth model predicts at its own internal resolution, so fine depth detail is limited even when detection runs in 4K.
Next steps: an accuracy study with GNSS-surveyed targets, profiling the two-GPU split, and measuring how accurate Monte Carlo distance intervals are on objects at known separations.

Stack

Python · PyTorch · OpenCV · NumPy · SciPy · pyproj · rasterio · SQLite · CUDA · Apple MPS

Pollutant Detection by Drone (Day & Night)

Mon, 01 Jul 2024 00:00:00 +0000

Context

Detect gas and hydrocarbon pollutants from a drone, day and night, to monitor mobile pollution sources (vessels, machinery) in maritime and industrial zones.

Pipeline

flowchart TB
 A["Drone — thermal/IR + visible"] --> B["Day / night acquisition"]
 B --> C["Pollutant signature detection"]
 C --> D["Localisation on moving targets"]
 D --> E["Maritime / industrial monitoring"]

Methods

Combined thermal/infrared and visible-spectrum imaging.
Detection and localisation of pollutant traces on moving targets.
Operational in both daytime and nighttime conditions.

Results

Detection of fuel leaks and hydrocarbon discharge in port areas and coastal waters.
Continuous monitoring capability for maritime and industrial zones.

Stack

Python · OpenCV · thermal/IR imaging · drone

Satellite-Based Sargassum Detection & Stranding Prediction

Sat, 01 Jun 2024 00:00:00 +0000

Context

Sargassum strandings disrupt Caribbean coastlines. The goal: detect algae rafts at sea early and predict where and how much will strand, so municipalities and environmental authorities can plan cleanup — and interventions at sea — in advance.

The problem: raw reflectance is hard to read

In raw multispectral satellite imagery, Sargassum is a faint signal on a large, variable ocean background. Rafts are thin, fragmented and often low-contrast: their edges blur into the surrounding water, and the overall structure of a band — where it starts, where it thins out, where it breaks up — is difficult to make out by eye or with generic indices.

Detecting that there is Sargassum is not enough. To act, one needs to see what it looks like.

Core contribution: an original contrast indicator

I designed a dedicated contrast indicator for Sargassum. Rather than simply flagging pixels, it is built to give a very clear reading of the data and of the shapes: rafts and bands stand out sharply from the water, with well-defined contours and their internal structure preserved. (Formulation details withheld for now.)

What it brings:

Clear delineation of rafts and of elongated Sargassum bands.
A readable picture of how the mass is organised, not just where it is.
A cleaner input for segmentation, quantification and drift modelling downstream.

Why shape matters

The geometry of the bands is the key information for operations:

Dispersion — is the Sargassum concentrated in a dense raft or scattered in many small patches?
Trailing (“traîne”) — how the band stretches and elongates behind its head, revealing how it is being pulled by currents and wind.
Orientation and drift direction — which way the band is heading, and therefore which stretches of coast are exposed.

Reading these shapes is what makes it possible to plan interventions on site (where to deploy collection at sea or barriers) and to anticipate strandings rather than react to them.

Pipeline

flowchart TB
 A["Satellite imagery — multispectral"] --> B["Cloud / water masking"]
 B --> C["Original Sargassum contrast indicator"]
 C --> D["Raft & band delineation — shape, dispersion, trailing"]
 D --> E["Quantification at sea"]
 E --> F["Drift / stranding model"]
 F --> G["Alerts & intervention planning"]

Methods

Multispectral satellite imagery analysis.
Original contrast indicator for a sharp reading of Sargassum rafts and bands.
Shape analysis of the bands: dispersion, elongation/trailing, orientation.
Raft detection and surface-area quantification at sea.
Predictive drift/stranding modelling to anticipate landing zones and volumes.

Results

Clear, readable maps of Sargassum rafts and bands at sea.
Early-warning capability for coastal municipalities.
Quantitative stranding forecasts supporting proactive cleanup and intervention planning.

Visuals and further details will be added.

Stack

Python · satellite imagery · spectral analysis · predictive modelling

Sub-Millimetre Crack Detection in Civil Structures

Fri, 01 Mar 2024 00:00:00 +0000

Context

Non-destructive structural health monitoring for bridges, buildings and infrastructure. The challenge is not only to detect cracks, but to classify them by width — without contact or scaffolding.

Width is what matters for inspection: depending on how open a crack is, the repair (filling) method is not the same. An inspection system therefore has to be very precise, because a wrong width class means a wrong repair specification.

Four width classes

Class	Crack width	Consequence
1	< 0.3 mm	Finest cracks — often narrower than a pixel; dedicated repair approach
2	0.3 – 0.7 mm	Different repair approach
3	0.7 – 1 mm	Different repair approach
4	≥ 1 mm	Different repair approach

Each class calls for its own filling/repair approach, so the classification directly drives the maintenance decision.

The sub-pixel challenge

The key difficulty is to go below the pixel size. A 0.3 mm crack does not even cover a full pixel at typical inspection resolutions — and the system must still detect it, even when the ground sampling distance is somewhat coarser than 0.3 mm.

Why is this possible at all? A crack thinner than a pixel does not vanish: it still occupies part of the pixels it crosses, and slightly alters their intensity compared with the surrounding surface. Taken individually, that partial-pixel signal is weak; but a crack is a continuous, elongated feature, so the same faint contrast repeats coherently along many adjacent pixels. Exploiting that contrast and continuity is what allows detection — and width estimation — beyond the nominal resolution of the image.

Acquisition: drone and smartphone

Drone photography — high-resolution UAV capture of structural surfaces, for large or hard-to-reach structures.
Smartphone photography — the approach also works on images taken with a mobile phone, which I have tested, making close-range inspection possible with everyday equipment.

Pipeline

flowchart TB
 A["Image capture — drone or smartphone"] --> B["Image tiling & calibration"]
 B --> C["Crack segmentation — CV / DL"]
 C --> D["Sub-pixel width estimation"]
 D --> E["Width classification — 4 classes"]
 E --> F["Defect map & inspection report"]

Methods

High-resolution capture of structural surfaces by drone or smartphone.
Computer-vision / deep-learning crack segmentation on concrete and masonry.
Metric calibration and sub-pixel analysis for crack width estimation.
Classification into four width classes matched to repair methods.

Results

Non-destructive, repeatable inspection of civil structures.
Geolocated defect maps supporting maintenance prioritisation.

Models and visuals will be added.

Stack

Python · OpenCV · deep learning · high-res RGB drone · smartphone imagery · photogrammetry