[{"content":"Context Personal digital traces (messaging, mail, photos, music listening, notes, calendar, health exports, code repositories) are among the most sensitive data a person owns, and also among the richest for studying behaviour — and feeling — over time. Sending them to a cloud LLM is not acceptable, so everything here runs on a single Apple-silicon desktop (M4, 32 GB unified memory), and no data leaves the machine.\nThe aim is an agent that answers questions about one\u0026rsquo;s own life while keeping a strict line between what is observed, what is computed, what is hypothesised by a model, and what is accepted as true by its owner.\nWhat is original Local agents over personal files are not new. Two aspects of this project are.\n1. A meaning graph on top of a measurement layer. Knowledge is held at two levels. An observatory stores facts and signals in long format (item, dimension, value, confidence, model, version). Above it, a meaning graph stores what those signals mean: relations (reified, so that a relationship can evolve and be measured), affective episodes (who felt what, when, how intensely, through which modality), themes, chapters of a life, and truths — statements with supporting and contradicting evidence that only the owner can validate. Every node and edge carries its provenance down to the source element, and an origin ranked by trust (human \u0026gt; declared \u0026gt; structural \u0026gt; rule/computed \u0026gt; vendor \u0026gt; model \u0026gt; inferred).\n2. Emotion measured across modalities, along unusual dimensions. The same fixed vocabulary of 53 emotions (Plutchik-based, each with valence and arousal coordinates) is applied to messages (local LLM), voice (five acoustic and multimodal layers fused with the transcript, with an explicit text/voice mismatch flag) and photos (deterministic colour and light measures, SigLIP embeddings, face signals treated as weak evidence). These sit in a catalogue of 929 dimensions in 88 groups that goes well beyond the usual metadata. A few examples of its less common entries:\nmodality dimension how it is defined photo colour temperature mean red–blue balance of the image (warm vs cool) photo colourfulness Hasler \u0026amp; Süsstrunk (2003) opponent-colour metric photo golden hour / blue hour sun elevation computed offline from date and position photo face size share of the frame taken by a face (a proxy for proximity) voice jitter, shimmer, harmonics-to-noise ratio openSMILE eGeMAPS prosody voice laughter, crying, sighs sound-event detection (AudioSet classes) voice text/voice mismatch voice valence and transcript valence differ by ≥ 0.6 message valence, arousal, tone per participant, constrained to the fixed grid message expressive punctuation, emoji palette deterministic text measures relation reply-latency asymmetry log ratio of the two parties\u0026rsquo; median reply delays relation initiatives conversations opened after 3 h of silence, per side relation trend slope of relationship intensity over 90 days (cooling \u0026lt; 0) period musical nostalgia mean age of the tracks listened to period creation / consumption time in creative tools ÷ time consuming From signals to meaning flowchart LR PH[\"Photo\"] --\u003e PM[\"Measureslight · contrast · colour temperaturecolourfulness · palette · sharpness\"] PH --\u003e PS[\"SigLIP embedding(image ↔ text similarity)\"] PH --\u003e PF[\"Vendor face signalssmile · expression · gaze · face size\"] VN[\"Voice note\"] --\u003e VT[\"Whisper transcript\"] VN --\u003e VL[\"Acoustic layersvalence/arousal/dominance · categoriessound events · prosody · audio-LLM\"] VT --\u003e VX[\"Text emotion of transcript\"] VL --\u003e VF[\"Weighted fusionconfidence = agreementtext/voice mismatch flag\"] VX --\u003e VF MS[\"Messages(pseudonymised: Me, P1, P2)\"] --\u003e ML[\"Local LLM, fixed gridemotion · intensity · valencearousal · tone · topics\"] PM --\u003e OB[(\"Observatory929 dimensions\")] PS --\u003e OB PF --\u003e OB VF --\u003e OB ML --\u003e OB OB -- \"emotion with enough confidence\" --\u003e AFF[\"Affective episode (AFF)who · emotion · when · intensity · modality\"] PF -. \"weak cue, confidence ≤ 0.4\" .-\u003e AFF ML -- \"topics\" --\u003e THM[\"Themes\"] OB -- \"aggregates\" --\u003e AG[\"Day / period indicatorsvalence · light · palette\"] AFF --\u003e G[\"Meaning graphrelations · episodes · themes · chapters · truths(all proposed)\"] THM --\u003e G AG --\u003e G G --\u003e HV((\"Ownervalidate · reject · correct\"))Each affective episode is a node linked to the person who felt it, to the emotion in the fixed vocabulary, and — when it exists — to the conversation or moment it occurred in. Inference rules then carry affect upwards (for instance, the affect of a chapter or of a routine), and strong aggregate correlations can become proposed truths with their measures as evidence.\nEmotion measurement in detail Vocabulary. The grid is fixed and versioned: Plutchik\u0026rsquo;s 8 primary emotions at 3 intensities, his 8 primary dyads, plus social and reflective emotions useful for a life (gratitude, tenderness, pride, shame, loneliness, nostalgia, wonder, relief, exhaustion…), 53 in total. Each carries starting coordinates in Russell\u0026rsquo;s circumplex — valence in [−1, 1], arousal in [0, 1] — meant to be calibrated later on the owner\u0026rsquo;s own validations. Plutchik oppositions are encoded, and health-app mood labels are mapped onto the grid.\nMessages. A local LLM (Qwen3-30B-A3B) reads each conversation with participants replaced by \u0026ldquo;Me\u0026rdquo;, \u0026ldquo;P1\u0026rdquo;, \u0026ldquo;P2\u0026rdquo;, messages framed as data it must not obey, and media reduced to placeholders such as \u0026ldquo;[voice note 23 s]\u0026rdquo;. Output is constrained by a JSON schema: for each participant, the dominant emotion from the grid (or none), intensity, valence, arousal, a tone from a closed list of 19, and a confidence; plus 1–3 topics that become themes. Long conversations are reduced to their beginning and end; short ones are batched ten per call. An affective episode is proposed only when confidence is at least 0.3.\nVoice. Each recording is converted to 16 kHz mono and passes through:\nWhisper large-v3-turbo (MLX) for the transcript; a wav2vec2 dimensional model giving valence, arousal and dominance on 15 s windows, weighted by window length, plus arousal variability; emotion2vec+ large for categorical emotion, mapped onto the grid (kept at ≥ 0.4); an AST AudioSet classifier for laughter, crying, sighs, shouting, whispering, music and ambience; openSMILE eGeMAPS prosody: pitch (semitones) and its variability, loudness, jitter, shimmer, harmonics-to-noise ratio, voiced segments per second, mean pause length; Qwen2.5-Omni-7B (llama.cpp, Metal), which listens to the audio and returns an emotion from the grid, a valence and short acoustic cues; the text-emotion model above, applied to the transcript. The categorical opinions (voice categories, audio-LLM, transcript) are combined by a weighted vote (weights 1.0, 1.0 and 0.8); valence is a weighted mean of the dimensional model, the audio-LLM and the transcript. Confidence is agreement between layers: one agreeing source gives 0.45, two give 0.65, three give 0.85, minus 0.1 for each source that disagrees. When the voice and the words disagree by 0.6 or more in valence (an \u0026ldquo;I\u0026rsquo;m fine\u0026rdquo; said sadly), a text/voice mismatch is flagged rather than averaged away. Laughter and crying scores are kept alongside.\nPhotos. On 256 px thumbnails, without any model: luminance, contrast, saturation, colour temperature, colourfulness, sharpness (variance of the Laplacian) and a 5-colour palette with named dominant hue. SigLIP so400m embeddings make the library searchable by text. Vendor face attributes (smile, smile type, expression, gaze, eye state, face size) are read as signals, but the schema states that a photographed smile is not a felt joy: the face modality is a weak cue whose confidence is capped at 0.4. Music mood is handled the same way: the mood of a track is never, on its own, evidence of the listener\u0026rsquo;s mood. At period level, valence is fused with a fixed precedence: declared \u0026gt; voice \u0026gt; text \u0026gt; face.\nHuman validation. Every episode, theme, chapter or truth is emitted as proposed. The PWA shows a validation queue — affective episodes ordered by confidence, with the evidence (the conversation or recording it came from) — and the owner validates, rejects or corrects. Only that human decision event turns a proposal into validated, an equivalence checked automatically. The planned next step is a small personal classifier on emotion2vec features, trained on these validations.\nArchitecture flowchart TB S[\"Sources (read-only snapshots)messaging · mail · photos · music · notes · calendar · exports\"] C[\"Connectorsdedicated ingestion process, sole holder of disk access\"] V[\"Vault (owner-only, 0700)content-addressed raw store\"] O[\"Observatoryfacts + signals (SQLite)\"] K[\"Meaning graphappend-only JSONL log → SQLite fold + FTS\"] N[\"Nightly / hourly pipelinesthumbnails · measures · SigLIP · emotions(GPU lock, one model at a time)\"] M[\"MCP server (stdio, no port)20 read-only / propose-only tools\"] H[\"Hermes Agentlocal LLM via Ollama\"] P[\"PWA (HTTPS + TOTP)chat · validation queue\"] U((\"Owner\")) S --\u003e C --\u003e V --\u003e O --\u003e K N --\u003e O O --\u003e M K --\u003e M M --\u003e H --\u003e P P -- \"validate / reject / correct\" --\u003e K U -- \"NetBird VPN\" --\u003e PRaw data is snapshotted (SQLite opened in mode=ro\u0026amp;immutable=1) and never written to; everything derived lives in a separate owner-only vault, and the code repository holds no data. The meaning graph is built on the atoms library\u0026rsquo;s knowledge-base module: an append-only, schema-validated JSONL event log folded into a disposable SQLite index.\nThe schema itself is data, verified mechanically: 27 node types and 50 predicates, 9 provenance origins, 31 locator kinds tying every claim to its source element, the 53-emotion vocabulary, 17 inference rules, and 42 sources behind the 929 dimensions (610 native, 126 vendor-computed, 86 aggregates, 74 deterministic measures, 29 model interpretations).\nSecurity model Source content is assumed adversarial (prompt injection in a message or mail), and the model may hallucinate or misuse tools. Defences are structural:\nNo ambient capabilities. The agent framework (Hermes Agent) has 20 built-in toolsets disabled (terminal, files, code execution, web, browser, delegation, memory, cloud speech…); no cloud provider, update check or external login; the only model endpoint is a loopback Ollama server. One narrow interface. Data is reachable only through a local MCP server over stdio, with closed schemas, validated arguments, explicit truncation and a content-free access log. Tools can propose, never decide. Privilege separation and pseudonymisation. Only ingestion holds full-disk access; analysis has no network. Third parties are opaque codes, and a real name in an identifier is rejected by the schema. Sensitivity levels (0–3) per element with a lower ceiling for remote sessions; remote access only through a NetBird VPN, TOTP and strict session cookies. Ethical exclusions: vendor-inferred ethnicity, skin tone and gender are never read. Models All inference is local; one large model is loaded at a time under a shared GPU lock.\ntask model runtime agent (tool use, ≥ 64k context) gpt-oss:20b, qwen3:8b fallback Ollama text emotions, voice-transcript emotions Qwen3-30B-A3B-Instruct-2507 (Q4) Ollama text / image embeddings bge-m3 · SigLIP so400m Ollama · PyTorch MPS image reading qwen3-vl:8b Ollama speech-to-text Whisper large-v3-turbo MLX voice emotion (5 layers) wav2vec2 dimensional · emotion2vec+ · AST · openSMILE eGeMAPS · Qwen2.5-Omni-7B PyTorch MPS · CPU · llama.cpp (Metal) Each model is smoke-tested on synthetic data only (a fictitious French sentence synthesised by a local TTS, then passed through the whole audio chain).\nEvaluation \u0026amp; results Agent answer quality. A 16-question benchmark (2 trap questions), with ground truth computed by calling the tools directly, scores tool choice, numerical fidelity (every number in the answer must appear in that turn\u0026rsquo;s tool outputs), correctness, trap handling and partial-period flagging: 73 % → 100 % after tool-description, argument-validation and configuration fixes. Server-side error correction. With a reduced 8-tool interface, simple questions went from 76 % to 90 % and cross-source questions from 73 % to 90 %; 20 deliberately unusual questions scored 89 % and exposed silent-zero failures, since fixed. Context and latency. Tool descriptions cut from 23.5 k to 9.5 k characters; a heavy aggregate query from 172 s to 0.1 s; all heavy benchmark queries ≤ 0.3 s. Cross-exploration benchmark. 60 placeholder-only scenarios with oracle code; oracles pass 61/61, with no leak at remote sensitivity ceilings 1 and 2. Verification. 56 named schema checks green; 631 native/vendor columns confirmed against real database structures; 1,235 Python unit tests, 15 JavaScript test files and 110 quality tests on real data that report counts only. Limitations \u0026amp; next steps Evaluation sets are small and authored by the system\u0026rsquo;s designer; they measure fidelity and regressions rather than general capability. Emotion outputs are hypotheses: the valence/arousal coordinates of the vocabulary are starting values, and personal calibration of voice emotion from validated examples is still pending, as is the larger target agent model. Agent answers take tens of seconds on a 20B model.\nStack Python · SQLite / FTS5 · Ollama · llama.cpp (Metal) · MLX · PyTorch MPS · Hermes Agent · MCP · SigLIP · Whisper · emotion2vec · openSMILE · NetBird · PWA · launchd\n","date":"2026-10-01T00:00:00Z","permalink":"/research/local-ai-agent/","title":"A Fully Local AI Agent with a Meaning Graph and Multimodal Emotion Measurement"},{"content":"Context Indoor sites are often documented with a handheld gimbal camera rather than a survey-grade scanner. The goal was to turn such video into usable 3D deliverables (dense cloud, textured mesh, plan and elevation orthoviews, a Gaussian Splatting scene and lightweight web exports) with no manual steps between capture and output. The pipeline runs natively on Windows on an RTX 5090 (Blackwell, sm_120) that is shared with other long-running GPU workloads, so it must never pre-empt them.\nApproach flowchart TB A[\"inbox/project/videos + GO file\"] --\u003e B[\"Keyframe extractionoptical-flow overlap + sharpness\"] B --\u003e C[\"COLMAP 4.2.1 SfMSIFT, global mapper\"] C --\u003e D{\"QC: registration, reprojection,trajectory continuity\"} D -- fail --\u003e C2[\"Fallback: incremental mapper\"] --\u003e D D -- pass --\u003e E[\"OpenMVS 2.4 (sm_120 build)densify, mesh, refine, texture\"] E --\u003e F[\"Ortho alignmentgravity, floor RANSAC, Manhattan yaw\"] F --\u003e G[\"Blender rendersorthos, axonometries, orbit\"] E --\u003e H[\"gsplat trainingMCMC or default strategy\"] G --\u003e I[\"Web exportsPotree, GLB, SOG\"] H --\u003e I I --\u003e J[\"outbox: deliverables, report.json, STATUS\"]Methods Overlap-driven keyframe selection. KLT tracking with a RANSAC similarity transform estimates overlap with the last keyframe. A frame is emitted when overlap drops below 0.80, tracking is lost or a time limit passes. The sharpest candidate (variance of the Laplacian) is the one kept. SfM with explicit QC. COLMAP\u0026rsquo;s global mapper is fast but non-deterministic. In testing it produced reconstructions that met registration and reprojection criteria but were geometrically incoherent. A trajectory-continuity check (jumps above 25× the median camera step) triggers an automatic fallback to the incremental mapper. Below 20 points per image the step fails. MVS hardening. The official OpenMVS Windows binary has only sm_89 kernels, so it was rebuilt for sm_120. The object ROI is off for rooms because it cropped the cloud. Refinement is dropped when it diverges, and seam leveling (which blackened faces) is off. Metric-ready orthoviews. The up axis comes from camera orientations. A RANSAC floor plane sets z = 0 and a histogram of wall-normal azimuths (mod 90°) sets Manhattan yaw. An optional two-point reference fixes scale. The 4×4 transform is stored so it can be applied again to the mesh and the splat. Gaussian Splatting. gsplat 1.6.0 was compiled for sm_120 with CUDA 13 under MSVC, which required local patches. Training uses the MCMC strategy with a Gaussian budget and falls back to the default 3DGS strategy when MCMC diverges. Operations. Projects start in watch-folder mode when a GO file appears. Stages can be resumed or redone. A GPU guard waits until enough VRAM has been free for a stable period, serialises GPU stages through a lock file and records peak VRAM per stage in report.json. Degraded QC produces a DONE_WITH_ALERTS status instead of a silent success. Results First real deployment: five locations inside a commercial showroom, one 4K handheld video per location, processed unattended in about 30 minutes per location.\nLocation Images SfM registered / points / reproj. Dense cloud Splat PSNR / SSIM 1 116 116/116 · 21,458 · 1.49 px 1.40 M 23.2 / 0.86 2 103 103/103 · 18,972 · 1.75 px 1.23 M 20.6 / 0.81 3 47 47/47 · 13,167 · 1.17 px 1.26 M 23.1 / 0.86 4 121 121/121 · 11,970 · 1.34 px 1.99 M 21.9 / 0.85 (default strategy) 5 64 64/64 · 12,106 · 1.48 px 0.91 M 21.8 / 0.83 All frames were registered in every location. Reprojection error was 1.17–1.75 px. Measured VRAM peaks: SIFT +1.3 GB, matching +1.6 GB, texturing +0.5 GB, Blender +5.4 GB, and splatting with 3 M Gaussians at 1920 px +25.2 GB. The sm_120 OpenMVS build densified the synthetic smoke test in 17 s. The CPU fallback took about 40 min. Metadata analysis showed shutter speeds of 1/50–1/100 s (about 14 px of motion blur at 4K) and automatic ISO and white balance. Exposure compensation in gsplat (ppisp) gave no gain (PSNR 20.5 vs 20.6). The quality limit is in the capture, not the solver. Limitations \u0026amp; next steps PSNR in the low 20s reflects motion blur and auto-exposure. The capture protocol now requires a 1/250–1/500 s shutter, locked exposure and white balance, and a dedicated floor pass. Merging videos from different locations caused false matches and an incoherent model, so each location is processed separately. Dark glossy tiled floors reconstruct poorly. Manhattan alignment is not yet reliable with furniture and glazing. CPU stages are the bottleneck on a 4-vCPU virtual machine. Planned: comparison with a commercial photogrammetry suite on the same frames, and retesting mesh refinement with the sm_120 build. Stack Python 3.11 · OpenCV · Open3D · NumPy · COLMAP 4.2.1 · OpenMVS 2.4 · gsplat 1.6.0 · PyTorch 2.14 (CUDA 13) · Blender 5.2 · FFmpeg · PotreeConverter · glTF-Transform · splat-transform\n","date":"2026-10-01T00:00:00Z","permalink":"/research/photogrammetry-splatting/","title":"Handheld Video to Photogrammetry and Gaussian Splatting"},{"content":"Context Fort-de-France (Martinique) has almost no local open data. The city\u0026rsquo;s own datasets were last updated in 2019, and there are no public pedestrian counters, open transit ridership figures or open road counts at city-centre scale. The project asks how far national open data and continuously archived feeds can go in describing the city centre as a system of points of interest (shops, administrations, schools, health, culture, entry gates) and flows (residents, commuters, pupils, visitors, cruise passengers, vehicles, buses) over a ten-year horizon (2016–2026).\nNo single source answers that question. The business register knows where establishments are but not who visits them. The census knows where people sleep but not where they spend the day. Transit feeds show the buses but not their passengers, and cruise schedules show the ships but not the people ashore. The approach is therefore to cross many heterogeneous sources on a common spatial and temporal frame, so that each one fills a blind spot of the others, and to check the result against the few measured signals that do exist.\nApproach The method comes from an earlier knowledge-base project of mine. Raw data is immutable and addressed by content. Every database is disposable and rebuilt from it. Every value keeps its provenance, and a set of invariants is checked on each rebuild.\nflowchart TB A[Source cataloguesources.yaml] --\u003e B[Immutable raw storesha256 + JSONL manifest] C[Scheduled collectorsMeta every 14 days · GTFS-RT 60 s] --\u003e B B --\u003e D[Versioned extractorsnormalised Parquet] D --\u003e E[DuckDB spatial warehouserebuilt, never edited] E --\u003e F[QA invariants I1–I6] E --\u003e G[Statisticsrequetes/*.sql] E --\u003e H[Desktop applicationmap + time selector] G --\u003e I[Calibration againstmeasured signals]Methods Acquisition. Each download is hashed (SHA-256) and logged in a JSONL manifest, and acquisition is idempotent. National files larger than 1 GB (Sirene business register) are read as remote HTTP-range extracts filtered to the municipality. The filtering query is recorded as part of the provenance. Ephemeral sources. Meta Movement Distribution keeps only 90 days online, so a scheduled task archives it every 14 days. A bus collector polls GTFS-RT vehicle positions every 60 s and predictions every 10 min, and zips each day with a SHA-256 checksum. Spatial frame. Data is stored in WGS84 and computed in EPSG:5490 on the INSEE 200 m grid. Points of interest are classified by one typology (NAF codes, OSM tags, stops), with family, weight and an hourly rhythm over seven time slots. Models. Hourly presence per grid cell combines residents (Filosofi 2021, distributed dasymetrically to buildings), employees, pupils and tourists. A Huff model estimates catchment. Cruise passengers onshore are modelled from ship schedules and capacities. An island-wide gravity model with BPR assignment covers visitors and through-traffic. Quality. Six invariants are checked, from provenance to the absence of personal data. Unconfirmed parameters go in a decision log. Data sources Crossing the sources is the core of the project. Business registers (Sirene, BODACC, URSSAF) give the stock and turnover of activities, but say nothing about the people around them. The census and tax data (RP, Filosofi) place residents, their income and housing vacancy, but only at night. Commuting files and the national travel survey (MOBPRO, MOBSCO, EMP) say who travels, why and when. Transit, accident and air-quality feeds (GTFS-RT, BAAC, Madininair) are the only measured traces of movement. Cruise schedules and ship capacities bring a population that no census counts. The geographic frames (IGN, OpenStreetMap, the INSEE 200 m grid) are what make all of these comparable cell by cell.\nThe catalogue lists 120 sources: 51 integrated, 32 to integrate, 7 to verify, 13 available on request only, 5 paid, 1 by partnership and 11 not retained. The tables below list every integrated source and summarise the others by theme.\nBusinesses, jobs \u0026amp; amenities Source What it contributes Period / frequency Status Sirene business register — INSEE Establishments, activity code (NAF), creation and closure dates, headcount bands Full history; stock rebuilt at any date 2016–2026 Integrated BODACC — DILA Business turnover and failures (sales of goodwill, insolvency proceedings) 2016–2026, daily Integrated Permanent amenities database (BPE), geolocated — INSEE Shops, services, health, administrations, education, sport, tourism 2024 and 2025 editions, annual Integrated Establishments and salaried headcount by municipality × activity — URSSAF / Acoss Private salaried employment by activity 2006–2025, annual Integrated Education directory and pupil numbers — MENJ / DEPP Schools, location, pupils per school year, staff Pupil numbers 2016–2025 Integrated FINESS — DREES / ARS Health and social facilities, pharmacies Extract of May 2026, bimonthly Integrated Overture Maps places — Overture Maps Foundation Points of interest with a confidence score (comparison layer) Frozen extract of release 2026-09-23, monthly Integrated OpenStreetMap (Overpass) — OpenStreetMap contributors Points of interest, shops, streets, pedestrian paths Current state Integrated Historic monuments (Mérimée / POP) — Ministry of Culture Heritage attractors (cathedral, Schœlcher library, Fort Saint-Louis) Current Integrated 23 more — e.g. Flores localised employment (INSEE), Foursquare Open Source Places, LOCOMVAC vacant commercial premises (DGFiP), Google Places Catalogued Population, income \u0026amp; housing Source What it contributes Period / frequency Status Census, IRIS-level files (RP) — INSEE Residents, age, occupations, dwellings (vacancy, car ownership), workforce RP 2016 → RP 2022, annual Integrated Filosofi by IRIS — INSEE Disposable and declared income, poverty 2016–2021, annual Integrated Filosofi 200 m grid files — INSEE Tax households, individuals, poverty, social housing and age per 200 m cell 2015, 2017, 2019, 2021 Integrated Filosofi 200 m grid, Géoplateforme WFS layer — INSEE Grid cell geometries 2015 layer Integrated IRCOM income tax by municipality — DGFiP Tax households, declared income 2016–2024, annual Integrated Cafdata — CNAF Recipients of minimum income, housing and family benefits 2020–2024 (2022 missing), annual Integrated Priority neighbourhoods (QPV) — ANCT / INSEE Priority urban policy areas, which include the city centre 2024 boundaries Integrated LOVAC — Cerema / DGALN Short-term and structural (≥ 2 years) vacancy of private housing 2020–2026 editions, annual Integrated RPLS social housing — SDES Social dwellings (address, surface, rent, vacancy) 2023 and 2025 Integrated Sitadel building permits — SDES Dwellings and premises authorised, started, demolished 2013–2026, monthly Integrated Geolocated DVF — DGFiP / Etalab Property transactions (price, surface, commercial premises) 2021–2025, rolling 5 years Integrated 8 more — e.g. census detail files and legal populations (INSEE), DV3F and land files (Cerema / DGFiP), WorldPop / GHSL gridded population Catalogued Mobility \u0026amp; transport Source What it contributes Period / frequency Status Commuting flows (MOBPRO) — INSEE Home → work flows between municipalities 2016–2022, annual Integrated School commuting (MOBSCO) — INSEE Home → study flows between municipalities 2016–2022, annual Integrated National travel survey 2018–2019 (EMP) — SDES Trip purposes, departure and return times, activity durations, distances 2018–2019 Integrated GTFS, urban network Centre zone (Mozaïk, BRT lines) — Martinique Transport Scheduled transit supply Archives since June 2023 Integrated GTFS-RT, continuous collection — Martinique Transport Observed bus speeds, buses in service, declared occupancy Since 7 October 2026; 60 s (positions), 10 min (predictions) Integrated GTFS, ferry shuttles — Martinique Transport Shuttles landing on the waterfront Versions since August 2024 Integrated Road injury accidents (BAAC) — ONISR Geolocated accidents (pedestrians, vehicles), with time 2016–2024 loaded Integrated Movement Distribution — Meta Data for Good / HDX Share of users at home, within 10 km, 10–100 km, beyond 100 km Daily; 90-day window, archived every 14 days Integrated Movement Range Maps — Meta Data for Good / HDX Change in movement, share of users staying put March 2020 → May 2022, daily (discontinued) Integrated BD TOPO main road network — IGN Island road graph for traffic assignment Current Integrated BD TOPO road sections — IGN Carriageway width, lanes and direction, used to measure kerbside parking Current Integrated OpenStreetMap car parks — OpenStreetMap contributors Car parks, type, declared capacity Current Integrated 18 more — e.g. ridership and ticket validations (Martinique Transport, on request), road counts (on request), Orange Flux Vision mobile data (paid), Waze for Cities / Strava Metro (partnership), a pedestrian counting campaign (to be created) Catalogued Tourism \u0026amp; cruise Source What it contributes Period / frequency Status Cruise call schedule, port of Fort-de-France — CruiseMapper Ship-by-ship calls with arrival and departure times March 2026 → March 2029; archived monthly, as past months disappear Integrated Ship pages — CruiseMapper Passenger capacity, crew, tonnage of ships in port Current Integrated Cruise season reviews — Martinique Tourism Authority (CMT), via the press Passengers and calls per season, cruise terminals Seasons 2024–25 and 2025–26 Integrated PortWatch — IMF / HDX Daily port calls from AIS 2019–2026, daily Integrated Monthly commercial air traffic — DGAC Arrivals at Martinique Aimé Césaire airport (tourist seasonality) 1990–2025, monthly Integrated Classified tourist accommodation — Atout France Classified hotels and residences, rooms and capacity Current stock Integrated Tourist accommodation capacity — INSEE Hotels by rating, campsites, residences, holiday villages 2015–2026, annual Integrated 7 more — e.g. DATAtourisme (ADN Tourisme), INSEE hotel occupancy survey, Inside Airbnb / AirDNA, port authority call data Catalogued Context: weather, air, calendar, safety Source What it contributes Period / frequency Status Climatological data — Météo-France Rain, temperature, wind (Fort-de-France Desaix and Le Lamentin stations) 2010–2026, hourly / daily Integrated Air quality — Madininair NO₂ as a traffic proxy, plus PM10, PM2.5, O₃ and H₂S from sargassum Hourly; monthly NO₂ 2016–2026 Integrated Recorded crime by municipality — SSMSI Recorded offences 2016–2025, annual Integrated Public holidays in Martinique — DINUM Calendar, including 22 May 2006–2031 Integrated School calendar — MENJ School holidays, including Carnival 2017–18 → 2026–27 Integrated Pageviews — Wikimedia Foundation Online interest in Fort-de-France, Carnival and heritage 2016–2026, monthly Integrated 4 more — e.g. NASA Black Marble night lights, Google Trends / GDELT, an events agenda (to be compiled) Catalogued Geography \u0026amp; base maps Source What it contributes Period / frequency Status Study perimeters — project City centre and extended centre (provisional, see D1) Version 0.1 Integrated Administrative boundaries API (geo.api.gouv.fr) — DINUM / IGN / INSEE Municipal boundary and legal population Current Integrated IRIS boundaries — IGN / INSEE Sub-municipal census units Current layer Integrated BD TOPO — IGN Buildings (height, use), roads, facilities Quarterly editions Integrated BD TOPO residential buildings, extended centre — IGN Use, height and footprint used to place households in buildings Current edition Integrated OpenStreetMap vector base map — OpenStreetMap contributors Streets, water, parks, coastline drawn in the application Current Integrated 9 more — e.g. Etalab cadastre, National Address Base, BD ORTHO aerial imagery (IGN), local urban plan (Géoportail de l\u0026rsquo;Urbanisme), natural risk maps (Géorisques) Catalogued Results The project started in October 2026 and is at an early, uncalibrated stage (application v0.6.0). All city-centre figures depend on a provisional perimeter (see below). Each result below states which sources it crosses: none of them could be produced from a single dataset.\nCatalogue. 120 sources are catalogued, of which 51 are integrated. Warehouse (first build). 117,161 entities, about 25,000 measures and 1,278 indicators, with all 6 invariants passing. Retail decline vs residents (measured, registers × census × tax data). Active personal-goods establishments (Sirene) fell from 498 (2016) to 363 (2026), a drop of 27 %, and the city centre\u0026rsquo;s share of the municipality in this sector fell from 75.5 % to 64.7 %. Over the same period, population in the city-centre IRIS (census) fell from 13,444 (2016) to 12,898 (2022), and housing vacancy rose from 24.9 % to 27.6 %. 47 % of city-centre households are below the poverty line, against 30 % for the municipality (Filosofi 2021). Read together, the three sources show a centre losing shops, residents and purchasing power at the same time. Day/night rhythm (modelled, six sources crossed). About 3,700 people are present at night and 11,200 in the morning, a ratio of 3.0. Night-time residents come from the Filosofi 200 m grid placed in BD TOPO buildings; daytime workers from Sirene headcount bands; pupils from the education directory; tourists from classified accommodation capacity. The morning slot is adjusted on the hourly NO₂ profile of traffic stations. Observed bus speed vs model (GTFS-RT × road graph × travel surveys). Over one full day, buses ran at 12.7 km/h at 05:00, falling to 7–8 km/h during the day and 10.6 km/h at 19:00. The assignment model, built from BD TOPO roads, MOBPRO commuting, EMP trip profiles and BPE amenities, predicts a 20 % slowdown at peak hours, whereas the buses show 45 %. The measured feed exposes what the model misses. Cruise check (schedules × ship capacities × reported counts). Onshore passengers are modelled from the CruiseMapper call schedule and ship capacities. On the one documented cruise day (24 April 2026), the estimates came within about 1–2 % of the passenger counts reported in the press (3,075 vs 3,112, and 1,995 vs 1,969). The fill rate used in the model was set on that same day, so this is a consistency check, not an independent validation. Limitations \u0026amp; next steps D1: the city-centre perimeter is provisional. It is a rectangle around the historic core plus a 1 km extended zone, used until an official perimeter is provided. Every city-centre figure depends on it. No pedestrian, ridership or road-count data is open. Hourly rhythms, attractiveness weights and parking assumptions are still uncalibrated. The measured controls so far are hourly NO₂ at traffic stations and observed bus speeds. The 2016–2026 business stock is rebuilt from creation and closure dates only. Next steps: build up several months of GTFS-RT data for measured congestion, request survey and counting data, and calibrate presence on field counts. Stack Python · DuckDB (spatial, httpfs) · Parquet · SQL · GTFS / GTFS-RT · pytest · ruff · Electron · React · TypeScript · ECharts · Leaflet · GitHub Actions\n","date":"2026-10-01T00:00:00Z","permalink":"/research/urban-flows-fort-de-france/","title":"Points of Interest and Urban Flows in Fort-de-France City Centre from Open Data"},{"content":"Context Consumer digital media decay on the scale of years. Unpowered SSDs lose charge, optical discs delaminate, formats go out of use and accounts get closed. Paper can last for centuries, needs no electricity and can be checked by eye. The question is how much information an ordinary A4 sheet can hold if it must still be read back reliably with a standard flatbed scanner and no special hardware.\nThe problem is hard because the print-and-scan channel is noisy in several ways at once. Toner spreads and clumps, density varies across the page, and scanners add rotation, anisotropic scaling, uneven paper feed, blur, vignetting and gamma. Paper also stretches non-linearly: at the target density, 0.2 % of stretch shifts the grid by about three cells. Common 2D codes such as QR and DataMatrix are designed for robustness rather than density. A long-term archive also has to remain decodable when today\u0026rsquo;s software no longer exists.\nApproach flowchart TB A[\"Source media: photo, video, voice, text\"] --\u003e B[\"Format-conservative reductionJPEG baseline · MJPEG/PCM AVI · PCM WAV · UTF-8\"] B --\u003e C[\"Archive: tar + zlib + SHA-256\"] C --\u003e D[\"Inter-page erasure codeCauchy MDS over GF(256): any K of N pages\"] D --\u003e E[\"Per-block Reed–Solomon + CRC-32page-wide interleaving, triplicated header\"] E --\u003e F[\"Multi-level cell rendering1-bit A4 PDF at 1200 dpi\"] F --\u003e G[\"Print → scan(simulated channel)\"] G --\u003e H[\"Fiducials → homography → local alignment markers\"] H --\u003e I[\"Grey-level calibration from the page itself\"] I --\u003e J[\"RS/CRC decode → erasure decode → SHA-256 check\"] J --\u003e K[\"Byte-exact files, or partial recovery\"] F -.-\u003e L[\"Printed appendix: format spec + decoder source in OCR-B with per-line CRC-16\"] L -.-\u003e J Left: page 1 of 10 of the photo archive, as sent to the printer (A4, 1200 dpi, a grid of 1,145 × 1,576 cells), downscaled. Right: the boxed top-left corner at printer resolution (about 10 × 9 mm, each printer dot drawn as 2 × 2 pixels): the solid corner fiducial, the start of the header band, and the first data cells. Each cell is 8 × 8 dots (0.169 mm) and its number of inked dots encodes one of 4 levels. The square at the corner of the grid is an alignment marker.\nA page of the printed appendix (page 11 of 30): the start of cdx_read.py (360 lines), set in OCR-B. Each line carries its number and its CRC-16, so a line misread by OCR can be found and checked by hand.\nMethods Multi-level cells instead of binary modules. Each cell carries 2 bits. Cells are drawn as clustered dots at the printer\u0026rsquo;s native resolution, so the driver does no halftoning and produces no moiré. Geometry recovery in stages. Four corner fiducials, one of them hollow to fix orientation, give a global homography. Alignment markers spread across the page then absorb local, non-linear paper stretch. Self-calibrating reader. The decoder estimates the printed grey levels and spatial shading from calibration structures on each page, so it does not depend on the printer\u0026rsquo;s dot settings. Two-level error control, a product code. Reed–Solomon and CRC-32 inside each page correct random and burst errors such as stains and scratches. A Cauchy MDS erasure code across pages, decoded block by block, lets any K of N pages rebuild an archive. Graceful failure. If the SHA-256 check fails, the decoder still extracts every complete file that comes before the damaged region. Self-describing archive. A printed appendix contains the format specification, a reading procedure and the full decoder source in OCR-B, with a line number and CRC-16 on each line. The decoder needs only NumPy, Pillow and reedsolo. Text uses DEFLATE (zlib), which can be reimplemented in a few hundred lines, rather than the stronger LZMA. Earlier prototype (codex_proto/). This version explored several physical media: paper, laser-marked stainless steel, intaglio plates and microfilm frames. It used 2- to 16-level cells, fountain-style parity pages, MLP-based grey-level readers, a printed optics calculator for reading microfilm, and lossy media reduction (parametric voice coding, wavelet-residual photo coding). Results All figures below come from simulation only. The software models laser defects (spot blur, toner grain, dot gain, banding, non-uniform density, textured paper), scanner defects (±1.2° rotation, ±0.3 % x/y scale, feed wobble, blur, vignetting, noise, gamma 2.2), coffee stains, scratches, and one deliberately lost page per archive. No physical print-and-scan test has been run yet.\nNet payload: 383,469 bytes per A4 page, about 59 bits/mm². End-to-end: 46 of 50 pages were scanned and all 4 of 4 archives were restored byte-identical (photo, video, voice, text). No Reed–Solomon codeword failed, and no inter-page block was lost. Error rates: clean pages needed about 0.007–0.008 % of bytes corrected. Pages with stains and/or scratches needed at most 0.90 %. Archive sizes: photo 2.86 MB → 10 pages (8 data + 2 parity); video (800×600, 24 s, MJPEG + 16 kHz PCM) 8.30 MB → 25 pages; voice 0.91 MB → 5 pages; text 5.58 MB → 10 pages. Text: the public-domain Mémoires d\u0026rsquo;outre-tombe (5.43 M characters) would take 1,357 typeset A4 pages. Encoded, it takes 6 data pages, a ratio of about 226. zlib compresses this prose by 2.62 and LZMA by 3.58. Partial recovery: an archive truncated to a quarter of its length still returned 1.42 MB of the 5.58 MB of text. Decoder robustness: the decoder also worked on scans rotated 90° and 180°, on 600 dpi scans and on JPEG (q92) scans. Decoder size: 731 lines of Python for the printed decoder modules. The same region of the photo page (380 × 300 dots, about 8 × 6.4 mm), enlarged. Left: as encoded. Right: after the simulated print-and-scan channel, resampled onto the page grid through the fiducial homography. Toner spread, blur and noise soften the cells, and the edge of a simulated coffee stain darkens the left part. The reader calibrates the grey levels locally, so it still separates the four levels. In the simulation run reported above, stained or scratched pages needed at most 0.90 % of bytes corrected.\nThe video archive: an underwater clip (school of fish over a reef) used as test footage. Top: the original 3840 × 2880 HEVC frames at 3.2 s, 12.8 s and 16.0 s, downscaled. Bottom: the same instants decoded from the simulated scans of the 25 pages (one deliberately lost). The paper channel itself loses nothing: the restored AVI is byte-identical to the archived one. What differs from the original is the reduction made before printing (800 × 600 MJPEG, 12.5 frames/s, 16 kHz mono sound) so that 24 s of video fit in 8.30 MB.\nClose-up of the 12.8 s frame. Left: a 1152 × 864-pixel area of the original, downscaled. Right: the matching 240 × 180-pixel area of the restored frame, enlarged with no smoothing. The fish and the reef stay recognisable. Fine texture and colour detail are lost, and MJPEG block artefacts appear. This is the price of fitting the clip into 25 pages.\nLimitations \u0026amp; next steps No physical test yet. The cell dot patterns were tuned in simulation. The next step is a real 1200 dpi print with a 1200 dpi scan to set them and measure the actual error margin. Small test set. The simulation covers one run over four archives. A sweep across defect severity, scan resolution and paper stock is still needed. Unmeasured physical properties. Long-term ageing of paper and toner, and how practical it is to OCR the decoder back from paper, have not been measured. Font dependency. The appendix body text relies on a system font. Stack Python · NumPy · SciPy · Pillow · reedsolo · ReportLab · soundfile · ffmpeg · Tesseract (OCR-B)\n","date":"2026-09-01T00:00:00Z","permalink":"/research/codex-paper-archive/","title":"Codex — High-Density Physical Archiving of Media on Paper"},{"content":"Context Production systems for document reconciliation and OCR, built for clients such as a wholesale distributor and an accounting firm, left several codebases that each solved the same sub-problems (page routing, OCR tuning, LLM extraction, field comparison) with their own bugs. The atoms library consolidates them into one package. Its central question is the one behind the original systems: can a document check be automated so that a human can audit and contest every verdict? Extracting a field is not enough. The system must state whether two documents agree, why, and from what evidence.\nApproach Everything is built from atoms. An atom has one responsibility, typed inputs and outputs, no global state, and injected dependencies (LLM client, OCR engine, storage). A pipeline is a composition of atoms, either written in code or declared as a YAML recipe that is validated when it loads, before anything runs.\nflowchart TB A[\"Input documents (PDF, scans, CSV, mail)\"] --\u003e B{\"Page routing: native text or OCR?\"} B -- \"native\" --\u003e C[\"Positioned native text\"] B -- \"scan\" --\u003e D[\"PaddleOCR (production settings)\"] C --\u003e E[\"Segmentation \u0026 classification\"] D --\u003e E E --\u003e F[\"LLM JSON extraction + verbatim grounding check\"] F --\u003e G[\"Cross-document reconciliation (invoice / delivery note / order)\"] G --\u003e H[\"Tri-state verdict + named reason + trace\"] H --\u003e I[\"Report / human review\"]The catalogue The 459 atoms are grouped into 32 families, one Python package each, ordered along the processing chain: capture the world, understand it, reconcile and decide, deliver, then operate in production. Two further packages assemble them: steps (77 named, parameterised steps) and recipes (the YAML loader and validator).\nStage Family Atoms Role Base core 25 pivot types, pipeline, configuration, resilience (retry, rate limits), files, concurrency Capture connect 40 resilient HTTP, mailboxes (Graph, IMAP), SFTP, OneDrive, read-only ERP source, open data, audit trail google 24 Gmail, Sheets, Drive, Apps Script, with resilient execution web 17 driver-agnostic web robots (Selenium, Playwright): waits, resumable skeleton, forms pdf 9 positioned native text, OCR-or-not decision, rendering, page handling, duplicates ocr 9 PaddleOCR with production settings, EasyOCR layout 3 page geometry: lines, columns, deskew, monospace rendering audio 16 chunked transcription, segment post-processing, microphone ↔ screen alignment vision 32 video / RTSP streams, detection, tracking, depth, segmentation, calibration vlm 10 screen reading with vision-language models: change detection, label location geo 11 geodesic measures, combined GPS fixes, line-distance filtering, spatial SQL calendar 4 public holidays, working days and opening hours Understand text 32 numbers and amounts, dates, normalisation, pattern-based fields, identifiers segment 7 pages → logical documents, typed document numbers, page nature classify 12 file name, sender, weighted patterns, TF-IDF, visual and LLM classifiers, cascades llm 24 provider-agnostic contract, providers, pool, prompts, structured extraction anchor 5 fields and tables located relative to printed labels, native text or OCR kb 8 traced knowledge base: validated event log folded into a SQLite index Reconcile match 25 EAN/GTIN, indexes, multi-criteria scores, optimal assignment, entity resolution compare 9 tri-state, explainable comparators for fields, records and sets check 9 generic tri-state checks: totals, VAT, balance, completeness, plausibility reconcile 12 set pairing, order → delivery → invoice chains, N-way, ledger and bank matching validate 14 rules engine, identifiers, amount consistency, corrected LLM verdicts, final decision rules 8 safe condition language, explainable first-match tables, escalation, state machines events 11 timestamped events: windows, expected-event watch, sessions, status transitions analytics 18 supply-chain analytics without pandas: demand, stock-outs, service level, safety stock Deliver export 8 CSV for French Excel, shared Excel workbooks, HTML / Markdown / PDF reports report 6 persistable discrepancies, summaries, review queue render 12 HTML deliverables: pre-rendered Mermaid, ECharts charts, GeoJSON maps diagram 6 process diagrams: draw.io / BPMN JSON in, Mermaid and BPMN XML out Operate ops 26 high availability (heartbeats, leader election), scheduled daemon, monitoring, pre-flight checks qa 7 editorial non-regression (golden files), simulation replay, inter-annotator agreement Named recipes Each of the 17 recipes is a YAML list of steps, named \u0026lt;family\u0026gt;.\u0026lt;atom\u0026gt;. Three examples:\ninvoice_multi_doc: a batch of multi-document PDFs becomes checked logical documents. io.collect → pdf.read → segment.documents → classify.documents → llm.extract → validate.ids → validate.amounts → validate.decide → export.csv → google.sheets.upsert → connect.notify. Each document ends as valid, to check, or invalid. doc_reconciliation: a file of invoices, credit notes and delivery notes, checked against orders. Guarded LLM extraction (llm.extract_docs) is cross-checked with pattern extraction (extract.regex, compare.extractions, reconcile.fields), then check.document and check.completeness, lines against the order reference (compare.reference), the order → delivery → invoice chain (reconcile.nway), and a discrepancy file plus review queue (report.discrepancies, report.review). 17 steps. meeting_transcript: a meeting recording becomes a timestamped transcript with speakers. audio.decode → audio.vad → audio.transcribe → audio.diarize → audio.speakers → audio.segments → export.csv → export.transcript → connect.notify. The others include delivery_vs_order, ledger_reconciliation, invoice_dedup, article_base_cleaning, stock_policy, mail_to_sheet, draft_mails, pdf_text_extract, rubric_eval, screen_session, camera_flow and process_from_transcript.\nThe Atlas The Atlas is a single offline HTML page generated from the code itself (syntax trees and YAML), with no hand-written data. It draws the catalogue as a periodic table: each family is an element with a symbol, each module an atom, each recipe a molecule. It also flags atoms that lack documentation, tests or a usage example. The interface is in French.\nThe \u0026ldquo;Capture\u0026rdquo; series of the periodic table. Each box is a family, with its symbol, number of modules and size in thousands of lines; each tile below it is an atom.\nThe \u0026ldquo;Reconcile\u0026rdquo; series: the families that pair, compare, check and decide.\nAn atom\u0026rsquo;s page (compare.fields): what it does, taken from its docstring, then each public name with its full signature.\nThe invoice_multi_doc recipe as a molecule: 11 steps, the context keys that link them, and the services they need (dashed when optional).\nMethods Tri-state results. Every comparison returns ok ∈ {True, False, None} (conforms / discrepancy / inconclusive) with a named reason code and a trace. A missing or unreadable value counts as inconclusive, never as an anomaly. Inconclusive results are still counted, not silently dropped. Hybrid native-text/OCR routing. Each PDF page is classified as native, OCR or empty, based on how much native text it holds and how much of it is covered by images. OCR runs only where it is needed. Grounded LLM extraction. Structured JSON extraction is validated against Pydantic schemas. Each extracted value is then checked verbatim against the source text, so altered or invented values are flagged. Cross-document reconciliation. Invoices, delivery notes and orders are compared line by line using named tolerances and dominance rules. For example, a matching line amount can override a price difference. Provenance. Every ported atom carries a Provenance: header and an entry tracing it to repo@commit:path:lines in the original production code. Bugs found while porting were fixed and each fix is covered by a non-regression test. Enforced architecture. Layering is checked by 30 import contracts: an atom imports only the core package. Strict typing, a ≥ 80 % coverage gate and a check that no client names appear in source code are also enforced. Device abstraction. Vision and audio atoms target cuda, mps or cpu via one setting. End-to-end evaluation. Tests run with real engines (PaddleOCR, Ollama qwen3:8b and qwen3-vl:8b) on data with known ground truth. Each metric is recorded against a threshold in a JSON/HTML report. Metrics that miss their threshold are kept and shown as failing. Results Codebase (current tree): 459 atom modules in 32 families, 77 parameterised steps, 17 YAML recipes, 30 import contracts. The suite collects 2,959 unit and recipe tests, plus 109 end-to-end tests that run separately. All 17 recipes have at least one end-to-end test, and 34 of the 35 Atlas elements are exercised end to end.\nSelected end-to-end scores (synthetic documents with known ground truth, local 8B models):\nTask Engine Measured Invoice + delivery note vs. order reconciliation qwen3:8b document types and fields 14/14; expected discrepancies 5/5; 0 false discrepancies on the compliant case LLM JSON extraction (4 native invoices) qwen3:8b valid JSON 4/4; exact fields 28/28; all deliberately altered values caught by grounding OCR on a photographed label PaddleOCR word recall 0.94 on the raw photo; a contrast-enhancement defect (0.65–0.71) was found and fixed (0.88–0.94) Table reconstruction from OCR PaddleOCR 100 % exact cells (proportional font), 95 % (monospace) Limitations \u0026amp; next steps Verbatim grounding cannot catch a value that exists in the document but was put in the wrong field: an absent field was wrongly filled in 1 of 8 cases. The 8B model judged a +50 % quantity gap a \u0026ldquo;minor\u0026rdquo; discrepancy, against the instruction. LLM verdicts are therefore normalised and kept downstream of deterministic rules. Scores come mostly from synthetic data. A real-document test set is being built outside the repository, and only cases that have been reviewed are scored. Next steps: measure the automatic-reconciliation rate and the false-positive rate on production data. Stack Python · Pydantic · PyYAML · PaddleOCR · Ollama (qwen3:8b, qwen3-vl:8b) · Mistral / Anthropic / OpenAI APIs · pytest · mypy · import-linter · ruff\n","date":"2026-09-01T00:00:00Z","permalink":"/research/processing-atoms/","title":"Explainable Processing Atoms for Documents \u0026 Data"},{"content":"Context Process-observation consulting produces flowcharts, interview notes, screen recordings and recommendations. Once a mission is delivered, that material is effectively dead: the next mission starts from a blank page, even when the answer was already found elsewhere. The research question is how to turn deliverables from documents into typed, sourced, queryable data, so that an assistant can answer \u0026ldquo;where have we seen this before, and what did we conclude?\u0026rdquo; while citing verifiable evidence.\nTwo requirements structure the work: traceability (every assertion points to its proof) and a strict separation between evidence and knowledge (what was observed is not what was generalised). Retrieval-augmented generation is fast to build but cites passages, not checkable assertions; a formal ontology is traceable but expensive. This project explores a practical path between the two.\nApproach flowchart TB A[\"`Raw mission sources immutable, hashed`\"] --\u003e B[\"`Append-only event log source of truth`\"] B --\u003e C[\"`Rebuildable SQLite base disposable`\"] C --\u003e D[Deterministic extraction] C --\u003e E[\"`LLM extraction proposer only`\"] D --\u003e F[\"`Proposed nodes and edges with provenance`\"] E --\u003e F F --\u003e G{\"`Human review outside MCP`\"} G --\u003e|validated| H[Knowledge graph] H --\u003e I[MCP: evidence] H --\u003e J[MCP: knowledge] H --\u003e K[MCP: ontology] I --\u003e L[AI assistant] J --\u003e L K --\u003e LThe core design choice is to split the interface into three MCP servers by epistemic status: evidence (attested, sourced observations), knowledge (generalisations), and ontology (the meta-model itself). The split prevents an assistant from presenting a generalisation as an observation.\nflowchart TB subgraph SCHEMA[\"SCHEMA · naming\"] ON[\"`ontology server How do we name it?`\"] end subgraph PROVEN[\"PROVEN · observed\"] EV[\"`evidence server What did we capture, and where?`\"] end subgraph GENERAL[\"GENERALISED · concluded\"] KN[\"`knowledge server What do we conclude? proposes, never validates`\"] end ON -. \"types\" .-\u003e EV EV == \"grounds every conclusion\" ==\u003e KN EV --\u003e AI[\"`AI assistant answers labelled by status`\"] KN --\u003e AI ON --\u003e AIThe graph is single; only the façades are split. The boundary that matters runs between evidence and knowledge: the evidence server only returns what was observed and where, the knowledge server returns conclusions that must chain back to that evidence, and no MCP tool can mark anything as validated.\nMethods Meta-model. A fixed core plus mission profiles and a versioned ontology (v0). Each node carries an origin (structural, deterministic, LLM-extracted) and a status (proposed → validated). Invariants (I1–I6). Provenance is mandatory on every derived node and edge; a recommendation must chain back to a sourced finding; LLM-extracted quotes must be literal substrings of their fragment; \u0026ldquo;validated\u0026rdquo; requires a recorded human decision. Raw sources are immutable; the database is disposable and rebuilt from the log. Competency questions before implementation. A v0 set of 33 questions, typed as factual, aggregate, traceability, contradiction, temporal, cross-mission and negative (requests the system must refuse, e.g. cross-client leakage), in an executable format with expected answers and references. Evaluation protocol. At least 30 natural-language tasks, scored automatically or by a judge. Gate thresholds were fixed up front: accuracy ≥ 0.8, valid citations ≥ 0.98, grounding ≥ 0.95, leakage = 0, median ≤ 6 tool calls per task; each question solvable in ≤ 3 calls; deterministic extraction precision ≥ 0.95 per type; inter-annotator κ ≥ 0.7. Phases P0–P6 with human gates. Implementation is delegated to an agent session that must stop at each gate and log open decisions. Black-box software mapping. The same method was applied to a commercial business-management software, without reading its source code. The target is a triplet of user, software and database truths linked by proven edges. Eleven heuristics (H1–H11) cover value matching, inverse and direct perturbation, calculated fields, conditional visibility, scenario diffs, invariant mining, input fuzzing, semantic priors (never treated as proof), documentation-to-test and symptom-to-rule. Inputs: the menu tree via AT-SPI, a screen survey, column profiling, and an event typology from the application audit log. A hypothesis is accepted only with independent counter-evidence; vendor-produced maps are held in quarantine as a scoring oracle. An illustrative excerpt of the typed graph (labels are fictitious): the recommendation is only admissible because the chain down to a literal fragment of a source is complete.\nflowchart TB R[\"`Recommendation single entry point`\"] -- responds to --\u003e C[\"`Finding double data entry at goods receipt`\"] O[\"`Observation “stock is re-keyed by hand”`\"] -- attests --\u003e C O -- extracted from --\u003e F[\"`Fragment interview notes, lines 40–52`\"] F -- cites --\u003e P[\"`Source file hashed, immutable`\"] C -- concerns --\u003e A[\"`Activity goods receipt`\"] A -- uses --\u003e S[\"`Screen receipt form`\"] S -- module of --\u003e Y[\"`System business software`\"]Status \u0026amp; results This is mainly a design with partial implementation, not a system in service.\nPilot (observation corpus, deterministic pass). 6,195 files (6.92 GB) acquired idempotently; 13,272 nodes and 6,022 edges extracted in 2.1 s at zero LLM cost; all six invariants green over 57,969 checks. Gold-set measurement. 145 annotation units plus 100 entity-resolution pairs. Human annotation was partial and the second annotator was an LLM. Human × LLM agreement: κ = 0.72 (software), 0.41 (screen), 0.31 (category), 0.52 (segment boundaries). Extractor precision against human reference: 0.74, 0.43, 0.04, 0.63. Gate G2 (≥ 0.95) not passed; the category result reflects a definitional mismatch, not a computation error. Software mapping. Schema of 427 tables with 0 declared foreign keys, triggers or views (all logic is applicative); 1,048 menu entries (899 leaves) mapped; 154 of 899 screens surveyed (17 %); 2,845 rule hypotheses reduced to 1,519 by a local LLM filter, with 1 decision error in a 50-item sample reviewed by the supervising agent. A read-only MCP prototype exists; no rule has yet been proven in the software. Work paused after the host VM was overloaded. Gold-set measurement against the gates fixed in advance: bars are the measured values per field, the orange line is the gate. Only one bar clears its gate.\n--- config: xyChart: width: 540 height: 320 themeVariables: xyChart: backgroundColor: \"transparent\" plotColorPalette: \"#3F86B8, #C46A2B\" --- xychart-beta title \"Human × LLM agreement κ (gate 0.7)\" x-axis [\"software\", \"screen\", \"category\", \"segments\"] y-axis \"κ\" 0 --\u003e 1 bar [0.72, 0.41, 0.31, 0.52] line [0.7, 0.7, 0.7, 0.7]--- config: xyChart: width: 540 height: 320 themeVariables: xyChart: backgroundColor: \"transparent\" plotColorPalette: \"#3F86B8, #C46A2B\" --- xychart-beta title \"Extractor precision vs human reference (gate 0.95)\" x-axis [\"software\", \"screen\", \"category\", \"segments\"] y-axis \"Precision\" 0 --\u003e 1 bar [0.74, 0.43, 0.04, 0.63] line [0.95, 0.95, 0.95, 0.95]The software mapping, as a funnel: the menu tree is fully mapped, but screen coverage and proof lag far behind, and the hypothesis pipeline has not yet produced a single proven rule.\n--- config: xyChart: width: 540 height: 360 themeVariables: xyChart: backgroundColor: \"transparent\" plotColorPalette: \"#3F86B8\" --- xychart-beta title \"Black-box mapping: from inventory to proof\" x-axis [\"Menu entries\", \"Leaves\", \"Screens\", \"Hypotheses\", \"Filtered\", \"Proven\"] y-axis \"Count\" 0 --\u003e 3000 bar [1048, 899, 154, 2845, 1519, 0]The MCP servers and end-to-end evaluation (P4–P5) are not implemented.\nLimitations \u0026amp; next steps Pilot results are below threshold and rest on partial human annotation. Next steps: redefine the category label, complete a fully human reference, re-measure G1/G2, then build the three servers and run the pre-registered evaluation. For the software mapping, the write-perturbation heuristics await written authorisation and a dedicated VM.\nStack Python 3.11 · JSONL event log · SQLite · YAML ontology · MCP · AT-SPI · Ollama · pytest\n","date":"2026-09-01T00:00:00Z","permalink":"/research/mission-knowledge-graph/","title":"Mission Knowledge Graphs: From Consulting Evidence to an Ontology Queried Through MCP"},{"content":"Context A client operating under strong security constraints manages several hundred sites and runs four recurring back-office processes every day: maintaining field equipment, handling recurring spurious equipment events, writing intervention reports for its customers, and checking that each site\u0026rsquo;s periodic health check has arrived. The inputs are scattered across three systems with very different access paths: PDF reports delivered by e-mail, a business web portal with no public API, and Excel workbooks maintained by hand over many years.\nThe goal: automate these processes on an office workstation while keeping a human operator as the final authority on every customer message.\nDesign constraints No language model at run time. Every decision comes from explicit rules and every sentence from a template. This makes the robot\u0026rsquo;s output reproducible: the same inputs always produce the same drafts, so a new version can be compared word for word against production, and any change traces back to a rule or a data source. No customer data goes to an external model service. A vision-language model was used once, offline, during design, to map recorded operator screen sessions; it is not part of the deployed system. Drafts, not sends. The robot only creates Outlook drafts; the operator reviews and sends them. A global kill switch forces draft mode regardless of per-process settings. Local execution, no new infrastructure. Python on a Windows workstation, scheduled tasks, a shared NAS folder for state and configuration. No plaintext secrets. Credentials and tokens live in the Windows Credential Manager (DPAPI); configuration is YAML the operator can edit without a rebuild. Architecture flowchart TB SCHED[\"Scheduled passes (morning / afternoon)\"] --\u003e ING MAIL[\"Mailbox (Microsoft Graph, read)\"] --\u003e ING[\"Ingestion PDF reports grouped by day\"] ING --\u003e PARSE[\"PDF parser (pypdf + regular expressions)\"] PORTAL[\"Business web portal Playwright login → session token → JSON endpoints replayed\"] --\u003e RULES XLS[\"Reference \u0026 history workbooks (Excel)\"] --\u003e RULES PARSE --\u003e RULES[\"Rule engine 4 processes, ordered rules \u0026 exclusions\"] RULES --\u003e TPL[\"Template composer French agreement, names, dates\"] TPL --\u003e DRAFTS[\"Outlook drafts (Graph, never sent)\"] RULES --\u003e TRACK[\"Tracking workbooks (atomic append)\"] DRAFTS --\u003e OP([\"Operator reviews \u0026 sends\"]) OP -. \"sent items: corrected recipients\" .-\u003e RULES NAS[(\"NAS: locks, heartbeats, journal, idempotence ledger\")] --- RULES RULES --\u003e HEALTH[\"Health alerts webhook + dead-man check\"]The four processes Equipment maintenance. Fault lines parsed from the daily PDF reports are matched against each device\u0026rsquo;s battery and repair history. An ordered decision tree (first match wins) chooses between ignoring, waiting, battery replacement, device replacement or radio-range testing, and drafts a technician work order; each device on a multi-device order keeps its own decision.\nRecurring spurious events. Event counts per site and per device over 24 hours are compared with thresholds to detect recurring spurious events. Portal data (an incident confirmed as real, the system being switched off minutes after a burst) overrides the counts; the outcome is an advisory e-mail to the customer, a work order, or nothing.\nIntervention reports. For each on-site intervention, the robot rebuilds a single timeline from the portal\u0026rsquo;s event log, call log and field-staff records, handles events that straddle midnight, and drafts a customer report with the media files attached.\nPeriodic health checks. Sites whose periodic health check is missing are identified, the status of each connectivity channel is checked on the portal, and a customer e-mail, a technician work order or an internal note is drafted.\nComposing an e-mail: from parameters to a draft Each draft is the end of a long chain of explicit decisions. Counted in the source code (October 2026), the four rule engines have 53 terminal outcomes (each place where a rule returns a decision), mapped onto 37 decision types and 13 draft builders.\nThe customer advisory e-mail of the spurious-events process shows the scale. A site reaches it only after up to 14 ordered tests in the rule engine (16 terminal outcomes for this process alone). The composer then reads 21 inputs — 17 fields of the site\u0026rsquo;s decision record and customer record, the recipient list, the attachments, and two learned resources (zone names from the portal and a spelling dictionary learned from the operator\u0026rsquo;s own history workbook). Between the decision and the final text there are 108 conditional branch points (if statements, conditional expressions and boolean operands, counted on the syntax tree of the composer and the ten helpers it calls, before the HTML pass).\nflowchart TB IN([\"1 · Rule engine — 14 ordered tests, first match wins. Input: daily event reports · portal journal · customer record · learned site profile\"]) --\u003e T0 T0[\"Blocking flag on the site record?\"] -- yes --\u003e X0[\"No e-mail\"] T0 -- no --\u003e T1[\"R6 · incident confirmed as real today?\"] T1 -- yes --\u003e X1[\"No e-mail\"] T1 -- no --\u003e T2[\"R17 · media events \u0026lt; 30 s apart across zones?\"] T2 -- yes --\u003e X2[\"Manual review\"] T2 -- no --\u003e T3[\"R12 · external cause confirmed on media?\"] T3 -- no --\u003e T4[\"R8 · ≥ 8 integrity events?\"] T4 -- yes --\u003e T4b[\"persisting ≥ 3 days?\"] T4b -- yes --\u003e X3[\"Technician work order\"] T4b -- no --\u003e X4[\"Keep watching\"] T4 -- no --\u003e T5[\"≥ 2 integrity events, ≥ half of the burst?\"] T5 -- yes --\u003e X5[\"No e-mail\"] T5 -- no --\u003e T6[\"R16 · ≥ 140 events, advisory ignored ≥ 4 weeks? (off by default)\"] T6 -- yes --\u003e X6[\"Billing notice\"] T6 -- no --\u003e T7[\"R15 · shared-building flag?\"] T7 -- yes --\u003e X7[\"Coordination e-mail\"] T7 -- no --\u003e T8[\"R5 · weekend + chronic-device flag?\"] T8 -- yes --\u003e X8[\"Temporary masking request\"] T8 -- no --\u003e T9[\"R1 · same device ≥ 7 events?\"] T9 -- no --\u003e T10[\"R2 · ≥ 5 devices and ≥ 8 events?\"] T10 -- no --\u003e T11[\"R4 · ≥ 8 events, ≥ 50 % 05:00–07:00?\"] T11 -- no --\u003e T12[\"R3 · ≥ 8 events, ≥ 60 % 12:00–18:00, ≥ 70 % in 4 h?\"] T12 -- no --\u003e T13[\"R1bis · site total ≥ 7 events?\"] T13 -- no --\u003e X9[\"No e-mail\"] T3 \u0026 T9 \u0026 T10 \u0026 T11 \u0026 T12 \u0026 T13 -- yes --\u003e ADV ADV([\"2 · Composer — advisory e-mail, rule id carried along. Customer name: legal form removed, re-cased, acronyms kept\"]) ADV --\u003e SUBJ[\"≥ 30 events?\"] SUBJ -- yes --\u003e S1[\"Subject with urgent prefix\"] SUBJ -- no --\u003e S2[\"Plain subject\"] S1 \u0026 S2 --\u003e GR[\"Contact cell\"] GR -- \"several people, or a role / organisation word\" --\u003e G1[\"Generic greeting\"] GR -- \"title + surname\" --\u003e G2[\"Greeting + normalised title (7 abbreviations) + SURNAME\"] GR -- \"surname only\" --\u003e G3[\"Greeting + SURNAME\"] G1 \u0026 G2 \u0026 G3 --\u003e DEV[\"Devices involved\"] DEV -- one --\u003e D1[\"Portal zone name if known, else cleaned label + jargon map + learned accents\"] DEV -- several --\u003e D2[\"Generic phrase + list of ≤ 6 by count, each with its device code if known\"] D1 \u0026 D2 --\u003e HND[\"System switched on/off within minutes of the burst? (R1–R4 only)\"] HND -- \"off just after\" --\u003e C1[\"Handling-error wording A\"] HND -- \"on just before\" --\u003e C2[\"Handling-error wording B\"] HND -- no --\u003e ACT[\"Site in activity? — all of: R1–R4 or R1bis · ≥ 2 devices · ≥ 10 events · no intervention in the window · professional site · ≥ 90 % in learned hours · media sensors only: ≥ 90 % while on\"] ACT -- yes --\u003e C3[\"Site-in-activity wording\"] ACT -- no --\u003e RUL[\"Rule that fired\"] RUL -- \"R1, R2\" --\u003e C4[\"Generic environmental cause\"] RUL -- \"R3, R4\" --\u003e C5[\"Staff-passage cause\"] RUL -- R12 --\u003e C6[\"External cause confirmed on media\"] RUL -- R1bis --\u003e C7[\"Cause to investigate on site\"] C4 \u0026 C5 --\u003e ATT[\"Media actually attached?\"] ATT -- yes --\u003e A1[\"+ reference to the attachments\"] ATT -- no --\u003e FLG A1 \u0026 C1 \u0026 C2 \u0026 C3 \u0026 C6 \u0026 C7 --\u003e FLG[\"Site flag: frequent contractor works?\"] FLG -- yes --\u003e F1[\"Request: switch off before works\"] FLG -- no --\u003e F2[\"Request: treat the cause\"] F1 \u0026 F2 --\u003e REC[\"Addresses in the customer record?\"] REC -- yes --\u003e RC1[\"All addresses found\"] REC -- no --\u003e RC2[\"Internal fallback mailbox\"] RC1 \u0026 RC2 --\u003e DRAFT([\"Plain text → Outlook HTML (unfolding, lists, fixed typography, signature) → drafts folder, never sent\"]) classDef q stroke-dasharray:4 3,stroke-width:2px; class T0,T1,T2,T3,T4,T4b,T5,T6,T7,T8,T9,T10,T11,T12,T13,SUBJ,GR,DEV,HND,ACT,RUL,ATT,FLG,REC q;The wording space of this single e-mail: 2 subject forms, 3 greeting forms, 2 opening-sentence forms, 7 reachable cause wordings (two of them with or without a reference to attached media), 2 requested actions, and an optional device list of 2 to 6 lines. Every threshold above lives in the YAML configuration, not in the code.\nParameter Source Effect on the e-mail Event count over 24 h daily reports figure in the opening sentence; ≥ 30 adds an urgent prefix to the subject Rule that fired rule engine selects the cause wording; gates the handling-error and site-in-activity checks Devices involved (cleaned, de-duplicated) daily reports one named device, or a generic phrase plus a list of up to 6 lines ordered by count Device code per report line daily reports appended in parentheses to each listed device, only when known Zone name of the main device portal replaces the raw report label when available Learned spellings operator\u0026rsquo;s history workbook restores accents and casing of device names Title and contact-name cell customer record one of 3 greeting forms; several people, a role or an organisation name fall back to the generic form Company name customer record legal form removed, re-cased with acronyms kept, used in subject and body Site type, learned opening hours, interventions in the window tracking workbook, portal journal enable or suppress the site-in-activity wording System switched on/off around the burst portal journal handling-error wording (2 variants), overriding every other cause Site flags customer record choice between the 2 requested actions Attachments present portal media reference to the attachments, or nothing Recipient addresses customer record every address found; otherwise an internal fallback mailbox The other drafts draw on the same shared French-language helpers: the preposition before a site name agrees with its first noun (6 forms, from chez to à l\u0026rsquo; and aux); report titles agree in gender and number; dates are written in full French form; a \u0026ldquo;Contact\u0026rdquo; heading becomes plural when several contacts are listed; and an 8-entry map turns technical equipment jargon into customer vocabulary.\nEngineering for reliability Idempotence. A daily ledger keyed on site, message type and subject hash prevents duplicate drafts; a cross-day register prevents re-issuing a work order before the previous intervention has happened. Missed days are caught up automatically. Concurrency on a shared NAS. Exclusive lock files (O_CREAT|O_EXCL) with a time-to-live for orphans; two workstations publish heartbeats, and the secondary takes over only when the primary has been silent for two minutes. Safe spreadsheet writes. Writes go through a temporary file and an atomic replace, with retries when the workbook is open by a user; inserted rows shift footer formulas so totals stay correct. Network robustness. Graph calls handle token refresh, HTTP 429 Retry-After and 5xx back-off. Explicit failure. A crash wrapper with no dependency on the main package sends a redacted alert and a dead-man ping even when imports fail; a sentinel raises an alert when the portal returns no data at all instead of reporting success. Shadow benchmarking. A sandbox build replays a day in read-only mode and diffs its drafts against production. The acceptance metric is the share of drafts the operator sends without changing a single word. Results 1,766 automated tests in the repository (October 2026), built on mock mail and portal adapters, with live-system tests marked separately. An audit of the corpus of received reports showed that 9.5 % of event lines were being silently discarded, because event codes and labels are concatenated in the extracted PDF text; the parser was rewritten to recognise the code forms actually emitted. Caching the fault-history workbook (1.49 MB, 15,533 rows) cut the equipment maintenance process on a replay of real reports from 161.1 s to 23.3 s; workbook opens fell from 26 to 3. First clean shadow comparison against production: 14 drafts on each side, 14 in common, 0 missed, 0 added. Stack Python 3.11 · Microsoft Graph · MSAL · httpx · Playwright · pypdf · openpyxl · pydantic · keyring · pytest · PyInstaller\n","date":"2026-06-01T00:00:00Z","permalink":"/research/deterministic-rpa/","title":"Deterministic Process Automation Under Strong Security Constraints"},{"content":"Context Safety recall campaigns depend on reaching every affected vehicle, but letters and phone calls leave a residual population of owners who never respond. In island and overseas territories the problem is sharper: network coverage is uneven, field logistics span several territories, and the registry of who drives what is incomplete. The practical question becomes a fleet-management one: which vehicles of the circulating fleet are still affected, where are they seen, and how far has each one progressed towards repair?\nApproach flowchart TB A[\"Recall listing(scheduled file pull or manual CSV)\"] --\u003e B[\"Central serverimport · versioning · model-name decoding\"] B --\u003e|\"incremental diff\"| C[\"Field units (laptops in agent vehicles)local recall copy\"] D[\"Camera stream\"] --\u003e E[\"Plate detection + OCR(on-device, offline)\"] E --\u003e F[\"Format-aware correction+ double-read confirmation\"] F --\u003e G{\"Match againstlocal recall list?\"} G --\u003e|\"no\"| H[\"Discarded — nothing stored\"] G --\u003e|\"yes\"| I[\"Alert to agent+ local queue\"] I --\u003e|\"sync when network returns\"| B B --\u003e J[\"Dashboardunits · tours · detections · incidents\"] J --\u003e K[\"Owner contact workflowidentified → contacted → repaired\"]Methods Data ingestion and versioning. The recall listing is pulled automatically on a schedule or imported by hand. Each import is diffed against the previous one with stable identifiers, versioned by hash, and only the delta is sent to the field units. Manufacturer type codes are translated into readable model names on import.\nMatching under OCR noise. Plate detection and character recognition run on the field laptop with ONNX models, with no network dependency. Two safeguards reduce false matches: (i) a positional correction step uses the known structure of national plate formats (current and legacy) to resolve classic OCR confusions (O/0, I/1, S/5, B/8, Z/2, G/6), rejecting any ambiguous reading; (ii) a double-read confirmation only counts a plate when it is read twice within a short time window. A smartphone web page offers a manual fallback (photo or typed plate, plus VIN check-digit validation).\nOffline-first synchronisation. Matches are queued locally and pushed to the central server on the next connection, so detections survive multi-day outages. Units report a heartbeat (software version, recall-list version, reading counters).\nField operations. Supervisors plan tours as ordered, geolocated checkpoints on a map, assign them to teams and units, reuse templates, and follow progress. Agents report incidents with a severity level. A contact workflow records the outcome of each owner approach, and the dashboard derives a conversion funnel (identified → approached → engaged → repaired) with weekly trends, the median delay to repair and a breakdown by territory.\nPrivacy by design. Plates that do not match the recall list are never stored or transmitted; retained records have fixed retention periods with automatic purge; actions are written to an audit log; dashboard access is role-based, including read-only accounts; data at rest is encrypted. A processing register was written alongside the code.\nDeployment \u0026amp; operations (DevOps) The system runs on two very different kinds of machine: Windows laptops in agents\u0026rsquo; vehicles, which may stay offline for days, and one server per instance. After the first weeks in the field, most of the engineering work went into making both sides easy to deploy, observe and recover, by a single person, without breaking field work.\nflowchart LR subgraph ADM[\"Admin workstation\"] GIT[\"Git repositorycommitted tree only\"] PROV[\"Provisioning scriptone config file per unit\"] BLD[\"Field buildPyInstaller bundle + ONNX models\"] end subgraph SRV[\"Server (Ubuntu LTS VPS)\"] CAD[\"CaddyTLS · HSTS · no access log\"] APP[\"Docker containerFastAPI API + dashboardloopback port only\"] VOL[(\"LUKS2 volumeSQLite · photos · archives\")] TIM[\"systemd timersmonitoring · backup · cleanup\"] end FLD[\"Field laptopslocal SQLite · sync loop\"] OBJ[(\"Off-site object storageage-encrypted backups\")] CHAT[\"Chat alertsstate changes + daily digests\"] GIT --\u003e|\"git archive over SSH → deploy script\"| APP PROV --\u003e|\"admin API\"| APP PROV --\u003e|\"config file + README\"| FLD BLD --\u003e|\"archive + one-click installer\"| FLD FLD \u003c--\u003e|\"HTTPS: heartbeat · pushes · recall diff\"| CAD CAD --\u003e APP APP --- VOL TIM --\u003e VOL TIM --\u003e|\"encrypted copy\"| OBJ TIM --\u003e CHATField laptops: packaging, installation and updates Packaging. The field application (Python backend, React interface) is frozen with PyInstaller into a one-folder Windows executable that embeds the ONNX models and the OCR configuration, so a unit needs no Python install and no network to read plates. Built-in --diag and --selftest modes check the bundled runtime and read a test image without starting the interface, which made start-up failures diagnosable on the field machine itself.\nOne-click installer. The build is delivered as an archive with a double-click INSTALLER.bat. It elevates itself and runs a PowerShell script that removes the \u0026ldquo;downloaded from the Internet\u0026rdquo; mark from the files (otherwise the bundled runtime fails to load), disables USB power saving so the GPS dongle is not suspended, and installs the unit\u0026rsquo;s configuration file where the executable reads it. The script refuses to guess: if several different configuration files are present, it lists them and asks which one is the new one. An optional reset clears the local database when a unit moves to another server; it shows the target server before asking for confirmation and warns that unsynchronised detections would be lost. The unit\u0026rsquo;s access token is never displayed. An update follows the same path: new build, re-run the installer.\nProvisioning. A Python script that uses only the standard library, so it runs on a bare Windows machine, creates the units from a CSV file through the server\u0026rsquo;s admin API and generates, for each unit, its configuration file (server address, unit code, display name, token) and a README to hand over with the laptop. The admin key is read from an environment variable or a masked prompt, never from a command-line argument.\nVersioning: software and recall list Software version. Each unit reports its application version in its heartbeat. The dashboard compares it with the expected release and flags the units that still need updating.\nRecall-list version. The recall list is versioned by hash. On each cycle, a unit compares its local hash with the server\u0026rsquo;s: identical, nothing to do; no local copy, full download; otherwise, the server returns only the rows added and removed since that hash (and the unit falls back to a full download if the server no longer knows its version). The change is applied in a single transaction, and the local hash only moves forward if everything was applied.\nThat last rule came from a field incident. A database constraint made the diff fail on one unit, the error was swallowed by the sync loop, and the unit kept sending heartbeats while its recall list stayed frozen for more than two weeks, so it alerted on vehicles that had already been repaired. Release 1.4.1 fixed the cause, added an explicit rollback, and shipped a schema migration that keeps existing data. It was tested with a 62-test run that included reproducing the bug and migrating a database with the old schema. Monitoring now raises a \u0026ldquo;recall sync stuck\u0026rdquo; alert when a unit has not received a published list after a grace period.\nIncremental, resilient synchronisation Every five minutes, or on demand, a unit runs a sync cycle made of independent steps: heartbeat, push of detections (in batches), GPS track, incidents, checkpoint updates, then pull of the recall list and the tours. Pushes come before pulls, so field data leaves first. A failing step no longer blocks the others: in an earlier version the cycle was monolithic, and a failed heartbeat once delayed detections by 22 minutes. Requests use short timeouts, downloads longer ones. The heartbeat carries the software version, the recall-list version, the number of pending detections and the reading counters, which is what the dashboard and the monitoring rely on.\nServer deployment Instance kit. Each deployment is an independent instance described by a short identifier, on an Ubuntu LTS virtual server. A bootstrap script, which is interactive and can be safely re-run, installs the system packages, Docker and Caddy from their official repositories. It enables unattended security updates without automatic reboot, restricts SSH to key authentication, enables a default-deny firewall and brute-force protection, and checks that the DNS record points to the server before applying the TLS configuration. Secrets live in root-only configuration files on the server, never in the repository.\nContainer and reverse proxy. A multi-stage Dockerfile builds the React dashboard with Node, then runs the FastAPI server on a slim Python image as an unprivileged user. Its health check allows a three-minute start period because the plate and OCR models are loaded before the server listens. The container is published on the loopback interface only. Caddy is the only public entry point: automatic Let\u0026rsquo;s Encrypt certificate, explicit HSTS, and no access log, because scan URLs contain plates. The application\u0026rsquo;s own access log is disabled for the same reason.\nDeploy and rollback. Code leaves the admin workstation only as a git archive of a committed tree: uncommitted changes are listed and not sent, and line endings are forced to LF. On the server, the new tree is unpacked next to the current one and swapped in, the previous one is kept, and the deployed commit is recorded. The deploy script then detaches from the SSH session, so a dropped connection interrupts neither the deployment nor a rollback. A lock prevents concurrent deployments. The script builds the image (labelled with its commit), keeps the previous image as rollback, recreates the container and waits until it reports healthy. It gives up after five minutes, or as soon as the container has restarted twice. On failure it rolls back automatically to the previous image and source tree, sets the failed code aside and sends an alert. Going back to any earlier version means re-deploying that commit. An image rollback does not undo a schema migration, so the documented procedure is to take a backup before any deployment that changes the schema.\nEncrypted volume and safe restarts Server data (database, contact photos, recall archives) lives on a LUKS2 file container mounted with restrictive options. Its passphrase is never stored on the machine, so after a reboot a person unlocks the volume. The container restarts on its own, but the server refuses to start unless a sentinel file on the encrypted volume is present. Without it, a reboot would have left an empty directory in place of the volume, and the application would have created a fresh, unencrypted database there and accepted field data into it. The guard was checked both ways on a throwaway container. A separate script grows the volume while it is mounted.\nBackups and recovery A daily systemd timer takes a consistent SQLite snapshot through the database\u0026rsquo;s backup API, archives it with the photos and recall archives, and encrypts the result with age to a public key. The private key is shown once, at creation, and kept off the server. Backups are kept locally and, optionally, copied to S3-compatible object storage with rclone, configured through environment variables only. A restore test mode decrypts a backup into a temporary directory on the encrypted volume, runs an integrity check, counts rows per table, then deletes everything. The apply mode moves the current state aside instead of deleting it. The documentation includes a step-by-step rebuild after total loss of the server. The continuity plan targets a recovery time of 4 hours and a maximum data loss of 24 hours.\nMonitoring A watchdog runs every 15 minutes and only posts to a chat webhook when a state changes (problem, then \u0026ldquo;resolved\u0026rdquo;). A fuller digest is sent twice a day, timed to the field shifts. It checks that the encrypted volume is mounted, the API health endpoint (with retries, because the daily import can block it for about a minute), the container health status, that the database is readable, disk usage on both the system disk and the encrypted volume, the size of the SQLite write-ahead log, TLS certificate expiry, backup age and size, units silent for more than 48 hours (not 24, since units are switched off at weekends) and units whose recall list is stuck. A separate watcher alerts when no recall import has happened for more than 30 hours. Several of these checks were added after real incidents: the encrypted volume once reached 90 % without any alert because only the system disk was watched. A weekly timer cleans up Docker build caches and keeps only the current and rollback images. A dedicated script truncates the SQLite log with a short lock wait, so field writes are never blocked for long.\nOffline resilience The field side is designed to keep working through a server outage: plate reading, matching against the local recall copy, alerts, tours already downloaded and the local statistics tab all work without a network, and the queue empties at the next connection. Only the dashboard (and the server-hosted smartphone page) is unavailable while the server is down.\nTesting There is no CI pipeline in the repository. Releases are tested before they ship, and the commits record those runs: 12/12 and 6/6 for the plate correction and confirmation stages, 62 tests for release 1.4.1. Behaviours that cannot be unit-tested, such as the volume guard or a live deployment, were checked on throwaway containers and against the production health endpoint.\nResults In early field operation, agents flagged 32 % of detections as misreadings over roughly one week. This measurement motivated the positional-correction and double-read confirmation stages, and the dashboard statistics now exclude detections marked as false positives. The new stages shipped with unit tests: 12/12 for plate correction and 6/6 for confirmation, settings and reading log. After the model-name mapping was extended, only one row of the production recall listing remained without a decoded model name. No end-to-end recognition rate has been measured yet.\nLimitations \u0026amp; next steps Recognition accuracy has not been measured against a ground-truth benchmark. The specification targets (read rate, latency) are design goals, not results. No legacy-format plate reading was observed at first. A dedicated counter now tracks these readings end to end, but the fix still needs a field check. A false match against a large listing still costs agent time. Calibrating the confidence threshold and confirmation window on labelled field video is the next step. Stack Python · FastAPI · SQLite · ONNX Runtime · fast-alpr (YOLOv9 plate detector + OCR) · OpenCV · PaddleOCR · React · TypeScript · Vite · Leaflet · Docker · Caddy · LUKS · systemd · age · rclone · PyInstaller · PowerShell\n","date":"2026-06-01T00:00:00Z","permalink":"/research/fleet-management/","title":"Field Identification of Recall-Affected Vehicles"},{"content":"Context A conference recorded on several microphones (speakers, room, music) produces hours of raw multitrack audio. Most editing time goes into locating material — who speaks when, on which track, where the silences are — before any cutting.\nOpen-source speech tooling (Whisper and its derivatives, pyannote, Silero VAD) solves transcription and \u0026ldquo;who spoke when\u0026rdquo; well, but stops at text. The question addressed here is whether that analysis can be delivered as an editing object: a multitrack timeline in AAF, the interchange format that audio workstations open natively, with tracks qualified and dead time removed.\nBefore → after One real job processed end to end: a conference recorded on 16 tracks. Everything on the right-hand side was produced automatically, with no manual cut; figures are read from the job\u0026rsquo;s analysis database and from the delivered AAF.\nInput (raw recording) Generated output (AAF timeline) Tracks 16 × 1 h 32 min 54 s (≈ 24 h 46 min of audio to review) 16 tracks kept in the AAF, 6 carrying clips Track roles 4 speech · 2 music (one auto-flagged as ambience) · 10 silent 10 silent tracks left empty, no manual triage Material 204 speech regions and 5 music passages detected 41 clips (37 speech, 4 music) Cuts — 40 hard cuts between sources, of which 33 microphone switches Transitions — 0 crossfades: straight cuts, each speech clip extended by a 0.3 s tail Dead time — 3 gaps ≥ 4.5 s rippled out (36.9 s); timeline 1 h 32 min 17 s Top: the 16 input tracks over the full recording, coloured by the detected role and activity; the four speech microphones are active almost all the time because each one also picks up its neighbours. Bottom: the generated edit, one lane per track as in the AAF — a single source is kept at any moment, ticks mark the 40 source switches, hatched areas in the top panel are the removed dead time.\nSpeech clips are few and long (median 79 s, longest 10 min 11 s): the montage follows the active speaker rather than chopping on every breath, and micro-switches shorter than 0.8 s are absorbed by the anti-flicker smoothing. Ten clips are still shorter than 6 s — short interjections, the first place to look when checking the edit.\nApproach flowchart TB A[\"Multitrack WAV recording\"] --\u003e B[\"Decode to 16 kHz mono + RMS energy per 100 ms\"] B --\u003e C[\"Voice activity detection (Silero)\"] C --\u003e D[\"Transcription (Whisper large-v3, chunked)\"] D --\u003e E[\"Forced word alignment (WhisperX / wav2vec2)\"] C --\u003e F[\"Speaker diarisation (pyannote)\"] E --\u003e G[\"Speaker assignment + segment post-processing\"] F --\u003e G B --\u003e H[\"Track role: speech / music / silent\"] G --\u003e H H --\u003e I[\"Montage: active microphone per 100 ms frame\"] I --\u003e J[\"AAF export (linked to original WAV names)\"] I --\u003e K[\"Mono mixdown WAV\"]Methods Transcription. Whisper large-v3 via faster-whisper (CTranslate2), run on fixed-length chunks inside an isolated, persistent worker process: the model is loaded once, and a native crash only loses one chunk, which is reported rather than silently dropped. Decoding uses beam search with Silero VAD filtering and condition_on_previous_text disabled to limit repetition loops.\nWord timing and speakers. Word-level timestamps come from WhisperX forced alignment (wav2vec2). Speaker turns come from pyannote diarisation and are assigned to words by temporal overlap.\nPost-processing. Pure filters remove known Whisper hallucinations (stock subtitle credits, \u0026ldquo;thanks for watching\u0026rdquo;-type phrases, music-only segments), non-Latin output on French audio and duplicates, then merge consecutive same-speaker segments.\nTrack qualification and montage. Each track is classified as speech, music or silent by combining frame energy, VAD coverage and transcript statistics. The montage selects, for every 100 ms frame, the microphone with the highest signal-to-noise ratio above its own noise floor (reducing cross-talk \u0026ldquo;bleed\u0026rdquo;), applies anti-flicker smoothing and ripples out global dead time.\nSynchronisation. Separate recorders are aligned by matching distinctive words across transcripts (robust median of offsets), after a coarse clock-based placement corrected for clock drift.\nExport. The edit list is written as a multitrack AAF with pyaaf2, either embedding the audio or linking to it.\nDeployment. A web portal accepts jobs and a separate compute worker polls it over HTTPS. Clients upload only compressed analysis copies together with a manifest of their original files; the delivered AAF references the original WAV names, so the editor relinks full-quality sources locally and the originals never leave the client\u0026rsquo;s machine.\nResults Benchmark (bench_stt.py) on real conference audio: one-minute speech windows selected by VAD on four speech tracks, Whisper large-v3 throughout, Apple Silicon laptop, warm timings excluding model loading. No ground-truth transcript was available, so quality figures are divergence from a pseudo-reference (WhisperX output), not absolute error.\nPipeline (CPU) Real-time factor openai-whisper 0.43 WhisperX 0.51 WhisperX + forced alignment 0.54 openai-whisper + word timestamps 0.57 faster-whisper (beam 5 + VAD) 0.66 faster-whisper + word timestamps 0.70 Word timing: native word timestamps deviate from forced alignment by a median of ~205 ms (faster-whisper) and ~210 ms (openai-whisper); forced alignment adds about 1.2 s per minute of audio. Transcript divergence: 7–11 % word-level divergence between pipelines, mostly punctuation, segmentation and normalisation on manual inspection. Repetitions: 1–2 % repeated content; no hallucinated passages observed on these clips. Track detection: the three music/silence tracks of the seven-track recording were identified by VAD and skipped automatically. Limitations \u0026amp; next steps Quality is measured against a pseudo-reference; a manually transcribed test set is needed to report true word error rate and diarisation error rate. Benchmark clips are short (one minute per track) and all single-speaker windows, so diarisation accuracy on multi-speaker overlap is not yet quantified. CTranslate2 has no Metal backend: on Apple Silicon, faster-whisper and WhisperX run on CPU, while CUDA is used where available. Device choice per stage was benchmarked rather than assumed; a CUDA comparison remains to be measured. AAF export targets Logic Pro import; other editors have not been systematically tested. Stack Python · faster-whisper · CTranslate2 · WhisperX · pyannote.audio · Silero VAD · PyTorch (MPS / CUDA) · ffmpeg · pyaaf2 · SQLite · PHP · MySQL\n","date":"2026-06-01T00:00:00Z","permalink":"/research/speech-to-edit/","title":"From Multitrack Recordings to an Editable Timeline: Transcription, Diarisation and AAF Export"},{"content":"Context Industrial sites are often covered by fixed cameras whose footage is only reviewed after an incident and yields no usable data. The aim was to turn existing cameras, and occasional drone footage, into a measuring instrument: for every detected object, a latitude/longitude position and metric distances rather than a pixel box.\nThis is hard for three reasons. A single camera gives no depth, so scene scale must be inferred. The usual georeferencing method, a ground-plane homography, assumes a flat floor and fails silently for raised objects or uneven terrain. And high-resolution video is expensive, while the common shortcut of downscaling to 640 px discards the geometric detail ground positions depend on.\nApproach flowchart TB A[\"Video sourcefile · RTSP · drone clip (up to 3840×2160)\"] --\u003e B[\"Object detector at native resolutiondetect · segment · pose · OBB\"] A --\u003e C[\"Metric monocular depth modeldepth map\"] B --\u003e D[\"Multi-object trackermotion model + association\"] B --\u003e E[\"Per-object depthmask pixels, outlier rejection\"] C --\u003e E E --\u003e F[\"Ray castingpixel ray + metric depth + camera pose\"] F --\u003e G[\"WGS84 positionoptional DEM for ground elevation\"] E --\u003e H[\"3D distancesbetween masks · along trajectories\"] D --\u003e H G --\u003e I[\"SQLite storedetections · tracks · GPS trajectories\"] H --\u003e I I --\u003e J[\"Analyticszones · flows · orthophoto overlay\"]Methods Detection. A YOLO-family detector, at several model sizes, in four modes (detect, segment, pose, oriented boxes). Inference runs at native resolution with no forced imgsz. Segmentation is the default because masks give cleaner depth samples than boxes. PyTorch, ONNX, TensorRT and CoreML weights are accepted (CUDA and Apple MPS). Monocular depth. A metric monocular depth model predicts a depth map, which is resampled to the frame size. Each object\u0026rsquo;s depth is a robust estimate over its mask pixels, after outlier rejection. A validation step compares predicted depth with reference zones whose distance is known, to detect calibration drift. Geolocation by ray casting. A ray is cast from the camera centre through the object\u0026rsquo;s pixel and stopped at the object\u0026rsquo;s metric depth. The resulting point is converted from the camera frame to a local North-East-Down frame, then to WGS84. Unlike a homography, this does not assume a flat ground. An optional DEM supplies ground elevation. Positions can be overlaid on a georeferenced orthophoto for visual checks. Tracking. A custom multi-object tracker with a motion model, confidence-aware association and a track-fusion step that links fragments using gap, heading and speed limits. When depth is available, each track stores a 3D trajectory, from which the distance travelled and the speed in m/s are derived. Inter-object distances. Later consolidated into a reusable vision library. Two segmented objects are back-projected into 3D, and the minimum point-to-point distance is reported. Overlapping masks are treated as contact. A Monte Carlo simulation perturbs depth, focal length and pixel positions to give a standard deviation and a 95 % interval. Zone membership uses the share of the mask inside the zone, with hysteresis, rather than a centre-point test. Engineering. The configuration supports a two-GPU split (detection on one device, depth estimation on the other), RTSP discovery and recording, and processing of several cameras in parallel. Results Measured on a single NVIDIA RTX 5090. The detection benchmark covers detection and tracking only, with depth disabled.\nSetting Video Throughput Output Largest detector, batch 1 3840×2160, 50 fps, 3,301 frames 5.1 fps 320,217 detections, 1,941 tracks Smallest detector, batch 1 3840×2160, 50 fps 13.0 fps 158,989 detections, 908 tracks Largest detector, batch 1 2304×1296, 15 fps, 5,000 frames 17.9 fps 73,502 detections, 2,706 tracks Smallest detector, batch 1 2304×1296, 15 fps 39.0 fps 51,890 detections, 1,499 tracks Full chain (detection + depth + tracking + storage) 2304×1296, 300 frames 2.02 fps 12.3 detections per frame In the full-chain run, depth estimation took 117.6 s of the 148.3 s total (about 80 %), and detection took 20.9 s. Depth estimation, not the detector, is the bottleneck. On 4K footage the largest detector runs about ten times slower than real time. That suits the intended use, which is analysis of recorded archives, not live alerting.\nLimitations \u0026amp; next steps No geolocation accuracy has been measured against surveyed ground truth yet. The ray-casting stage was added after the timing runs above, so its cost is not included in them. The metric scale depends on the depth model and on the camera pose and intrinsics. Errors in tilt or focal length propagate directly into ground positions. The depth model predicts at its own internal resolution, so fine depth detail is limited even when detection runs in 4K. Next steps: an accuracy study with GNSS-surveyed targets, profiling the two-GPU split, and measuring how accurate Monte Carlo distance intervals are on objects at known separations. Stack Python · PyTorch · OpenCV · NumPy · SciPy · pyproj · rasterio · SQLite · CUDA · Apple MPS\n","date":"2026-01-01T00:00:00Z","permalink":"/research/video-geolocation/","title":"Geolocated Video Analysis and Inter-Object Distance Measurement"},{"content":"Context Detect gas and hydrocarbon pollutants from a drone, day and night, to monitor mobile pollution sources (vessels, machinery) in maritime and industrial zones.\nPipeline flowchart TB A[\"Drone — thermal/IR + visible\"] --\u003e B[\"Day / night acquisition\"] B --\u003e C[\"Pollutant signature detection\"] C --\u003e D[\"Localisation on moving targets\"] D --\u003e E[\"Maritime / industrial monitoring\"]Methods Combined thermal/infrared and visible-spectrum imaging. Detection and localisation of pollutant traces on moving targets. Operational in both daytime and nighttime conditions. Results Detection of fuel leaks and hydrocarbon discharge in port areas and coastal waters. Continuous monitoring capability for maritime and industrial zones. Stack Python · OpenCV · thermal/IR imaging · drone\n","date":"2024-07-01T00:00:00Z","permalink":"/research/pollutant-detection/","title":"Pollutant Detection by Drone (Day \u0026 Night)"},{"content":"Context Sargassum strandings disrupt Caribbean coastlines. The goal: detect algae rafts at sea early and predict where and how much will strand, so municipalities and environmental authorities can plan cleanup — and interventions at sea — in advance.\nThe problem: raw reflectance is hard to read In raw multispectral satellite imagery, Sargassum is a faint signal on a large, variable ocean background. Rafts are thin, fragmented and often low-contrast: their edges blur into the surrounding water, and the overall structure of a band — where it starts, where it thins out, where it breaks up — is difficult to make out by eye or with generic indices.\nDetecting that there is Sargassum is not enough. To act, one needs to see what it looks like.\nCore contribution: an original contrast indicator I designed a dedicated contrast indicator for Sargassum. Rather than simply flagging pixels, it is built to give a very clear reading of the data and of the shapes: rafts and bands stand out sharply from the water, with well-defined contours and their internal structure preserved. (Formulation details withheld for now.)\nWhat it brings:\nClear delineation of rafts and of elongated Sargassum bands. A readable picture of how the mass is organised, not just where it is. A cleaner input for segmentation, quantification and drift modelling downstream. Why shape matters The geometry of the bands is the key information for operations:\nDispersion — is the Sargassum concentrated in a dense raft or scattered in many small patches? Trailing (\u0026ldquo;traîne\u0026rdquo;) — how the band stretches and elongates behind its head, revealing how it is being pulled by currents and wind. Orientation and drift direction — which way the band is heading, and therefore which stretches of coast are exposed. Reading these shapes is what makes it possible to plan interventions on site (where to deploy collection at sea or barriers) and to anticipate strandings rather than react to them.\nPipeline flowchart TB A[\"Satellite imagery — multispectral\"] --\u003e B[\"Cloud / water masking\"] B --\u003e C[\"Original Sargassum contrast indicator\"] C --\u003e D[\"Raft \u0026 band delineation — shape, dispersion, trailing\"] D --\u003e E[\"Quantification at sea\"] E --\u003e F[\"Drift / stranding model\"] F --\u003e G[\"Alerts \u0026 intervention planning\"]Methods Multispectral satellite imagery analysis. Original contrast indicator for a sharp reading of Sargassum rafts and bands. Shape analysis of the bands: dispersion, elongation/trailing, orientation. Raft detection and surface-area quantification at sea. Predictive drift/stranding modelling to anticipate landing zones and volumes. Results Clear, readable maps of Sargassum rafts and bands at sea. Early-warning capability for coastal municipalities. Quantitative stranding forecasts supporting proactive cleanup and intervention planning. Visuals and further details will be added.\nStack Python · satellite imagery · spectral analysis · predictive modelling\n","date":"2024-06-01T00:00:00Z","permalink":"/research/sargassum-detection/","title":"Satellite-Based Sargassum Detection \u0026 Stranding Prediction"},{"content":"Context Non-destructive structural health monitoring for bridges, buildings and infrastructure. The challenge is not only to detect cracks, but to classify them by width — without contact or scaffolding.\nWidth is what matters for inspection: depending on how open a crack is, the repair (filling) method is not the same. An inspection system therefore has to be very precise, because a wrong width class means a wrong repair specification.\nFour width classes Class Crack width Consequence 1 \u0026lt; 0.3 mm Finest cracks — often narrower than a pixel; dedicated repair approach 2 0.3 – 0.7 mm Different repair approach 3 0.7 – 1 mm Different repair approach 4 ≥ 1 mm Different repair approach Each class calls for its own filling/repair approach, so the classification directly drives the maintenance decision.\nThe sub-pixel challenge The key difficulty is to go below the pixel size. A 0.3 mm crack does not even cover a full pixel at typical inspection resolutions — and the system must still detect it, even when the ground sampling distance is somewhat coarser than 0.3 mm.\nWhy is this possible at all? A crack thinner than a pixel does not vanish: it still occupies part of the pixels it crosses, and slightly alters their intensity compared with the surrounding surface. Taken individually, that partial-pixel signal is weak; but a crack is a continuous, elongated feature, so the same faint contrast repeats coherently along many adjacent pixels. Exploiting that contrast and continuity is what allows detection — and width estimation — beyond the nominal resolution of the image.\nAcquisition: drone and smartphone Drone photography — high-resolution UAV capture of structural surfaces, for large or hard-to-reach structures. Smartphone photography — the approach also works on images taken with a mobile phone, which I have tested, making close-range inspection possible with everyday equipment. Pipeline flowchart TB A[\"Image capture — drone or smartphone\"] --\u003e B[\"Image tiling \u0026 calibration\"] B --\u003e C[\"Crack segmentation — CV / DL\"] C --\u003e D[\"Sub-pixel width estimation\"] D --\u003e E[\"Width classification — 4 classes\"] E --\u003e F[\"Defect map \u0026 inspection report\"]Methods High-resolution capture of structural surfaces by drone or smartphone. Computer-vision / deep-learning crack segmentation on concrete and masonry. Metric calibration and sub-pixel analysis for crack width estimation. Classification into four width classes matched to repair methods. Results Non-destructive, repeatable inspection of civil structures. Geolocated defect maps supporting maintenance prioritisation. Models and visuals will be added.\nStack Python · OpenCV · deep learning · high-res RGB drone · smartphone imagery · photogrammetry\n","date":"2024-03-01T00:00:00Z","permalink":"/research/crack-detection/","title":"Sub-Millimetre Crack Detection in Civil Structures"}]