Provenance on Boris Dupoux

Explainable Processing Atoms for Documents & Data

Tue, 01 Sep 2026 00:00:00 +0000

Context

Production systems for document reconciliation and OCR, built for clients such as a wholesale distributor and an accounting firm, left several codebases that each solved the same sub-problems (page routing, OCR tuning, LLM extraction, field comparison) with their own bugs. The atoms library consolidates them into one package. Its central question is the one behind the original systems: can a document check be automated so that a human can audit and contest every verdict? Extracting a field is not enough. The system must state whether two documents agree, why, and from what evidence.

Approach

Everything is built from atoms. An atom has one responsibility, typed inputs and outputs, no global state, and injected dependencies (LLM client, OCR engine, storage). A pipeline is a composition of atoms, either written in code or declared as a YAML recipe that is validated when it loads, before anything runs.

flowchart TB
 A["Input documents (PDF, scans, CSV, mail)"] --> B{"Page routing: native text or OCR?"}
 B -- "native" --> C["Positioned native text"]
 B -- "scan" --> D["PaddleOCR (production settings)"]
 C --> E["Segmentation & classification"]
 D --> E
 E --> F["LLM JSON extraction + verbatim grounding check"]
 F --> G["Cross-document reconciliation (invoice / delivery note / order)"]
 G --> H["Tri-state verdict + named reason + trace"]
 H --> I["Report / human review"]

The catalogue

The 459 atoms are grouped into 32 families, one Python package each, ordered along the processing chain: capture the world, understand it, reconcile and decide, deliver, then operate in production. Two further packages assemble them: steps (77 named, parameterised steps) and recipes (the YAML loader and validator).

Stage	Family	Atoms	Role
Base	`core`	25	pivot types, pipeline, configuration, resilience (retry, rate limits), files, concurrency
Capture	`connect`	40	resilient HTTP, mailboxes (Graph, IMAP), SFTP, OneDrive, read-only ERP source, open data, audit trail
	`google`	24	Gmail, Sheets, Drive, Apps Script, with resilient execution
	`web`	17	driver-agnostic web robots (Selenium, Playwright): waits, resumable skeleton, forms
	`pdf`	9	positioned native text, OCR-or-not decision, rendering, page handling, duplicates
	`ocr`	9	PaddleOCR with production settings, EasyOCR
	`layout`	3	page geometry: lines, columns, deskew, monospace rendering
	`audio`	16	chunked transcription, segment post-processing, microphone ↔ screen alignment
	`vision`	32	video / RTSP streams, detection, tracking, depth, segmentation, calibration
	`vlm`	10	screen reading with vision-language models: change detection, label location
	`geo`	11	geodesic measures, combined GPS fixes, line-distance filtering, spatial SQL
	`calendar`	4	public holidays, working days and opening hours
Understand	`text`	32	numbers and amounts, dates, normalisation, pattern-based fields, identifiers
	`segment`	7	pages → logical documents, typed document numbers, page nature
	`classify`	12	file name, sender, weighted patterns, TF-IDF, visual and LLM classifiers, cascades
	`llm`	24	provider-agnostic contract, providers, pool, prompts, structured extraction
	`anchor`	5	fields and tables located relative to printed labels, native text or OCR
	`kb`	8	traced knowledge base: validated event log folded into a SQLite index
Reconcile	`match`	25	EAN/GTIN, indexes, multi-criteria scores, optimal assignment, entity resolution
	`compare`	9	tri-state, explainable comparators for fields, records and sets
	`check`	9	generic tri-state checks: totals, VAT, balance, completeness, plausibility
	`reconcile`	12	set pairing, order → delivery → invoice chains, N-way, ledger and bank matching
	`validate`	14	rules engine, identifiers, amount consistency, corrected LLM verdicts, final decision
	`rules`	8	safe condition language, explainable first-match tables, escalation, state machines
	`events`	11	timestamped events: windows, expected-event watch, sessions, status transitions
	`analytics`	18	supply-chain analytics without pandas: demand, stock-outs, service level, safety stock
Deliver	`export`	8	CSV for French Excel, shared Excel workbooks, HTML / Markdown / PDF reports
	`report`	6	persistable discrepancies, summaries, review queue
	`render`	12	HTML deliverables: pre-rendered Mermaid, ECharts charts, GeoJSON maps
	`diagram`	6	process diagrams: draw.io / BPMN JSON in, Mermaid and BPMN XML out
Operate	`ops`	26	high availability (heartbeats, leader election), scheduled daemon, monitoring, pre-flight checks
	`qa`	7	editorial non-regression (golden files), simulation replay, inter-annotator agreement

Named recipes

Each of the 17 recipes is a YAML list of steps, named <family>.<atom>. Three examples:

invoice_multi_doc: a batch of multi-document PDFs becomes checked logical documents. io.collect → pdf.read → segment.documents → classify.documents → llm.extract → validate.ids → validate.amounts → validate.decide → export.csv → google.sheets.upsert → connect.notify. Each document ends as valid, to check, or invalid.
doc_reconciliation: a file of invoices, credit notes and delivery notes, checked against orders. Guarded LLM extraction (llm.extract_docs) is cross-checked with pattern extraction (extract.regex, compare.extractions, reconcile.fields), then check.document and check.completeness, lines against the order reference (compare.reference), the order → delivery → invoice chain (reconcile.nway), and a discrepancy file plus review queue (report.discrepancies, report.review). 17 steps.
meeting_transcript: a meeting recording becomes a timestamped transcript with speakers. audio.decode → audio.vad → audio.transcribe → audio.diarize → audio.speakers → audio.segments → export.csv → export.transcript → connect.notify.

The others include delivery_vs_order, ledger_reconciliation, invoice_dedup, article_base_cleaning, stock_policy, mail_to_sheet, draft_mails, pdf_text_extract, rubric_eval, screen_session, camera_flow and process_from_transcript.

The Atlas

The Atlas is a single offline HTML page generated from the code itself (syntax trees and YAML), with no hand-written data. It draws the catalogue as a periodic table: each family is an element with a symbol, each module an atom, each recipe a molecule. It also flags atoms that lack documentation, tests or a usage example. The interface is in French.

The “Capture” series of the periodic table. Each box is a family, with its symbol, number of modules and size in thousands of lines; each tile below it is an atom.

The “Reconcile” series: the families that pair, compare, check and decide.

An atom’s page (compare.fields): what it does, taken from its docstring, then each public name with its full signature.

The invoice_multi_doc recipe as a molecule: 11 steps, the context keys that link them, and the services they need (dashed when optional).

Methods

Tri-state results. Every comparison returns ok ∈ {True, False, None} (conforms / discrepancy / inconclusive) with a named reason code and a trace. A missing or unreadable value counts as inconclusive, never as an anomaly. Inconclusive results are still counted, not silently dropped.
Hybrid native-text/OCR routing. Each PDF page is classified as native, OCR or empty, based on how much native text it holds and how much of it is covered by images. OCR runs only where it is needed.
Grounded LLM extraction. Structured JSON extraction is validated against Pydantic schemas. Each extracted value is then checked verbatim against the source text, so altered or invented values are flagged.
Cross-document reconciliation. Invoices, delivery notes and orders are compared line by line using named tolerances and dominance rules. For example, a matching line amount can override a price difference.
Provenance. Every ported atom carries a Provenance: header and an entry tracing it to repo@commit:path:lines in the original production code. Bugs found while porting were fixed and each fix is covered by a non-regression test.
Enforced architecture. Layering is checked by 30 import contracts: an atom imports only the core package. Strict typing, a ≥ 80 % coverage gate and a check that no client names appear in source code are also enforced.
Device abstraction. Vision and audio atoms target cuda, mps or cpu via one setting.
End-to-end evaluation. Tests run with real engines (PaddleOCR, Ollama qwen3:8b and qwen3-vl:8b) on data with known ground truth. Each metric is recorded against a threshold in a JSON/HTML report. Metrics that miss their threshold are kept and shown as failing.

Results

Codebase (current tree): 459 atom modules in 32 families, 77 parameterised steps, 17 YAML recipes, 30 import contracts. The suite collects 2,959 unit and recipe tests, plus 109 end-to-end tests that run separately. All 17 recipes have at least one end-to-end test, and 34 of the 35 Atlas elements are exercised end to end.

Selected end-to-end scores (synthetic documents with known ground truth, local 8B models):

Task	Engine	Measured
Invoice + delivery note vs. order reconciliation	`qwen3:8b`	document types and fields 14/14; expected discrepancies 5/5; 0 false discrepancies on the compliant case
LLM JSON extraction (4 native invoices)	`qwen3:8b`	valid JSON 4/4; exact fields 28/28; all deliberately altered values caught by grounding
OCR on a photographed label	PaddleOCR	word recall 0.94 on the raw photo; a contrast-enhancement defect (0.65–0.71) was found and fixed (0.88–0.94)
Table reconstruction from OCR	PaddleOCR	100 % exact cells (proportional font), 95 % (monospace)

Limitations & next steps

Verbatim grounding cannot catch a value that exists in the document but was put in the wrong field: an absent field was wrongly filled in 1 of 8 cases.
The 8B model judged a +50 % quantity gap a “minor” discrepancy, against the instruction. LLM verdicts are therefore normalised and kept downstream of deterministic rules.
Scores come mostly from synthetic data. A real-document test set is being built outside the repository, and only cases that have been reviewed are scored.
Next steps: measure the automatic-reconciliation rate and the false-positive rate on production data.

Stack

Python · Pydantic · PyYAML · PaddleOCR · Ollama (qwen3:8b, qwen3-vl:8b) · Mistral / Anthropic / OpenAI APIs · pytest · mypy · import-linter · ruff

Mission Knowledge Graphs: From Consulting Evidence to an Ontology Queried Through MCP

Tue, 01 Sep 2026 00:00:00 +0000

Context

Process-observation consulting produces flowcharts, interview notes, screen recordings and recommendations. Once a mission is delivered, that material is effectively dead: the next mission starts from a blank page, even when the answer was already found elsewhere. The research question is how to turn deliverables from documents into typed, sourced, queryable data, so that an assistant can answer “where have we seen this before, and what did we conclude?” while citing verifiable evidence.

Two requirements structure the work: traceability (every assertion points to its proof) and a strict separation between evidence and knowledge (what was observed is not what was generalised). Retrieval-augmented generation is fast to build but cites passages, not checkable assertions; a formal ontology is traceable but expensive. This project explores a practical path between the two.

Approach

flowchart TB
 A["`Raw mission sources
immutable, hashed`"] --> B["`Append-only event log
source of truth`"]
 B --> C["`Rebuildable SQLite base
disposable`"]
 C --> D[Deterministic extraction]
 C --> E["`LLM extraction
proposer only`"]
 D --> F["`Proposed nodes and edges
with provenance`"]
 E --> F
 F --> G{"`Human review
outside MCP`"}
 G -->|validated| H[Knowledge graph]
 H --> I[MCP: evidence]
 H --> J[MCP: knowledge]
 H --> K[MCP: ontology]
 I --> L[AI assistant]
 J --> L
 K --> L

The core design choice is to split the interface into three MCP servers by epistemic status: evidence (attested, sourced observations), knowledge (generalisations), and ontology (the meta-model itself). The split prevents an assistant from presenting a generalisation as an observation.

flowchart TB
 subgraph SCHEMA["SCHEMA · naming"]
 ON["`ontology server
How do we name it?`"]
 end
 subgraph PROVEN["PROVEN · observed"]
 EV["`evidence server
What did we capture, and where?`"]
 end
 subgraph GENERAL["GENERALISED · concluded"]
 KN["`knowledge server
What do we conclude?
proposes, never validates`"]
 end
 ON -. "types" .-> EV
 EV == "grounds every conclusion" ==> KN
 EV --> AI["`AI assistant
answers labelled by status`"]
 KN --> AI
 ON --> AI

The graph is single; only the façades are split. The boundary that matters runs between evidence and knowledge: the evidence server only returns what was observed and where, the knowledge server returns conclusions that must chain back to that evidence, and no MCP tool can mark anything as validated.

Methods

Meta-model. A fixed core plus mission profiles and a versioned ontology (v0). Each node carries an origin (structural, deterministic, LLM-extracted) and a status (proposed → validated).
Invariants (I1–I6). Provenance is mandatory on every derived node and edge; a recommendation must chain back to a sourced finding; LLM-extracted quotes must be literal substrings of their fragment; “validated” requires a recorded human decision. Raw sources are immutable; the database is disposable and rebuilt from the log.
Competency questions before implementation. A v0 set of 33 questions, typed as factual, aggregate, traceability, contradiction, temporal, cross-mission and negative (requests the system must refuse, e.g. cross-client leakage), in an executable format with expected answers and references.
Evaluation protocol. At least 30 natural-language tasks, scored automatically or by a judge. Gate thresholds were fixed up front: accuracy ≥ 0.8, valid citations ≥ 0.98, grounding ≥ 0.95, leakage = 0, median ≤ 6 tool calls per task; each question solvable in ≤ 3 calls; deterministic extraction precision ≥ 0.95 per type; inter-annotator κ ≥ 0.7.
Phases P0–P6 with human gates. Implementation is delegated to an agent session that must stop at each gate and log open decisions.
Black-box software mapping. The same method was applied to a commercial business-management software, without reading its source code. The target is a triplet of user, software and database truths linked by proven edges. Eleven heuristics (H1–H11) cover value matching, inverse and direct perturbation, calculated fields, conditional visibility, scenario diffs, invariant mining, input fuzzing, semantic priors (never treated as proof), documentation-to-test and symptom-to-rule. Inputs: the menu tree via AT-SPI, a screen survey, column profiling, and an event typology from the application audit log. A hypothesis is accepted only with independent counter-evidence; vendor-produced maps are held in quarantine as a scoring oracle.

An illustrative excerpt of the typed graph (labels are fictitious): the recommendation is only admissible because the chain down to a literal fragment of a source is complete.

flowchart TB
 R["`Recommendation
single entry point`"] -- responds to --> C["`Finding
double data entry at goods receipt`"]
 O["`Observation
“stock is re-keyed by hand”`"] -- attests --> C
 O -- extracted from --> F["`Fragment
interview notes, lines 40–52`"]
 F -- cites --> P["`Source file
hashed, immutable`"]
 C -- concerns --> A["`Activity
goods receipt`"]
 A -- uses --> S["`Screen
receipt form`"]
 S -- module of --> Y["`System
business software`"]

Status & results

This is mainly a design with partial implementation, not a system in service.

Pilot (observation corpus, deterministic pass). 6,195 files (6.92 GB) acquired idempotently; 13,272 nodes and 6,022 edges extracted in 2.1 s at zero LLM cost; all six invariants green over 57,969 checks.
Gold-set measurement. 145 annotation units plus 100 entity-resolution pairs. Human annotation was partial and the second annotator was an LLM. Human × LLM agreement: κ = 0.72 (software), 0.41 (screen), 0.31 (category), 0.52 (segment boundaries). Extractor precision against human reference: 0.74, 0.43, 0.04, 0.63. Gate G2 (≥ 0.95) not passed; the category result reflects a definitional mismatch, not a computation error.
Software mapping. Schema of 427 tables with 0 declared foreign keys, triggers or views (all logic is applicative); 1,048 menu entries (899 leaves) mapped; 154 of 899 screens surveyed (17 %); 2,845 rule hypotheses reduced to 1,519 by a local LLM filter, with 1 decision error in a 50-item sample reviewed by the supervising agent. A read-only MCP prototype exists; no rule has yet been proven in the software. Work paused after the host VM was overloaded.

Gold-set measurement against the gates fixed in advance: bars are the measured values per field, the orange line is the gate. Only one bar clears its gate.

---
config:
 xyChart:
 width: 540
 height: 320
 themeVariables:
 xyChart:
 backgroundColor: "transparent"
 plotColorPalette: "#3F86B8, #C46A2B"
---
xychart-beta
 title "Human × LLM agreement κ (gate 0.7)"
 x-axis ["software", "screen", "category", "segments"]
 y-axis "κ" 0 --> 1
 bar [0.72, 0.41, 0.31, 0.52]
 line [0.7, 0.7, 0.7, 0.7]

---
config:
 xyChart:
 width: 540
 height: 320
 themeVariables:
 xyChart:
 backgroundColor: "transparent"
 plotColorPalette: "#3F86B8, #C46A2B"
---
xychart-beta
 title "Extractor precision vs human reference (gate 0.95)"
 x-axis ["software", "screen", "category", "segments"]
 y-axis "Precision" 0 --> 1
 bar [0.74, 0.43, 0.04, 0.63]
 line [0.95, 0.95, 0.95, 0.95]

The software mapping, as a funnel: the menu tree is fully mapped, but screen coverage and proof lag far behind, and the hypothesis pipeline has not yet produced a single proven rule.

---
config:
 xyChart:
 width: 540
 height: 360
 themeVariables:
 xyChart:
 backgroundColor: "transparent"
 plotColorPalette: "#3F86B8"
---
xychart-beta
 title "Black-box mapping: from inventory to proof"
 x-axis ["Menu entries", "Leaves", "Screens", "Hypotheses", "Filtered", "Proven"]
 y-axis "Count" 0 --> 3000
 bar [1048, 899, 154, 2845, 1519, 0]

The MCP servers and end-to-end evaluation (P4–P5) are not implemented.

Limitations & next steps

Pilot results are below threshold and rest on partial human annotation. Next steps: redefine the category label, complete a fully human reference, re-measure G1/G2, then build the three servers and run the pre-registered evaluation. For the software mapping, the write-perturbation heuristics await written authorisation and a dedicated VM.

Stack

Python 3.11 · JSONL event log · SQLite · YAML ontology · MCP · AT-SPI · Ollama · pytest