Explainable Processing Atoms for Documents & Data

Tue, 01 Sep 2026 00:00:00 +0000

Context

Production systems for document reconciliation and OCR, built for clients such as a wholesale distributor and an accounting firm, left several codebases that each solved the same sub-problems (page routing, OCR tuning, LLM extraction, field comparison) with their own bugs. The atoms library consolidates them into one package. Its central question is the one behind the original systems: can a document check be automated so that a human can audit and contest every verdict? Extracting a field is not enough. The system must state whether two documents agree, why, and from what evidence.

Approach

Everything is built from atoms. An atom has one responsibility, typed inputs and outputs, no global state, and injected dependencies (LLM client, OCR engine, storage). A pipeline is a composition of atoms, either written in code or declared as a YAML recipe that is validated when it loads, before anything runs.

flowchart TB
 A["Input documents (PDF, scans, CSV, mail)"] --> B{"Page routing: native text or OCR?"}
 B -- "native" --> C["Positioned native text"]
 B -- "scan" --> D["PaddleOCR (production settings)"]
 C --> E["Segmentation & classification"]
 D --> E
 E --> F["LLM JSON extraction + verbatim grounding check"]
 F --> G["Cross-document reconciliation (invoice / delivery note / order)"]
 G --> H["Tri-state verdict + named reason + trace"]
 H --> I["Report / human review"]

The catalogue

The 459 atoms are grouped into 32 families, one Python package each, ordered along the processing chain: capture the world, understand it, reconcile and decide, deliver, then operate in production. Two further packages assemble them: steps (77 named, parameterised steps) and recipes (the YAML loader and validator).

Stage	Family	Atoms	Role
Base	`core`	25	pivot types, pipeline, configuration, resilience (retry, rate limits), files, concurrency
Capture	`connect`	40	resilient HTTP, mailboxes (Graph, IMAP), SFTP, OneDrive, read-only ERP source, open data, audit trail
	`google`	24	Gmail, Sheets, Drive, Apps Script, with resilient execution
	`web`	17	driver-agnostic web robots (Selenium, Playwright): waits, resumable skeleton, forms
	`pdf`	9	positioned native text, OCR-or-not decision, rendering, page handling, duplicates
	`ocr`	9	PaddleOCR with production settings, EasyOCR
	`layout`	3	page geometry: lines, columns, deskew, monospace rendering
	`audio`	16	chunked transcription, segment post-processing, microphone ↔ screen alignment
	`vision`	32	video / RTSP streams, detection, tracking, depth, segmentation, calibration
	`vlm`	10	screen reading with vision-language models: change detection, label location
	`geo`	11	geodesic measures, combined GPS fixes, line-distance filtering, spatial SQL
	`calendar`	4	public holidays, working days and opening hours
Understand	`text`	32	numbers and amounts, dates, normalisation, pattern-based fields, identifiers
	`segment`	7	pages → logical documents, typed document numbers, page nature
	`classify`	12	file name, sender, weighted patterns, TF-IDF, visual and LLM classifiers, cascades
	`llm`	24	provider-agnostic contract, providers, pool, prompts, structured extraction
	`anchor`	5	fields and tables located relative to printed labels, native text or OCR
	`kb`	8	traced knowledge base: validated event log folded into a SQLite index
Reconcile	`match`	25	EAN/GTIN, indexes, multi-criteria scores, optimal assignment, entity resolution
	`compare`	9	tri-state, explainable comparators for fields, records and sets
	`check`	9	generic tri-state checks: totals, VAT, balance, completeness, plausibility
	`reconcile`	12	set pairing, order → delivery → invoice chains, N-way, ledger and bank matching
	`validate`	14	rules engine, identifiers, amount consistency, corrected LLM verdicts, final decision
	`rules`	8	safe condition language, explainable first-match tables, escalation, state machines
	`events`	11	timestamped events: windows, expected-event watch, sessions, status transitions
	`analytics`	18	supply-chain analytics without pandas: demand, stock-outs, service level, safety stock
Deliver	`export`	8	CSV for French Excel, shared Excel workbooks, HTML / Markdown / PDF reports
	`report`	6	persistable discrepancies, summaries, review queue
	`render`	12	HTML deliverables: pre-rendered Mermaid, ECharts charts, GeoJSON maps
	`diagram`	6	process diagrams: draw.io / BPMN JSON in, Mermaid and BPMN XML out
Operate	`ops`	26	high availability (heartbeats, leader election), scheduled daemon, monitoring, pre-flight checks
	`qa`	7	editorial non-regression (golden files), simulation replay, inter-annotator agreement

Named recipes

Each of the 17 recipes is a YAML list of steps, named <namespace>.<step>. Three examples:

invoice_multi_doc: a batch of multi-document PDFs becomes checked logical documents. io.collect → pdf.read → segment.documents → classify.documents → llm.extract → validate.ids → validate.amounts → validate.decide → export.csv → google.sheets.upsert → connect.notify. Each document ends as valid, to check, or invalid.
doc_reconciliation: a file of invoices, credit notes and delivery notes, checked against orders. Guarded LLM extraction (llm.extract_docs) is cross-checked with pattern extraction (extract.regex, compare.extractions, reconcile.fields), then check.document and check.completeness, lines against the order reference (compare.reference), the order → delivery → invoice chain (reconcile.nway), and a discrepancy file plus review queue (report.discrepancies, report.review). 17 steps.
meeting_transcript: a meeting recording becomes a timestamped transcript with speakers. audio.decode → audio.vad → audio.transcribe → audio.diarize → audio.speakers → audio.segments → export.csv → export.transcript → connect.notify.

The others include delivery_vs_order, ledger_reconciliation, invoice_dedup, article_base_cleaning, stock_policy, mail_to_sheet, draft_mails, pdf_text_extract, rubric_eval, screen_session, camera_flow and process_from_transcript.

The Atlas

The Atlas is a single offline HTML page generated from the code itself (syntax trees and YAML), with no hand-written data. It draws the catalogue as a periodic table: each family is an element with a symbol, each module an atom, each recipe a molecule. It also flags atoms that lack documentation, tests or a usage example. The interface is in French.

The “Capture” series of the periodic table. Each box is a family, with its symbol, number of modules and size in thousands of lines; each tile below it is an atom.

The “Reconcile” series: the families that pair, compare, check and decide.

An atom’s page (compare.fields): what it does, taken from its docstring, then each public name with its full signature.

The invoice_multi_doc recipe as a molecule: 11 steps, the context keys that link them, and the services they need (dashed when optional).

Methods

Tri-state results. Every comparison returns ok ∈ {True, False, None} (conforms / discrepancy / inconclusive) with a named reason code and a trace. A missing or unreadable value counts as inconclusive, never as an anomaly. Inconclusive results are still counted, not silently dropped.
Hybrid native-text/OCR routing. Each PDF page is classified as native, OCR or empty, based on how much native text it holds and how much of it is covered by images. OCR runs only where it is needed.
Grounded LLM extraction. Structured JSON extraction is validated against Pydantic schemas. Each extracted value is then checked verbatim against the source text, so altered or invented values are flagged: 16/16 synthetic alterations were rejected by the verbatim check (true by construction).
Cross-document reconciliation. Invoices, delivery notes and orders are compared line by line using named tolerances and dominance rules. For example, a matching line amount can override a price difference.
Provenance. Every ported atom carries a Provenance: header and an entry tracing it to repo@commit:path:lines in the original production code. Bugs found while porting were fixed and each fix is covered by a non-regression test.
Enforced architecture. Layering is checked by 30 import contracts: an atom imports only the core package. Strict typing, a ≥ 80 % coverage gate and a check that no client names appear in source code are also enforced.
Device abstraction. Vision and audio atoms target cuda, mps or cpu via one setting.
End-to-end evaluation. Tests run with real engines (PaddleOCR, Ollama qwen3:8b and qwen3-vl:8b) on data with known ground truth. Each metric is recorded against a threshold in a JSON/HTML report. Metrics that miss their threshold are kept and shown as failing.

Results

Codebase (current tree): 459 atom modules in 32 families, 77 parameterised steps, 17 YAML recipes, 30 import contracts. The suite collects 2,959 unit and recipe tests, plus, run separately, 99 end-to-end tests with real engines and 10 live-web tests; these counts are of collected tests, as pass status is not logged. All 17 recipes have at least one end-to-end test, and 34 of 35 Atlas elements (all but anchor) are touched end to end, but only about 41 % of modules.

Selected end-to-end scores (synthetic documents with known ground truth, local 8B models):

Task	Engine	Measured
Invoice + delivery note vs. order reconciliation	`qwen3:8b`	document types and fields 14/14 (2 synthetic dossiers, 1 run); all 5 expected signals of the one injected discrepancy (a short delivery billed in full), 1 synthetic dossier; 0 error-level discrepancies on 1 compliant synthetic dossier (warnings not counted)
LLM JSON extraction (4 native invoices)	`qwen3:8b`	valid JSON 4/4; exact fields 28/28 (4 synthetic invoices × 7 fields); 16/16 synthetic alterations rejected by the verbatim check (true by construction)
OCR on a label	PaddleOCR	synthetic label with simulated photo degradation (17 words, 2 scenes): recall 0.94 raw; an over-saturation defect in enhancement (0.65–0.71) was fixed (0.88–0.94), with no net gain over the raw image
Table reconstruction from OCR (one synthetic 10×4 table)	PaddleOCR	100 % exact cells (proportional font), 95 % (monospace)

Limitations & next steps

Verbatim grounding cannot catch a value that exists in the document but was put in the wrong field: an absent field was wrongly filled in 1 of 8 cases.
The 8B model judged a +50 % quantity gap a “minor” discrepancy, against the instruction. LLM verdicts are therefore normalised and kept downstream of deterministic rules.
Scores come mostly from synthetic data. A real-document test set is being built outside the repository, and only cases that have been reviewed are scored.
First runs on real documents (about 10–25 cases per recipe, partly unreviewed) show strong text extraction but weak duplicate detection and document splitting; reviewed scores will be published.

Stack

Python · Pydantic · PyYAML · PaddleOCR · Ollama (qwen3:8b, qwen3-vl:8b) · Mistral / Anthropic / OpenAI APIs · pytest · mypy · import-linter · ruff

Explainability on Boris Dupoux