Context
Process-observation consulting produces flowcharts, interview notes, screen recordings and recommendations. Once a mission is delivered, that material is effectively dead: the next mission starts from a blank page, even when the answer was already found elsewhere. The research question is how to turn deliverables from documents into typed, sourced, queryable data, so that an assistant can answer “where have we seen this before, and what did we conclude?” while citing verifiable evidence.
Two requirements structure the work: traceability (every assertion points to its proof) and a strict separation between evidence and knowledge (what was observed is not what was generalised). Retrieval-augmented generation is fast to build but cites passages, not checkable assertions; a formal ontology is traceable but expensive. This project explores a practical path between the two.
Approach
The core design choice is to split the interface into three MCP servers by epistemic status: evidence (attested, sourced observations), knowledge (generalisations), and ontology (the meta-model itself). The split prevents an assistant from presenting a generalisation as an observation.
The graph is single; only the façades are split. The boundary that matters runs between evidence and knowledge: the evidence server only returns what was observed and where, the knowledge server returns conclusions that must chain back to that evidence, and no MCP tool can mark anything as validated.
Methods
- Meta-model. A fixed core plus mission profiles and a versioned ontology (v0). Each node carries an origin (structural, deterministic, LLM-extracted) and a status (proposed → validated).
- Invariants (I1–I6). Provenance is mandatory on every derived node and edge; a recommendation must chain back to a sourced finding; LLM-extracted quotes must be literal substrings of their fragment; “validated” requires a recorded human decision. Raw sources are immutable; the database is disposable and rebuilt from the log.
- Competency questions before implementation. A v0 set of 33 questions, typed as factual, aggregate, traceability, contradiction, temporal, cross-mission and negative (requests the system must refuse, e.g. cross-client leakage), in an executable format with expected answers and references.
- Evaluation protocol. At least 30 natural-language tasks, scored automatically or by a judge. Gate thresholds were fixed up front: accuracy ≥ 0.8, valid citations ≥ 0.98, grounding ≥ 0.95, leakage = 0, median ≤ 6 tool calls per task; each question solvable in ≤ 3 calls; deterministic extraction precision ≥ 0.95 per type; inter-annotator κ ≥ 0.7.
- Phases P0–P6 with human gates. Implementation is delegated to an agent session that must stop at each gate and log open decisions.
- Black-box software mapping. The same method was applied to a commercial business-management software, without reading its source code. The target is a triplet of user, software and database truths linked by proven edges. Eleven heuristics (H1–H11) cover value matching, inverse and direct perturbation, calculated fields, conditional visibility, scenario diffs, invariant mining, input fuzzing, semantic priors (never treated as proof), documentation-to-test and symptom-to-rule. Inputs: the menu tree via AT-SPI, a screen survey, column profiling, and an event typology from the application audit log. A hypothesis is accepted only with independent counter-evidence; vendor-produced maps are held in quarantine as a scoring oracle.
An illustrative excerpt of the typed graph (labels are fictitious): the recommendation is only admissible because the chain down to a literal fragment of a source is complete.
Status & results
This is mainly a design with partial implementation, not a system in service.
- Pilot (observation corpus, deterministic pass). 6,195 files (6.92 GB) acquired idempotently; 13,272 nodes and 6,022 edges extracted in 2.1 s at zero LLM cost; all six invariants green over 57,969 checks.
- Gold-set measurement. 145 annotation units plus 100 entity-resolution pairs. Human annotation was partial and the second annotator was an LLM. Human × LLM agreement: κ = 0.72 (software), 0.41 (screen), 0.31 (category), 0.52 (segment boundaries). Extractor precision against human reference: 0.74, 0.43, 0.04, 0.63. Gate G2 (≥ 0.95) not passed; the category result reflects a definitional mismatch, not a computation error.
- Software mapping. Schema of 427 tables with 0 declared foreign keys, triggers or views (all logic is applicative); 1,048 menu entries (899 leaves) mapped; 154 of 899 screens surveyed (17 %); 2,845 rule hypotheses reduced to 1,519 by a local LLM filter, with 1 decision error in a 50-item sample reviewed by the supervising agent. A read-only MCP prototype exists; no rule has yet been proven in the software. Work paused after the host VM was overloaded.
Gold-set measurement against the gates fixed in advance: bars are the measured values per field, the orange line is the gate. Only one bar clears its gate.
The software mapping, as a funnel: the menu tree is fully mapped, but screen coverage and proof lag far behind, and the hypothesis pipeline has not yet produced a single proven rule.
The MCP servers and end-to-end evaluation (P4–P5) are not implemented.
Limitations & next steps
Pilot results are below threshold and rest on partial human annotation. Next steps: redefine the category label, complete a fully human reference, re-measure G1/G2, then build the three servers and run the pre-registered evaluation. For the software mapping, the write-perturbation heuristics await written authorisation and a dedicated VM.
Stack
Python 3.11 · JSONL event log · SQLite · YAML ontology · MCP · AT-SPI · Ollama · pytest