Mission Knowledge Graphs: From Consulting Evidence to an Ontology Queried Through MCP

Tue, 01 Sep 2026 00:00:00 +0000

Context

Process-observation consulting produces flowcharts, interview notes, screen recordings and recommendations. Once a mission is delivered, that material is effectively dead: the next mission starts from a blank page, even when the answer was already found elsewhere. The research question is how to turn deliverables from documents into typed, sourced, queryable data, so that an assistant can answer “where have we seen this before, and what did we conclude?” while citing verifiable evidence.

Two requirements structure the work: traceability (every assertion points to its proof) and a strict separation between evidence and knowledge (what was observed is not what was generalised). Retrieval-augmented generation is fast to build but cites passages, not checkable assertions; a formal ontology is traceable but expensive. This project explores a practical path between the two.

Approach

flowchart TB
 A["`Raw mission sources
immutable, hashed`"] --> B["`Append-only event log
source of truth`"]
 B --> C["`Rebuildable SQLite base
disposable`"]
 C --> D[Deterministic extraction]
 C --> E["`LLM extraction
proposer only`"]
 D --> F["`Proposed nodes and edges
with provenance`"]
 E --> F
 F --> G{"`Human review
outside MCP`"}
 G -->|validated| H[Knowledge graph]
 H --> I[MCP: evidence]
 H --> J[MCP: knowledge]
 H --> K[MCP: ontology]
 I --> L[AI assistant]
 J --> L
 K --> L

The core design choice is to split the interface into three MCP servers by epistemic status: evidence (attested, sourced observations), knowledge (generalisations), and ontology (the meta-model itself). The split prevents an assistant from presenting a generalisation as an observation.

flowchart TB
 subgraph SCHEMA["SCHEMA · naming"]
 ON["`ontology server
How do we name it?`"]
 end
 subgraph PROVEN["PROVEN · observed"]
 EV["`evidence server
What did we capture, and where?`"]
 end
 subgraph GENERAL["GENERALISED · concluded"]
 KN["`knowledge server
What do we conclude?
proposes, never validates`"]
 end
 ON -. "types" .-> EV
 EV == "grounds every conclusion" ==> KN
 EV --> AI["`AI assistant
answers labelled by status`"]
 KN --> AI
 ON --> AI

The graph is single; only the façades are split. The boundary that matters runs between evidence and knowledge: the evidence server only returns what was observed and where, the knowledge server returns conclusions that must chain back to that evidence, and no MCP tool can mark anything as validated.

Methods

Meta-model. A fixed core plus mission profiles and a versioned ontology (v0). Each node carries an origin (structural, deterministic, LLM-extracted) and a status (proposed → validated).
Invariants (I1–I6). Provenance is mandatory on every derived node and edge; a recommendation must chain back to a sourced finding; LLM-extracted quotes must be literal substrings of their fragment; “validated” requires a recorded human decision. Raw sources are immutable; the database is disposable and rebuilt from the log.
Competency questions before implementation. A v0 set of 33 questions, typed as factual, aggregate, traceability, contradiction, temporal, cross-mission and negative (requests the system must refuse, e.g. cross-client leakage), in an executable format with expected answers and references.
Evaluation protocol. At least 30 natural-language tasks, scored automatically or by a judge. Gate thresholds were fixed up front: accuracy ≥ 0.8, valid citations ≥ 0.98, grounding ≥ 0.95, leakage = 0, median ≤ 6 tool calls per task; each question solvable in ≤ 3 calls; deterministic extraction precision ≥ 0.95 per type; inter-annotator κ ≥ 0.7.
Phases P0–P6 with human gates. Implementation is delegated to an agent session that must stop at each gate and log open decisions.
Black-box software mapping. The same method was applied to a commercial business-management software, without reading its source code. The target is a triplet of user, software and database truths linked by proven edges. Eleven heuristics (H1–H11) cover value matching, inverse and direct perturbation, calculated fields, conditional visibility, scenario diffs, invariant mining, input fuzzing, semantic priors (never treated as proof), documentation-to-test and symptom-to-rule. Inputs: the menu tree via AT-SPI, a screen survey, column profiling, and an event typology from the application audit log. A hypothesis is accepted only with independent counter-evidence; vendor-produced maps are held in quarantine as a scoring oracle.

An illustrative excerpt of the typed graph (labels are fictitious): the recommendation is only admissible because the chain down to a literal fragment of a source is complete.

flowchart TB
 R["`Recommendation
single entry point`"] -- responds to --> C["`Finding
double data entry at goods receipt`"]
 O["`Observation
“stock is re-keyed by hand”`"] -- attests --> C
 O -- extracted from --> F["`Fragment
interview notes, lines 40–52`"]
 F -- cites --> P["`Source file
hashed, immutable`"]
 C -- concerns --> A["`Activity
goods receipt`"]
 A -- uses --> S["`Screen
receipt form`"]
 S -- module of --> Y["`System
business software`"]

Status & results

This is mainly a design with partial implementation, not a system in service.

Pilot (observation corpus, deterministic pass). 6,195 files (6.92 GB) acquired idempotently; 13,272 nodes and 6,022 edges extracted in 2.1 s at zero LLM cost; all six invariants green over 57,969 checks.
Gold-set measurement. 145 annotation units plus 100 entity-resolution pairs. Human annotation was partial and the second annotator was an LLM. Human × LLM agreement: κ = 0.72 (software), 0.41 (screen), 0.31 (category), 0.52 (segment boundaries). Extractor precision against human reference: 0.74, 0.43, 0.04, 0.63. Gate G2 (≥ 0.95) not passed; the category result reflects a definitional mismatch, not a computation error.
Software mapping. Schema of 427 tables with 0 declared foreign keys, triggers or views (all logic is applicative); 1,048 menu entries (899 leaves) mapped; 154 of 899 screens surveyed (17 %); 2,845 rule hypotheses reduced to 1,519 by a local LLM filter, with 1 decision error in a 50-item sample reviewed by the supervising agent. A read-only MCP prototype exists; no rule has yet been proven in the software. Work paused after the host VM was overloaded.

Gold-set measurement against the gates fixed in advance: bars are the measured values per field, the orange line is the gate. Only one bar clears its gate.

---
config:
 xyChart:
 width: 540
 height: 320
 themeVariables:
 xyChart:
 backgroundColor: "transparent"
 plotColorPalette: "#3F86B8, #C46A2B"
---
xychart-beta
 title "Human × LLM agreement κ (gate 0.7)"
 x-axis ["software", "screen", "category", "segments"]
 y-axis "κ" 0 --> 1
 bar [0.72, 0.41, 0.31, 0.52]
 line [0.7, 0.7, 0.7, 0.7]

---
config:
 xyChart:
 width: 540
 height: 320
 themeVariables:
 xyChart:
 backgroundColor: "transparent"
 plotColorPalette: "#3F86B8, #C46A2B"
---
xychart-beta
 title "Extractor precision vs human reference (gate 0.95)"
 x-axis ["software", "screen", "category", "segments"]
 y-axis "Precision" 0 --> 1
 bar [0.74, 0.43, 0.04, 0.63]
 line [0.95, 0.95, 0.95, 0.95]

The software mapping, as a funnel: the menu tree is fully mapped, but screen coverage and proof lag far behind, and the hypothesis pipeline has not yet produced a single proven rule.

---
config:
 xyChart:
 width: 540
 height: 360
 themeVariables:
 xyChart:
 backgroundColor: "transparent"
 plotColorPalette: "#3F86B8"
---
xychart-beta
 title "Black-box mapping: from inventory to proof"
 x-axis ["Menu entries", "Leaves", "Screens", "Hypotheses", "Filtered", "Proven"]
 y-axis "Count" 0 --> 3000
 bar [1048, 899, 154, 2845, 1519, 0]

The MCP servers and end-to-end evaluation (P4–P5) are not implemented.

Limitations & next steps

Pilot results are below threshold and rest on partial human annotation. Next steps: redefine the category label, complete a fully human reference, re-measure G1/G2, then build the three servers and run the pre-registered evaluation. For the software mapping, the write-perturbation heuristics await written authorisation and a dedicated VM.

Stack

Python 3.11 · JSONL event log · SQLite · YAML ontology · MCP · AT-SPI · Ollama · pytest