Mcp on Boris Dupoux

A Fully Local AI Agent with a Meaning Graph and Multimodal Emotion Measurement

Thu, 01 Oct 2026 00:00:00 +0000

Context

Personal digital traces (messaging, mail, photos, music listening, notes, calendar, health exports, code repositories) are among the most sensitive data a person owns, and also among the richest for studying behaviour — and feeling — over time. Sending them to a cloud LLM is not acceptable, so everything here runs on a single Apple-silicon desktop (M4, 32 GB unified memory), and no data leaves the machine.

The aim is an agent that answers questions about one’s own life while keeping a strict line between what is observed, what is computed, what is hypothesised by a model, and what is accepted as true by its owner.

What is original

Local agents over personal files are not new. Two aspects of this project are.

1. A meaning graph on top of a measurement layer. Knowledge is held at two levels. An observatory stores facts and signals in long format (item, dimension, value, confidence, model, version). Above it, a meaning graph stores what those signals mean: relations (reified, so that a relationship can evolve and be measured), affective episodes (who felt what, when, how intensely, through which modality), themes, chapters of a life, and truths — statements with supporting and contradicting evidence that only the owner can validate. Every node and edge carries its provenance down to the source element, and an origin ranked by trust (human > declared > structural > rule/computed > vendor > model > inferred).

2. Emotion measured across modalities, along unusual dimensions. The same fixed vocabulary of 53 emotions (Plutchik-based, each with valence and arousal coordinates) is applied to messages (local LLM), voice (five acoustic and multimodal layers fused with the transcript, with an explicit text/voice mismatch flag) and photos (deterministic colour and light measures, SigLIP embeddings, face signals treated as weak evidence). These sit in a catalogue of 929 dimensions in 88 groups that goes well beyond the usual metadata. A few examples of its less common entries:

modality	dimension	how it is defined
photo	colour temperature	mean red–blue balance of the image (warm vs cool)
photo	colourfulness	Hasler & Süsstrunk (2003) opponent-colour metric
photo	golden hour / blue hour	sun elevation computed offline from date and position
photo	face size	share of the frame taken by a face (a proxy for proximity)
voice	jitter, shimmer, harmonics-to-noise ratio	openSMILE eGeMAPS prosody
voice	laughter, crying, sighs	sound-event detection (AudioSet classes)
voice	text/voice mismatch	voice valence and transcript valence differ by ≥ 0.6
message	valence, arousal, tone	per participant, constrained to the fixed grid
message	expressive punctuation, emoji palette	deterministic text measures
relation	reply-latency asymmetry	log ratio of the two parties’ median reply delays
relation	initiatives	conversations opened after 3 h of silence, per side
relation	trend	slope of relationship intensity over 90 days (cooling < 0)
period	musical nostalgia	mean age of the tracks listened to
period	creation / consumption	time in creative tools ÷ time consuming

From signals to meaning

flowchart LR
 PH["Photo"] --> PM["Measures
light · contrast · colour temperature
colourfulness · palette · sharpness"]
 PH --> PS["SigLIP embedding
(image ↔ text similarity)"]
 PH --> PF["Vendor face signals
smile · expression · gaze · face size"]
 VN["Voice note"] --> VT["Whisper transcript"]
 VN --> VL["Acoustic layers
valence/arousal/dominance · categories
sound events · prosody · audio-LLM"]
 VT --> VX["Text emotion of transcript"]
 VL --> VF["Weighted fusion
confidence = agreement
text/voice mismatch flag"]
 VX --> VF
 MS["Messages
(pseudonymised: Me, P1, P2)"] --> ML["Local LLM, fixed grid
emotion · intensity · valence
arousal · tone · topics"]
 PM --> OB[("Observatory
929 dimensions")]
 PS --> OB
 PF --> OB
 VF --> OB
 ML --> OB
 OB -- "emotion with enough confidence" --> AFF["Affective episode (AFF)
who · emotion · when · intensity · modality"]
 PF -. "weak cue, confidence ≤ 0.4" .-> AFF
 ML -- "topics" --> THM["Themes"]
 OB -- "aggregates" --> AG["Day / period indicators
valence · light · palette"]
 AFF --> G["Meaning graph
relations · episodes · themes · chapters · truths
(all proposed)"]
 THM --> G
 AG --> G
 G --> HV(("Owner
validate · reject · correct"))

Each affective episode is a node linked to the person who felt it, to the emotion in the fixed vocabulary, and — when it exists — to the conversation or moment it occurred in. Inference rules then carry affect upwards (for instance, the affect of a chapter or of a routine), and strong aggregate correlations can become proposed truths with their measures as evidence.

Emotion measurement in detail

Vocabulary. The grid is fixed and versioned: Plutchik’s 8 primary emotions at 3 intensities, his 8 primary dyads, plus social and reflective emotions useful for a life (gratitude, tenderness, pride, shame, loneliness, nostalgia, wonder, relief, exhaustion…), 53 in total. Each carries starting coordinates in Russell’s circumplex — valence in [−1, 1], arousal in [0, 1] — meant to be calibrated later on the owner’s own validations. Plutchik oppositions are encoded, and health-app mood labels are mapped onto the grid.

Messages. A local LLM (Qwen3-30B-A3B) reads each conversation with participants replaced by “Me”, “P1”, “P2”, messages framed as data it must not obey, and media reduced to placeholders such as “[voice note 23 s]”. Output is constrained by a JSON schema: for each participant, the dominant emotion from the grid (or none), intensity, valence, arousal, a tone from a closed list of 19, and a confidence; plus 1–3 topics that become themes. Long conversations are reduced to their beginning and end; short ones are batched ten per call. An affective episode is proposed only when confidence is at least 0.3.

Voice. Each recording is converted to 16 kHz mono and passes through:

Whisper large-v3-turbo (MLX) for the transcript;
a wav2vec2 dimensional model giving valence, arousal and dominance on 15 s windows, weighted by window length, plus arousal variability;
emotion2vec+ large for categorical emotion, mapped onto the grid (kept at ≥ 0.4);
an AST AudioSet classifier for laughter, crying, sighs, shouting, whispering, music and ambience;
openSMILE eGeMAPS prosody: pitch (semitones) and its variability, loudness, jitter, shimmer, harmonics-to-noise ratio, voiced segments per second, mean pause length;
Qwen2.5-Omni-7B (llama.cpp, Metal), which listens to the audio and returns an emotion from the grid, a valence and short acoustic cues;
the text-emotion model above, applied to the transcript.

The categorical opinions (voice categories, audio-LLM, transcript) are combined by a weighted vote (weights 1.0, 1.0 and 0.8); valence is a weighted mean of the dimensional model, the audio-LLM and the transcript. Confidence is agreement between layers: one agreeing source gives 0.45, two give 0.65, three give 0.85, minus 0.1 for each source that disagrees. When the voice and the words disagree by 0.6 or more in valence (an “I’m fine” said sadly), a text/voice mismatch is flagged rather than averaged away. Laughter and crying scores are kept alongside.

Photos. On 256 px thumbnails, without any model: luminance, contrast, saturation, colour temperature, colourfulness, sharpness (variance of the Laplacian) and a 5-colour palette with named dominant hue. SigLIP so400m embeddings make the library searchable by text. Vendor face attributes (smile, smile type, expression, gaze, eye state, face size) are read as signals, but the schema states that a photographed smile is not a felt joy: the face modality is a weak cue whose confidence is capped at 0.4. Music mood is handled the same way: the mood of a track is never, on its own, evidence of the listener’s mood. At period level, valence is fused with a fixed precedence: declared > voice > text > face.

Human validation. Every episode, theme, chapter or truth is emitted as proposed. The PWA shows a validation queue — affective episodes ordered by confidence, with the evidence (the conversation or recording it came from) — and the owner validates, rejects or corrects. Only that human decision event turns a proposal into validated, an equivalence checked automatically. The planned next step is a small personal classifier on emotion2vec features, trained on these validations.

Architecture

flowchart TB
 S["Sources (read-only snapshots)
messaging · mail · photos · music · notes · calendar · exports"]
 C["Connectors
dedicated ingestion process, sole holder of disk access"]
 V["Vault (owner-only, 0700)
content-addressed raw store"]
 O["Observatory
facts + signals (SQLite)"]
 K["Meaning graph
append-only JSONL log → SQLite fold + FTS"]
 N["Nightly / hourly pipelines
thumbnails · measures · SigLIP · emotions
(GPU lock, one model at a time)"]
 M["MCP server (stdio, no port)
20 read-only / propose-only tools"]
 H["Hermes Agent
local LLM via Ollama"]
 P["PWA (HTTPS + TOTP)
chat · validation queue"]
 U(("Owner"))
 S --> C --> V --> O --> K
 N --> O
 O --> M
 K --> M
 M --> H --> P
 P -- "validate / reject / correct" --> K
 U -- "NetBird VPN" --> P

Raw data is snapshotted (SQLite opened in mode=ro&immutable=1) and never written to; everything derived lives in a separate owner-only vault, and the code repository holds no data. The meaning graph is built on the atoms library’s knowledge-base module: an append-only, schema-validated JSONL event log folded into a disposable SQLite index.

The schema itself is data, verified mechanically: 27 node types and 50 predicates, 9 provenance origins, 31 locator kinds tying every claim to its source element, the 53-emotion vocabulary, 17 inference rules, and 42 sources behind the 929 dimensions (610 native, 126 vendor-computed, 86 aggregates, 74 deterministic measures, 29 model interpretations).

Security model

Source content is assumed adversarial (prompt injection in a message or mail), and the model may hallucinate or misuse tools. Defences are structural:

No ambient capabilities. The agent framework (Hermes Agent) has 20 built-in toolsets disabled (terminal, files, code execution, web, browser, delegation, memory, cloud speech…); no cloud provider, update check or external login; the only model endpoint is a loopback Ollama server.
One narrow interface. Data is reachable only through a local MCP server over stdio, with closed schemas, validated arguments, explicit truncation and a content-free access log. Tools can propose, never decide.
Privilege separation and pseudonymisation. Only ingestion holds full-disk access; analysis has no network. Third parties are opaque codes, and a real name in an identifier is rejected by the schema.
Sensitivity levels (0–3) per element with a lower ceiling for remote sessions; remote access only through a NetBird VPN, TOTP and strict session cookies.
Ethical exclusions: vendor-inferred ethnicity, skin tone and gender are never read.

Models

All inference is local; one large model is loaded at a time under a shared GPU lock.

task	model	runtime
agent (tool use, ≥ 64k context)	gpt-oss:20b, qwen3:8b fallback	Ollama
text emotions, voice-transcript emotions	Qwen3-30B-A3B-Instruct-2507 (Q4)	Ollama
text / image embeddings	bge-m3 · SigLIP so400m	Ollama · PyTorch MPS
image reading	qwen3-vl:8b	Ollama
speech-to-text	Whisper large-v3-turbo	MLX
voice emotion (5 layers)	wav2vec2 dimensional · emotion2vec+ · AST · openSMILE eGeMAPS · Qwen2.5-Omni-7B	PyTorch MPS · CPU · llama.cpp (Metal)

Each model is smoke-tested on synthetic data only (a fictitious French sentence synthesised by a local TTS, then passed through the whole audio chain).

Evaluation & results

Agent answer quality. A 16-question benchmark (2 trap questions), with ground truth computed by calling the tools directly, scores tool choice, numerical fidelity (every number in the answer must appear in that turn’s tool outputs), correctness, trap handling and partial-period flagging: 73 % → 100 % after tool-description, argument-validation and configuration fixes.
Server-side error correction. With a reduced 8-tool interface, simple questions went from 76 % to 90 % and cross-source questions from 73 % to 90 %; 20 deliberately unusual questions scored 89 % and exposed silent-zero failures, since fixed.
Context and latency. Tool descriptions cut from 23.5 k to 9.5 k characters; a heavy aggregate query from 172 s to 0.1 s; all heavy benchmark queries ≤ 0.3 s.
Cross-exploration benchmark. 60 placeholder-only scenarios with oracle code; oracles pass 61/61, with no leak at remote sensitivity ceilings 1 and 2.
Verification. 56 named schema checks green; 631 native/vendor columns confirmed against real database structures; 1,235 Python unit tests, 15 JavaScript test files and 110 quality tests on real data that report counts only.

Limitations & next steps

Evaluation sets are small and authored by the system’s designer; they measure fidelity and regressions rather than general capability. Emotion outputs are hypotheses: the valence/arousal coordinates of the vocabulary are starting values, and personal calibration of voice emotion from validated examples is still pending, as is the larger target agent model. Agent answers take tens of seconds on a 20B model.

Stack

Python · SQLite / FTS5 · Ollama · llama.cpp (Metal) · MLX · PyTorch MPS · Hermes Agent · MCP · SigLIP · Whisper · emotion2vec · openSMILE · NetBird · PWA · launchd

Mission Knowledge Graphs: From Consulting Evidence to an Ontology Queried Through MCP

Tue, 01 Sep 2026 00:00:00 +0000

Context

Process-observation consulting produces flowcharts, interview notes, screen recordings and recommendations. Once a mission is delivered, that material is effectively dead: the next mission starts from a blank page, even when the answer was already found elsewhere. The research question is how to turn deliverables from documents into typed, sourced, queryable data, so that an assistant can answer “where have we seen this before, and what did we conclude?” while citing verifiable evidence.

Two requirements structure the work: traceability (every assertion points to its proof) and a strict separation between evidence and knowledge (what was observed is not what was generalised). Retrieval-augmented generation is fast to build but cites passages, not checkable assertions; a formal ontology is traceable but expensive. This project explores a practical path between the two.

Approach

flowchart TB
 A["`Raw mission sources
immutable, hashed`"] --> B["`Append-only event log
source of truth`"]
 B --> C["`Rebuildable SQLite base
disposable`"]
 C --> D[Deterministic extraction]
 C --> E["`LLM extraction
proposer only`"]
 D --> F["`Proposed nodes and edges
with provenance`"]
 E --> F
 F --> G{"`Human review
outside MCP`"}
 G -->|validated| H[Knowledge graph]
 H --> I[MCP: evidence]
 H --> J[MCP: knowledge]
 H --> K[MCP: ontology]
 I --> L[AI assistant]
 J --> L
 K --> L

The core design choice is to split the interface into three MCP servers by epistemic status: evidence (attested, sourced observations), knowledge (generalisations), and ontology (the meta-model itself). The split prevents an assistant from presenting a generalisation as an observation.

flowchart TB
 subgraph SCHEMA["SCHEMA · naming"]
 ON["`ontology server
How do we name it?`"]
 end
 subgraph PROVEN["PROVEN · observed"]
 EV["`evidence server
What did we capture, and where?`"]
 end
 subgraph GENERAL["GENERALISED · concluded"]
 KN["`knowledge server
What do we conclude?
proposes, never validates`"]
 end
 ON -. "types" .-> EV
 EV == "grounds every conclusion" ==> KN
 EV --> AI["`AI assistant
answers labelled by status`"]
 KN --> AI
 ON --> AI

The graph is single; only the façades are split. The boundary that matters runs between evidence and knowledge: the evidence server only returns what was observed and where, the knowledge server returns conclusions that must chain back to that evidence, and no MCP tool can mark anything as validated.

Methods

Meta-model. A fixed core plus mission profiles and a versioned ontology (v0). Each node carries an origin (structural, deterministic, LLM-extracted) and a status (proposed → validated).
Invariants (I1–I6). Provenance is mandatory on every derived node and edge; a recommendation must chain back to a sourced finding; LLM-extracted quotes must be literal substrings of their fragment; “validated” requires a recorded human decision. Raw sources are immutable; the database is disposable and rebuilt from the log.
Competency questions before implementation. A v0 set of 33 questions, typed as factual, aggregate, traceability, contradiction, temporal, cross-mission and negative (requests the system must refuse, e.g. cross-client leakage), in an executable format with expected answers and references.
Evaluation protocol. At least 30 natural-language tasks, scored automatically or by a judge. Gate thresholds were fixed up front: accuracy ≥ 0.8, valid citations ≥ 0.98, grounding ≥ 0.95, leakage = 0, median ≤ 6 tool calls per task; each question solvable in ≤ 3 calls; deterministic extraction precision ≥ 0.95 per type; inter-annotator κ ≥ 0.7.
Phases P0–P6 with human gates. Implementation is delegated to an agent session that must stop at each gate and log open decisions.
Black-box software mapping. The same method was applied to a commercial business-management software, without reading its source code. The target is a triplet of user, software and database truths linked by proven edges. Eleven heuristics (H1–H11) cover value matching, inverse and direct perturbation, calculated fields, conditional visibility, scenario diffs, invariant mining, input fuzzing, semantic priors (never treated as proof), documentation-to-test and symptom-to-rule. Inputs: the menu tree via AT-SPI, a screen survey, column profiling, and an event typology from the application audit log. A hypothesis is accepted only with independent counter-evidence; vendor-produced maps are held in quarantine as a scoring oracle.

An illustrative excerpt of the typed graph (labels are fictitious): the recommendation is only admissible because the chain down to a literal fragment of a source is complete.

flowchart TB
 R["`Recommendation
single entry point`"] -- responds to --> C["`Finding
double data entry at goods receipt`"]
 O["`Observation
“stock is re-keyed by hand”`"] -- attests --> C
 O -- extracted from --> F["`Fragment
interview notes, lines 40–52`"]
 F -- cites --> P["`Source file
hashed, immutable`"]
 C -- concerns --> A["`Activity
goods receipt`"]
 A -- uses --> S["`Screen
receipt form`"]
 S -- module of --> Y["`System
business software`"]

Status & results

This is mainly a design with partial implementation, not a system in service.

Pilot (observation corpus, deterministic pass). 6,195 files (6.92 GB) acquired idempotently; 13,272 nodes and 6,022 edges extracted in 2.1 s at zero LLM cost; all six invariants green over 57,969 checks.
Gold-set measurement. 145 annotation units plus 100 entity-resolution pairs. Human annotation was partial and the second annotator was an LLM. Human × LLM agreement: κ = 0.72 (software), 0.41 (screen), 0.31 (category), 0.52 (segment boundaries). Extractor precision against human reference: 0.74, 0.43, 0.04, 0.63. Gate G2 (≥ 0.95) not passed; the category result reflects a definitional mismatch, not a computation error.
Software mapping. Schema of 427 tables with 0 declared foreign keys, triggers or views (all logic is applicative); 1,048 menu entries (899 leaves) mapped; 154 of 899 screens surveyed (17 %); 2,845 rule hypotheses reduced to 1,519 by a local LLM filter, with 1 decision error in a 50-item sample reviewed by the supervising agent. A read-only MCP prototype exists; no rule has yet been proven in the software. Work paused after the host VM was overloaded.

Gold-set measurement against the gates fixed in advance: bars are the measured values per field, the orange line is the gate. Only one bar clears its gate.

---
config:
 xyChart:
 width: 540
 height: 320
 themeVariables:
 xyChart:
 backgroundColor: "transparent"
 plotColorPalette: "#3F86B8, #C46A2B"
---
xychart-beta
 title "Human × LLM agreement κ (gate 0.7)"
 x-axis ["software", "screen", "category", "segments"]
 y-axis "κ" 0 --> 1
 bar [0.72, 0.41, 0.31, 0.52]
 line [0.7, 0.7, 0.7, 0.7]

---
config:
 xyChart:
 width: 540
 height: 320
 themeVariables:
 xyChart:
 backgroundColor: "transparent"
 plotColorPalette: "#3F86B8, #C46A2B"
---
xychart-beta
 title "Extractor precision vs human reference (gate 0.95)"
 x-axis ["software", "screen", "category", "segments"]
 y-axis "Precision" 0 --> 1
 bar [0.74, 0.43, 0.04, 0.63]
 line [0.95, 0.95, 0.95, 0.95]

The software mapping, as a funnel: the menu tree is fully mapped, but screen coverage and proof lag far behind, and the hypothesis pipeline has not yet produced a single proven rule.

---
config:
 xyChart:
 width: 540
 height: 360
 themeVariables:
 xyChart:
 backgroundColor: "transparent"
 plotColorPalette: "#3F86B8"
---
xychart-beta
 title "Black-box mapping: from inventory to proof"
 x-axis ["Menu entries", "Leaves", "Screens", "Hypotheses", "Filtered", "Proven"]
 y-axis "Count" 0 --> 3000
 bar [1048, 899, 154, 2845, 1519, 0]

The MCP servers and end-to-end evaluation (P4–P5) are not implemented.

Limitations & next steps

Pilot results are below threshold and rest on partial human annotation. Next steps: redefine the category label, complete a fully human reference, re-measure G1/G2, then build the three servers and run the pre-registered evaluation. For the software mapping, the write-perturbation heuristics await written authorisation and a dedicated VM.

Stack

Python 3.11 · JSONL event log · SQLite · YAML ontology · MCP · AT-SPI · Ollama · pytest