A Fully Local AI Agent with a Meaning Graph and Multimodal Emotion Measurement

Thu, 01 Oct 2026 00:00:00 +0000

Context

Personal digital traces (messaging, mail, photos, music listening, notes, calendar, health exports, code repositories) are among the most sensitive data a person owns, and also among the richest for studying behaviour — and feeling — over time. Sending them to a cloud LLM is not acceptable, so everything here runs on a single Apple-silicon desktop (M4, 32 GB unified memory), and no data leaves the machine.

The aim is an agent that answers questions about one’s own life while keeping a strict line between what is observed, what is computed, what is hypothesised by a model, and what is accepted as true by its owner.

What is original

Local agents over personal files are not new. Two aspects of this project are.

1. A meaning graph on top of a measurement layer. Knowledge is held at two levels. An observatory stores facts and signals in long format (item, dimension, value, confidence, model, version). Above it, a meaning graph stores what those signals mean: relations (reified, so that a relationship can evolve and be measured), affective episodes (who felt what, when, how intensely, through which modality), themes, chapters of a life, and truths — statements with supporting and contradicting evidence that only the owner can validate. Every node and edge carries its provenance down to the source element, and an origin ranked by trust (human > declared > structural > rule/computed > vendor > model > inferred).

2. Emotion measured across modalities, along unusual dimensions. The same fixed vocabulary of 53 emotions (Plutchik-based, each with valence and arousal coordinates) is applied to messages (local LLM), voice (five acoustic and multimodal layers fused with the transcript, with an explicit text/voice mismatch flag) and photos (deterministic colour and light measures, SigLIP embeddings, face signals treated as weak evidence). These sit in a catalogue of 929 dimensions in 88 groups that goes well beyond the usual metadata. A few examples of its less common entries:

modality	dimension	how it is defined
photo	colour temperature	mean red–blue balance of the image (warm vs cool)
photo	colourfulness	Hasler & Süsstrunk (2003) opponent-colour metric
photo	golden hour / blue hour	sun elevation computed offline from date and position
photo	face size	share of the frame taken by a face (a proxy for proximity)
voice	jitter, shimmer, harmonics-to-noise ratio	openSMILE eGeMAPS prosody
voice	laughter, crying, sighs	sound-event detection (AudioSet classes)
voice	text/voice mismatch	voice valence and transcript valence differ by ≥ 0.6
message	valence, arousal, tone	per participant, constrained to the fixed grid
message	expressive punctuation, emoji palette	deterministic text measures
relation	reply-latency asymmetry	log ratio of the two parties’ median reply delays
relation	initiatives	conversations opened after 3 h of silence, per side
relation	trend	slope of relationship intensity over 90 days (cooling < 0)
period	musical nostalgia	mean age of the tracks listened to
period	creation / consumption	time in creative tools ÷ time consuming

From signals to meaning

flowchart LR
 PH["Photo"] --> PM["Measures
light · contrast · colour temperature
colourfulness · palette · sharpness"]
 PH --> PS["SigLIP embedding
(image ↔ text similarity)"]
 PH --> PF["Vendor face signals
smile · expression · gaze · face size"]
 VN["Voice note"] --> VT["Whisper transcript"]
 VN --> VL["Acoustic layers
valence/arousal/dominance · categories
sound events · prosody · audio-LLM"]
 VT --> VX["Text emotion of transcript"]
 VL --> VF["Weighted fusion
confidence = agreement
text/voice mismatch flag"]
 VX --> VF
 MS["Messages
(pseudonymised: Me, P1, P2)"] --> ML["Local LLM, fixed grid
emotion · intensity · valence
arousal · tone · topics"]
 PM --> OB[("Observatory
929 dimensions")]
 PS --> OB
 PF --> OB
 VF --> OB
 ML --> OB
 OB -- "emotion with enough confidence" --> AFF["Affective episode (AFF)
who · emotion · when · intensity · modality"]
 PF -. "weak cue, confidence ≤ 0.4" .-> AFF
 ML -- "topics" --> THM["Themes"]
 OB -- "aggregates" --> AG["Day / period indicators
valence · light · palette"]
 AFF --> G["Meaning graph
relations · episodes · themes · chapters · truths
(all proposed)"]
 THM --> G
 AG --> G
 G --> HV(("Owner
validate · reject · correct"))

Each affective episode is a node linked to the person who felt it, to the emotion in the fixed vocabulary, and — when it exists — to the conversation or moment it occurred in. Inference rules then carry affect upwards (for instance, the affect of a chapter or of a routine), and strong aggregate correlations can become proposed truths with their measures as evidence.

Emotion measurement in detail

Vocabulary. The grid is fixed and versioned: Plutchik’s 8 primary emotions at 3 intensities, his 8 primary dyads, plus social and reflective emotions useful for a life (gratitude, tenderness, pride, shame, loneliness, nostalgia, wonder, relief, exhaustion…), 53 in total. Each carries starting coordinates in Russell’s circumplex — valence in [−1, 1], arousal in [0, 1] — meant to be calibrated later on the owner’s own validations. Plutchik oppositions are encoded, and health-app mood labels are mapped onto the grid.

Messages. A local LLM (Qwen3-30B-A3B) reads each conversation with participants replaced by “Me”, “P1”, “P2”, messages framed as data it must not obey, and media reduced to placeholders such as “[voice note 23 s]”. Output is constrained by a JSON schema: for each participant, the dominant emotion from the grid (or none), intensity, valence, arousal, a tone from a closed list of 19, and a confidence; plus 1–3 topics that become themes. Long conversations are reduced to their beginning and end; short ones are batched ten per call. An affective episode is proposed only when confidence is at least 0.3.

Voice. Each recording is converted to 16 kHz mono and passes through:

Whisper large-v3-turbo (MLX) for the transcript;
a wav2vec2 dimensional model giving valence, arousal and dominance on 15 s windows, weighted by window length, plus arousal variability;
emotion2vec+ large for categorical emotion, mapped onto the grid (kept at ≥ 0.4);
an AST AudioSet classifier for laughter, crying, sighs, shouting, whispering, music and ambience;
openSMILE eGeMAPS prosody: pitch (semitones) and its variability, loudness, jitter, shimmer, harmonics-to-noise ratio, voiced segments per second, mean pause length;
Qwen2.5-Omni-7B (llama.cpp, Metal), which listens to the audio and returns an emotion from the grid, a valence and short acoustic cues;
the text-emotion model above, applied to the transcript.

The categorical opinions (voice categories, audio-LLM, transcript) are combined by a weighted vote (weights 1.0, 1.0 and 0.8); valence is a weighted mean of the dimensional model, the audio-LLM and the transcript. Confidence is agreement between layers: one agreeing source gives 0.45, two give 0.65, three give 0.85, minus 0.1 for each source that disagrees. When the voice and the words disagree by 0.6 or more in valence (an “I’m fine” said sadly), a text/voice mismatch is flagged rather than averaged away. Laughter and crying scores are kept alongside.

Photos. On 256 px thumbnails, without any model: luminance, contrast, saturation, colour temperature, colourfulness, sharpness (variance of the Laplacian) and a 5-colour palette with named dominant hue. SigLIP so400m embeddings make the library searchable by text. Vendor face attributes (smile, smile type, expression, gaze, eye state, face size) are read as signals, but the schema states that a photographed smile is not a felt joy: the face modality is a weak cue whose confidence is capped at 0.4. Music mood is handled the same way: the mood of a track is never, on its own, evidence of the listener’s mood. At period level, valence is fused with a fixed precedence: declared > voice > text > face.

Human validation. Every episode, theme, chapter or truth is emitted as proposed. The PWA shows a validation queue — affective episodes ordered by confidence, with the evidence (the conversation or recording it came from) — and the owner validates, rejects or corrects. Only that human decision event turns a proposal into validated, an equivalence checked automatically. The planned next step is a small personal classifier on emotion2vec features, trained on these validations.

Architecture

flowchart TB
 S["Sources (read-only snapshots)
messaging · mail · photos · music · notes · calendar · exports"]
 C["Connectors
dedicated ingestion process, sole holder of disk access"]
 V["Vault (owner-only, 0700)
content-addressed raw store"]
 O["Observatory
facts + signals (SQLite)"]
 K["Meaning graph
append-only JSONL log → SQLite fold + FTS"]
 N["Nightly / hourly pipelines
thumbnails · measures · SigLIP · emotions
(GPU lock, one model at a time)"]
 M["MCP server (stdio, no port)
20 read-only / propose-only tools"]
 H["Hermes Agent
local LLM via Ollama"]
 P["PWA (HTTPS + TOTP)
chat · validation queue"]
 U(("Owner"))
 S --> C --> V --> O --> K
 N --> O
 O --> M
 K --> M
 M --> H --> P
 P -- "validate / reject / correct" --> K
 U -- "NetBird VPN" --> P

Raw data is snapshotted (SQLite opened in mode=ro&immutable=1) and never written to; everything derived lives in a separate owner-only vault, and the code repository holds no data. The meaning graph is built on the atoms library’s knowledge-base module: an append-only, schema-validated JSONL event log folded into a disposable SQLite index.

The schema itself is data, verified mechanically: 27 node types and 50 predicates, 9 provenance origins, 31 locator kinds tying every claim to its source element, the 53-emotion vocabulary, 17 inference rules, and 42 sources behind the 929 dimensions (610 native, 126 vendor-computed, 86 aggregates, 74 deterministic measures, 29 model interpretations).

Security model

Source content is assumed adversarial (prompt injection in a message or mail), and the model may hallucinate or misuse tools. Defences are structural:

No ambient capabilities. The agent framework (Hermes Agent) has 20 built-in toolsets disabled (terminal, files, code execution, web, browser, delegation, memory, cloud speech…); no cloud provider, update check or external login; the only model endpoint is a loopback Ollama server.
One narrow interface. Data is reachable only through a local MCP server over stdio, with closed schemas, validated arguments, explicit truncation and a content-free access log. Tools can propose, never decide.
Privilege separation and pseudonymisation. Only ingestion holds full-disk access; analysis has no network. Third parties are opaque codes, and a real name in an identifier is rejected by the schema.
Sensitivity levels (0–3) per element with a lower ceiling for remote sessions; remote access only through a NetBird VPN, TOTP and strict session cookies.
Ethical exclusions: vendor-inferred ethnicity, skin tone and gender are never read.

Models

All inference is local; one large model is loaded at a time under a shared GPU lock.

task	model	runtime
agent (tool use, ≥ 64k context)	gpt-oss:20b, qwen3:8b fallback	Ollama
text emotions, voice-transcript emotions	Qwen3-30B-A3B-Instruct-2507 (Q4)	Ollama
text / image embeddings	bge-m3 · SigLIP so400m	Ollama · PyTorch MPS
image reading	qwen3-vl:8b	Ollama
speech-to-text	Whisper large-v3-turbo	MLX
voice emotion (5 layers)	wav2vec2 dimensional · emotion2vec+ · AST · openSMILE eGeMAPS · Qwen2.5-Omni-7B	PyTorch MPS · CPU · llama.cpp (Metal)

Each model is smoke-tested on synthetic data only (a fictitious French sentence synthesised by a local TTS, then passed through the whole audio chain).

Evaluation & results

Agent answer quality. A 16-question benchmark (2 trap questions), with ground truth computed by calling the tools directly, scores tool choice, numerical fidelity (every number in the answer must appear in that turn’s tool outputs), correctness, trap handling and partial-period flagging: 73 % → 100 % after tool-description, argument-validation and configuration fixes.
Server-side error correction. With a reduced 8-tool interface, simple questions went from 76 % to 90 % and cross-source questions from 73 % to 90 %; 20 deliberately unusual questions scored 89 % and exposed silent-zero failures, since fixed.
Context and latency. Tool descriptions cut from 23.5 k to 9.5 k characters; a heavy aggregate query from 172 s to 0.1 s; all heavy benchmark queries ≤ 0.3 s.
Cross-exploration benchmark. 60 placeholder-only scenarios with oracle code; oracles pass 61/61, with no leak at remote sensitivity ceilings 1 and 2.
Verification. 56 named schema checks green; 631 native/vendor columns confirmed against real database structures; 1,235 Python unit tests, 15 JavaScript test files and 110 quality tests on real data that report counts only.

Limitations & next steps

Evaluation sets are small and authored by the system’s designer; they measure fidelity and regressions rather than general capability. Emotion outputs are hypotheses: the valence/arousal coordinates of the vocabulary are starting values, and personal calibration of voice emotion from validated examples is still pending, as is the larger target agent model. Agent answers take tens of seconds on a 20B model.

Stack

Python · SQLite / FTS5 · Ollama · llama.cpp (Metal) · MLX · PyTorch MPS · Hermes Agent · MCP · SigLIP · Whisper · emotion2vec · openSMILE · NetBird · PWA · launchd

Emotion Recognition on Boris Dupoux