Automation on Boris Dupoux

Deterministic Process Automation Under Strong Security Constraints

Mon, 01 Jun 2026 00:00:00 +0000

Context

A client operating under strong security constraints manages several hundred sites and runs four recurring back-office processes every day: maintaining field equipment, handling recurring spurious equipment events, writing intervention reports for its customers, and checking that each site’s periodic health check has arrived. The inputs are scattered across three systems with very different access paths: PDF reports delivered by e-mail, a business web portal with no public API, and Excel workbooks maintained by hand over many years.

The goal: automate these processes on an office workstation while keeping a human operator as the final authority on every customer message.

Design constraints

No language model at run time. Every decision comes from explicit rules and every sentence from a template. This makes the robot’s output reproducible: the same inputs always produce the same drafts, so a new version can be compared word for word against production, and any change traces back to a rule or a data source. No customer data goes to an external model service. A vision-language model was used once, offline, during design, to map recorded operator screen sessions; it is not part of the deployed system.
Drafts, not sends. The robot only creates Outlook drafts; the operator reviews and sends them. A global kill switch forces draft mode regardless of per-process settings.
Local execution, no new infrastructure. Python on a Windows workstation, scheduled tasks, a shared NAS folder for state and configuration.
No plaintext secrets. Credentials and tokens live in the Windows Credential Manager (DPAPI); configuration is YAML the operator can edit without a rebuild.

Architecture

flowchart TB
 SCHED["Scheduled passes (morning / afternoon)"] --> ING
 MAIL["Mailbox (Microsoft Graph, read)"] --> ING["Ingestion PDF reports grouped by day"]
 ING --> PARSE["PDF parser (pypdf + regular expressions)"]
 PORTAL["Business web portal Playwright login → session token → JSON endpoints replayed"] --> RULES
 XLS["Reference & history workbooks (Excel)"] --> RULES
 PARSE --> RULES["Rule engine 4 processes, ordered rules & exclusions"]
 RULES --> TPL["Template composer French agreement, names, dates"]
 TPL --> DRAFTS["Outlook drafts (Graph, never sent)"]
 RULES --> TRACK["Tracking workbooks (atomic append)"]
 DRAFTS --> OP(["Operator reviews & sends"])
 OP -. "sent items: corrected recipients" .-> RULES
 NAS[("NAS: locks, heartbeats, journal, idempotence ledger")] --- RULES
 RULES --> HEALTH["Health alerts webhook + dead-man check"]

The four processes

Equipment maintenance. Fault lines parsed from the daily PDF reports are matched against each device’s battery and repair history. An ordered decision tree (first match wins) chooses between ignoring, waiting, battery replacement, device replacement or radio-range testing, and drafts a technician work order; each device on a multi-device order keeps its own decision.

Recurring spurious events. Event counts per site and per device over 24 hours are compared with thresholds to detect recurring spurious events. Portal data (an incident confirmed as real, the system being switched off minutes after a burst) overrides the counts; the outcome is an advisory e-mail to the customer, a work order, or nothing.

Intervention reports. For each on-site intervention, the robot rebuilds a single timeline from the portal’s event log, call log and field-staff records, handles events that straddle midnight, and drafts a customer report with the media files attached.

Periodic health checks. Sites whose periodic health check is missing are identified, the status of each connectivity channel is checked on the portal, and a customer e-mail, a technician work order or an internal note is drafted.

Composing an e-mail: from parameters to a draft

Each draft is the end of a long chain of explicit decisions. Counted in the source code (October 2026), the four rule engines have 53 terminal outcomes (each place where a rule returns a decision), mapped onto 37 decision types and 13 draft builders.

The customer advisory e-mail of the spurious-events process shows the scale. A site reaches it only after up to 14 ordered tests in the rule engine (16 terminal outcomes for this process alone). The composer then reads 21 inputs — 17 fields of the site’s decision record and customer record, the recipient list, the attachments, and two learned resources (zone names from the portal and a spelling dictionary learned from the operator’s own history workbook). Between the decision and the final text there are 108 conditional branch points (if statements, conditional expressions and boolean operands, counted on the syntax tree of the composer and the ten helpers it calls, before the HTML pass).

flowchart TB
 IN(["1 · Rule engine — 14 ordered tests, first match wins. Input: daily event reports · portal journal · customer record · learned site profile"]) --> T0

 T0["Blocking flag on the site record?"] -- yes --> X0["No e-mail"]
 T0 -- no --> T1["R6 · incident confirmed as real today?"]
 T1 -- yes --> X1["No e-mail"]
 T1 -- no --> T2["R17 · media events < 30 s apart across zones?"]
 T2 -- yes --> X2["Manual review"]
 T2 -- no --> T3["R12 · external cause confirmed on media?"]
 T3 -- no --> T4["R8 · ≥ 8 integrity events?"]
 T4 -- yes --> T4b["persisting ≥ 3 days?"]
 T4b -- yes --> X3["Technician work order"]
 T4b -- no --> X4["Keep watching"]
 T4 -- no --> T5["≥ 2 integrity events, ≥ half of the burst?"]
 T5 -- yes --> X5["No e-mail"]
 T5 -- no --> T6["R16 · ≥ 140 events, advisory ignored ≥ 4 weeks? (off by default)"]
 T6 -- yes --> X6["Billing notice"]
 T6 -- no --> T7["R15 · shared-building flag?"]
 T7 -- yes --> X7["Coordination e-mail"]
 T7 -- no --> T8["R5 · weekend + chronic-device flag?"]
 T8 -- yes --> X8["Temporary masking request"]
 T8 -- no --> T9["R1 · same device ≥ 7 events?"]
 T9 -- no --> T10["R2 · ≥ 5 devices and ≥ 8 events?"]
 T10 -- no --> T11["R4 · ≥ 8 events, ≥ 50 % 05:00–07:00?"]
 T11 -- no --> T12["R3 · ≥ 8 events, ≥ 60 % 12:00–18:00, ≥ 70 % in 4 h?"]
 T12 -- no --> T13["R1bis · site total ≥ 7 events?"]
 T13 -- no --> X9["No e-mail"]

 T3 & T9 & T10 & T11 & T12 & T13 -- yes --> ADV

 ADV(["2 · Composer — advisory e-mail, rule id carried along. Customer name: legal form removed, re-cased, acronyms kept"])
 ADV --> SUBJ["≥ 30 events?"]
 SUBJ -- yes --> S1["Subject with urgent prefix"]
 SUBJ -- no --> S2["Plain subject"]
 S1 & S2 --> GR["Contact cell"]
 GR -- "several people, or a role / organisation word" --> G1["Generic greeting"]
 GR -- "title + surname" --> G2["Greeting + normalised title (7 abbreviations) + SURNAME"]
 GR -- "surname only" --> G3["Greeting + SURNAME"]
 G1 & G2 & G3 --> DEV["Devices involved"]
 DEV -- one --> D1["Portal zone name if known, else cleaned label + jargon map + learned accents"]
 DEV -- several --> D2["Generic phrase + list of ≤ 6 by count, each with its device code if known"]
 D1 & D2 --> HND["System switched on/off within minutes of the burst? (R1–R4 only)"]
 HND -- "off just after" --> C1["Handling-error wording A"]
 HND -- "on just before" --> C2["Handling-error wording B"]
 HND -- no --> ACT["Site in activity? — all of: R1–R4 or R1bis · ≥ 2 devices · ≥ 10 events · no intervention in the window · professional site · ≥ 90 % in learned hours · media sensors only: ≥ 90 % while on"]
 ACT -- yes --> C3["Site-in-activity wording"]
 ACT -- no --> RUL["Rule that fired"]
 RUL -- "R1, R2" --> C4["Generic environmental cause"]
 RUL -- "R3, R4" --> C5["Staff-passage cause"]
 RUL -- R12 --> C6["External cause confirmed on media"]
 RUL -- R1bis --> C7["Cause to investigate on site"]
 C4 & C5 --> ATT["Media actually attached?"]
 ATT -- yes --> A1["+ reference to the attachments"]
 ATT -- no --> FLG
 A1 & C1 & C2 & C3 & C6 & C7 --> FLG["Site flag: frequent contractor works?"]
 FLG -- yes --> F1["Request: switch off before works"]
 FLG -- no --> F2["Request: treat the cause"]
 F1 & F2 --> REC["Addresses in the customer record?"]
 REC -- yes --> RC1["All addresses found"]
 REC -- no --> RC2["Internal fallback mailbox"]
 RC1 & RC2 --> DRAFT(["Plain text → Outlook HTML (unfolding, lists, fixed typography, signature) → drafts folder, never sent"])
 classDef q stroke-dasharray:4 3,stroke-width:2px;
 class T0,T1,T2,T3,T4,T4b,T5,T6,T7,T8,T9,T10,T11,T12,T13,SUBJ,GR,DEV,HND,ACT,RUL,ATT,FLG,REC q;

The wording space of this single e-mail: 2 subject forms, 3 greeting forms, 2 opening-sentence forms, 7 reachable cause wordings (two of them with or without a reference to attached media), 2 requested actions, and an optional device list of 2 to 6 lines. Every threshold above lives in the YAML configuration, not in the code.

Parameter	Source	Effect on the e-mail
Event count over 24 h	daily reports	figure in the opening sentence; ≥ 30 adds an urgent prefix to the subject
Rule that fired	rule engine	selects the cause wording; gates the handling-error and site-in-activity checks
Devices involved (cleaned, de-duplicated)	daily reports	one named device, or a generic phrase plus a list of up to 6 lines ordered by count
Device code per report line	daily reports	appended in parentheses to each listed device, only when known
Zone name of the main device	portal	replaces the raw report label when available
Learned spellings	operator’s history workbook	restores accents and casing of device names
Title and contact-name cell	customer record	one of 3 greeting forms; several people, a role or an organisation name fall back to the generic form
Company name	customer record	legal form removed, re-cased with acronyms kept, used in subject and body
Site type, learned opening hours, interventions in the window	tracking workbook, portal journal	enable or suppress the site-in-activity wording
System switched on/off around the burst	portal journal	handling-error wording (2 variants), overriding every other cause
Site flags	customer record	choice between the 2 requested actions
Attachments present	portal media	reference to the attachments, or nothing
Recipient addresses	customer record	every address found; otherwise an internal fallback mailbox

The other drafts draw on the same shared French-language helpers: the preposition before a site name agrees with its first noun (6 forms, from chez to à l’ and aux); report titles agree in gender and number; dates are written in full French form; a “Contact” heading becomes plural when several contacts are listed; and an 8-entry map turns technical equipment jargon into customer vocabulary.

Engineering for reliability

Idempotence. A daily ledger keyed on site, message type and subject hash prevents duplicate drafts; a cross-day register prevents re-issuing a work order before the previous intervention has happened. Missed days are caught up automatically.
Concurrency on a shared NAS. Exclusive lock files (O_CREAT|O_EXCL) with a time-to-live for orphans; two workstations publish heartbeats, and the secondary takes over only when the primary has been silent for two minutes.
Safe spreadsheet writes. Writes go through a temporary file and an atomic replace, with retries when the workbook is open by a user; inserted rows shift footer formulas so totals stay correct.
Network robustness. Graph calls handle token refresh, HTTP 429 Retry-After and 5xx back-off.
Explicit failure. A crash wrapper with no dependency on the main package sends a redacted alert and a dead-man ping even when imports fail; a sentinel raises an alert when the portal returns no data at all instead of reporting success.
Shadow benchmarking. A sandbox build replays a day in read-only mode and diffs its drafts against production. The acceptance metric is the share of drafts the operator sends without changing a single word.

Results

1,766 automated tests in the repository (October 2026), built on mock mail and portal adapters, with live-system tests marked separately.
An audit of the corpus of received reports showed that 9.5 % of event lines were being silently discarded, because event codes and labels are concatenated in the extracted PDF text; the parser was rewritten to recognise the code forms actually emitted.
Caching the fault-history workbook (1.49 MB, 15,533 rows) cut the equipment maintenance process on a replay of real reports from 161.1 s to 23.3 s; workbook opens fell from 26 to 3.
First clean shadow comparison against production: 14 drafts on each side, 14 in common, 0 missed, 0 added.

Stack

Python 3.11 · Microsoft Graph · MSAL · httpx · Playwright · pypdf · openpyxl · pydantic · keyring · pytest · PyInstaller

Field Identification of Recall-Affected Vehicles

Mon, 01 Jun 2026 00:00:00 +0000

Context

Safety recall campaigns depend on reaching every affected vehicle, but letters and phone calls leave a residual population of owners who never respond. In island and overseas territories the problem is sharper: network coverage is uneven, field logistics span several territories, and the registry of who drives what is incomplete. The practical question becomes a fleet-management one: which vehicles of the circulating fleet are still affected, where are they seen, and how far has each one progressed towards repair?

Approach

flowchart TB
 A["Recall listing
(scheduled file pull or manual CSV)"] --> B["Central server
import · versioning · model-name decoding"]
 B -->|"incremental diff"| C["Field units (laptops in agent vehicles)
local recall copy"]
 D["Camera stream"] --> E["Plate detection + OCR
(on-device, offline)"]
 E --> F["Format-aware correction
+ double-read confirmation"]
 F --> G{"Match against
local recall list?"}
 G -->|"no"| H["Discarded — nothing stored"]
 G -->|"yes"| I["Alert to agent
+ local queue"]
 I -->|"sync when network returns"| B
 B --> J["Dashboard
units · tours · detections · incidents"]
 J --> K["Owner contact workflow
identified → contacted → repaired"]

Methods

Data ingestion and versioning. The recall listing is pulled automatically on a schedule or imported by hand. Each import is diffed against the previous one with stable identifiers, versioned by hash, and only the delta is sent to the field units. Manufacturer type codes are translated into readable model names on import.

Matching under OCR noise. Plate detection and character recognition run on the field laptop with ONNX models, with no network dependency. Two safeguards reduce false matches: (i) a positional correction step uses the known structure of national plate formats (current and legacy) to resolve classic OCR confusions (O/0, I/1, S/5, B/8, Z/2, G/6), rejecting any ambiguous reading; (ii) a double-read confirmation only counts a plate when it is read twice within a short time window. A smartphone web page offers a manual fallback (photo or typed plate, plus VIN check-digit validation).

Offline-first synchronisation. Matches are queued locally and pushed to the central server on the next connection, so detections survive multi-day outages. Units report a heartbeat (software version, recall-list version, reading counters).

Field operations. Supervisors plan tours as ordered, geolocated checkpoints on a map, assign them to teams and units, reuse templates, and follow progress. Agents report incidents with a severity level. A contact workflow records the outcome of each owner approach, and the dashboard derives a conversion funnel (identified → approached → engaged → repaired) with weekly trends, the median delay to repair and a breakdown by territory.

Privacy by design. Plates that do not match the recall list are never stored or transmitted; retained records have fixed retention periods with automatic purge; actions are written to an audit log; dashboard access is role-based, including read-only accounts; data at rest is encrypted. A processing register was written alongside the code.

Deployment & operations (DevOps)

The system runs on two very different kinds of machine: Windows laptops in agents’ vehicles, which may stay offline for days, and one server per instance. After the first weeks in the field, most of the engineering work went into making both sides easy to deploy, observe and recover, by a single person, without breaking field work.

flowchart LR
 subgraph ADM["Admin workstation"]
 GIT["Git repository
committed tree only"]
 PROV["Provisioning script
one config file per unit"]
 BLD["Field build
PyInstaller bundle + ONNX models"]
 end
 subgraph SRV["Server (Ubuntu LTS VPS)"]
 CAD["Caddy
TLS · HSTS · no access log"]
 APP["Docker container
FastAPI API + dashboard
loopback port only"]
 VOL[("LUKS2 volume
SQLite · photos · archives")]
 TIM["systemd timers
monitoring · backup · cleanup"]
 end
 FLD["Field laptops
local SQLite · sync loop"]
 OBJ[("Off-site object storage
age-encrypted backups")]
 CHAT["Chat alerts
state changes + daily digests"]
 GIT -->|"git archive over SSH → deploy script"| APP
 PROV -->|"admin API"| APP
 PROV -->|"config file + README"| FLD
 BLD -->|"archive + one-click installer"| FLD
 FLD <-->|"HTTPS: heartbeat · pushes · recall diff"| CAD
 CAD --> APP
 APP --- VOL
 TIM --> VOL
 TIM -->|"encrypted copy"| OBJ
 TIM --> CHAT

Field laptops: packaging, installation and updates

Packaging. The field application (Python backend, React interface) is frozen with PyInstaller into a one-folder Windows executable that embeds the ONNX models and the OCR configuration, so a unit needs no Python install and no network to read plates. Built-in --diag and --selftest modes check the bundled runtime and read a test image without starting the interface, which made start-up failures diagnosable on the field machine itself.

One-click installer. The build is delivered as an archive with a double-click INSTALLER.bat. It elevates itself and runs a PowerShell script that removes the “downloaded from the Internet” mark from the files (otherwise the bundled runtime fails to load), disables USB power saving so the GPS dongle is not suspended, and installs the unit’s configuration file where the executable reads it. The script refuses to guess: if several different configuration files are present, it lists them and asks which one is the new one. An optional reset clears the local database when a unit moves to another server; it shows the target server before asking for confirmation and warns that unsynchronised detections would be lost. The unit’s access token is never displayed. An update follows the same path: new build, re-run the installer.

Provisioning. A Python script that uses only the standard library, so it runs on a bare Windows machine, creates the units from a CSV file through the server’s admin API and generates, for each unit, its configuration file (server address, unit code, display name, token) and a README to hand over with the laptop. The admin key is read from an environment variable or a masked prompt, never from a command-line argument.

Versioning: software and recall list

Software version. Each unit reports its application version in its heartbeat. The dashboard compares it with the expected release and flags the units that still need updating.

Recall-list version. The recall list is versioned by hash. On each cycle, a unit compares its local hash with the server’s: identical, nothing to do; no local copy, full download; otherwise, the server returns only the rows added and removed since that hash (and the unit falls back to a full download if the server no longer knows its version). The change is applied in a single transaction, and the local hash only moves forward if everything was applied.

That last rule came from a field incident. A database constraint made the diff fail on one unit, the error was swallowed by the sync loop, and the unit kept sending heartbeats while its recall list stayed frozen for more than two weeks, so it alerted on vehicles that had already been repaired. Release 1.4.1 fixed the cause, added an explicit rollback, and shipped a schema migration that keeps existing data. It was tested with a 62-test run that included reproducing the bug and migrating a database with the old schema. Monitoring now raises a “recall sync stuck” alert when a unit has not received a published list after a grace period.

Incremental, resilient synchronisation

Every five minutes, or on demand, a unit runs a sync cycle made of independent steps: heartbeat, push of detections (in batches), GPS track, incidents, checkpoint updates, then pull of the recall list and the tours. Pushes come before pulls, so field data leaves first. A failing step no longer blocks the others: in an earlier version the cycle was monolithic, and a failed heartbeat once delayed detections by 22 minutes. Requests use short timeouts, downloads longer ones. The heartbeat carries the software version, the recall-list version, the number of pending detections and the reading counters, which is what the dashboard and the monitoring rely on.

Server deployment

Instance kit. Each deployment is an independent instance described by a short identifier, on an Ubuntu LTS virtual server. A bootstrap script, which is interactive and can be safely re-run, installs the system packages, Docker and Caddy from their official repositories. It enables unattended security updates without automatic reboot, restricts SSH to key authentication, enables a default-deny firewall and brute-force protection, and checks that the DNS record points to the server before applying the TLS configuration. Secrets live in root-only configuration files on the server, never in the repository.

Container and reverse proxy. A multi-stage Dockerfile builds the React dashboard with Node, then runs the FastAPI server on a slim Python image as an unprivileged user. Its health check allows a three-minute start period because the plate and OCR models are loaded before the server listens. The container is published on the loopback interface only. Caddy is the only public entry point: automatic Let’s Encrypt certificate, explicit HSTS, and no access log, because scan URLs contain plates. The application’s own access log is disabled for the same reason.

Deploy and rollback. Code leaves the admin workstation only as a git archive of a committed tree: uncommitted changes are listed and not sent, and line endings are forced to LF. On the server, the new tree is unpacked next to the current one and swapped in, the previous one is kept, and the deployed commit is recorded. The deploy script then detaches from the SSH session, so a dropped connection interrupts neither the deployment nor a rollback. A lock prevents concurrent deployments. The script builds the image (labelled with its commit), keeps the previous image as rollback, recreates the container and waits until it reports healthy. It gives up after five minutes, or as soon as the container has restarted twice. On failure it rolls back automatically to the previous image and source tree, sets the failed code aside and sends an alert. Going back to any earlier version means re-deploying that commit. An image rollback does not undo a schema migration, so the documented procedure is to take a backup before any deployment that changes the schema.

Encrypted volume and safe restarts

Server data (database, contact photos, recall archives) lives on a LUKS2 file container mounted with restrictive options. Its passphrase is never stored on the machine, so after a reboot a person unlocks the volume. The container restarts on its own, but the server refuses to start unless a sentinel file on the encrypted volume is present. Without it, a reboot would have left an empty directory in place of the volume, and the application would have created a fresh, unencrypted database there and accepted field data into it. The guard was checked both ways on a throwaway container. A separate script grows the volume while it is mounted.

Backups and recovery

A daily systemd timer takes a consistent SQLite snapshot through the database’s backup API, archives it with the photos and recall archives, and encrypts the result with age to a public key. The private key is shown once, at creation, and kept off the server. Backups are kept locally and, optionally, copied to S3-compatible object storage with rclone, configured through environment variables only. A restore test mode decrypts a backup into a temporary directory on the encrypted volume, runs an integrity check, counts rows per table, then deletes everything. The apply mode moves the current state aside instead of deleting it. The documentation includes a step-by-step rebuild after total loss of the server. The continuity plan targets a recovery time of 4 hours and a maximum data loss of 24 hours.

Monitoring

A watchdog runs every 15 minutes and only posts to a chat webhook when a state changes (problem, then “resolved”). A fuller digest is sent twice a day, timed to the field shifts. It checks that the encrypted volume is mounted, the API health endpoint (with retries, because the daily import can block it for about a minute), the container health status, that the database is readable, disk usage on both the system disk and the encrypted volume, the size of the SQLite write-ahead log, TLS certificate expiry, backup age and size, units silent for more than 48 hours (not 24, since units are switched off at weekends) and units whose recall list is stuck. A separate watcher alerts when no recall import has happened for more than 30 hours. Several of these checks were added after real incidents: the encrypted volume once reached 90 % without any alert because only the system disk was watched. A weekly timer cleans up Docker build caches and keeps only the current and rollback images. A dedicated script truncates the SQLite log with a short lock wait, so field writes are never blocked for long.

Offline resilience

The field side is designed to keep working through a server outage: plate reading, matching against the local recall copy, alerts, tours already downloaded and the local statistics tab all work without a network, and the queue empties at the next connection. Only the dashboard (and the server-hosted smartphone page) is unavailable while the server is down.

Testing

There is no CI pipeline in the repository. Releases are tested before they ship, and the commits record those runs: 12/12 and 6/6 for the plate correction and confirmation stages, 62 tests for release 1.4.1. Behaviours that cannot be unit-tested, such as the volume guard or a live deployment, were checked on throwaway containers and against the production health endpoint.

Results

In early field operation, agents flagged 32 % of detections as misreadings over roughly one week. This measurement motivated the positional-correction and double-read confirmation stages, and the dashboard statistics now exclude detections marked as false positives.
The new stages shipped with unit tests: 12/12 for plate correction and 6/6 for confirmation, settings and reading log.
After the model-name mapping was extended, only one row of the production recall listing remained without a decoded model name.

No end-to-end recognition rate has been measured yet.

Limitations & next steps

Recognition accuracy has not been measured against a ground-truth benchmark. The specification targets (read rate, latency) are design goals, not results.
No legacy-format plate reading was observed at first. A dedicated counter now tracks these readings end to end, but the fix still needs a field check.
A false match against a large listing still costs agent time. Calibrating the confidence threshold and confirmation window on labelled field video is the next step.

Stack

Python · FastAPI · SQLite · ONNX Runtime · fast-alpr (YOLOv9 plate detector + OCR) · OpenCV · PaddleOCR · React · TypeScript · Vite · Leaflet · Docker · Caddy · LUKS · systemd · age · rclone · PyInstaller · PowerShell

From Multitrack Recordings to an Editable Timeline: Transcription, Diarisation and AAF Export

Mon, 01 Jun 2026 00:00:00 +0000

Context

A conference recorded on several microphones (speakers, room, music) produces hours of raw multitrack audio. Most editing time goes into locating material — who speaks when, on which track, where the silences are — before any cutting.

Open-source speech tooling (Whisper and its derivatives, pyannote, Silero VAD) solves transcription and “who spoke when” well, but stops at text. The question addressed here is whether that analysis can be delivered as an editing object: a multitrack timeline in AAF, the interchange format that audio workstations open natively, with tracks qualified and dead time removed.

Before → after

One real job processed end to end: a conference recorded on 16 tracks. Everything on the right-hand side was produced automatically, with no manual cut; figures are read from the job’s analysis database and from the delivered AAF.

	Input (raw recording)	Generated output (AAF timeline)
Tracks	16 × 1 h 32 min 54 s (≈ 24 h 46 min of audio to review)	16 tracks kept in the AAF, 6 carrying clips
Track roles	4 speech · 2 music (one auto-flagged as ambience) · 10 silent	10 silent tracks left empty, no manual triage
Material	204 speech regions and 5 music passages detected	41 clips (37 speech, 4 music)
Cuts	—	40 hard cuts between sources, of which 33 microphone switches
Transitions	—	0 crossfades: straight cuts, each speech clip extended by a 0.3 s tail
Dead time	—	3 gaps ≥ 4.5 s rippled out (36.9 s); timeline 1 h 32 min 17 s

Top: the 16 input tracks over the full recording, coloured by the detected role and activity; the four speech microphones are active almost all the time because each one also picks up its neighbours. Bottom: the generated edit, one lane per track as in the AAF — a single source is kept at any moment, ticks mark the 40 source switches, hatched areas in the top panel are the removed dead time.

Speech clips are few and long (median 79 s, longest 10 min 11 s): the montage follows the active speaker rather than chopping on every breath, and micro-switches shorter than 0.8 s are absorbed by the anti-flicker smoothing. Ten clips are still shorter than 6 s — short interjections, the first place to look when checking the edit.

Approach

flowchart TB
 A["Multitrack WAV recording"] --> B["Decode to 16 kHz mono + RMS energy per 100 ms"]
 B --> C["Voice activity detection (Silero)"]
 C --> D["Transcription (Whisper large-v3, chunked)"]
 D --> E["Forced word alignment (WhisperX / wav2vec2)"]
 C --> F["Speaker diarisation (pyannote)"]
 E --> G["Speaker assignment + segment post-processing"]
 F --> G
 B --> H["Track role: speech / music / silent"]
 G --> H
 H --> I["Montage: active microphone per 100 ms frame"]
 I --> J["AAF export (linked to original WAV names)"]
 I --> K["Mono mixdown WAV"]

Methods

Transcription. Whisper large-v3 via faster-whisper (CTranslate2), run on fixed-length chunks inside an isolated, persistent worker process: the model is loaded once, and a native crash only loses one chunk, which is reported rather than silently dropped. Decoding uses beam search with Silero VAD filtering and condition_on_previous_text disabled to limit repetition loops.

Word timing and speakers. Word-level timestamps come from WhisperX forced alignment (wav2vec2). Speaker turns come from pyannote diarisation and are assigned to words by temporal overlap.

Post-processing. Pure filters remove known Whisper hallucinations (stock subtitle credits, “thanks for watching”-type phrases, music-only segments), non-Latin output on French audio and duplicates, then merge consecutive same-speaker segments.

Track qualification and montage. Each track is classified as speech, music or silent by combining frame energy, VAD coverage and transcript statistics. The montage selects, for every 100 ms frame, the microphone with the highest signal-to-noise ratio above its own noise floor (reducing cross-talk “bleed”), applies anti-flicker smoothing and ripples out global dead time.

Synchronisation. Separate recorders are aligned by matching distinctive words across transcripts (robust median of offsets), after a coarse clock-based placement corrected for clock drift.

Export. The edit list is written as a multitrack AAF with pyaaf2, either embedding the audio or linking to it.

Deployment. A web portal accepts jobs and a separate compute worker polls it over HTTPS. Clients upload only compressed analysis copies together with a manifest of their original files; the delivered AAF references the original WAV names, so the editor relinks full-quality sources locally and the originals never leave the client’s machine.

Results

Benchmark (bench_stt.py) on real conference audio: one-minute speech windows selected by VAD on four speech tracks, Whisper large-v3 throughout, Apple Silicon laptop, warm timings excluding model loading. No ground-truth transcript was available, so quality figures are divergence from a pseudo-reference (WhisperX output), not absolute error.

Pipeline (CPU)	Real-time factor
openai-whisper	0.43
WhisperX	0.51
WhisperX + forced alignment	0.54
openai-whisper + word timestamps	0.57
faster-whisper (beam 5 + VAD)	0.66
faster-whisper + word timestamps	0.70

Word timing: native word timestamps deviate from forced alignment by a median of ~205 ms (faster-whisper) and ~210 ms (openai-whisper); forced alignment adds about 1.2 s per minute of audio.
Transcript divergence: 7–11 % word-level divergence between pipelines, mostly punctuation, segmentation and normalisation on manual inspection.
Repetitions: 1–2 % repeated content; no hallucinated passages observed on these clips.
Track detection: the three music/silence tracks of the seven-track recording were identified by VAD and skipped automatically.

Limitations & next steps

Quality is measured against a pseudo-reference; a manually transcribed test set is needed to report true word error rate and diarisation error rate.
Benchmark clips are short (one minute per track) and all single-speaker windows, so diarisation accuracy on multi-speaker overlap is not yet quantified.
CTranslate2 has no Metal backend: on Apple Silicon, faster-whisper and WhisperX run on CPU, while CUDA is used where available. Device choice per stage was benchmarked rather than assumed; a CUDA comparison remains to be measured.
AAF export targets Logic Pro import; other editors have not been systematically tested.

Stack

Python · faster-whisper · CTranslate2 · WhisperX · pyannote.audio · Silero VAD · PyTorch (MPS / CUDA) · ffmpeg · pyaaf2 · SQLite · PHP · MySQL