From Multitrack Recordings to an Editable Timeline: Transcription, Diarisation and AAF Export

Mon, 01 Jun 2026 00:00:00 +0000

Context

A conference recorded on several microphones (speakers, room, music) produces hours of raw multitrack audio. Most editing time goes into locating material — who speaks when, on which track, where the silences are — before any cutting.

Open-source speech tooling (Whisper and its derivatives, pyannote, Silero VAD) solves transcription and “who spoke when” well, but stops at text. The question addressed here is whether that analysis can be delivered as an editing object: a multitrack timeline in AAF, the interchange format that audio workstations open natively, with tracks qualified and dead time removed.

Before → after

One real job processed end to end: a conference recorded on 16 tracks. Everything on the right-hand side was produced automatically, with no manual cut; figures are read from the job’s analysis database and from the delivered AAF. This job was transcribed with WhisperX (with pyannote diarisation), not faster-whisper. Edit quality has not yet been evaluated against a human edit (n = 1 conference).

	Input (raw recording)	Generated output (AAF timeline)
Tracks	16 × 1 h 32 min 54 s (≈ 24 h 46 min of audio); 10 of the 16 tracks were near-digital silence, 6 carried signal (≈ 9 h 17 min)	16 tracks kept in the AAF, 6 carrying clips
Track roles	4 speech · 2 music (one auto-flagged as ambience) · 10 silent	10 silent tracks left empty, no manual triage
Material	204 speech regions and 5 music passages detected	41 clips (37 speech, 4 music)
Cuts	—	40 hard cuts between sources, of which 33 microphone switches
Transitions	—	0 crossfades: straight cuts, each speech clip extended by a 0.3 s tail
Dead time	—	head/tail trimmed and one internal 16 s gap removed (36.9 s total); timeline 1 h 32 min 17 s

Top: the 16 input tracks over the full recording, coloured by the detected role and activity; the four speech microphones are active almost all the time because each one also picks up its neighbours. Bottom: the generated edit, one lane per track as in the AAF — a single source is kept at any moment, ticks mark the 40 source switches, hatched areas in the top panel are the removed dead time.

Speech clips are few and long (median 79 s, longest 10 min 11 s): the montage follows the active speaker rather than chopping on every breath, and micro-switches shorter than 0.8 s are absorbed by the anti-flicker smoothing. Ten clips are still shorter than 6 s — short interjections, the first place to look when checking the edit.

The silent tracks still produced stock Whisper hallucinations in their raw transcripts; they are excluded from the edit by track classification.

Approach

flowchart TB
 A["Multitrack WAV recording"] --> B["Decode to 16 kHz mono + RMS energy per 100 ms"]
 B --> C["Voice activity detection (Silero)"]
 C --> D["Transcription (Whisper large-v3, chunked)"]
 D --> E["Forced word alignment (WhisperX / wav2vec2)"]
 C --> F["Speaker diarisation (pyannote)"]
 E --> G["Speaker assignment + segment post-processing"]
 F --> G
 B --> H["Track role: speech / music / silent"]
 G --> H
 H --> I["Montage: active microphone per 100 ms frame"]
 I --> J["AAF export (linked to original WAV names)"]
 I --> K["Mono mixdown WAV"]

Methods

Transcription. Whisper large-v3, either through WhisperX (used for the case study above) or via faster-whisper (CTranslate2). The faster-whisper path runs on fixed-length chunks inside an isolated, persistent worker process: the model is loaded once, and a native crash only loses one chunk, which is reported rather than silently dropped. Decoding uses beam search with Silero VAD filtering and condition_on_previous_text disabled to limit repetition loops.

Word timing and speakers. Word-level timestamps come from WhisperX forced alignment (wav2vec2). Speaker turns come from pyannote diarisation and are assigned to words by temporal overlap.

Post-processing. Pure filters remove known Whisper hallucinations (stock subtitle credits, “thanks for watching”-type phrases, music-only segments), non-Latin output on French audio and duplicates, then merge consecutive same-speaker segments.

Track qualification and montage. Each track is classified as speech, music or silent by combining frame energy, VAD coverage and transcript statistics. The montage selects, per 100 ms frame, the microphone using each speech segment’s signal-to-noise ratio above the track’s noise floor (reducing cross-talk “bleed”), applies anti-flicker smoothing and ripples out global dead time.

Synchronisation. Separate recorders are aligned by matching distinctive words across transcripts (robust median of offsets), after a coarse clock-based placement corrected for clock drift.

Export. The edit list is written as a multitrack AAF with pyaaf2, either embedding the audio or linking to it.

Deployment. A web portal accepts jobs and a separate compute worker polls it over HTTPS. Clients upload only compressed analysis copies together with a manifest of their original files; the delivered AAF references the original WAV names, so the editor relinks full-quality sources locally and the originals never leave the client’s machine.

Results

Benchmark (bench_stt.py) on real conference audio from a separate seven-track recording, distinct from the 16-track case study above: one-minute speech windows selected by VAD on four speech tracks, Whisper large-v3 throughout, Apple Silicon laptop, warm timings excluding model loading. No ground-truth transcript was available, so quality figures are divergence from a pseudo-reference (WhisperX output), not absolute error.

Real-time factors (4 × 1 min, single run; decoding settings differ per pipeline):

Pipeline (CPU)	Real-time factor
openai-whisper	0.43
WhisperX	0.51
WhisperX + forced alignment	0.54
openai-whisper + word timestamps	0.57
faster-whisper (beam 5 + VAD)	0.66
faster-whisper + word timestamps	0.70

Word timing: native word timestamps deviate from forced alignment by a median of ~205 ms (faster-whisper) and ~210 ms (openai-whisper); forced alignment adds about 1.2 s per minute of audio.
Transcript divergence: 7–11 % word-level divergence between pipelines (pseudo-reference, 4 × 1-minute clips of the separate 7-track recording). The metric strips case and punctuation, so on manual inspection the divergence is largely word segmentation, spelling of names and clip-edge effects.
Repetitions: 1.1–2.4 % repeated content; no hallucinated passages observed on these clips.
Track detection: the three music/silence tracks of the seven-track recording were identified by VAD and skipped automatically.

Limitations & next steps

Quality is measured against a pseudo-reference; a manually transcribed test set is needed to report true word error rate and diarisation error rate.
Edit quality has not yet been evaluated against a human edit (n = 1 conference).
Benchmark clips are short (one minute per track) and all single-speaker windows, so diarisation accuracy on multi-speaker overlap is not yet quantified.
CTranslate2 has no Metal backend: on Apple Silicon, faster-whisper and WhisperX run on CPU, while CUDA is used where available. Device choice per stage was benchmarked rather than assumed; a CUDA comparison remains to be measured.
AAF export targets Logic Pro import; other editors have not been systematically tested.

Stack

Python · faster-whisper · CTranslate2 · WhisperX · pyannote.audio · Silero VAD · PyTorch (MPS / CUDA) · ffmpeg · pyaaf2 · SQLite · PHP · MySQL

Diarisation on Boris Dupoux