Can we trust the machine's transcript?

Why this is the hard, fair test

Think-aloud over live music is exactly the condition that breaks speech recognition: the speech is intermittent and buried under violin, piano, voice, or đàn tranh, and the model is tempted to fill the instrumental or quiet passages with invented text. We run each engine two ways — on the raw room recording and on an isolated voice stem (RoFormer removes the instrument) — and attach to every transcript segment the same audio reference: how much real voice is actually present at that moment (PANNs speech on the stem) and how much music (PANNs music on the room). Because the reference describes the audio, not the engine, it is a fair common ground for comparing machines.

Which machine hallucinates most

The engine people reach for first is not the one to trust. Whisper turbo (raw) transcribes over non-voice audio 49% of its time (≈ 702 minutes across the cohort); Deepgram Nova-3 (voice stem) is the cleanest at 29%. Whisper — the most widely used — loops and fabricates far more than the commercial engines, and its voice-stem run trades music-hallucination for a flood of subtitle-junk.

Sorted cleanest → dirtiest by the share of transcribed time spent over non-voice audio. Rates are given per session (loops) and per 1 000 words (artifacts) so engines with different coverage compare fairly — Whisper large-v3 has a raw transcript for only 12 sessions, the rest have all 26. Reference: v=voice-stem PANNs speech · m=room PANNs music (shared across engines). "Over non-voice" is an indicator weighted continuously by (1 − voice-presence), not an exact hallucination clock.

Why AssemblyAI shows the fewest fabrications — and why that is not the same as "most accurate"

What the ranking actually measures. Every number in the table is a fabrication signal — looped lines, caption-junk phrases, and text emitted where the voice stem shows no voice. AssemblyAI Universal-2 and Deepgram Nova-3 score low on all three; the two Whisper models score high. That is a real, measured difference, but it is a statement about how much each engine invents, not about how correctly it transcribes the words that were truly spoken.

The likely mechanism (why the commercial engines invent less). Whisper is an encoder–decoder language model with no built-in voice-activity gate: handed an instrumental or silent passage, it still tries to decode words, and its autoregressive decoder can fall into repetition loops — the 500-plus loops and the © transcript Emily Beynon credit are textbook Whisper failure modes, and it learned the "Thank you / thanks for watching" phrases from caption-heavy training data. AssemblyAI and Deepgram are production systems built around strong voice-activity detection and endpointing: when they are not confident speech is present, they tend to emit nothing. Emitting nothing over the instrument is exactly what removes the fabrications we count.

Why "cleanest" is not "most correct." The same caution has a cost: an engine that stays silent when unsure can also drop real but quiet speech — a false-clean, not a true-accurate. We cannot crown any engine "correct," because we have no human ground-truth transcript to score against (no word-error-rate). So "AssemblyAI is cleanest" means precisely "it fabricates the least of what we can detect," and no more.

What we actually trust, then. Not any single engine, but agreement across independent engines over audio where a voice is present. When AssemblyAI, Deepgram, and Whisper — which never see each other's output — all print the same line while the voice-presence signal is high, that convergence is strong evidence the words are real. That is why the reading transcript below is built on cross-engine agreement, and why AssemblyAI is used only as a practical backbone (the fewest fabrications to wade through), never as an oracle.

How we decide — the coding scheme

To stay honest we separate what is measured (raw, automatic) from what is inferred (a verdict, with a reason, checkable against the audio). Four layers. Your three words map onto the verdict layer: nó thật → real, nó giả → fabricated, bị tịt tiếng nên fill vào → dead-audio.

One session, adjudicated line by line — Jennifer

We routed the 45 most-flagged segments of Jennifer's session (violin + piano, 83 min) to an LLM, which classified each with a one-line reason. The load-bearing finding: fabrications and real speech often carry identical text — only the voice signal separates them.

Read it like the studio timeline — original, filtered, coded

One shared time axis, drawn the way the study's own timeline is. Bottom-up it stacks ORIGINAL (each engine a lane of blocks at their exact start–end times) → FILTER (the machine's automatic flags, as bracketed spans) → CODE (the adjudicated verdict, where a human/LLM has judged) — granular at the bottom, grouped upward with brackets. The top lane is the person: their own voice/music track, from the session audio, independent of any engine — the anchor everything hangs on. Drag the overview to jump; use the zoom to go from whole-session shape to word-by-word reading.

Verify every step yourself

Nothing here is a black box. Each stage is a small script you can open and re-run; the reader above is just the last stage made visible. This is the whole path from a raw audio file to a line on the screen:

1. Raw transcripts. Each engine's per-clip output sits untouched on the data drive (transcripts_<engine>/, …_vocalstem/ for the stem). We never edit them.
2. Load into segments. hallucination.py reads each format (Whisper JSON, Deepgram sentences, AssemblyAI words→sentences) into one shape {start, end, text} — every engine starts its first line "Weather is really bad for tuning." at ~13 s, so the parsing lines up.
3. Place on the session clock. Each clip's offset comes from alignment.json, matched by its VID id (not by sort order), so a clip that started 11 min in lands 11 min in.
4. Attach the human's voice track. audio_classification_vocals.json / audio_classification.json (PANNs) give voice- and music-presence for every second, computed once and shared by all engines — the anchor column.
5. Measure, threshold-free. analyze() counts repetition loops (≥3 identical in a row) and artifact phrases (a published junk list), and weights transcription by (1−voice-presence) — no tuned cut-offs.
6. Aggregate + align for reading. build_multiengine.py writes one JSON per session (multiengine/) with all engines on the shared clock plus the 1 s voice track; the cohort table above is results/COHORT_engines.json.
7. Judge, out loud. adjudicate.py routes the flagged lines and an LLM assigns real / fabricated / dead-audio with a one-line reason (adjudication/) — always overturnable by listening. The full dumps in dumps/ show every line, verdict-free.

What this supports, honestly

The point is not that ASR is useless — most words in every engine are real speech, and the cleaner commercial engines are genuinely usable as a first draft. The point is that even the strongest engines fabricate content — invented credits, looped subtitle-phrases, foreign-language junk over the instrument — at rates that make an unreviewed transcript unsafe as a record of what a person actually said. A line like © transcript Emily Beynon entering the dataset as if a participant said it is the exact risk: a fabricated attribution silently coded as data. And because fabrication and real speech can carry identical words, the text cannot police itself. For verbatim human-subjects analysis, a human must transcribe — or verify every line against the audio — before the text is treated as evidence. That is the evidence behind the requirement.

PANNs also gives the whole-session trend of talking vs. music vs. both at once — but "talking" there is room talk (researcher + participant combined), never attributed to a person, until speaker diarization / voice-ID is added. Still to come: speaker-attribution error, and feeding the human-verified transcripts into the separate coding / portraiture lane.