Sandi Verdev

Case study

Turning 2-hour mixed-language lessons into speaker-labeled transcripts

Own project · Python, FastAPI, GPU · Oct – Nov 2026

Problem

Online language lessons mix Slovenian, Italian and English, often in one sentence, with a teacher and several students. Out of the box, Whisper-style transcription detects the language once per file and doesn't label who is speaking.

An LLM clean-up tends to "fix" the students' mistakes, which are the point of the lesson.

What I did

  • Kept the teacher and student audio tracks separate, so the track tells who speaks; on single-track audio, pyannote separates speakers.
  • Detected the language per chunk of up to 30 seconds instead of once per file.
  • Made the LLM clean-up a separate, optional step. The raw transcript is always kept; the clean-up only fixes mishearings, keeps the speakers' own words and the students' mistakes, and marks unclear parts instead of guessing.
  • Picked the clean-up model by testing candidates against the teacher's own transcript of 14 minutes. A "correct, don't rewrite" rule cut character errors by about 22% and the run-to-run swing from 5 points to 0.3.
  • Sandboxed the clean-up model: it gets no tools, so an instruction spoken in a recording can't make it run commands or touch files.
  • Built the echo-cancellation library from source and read its code before using it, since it comes from a little-known publisher, and pinned it to the audited version.
  • Ran one job at a time on one GPU: models load per job and unload after a minute idle, and a re-uploaded file is answered from the cache.

Result

Raw and cleaned transcript, two real excerptsTwo excerpts from a lesson. First: the student says 'un poco ricci', which the clean-up keeps; the teacher's 'porici' is fixed to 'Un po' ricci', and the student repeats it. Second: a student's Italian 'lujalna zepikolo', heard as Slovenian, becomes 'Lui ha un naso piccolo' in Italian, and 'Pikolo' becomes 'Piccolo'.Raw transcriptAfter the clean-upMishearing fixed, the student's own words keptStudentITun poco ricciITUn poco ricci.TeacherITporiciITUn po' ricci.StudentITun po' ricciITUn po' ricci.Wrong language and mishearing fixedStudentSLlujalna zepikolo.ITLui ha un naso piccolo.TeacherITun naso piccoloITUn naso piccolo.StudentSLPikolo.ITPiccolo.
Real lines from a lesson. The tag shows the language each line was heard in. Bold: changed by the clean-up. The student's "un poco" (Italian says "un po'") stays as said; the teacher corrects it in the lesson.

Stack

  • Python
  • FastAPI
  • faster-whisper (large-v3)
  • pyannote
  • WebRTC echo cancellation
  • LLM clean-up
  • CUDA