Problem
Online language lessons mix Slovenian, Italian and English, often in one sentence, with a teacher and several students. Out of the box, Whisper-style transcription detects the language once per file and doesn't label who is speaking.
An LLM clean-up tends to "fix" the students' mistakes, which are the point of the lesson.
What I did
- Kept the teacher and student audio tracks separate, so the track tells who speaks; on single-track audio, pyannote separates speakers.
- Detected the language per chunk of up to 30 seconds instead of once per file.
- Made the LLM clean-up a separate, optional step. The raw transcript is always kept; the clean-up only fixes mishearings, keeps the speakers' own words and the students' mistakes, and marks unclear parts instead of guessing.
- Picked the clean-up model by testing candidates against the teacher's own transcript of 14 minutes. A "correct, don't rewrite" rule cut character errors by about 22% and the run-to-run swing from 5 points to 0.3.
- Sandboxed the clean-up model: it gets no tools, so an instruction spoken in a recording can't make it run commands or touch files.
- Built the echo-cancellation library from source and read its code before using it, since it comes from a little-known publisher, and pinned it to the audited version.
- Ran one job at a time on one GPU: models load per job and unload after a minute idle, and a re-uploaded file is answered from the cache.
Result
Swipe sideways to see the whole chart →
Stack
- Python
- FastAPI
- faster-whisper (large-v3)
- pyannote
- WebRTC echo cancellation
- LLM clean-up
- CUDA