Fine-tuning STT to my speech: 24% to 6% WER Part 2

Earlier I finetuned Whisper to my speech and got Whisper Turbo down to ~6.5% WER, but I faced some issues and it was not that usable for me. Despite being v3-turbo, it was still very slow for day to day use.

With so many new models coming out, I started looking around what I could use to make my workload faster. I had few parameters in mind:

1. Fast inference
2. Easy to finetune
3. <= 6% WER on my speech

WER (Word Error Rate) is the % of words the model gets wrong, lower is better.

Some options I considered:

1. Whisper Small
2. Nvidia Parakeet
3. Moonshine

Whisper Small is a small model that is fast and easy to finetune but if Turbo got 6% WER, I don’t think we can get better than that with Small. Moonshine might be good but has few resources on finetuning. Parakeet is quite popular and has lot of resources so I decided to go with it.

Parakeet Architecture:

While whisper is a full attention encoder, Parakeet encoder is a Conformer, a hybrid of attention and convolution layers. Its decoder is a small LSTM plus a joint network. The easiest learning to borrow from our previous whisper finetuning is that we need to freeze the decoder (and joint) and only train the encoder.

Another big difference is Whisper always pads the audio to 30s and then process the whole 30s but Parakeet processes only the actual audio length, another reason for faster inference.

Finetuning:

The dataset remains the same, 720 Harvard sentences which I have recorded. More details here. 648 (90%) in training and 72 (10%) in test set.

Base Results: Parakeet scores 22-25% WER on my speech. Much better than Whisper Turbo (53%) but yeah not usable directly.

Training:

While I trained Whisper using LoRA, for Parakeet I decided to finetune the whole encoder. The decision was quite simple - the standard finetuning process for Parakeet is finetuning the model rather than LoRA adapter. For Whisper, it’s LoRA adapter imo because it’s a slow model due to 30s padding so finetuning full model is hard on limited RAM.

1st Attempt:

Learning Rate - 2e-5 WER on all 720 clips - 6.7% Test WER - 11%

1st run was quite impressive, WER looked good while things being much faster. I decided to give it a try on 1 new sentence and it was a disaster around 40% WER on that one. I thought of it as one off case so tried again, 50% this time.

This doesn’t make sense, so started doing some digging and here’s the interesting insight:

All the samples in my training set were ~2s long with no silence at the start or end. In real life audio, there’s always some silence around speech, you press the record button and then start speaking so like some delay is always there. Similarly you think, so some delay comes there as well. My training audio was complete content and no reality.

Parakeet learned the pattern quite well and new sentence with silence it was a disaster. I tried sending another fast sentence just like my hardvard sentences and it went through perfectly. So now we know the cause and need to find a way to fix it.

Ideal way out - Start recording fresh, be more human while recording rather than a bot speaking in a lab. But I was not ready to spend another 2 hours in a silent room speaking 720 sentences. Claude to my rescue again - It suggested me to record 60s of silence and it can add a copy of each recording with the silence in start and end.

So now the training set had both, 648 original clips and 648 copies with random 0.2-1.5s of my room silence at start and end, 1,296 in total. The test set got the same padded copy, and I picked the best run based on WER with silence since that’s closer to real life.

Sounds good enough, so I gave it a try and here’s how the 2nd run look like -

Holy 2.96% WER on all 720 clips and 6.14% on test set. We improved even more than expected. Just to check things better, I add the recorded silence on the test set to do another measurement.

1st run scored 43% (1s silence at the end) to 89% (silence on both sides, mostly empty output) WER but the 2nd run scored 5.29% WER, so silence was working. We had good WER with silence and without silence. Passed my 2 sentences and it passed with flying colors.

Here’s both runs side by side:

Note: 2nd run changed 3 things, not just silence. 2x the clips (so 2x the steps) and a lower learning rate (2e-5 to 1e-5, so it forgets less of what it already knew).

Looks like we have a winner quite easily. I wanted to use it in daily workflow first before tweaking more parameters.

Inference:

I had OpenWhispr installed on my system, so I decided to use it to test this model only as they support Parakeet by default through sherpa-onnx (ONNX runtime). So I needed to convert the model to ONNX. By default the float models are quite big, so it was ideal to convert it to int8.

Conversion did bump up the WER a bit (6.7% to 7.9% on test set, measured with sherpa-onnx which scores a bit differently than the 6.14% from training) but it made the encoder 4x smaller (2.4GB to 622MB) so overall good deal.

OpenWhispr doesn’t support custom models, I am a rare user I get it. But the fix is quite simple, I just downloaded Parakeet in OpenWhispr and then replaced its onnx files (encoder, decoder, joiner and tokens) with my own.

Results:

It’s fast enough that I can use it in daily workflows. On my M4 Pro it takes ~0.06s to transcribe a 2-3s clip on CPU (int8, 4 threads). It’s been 2 days since I started using it and I am quite happy with the results. Let’s see how it goes in long term.

All this training was done on Kaggle free tier this time.