Robust Speech Recognition via Large-Scale Weak Supervision

# Robust Speech Recognition via Large-Scale Weak Supervision Alec Radford * 1 Jong Wook Kim * 1 Tao Xu 1 Greg Brockman 1 Christine McLeavey 1 Ilya Sutskever 1 Abstract We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness.

We are releasing models and inference code to serve as a foundation for further work on robust speech processing. 1. Introduction Progress in speech recognition has been energized by the development of unsupervised pre-training techniques exem-plified by Wav2Vec 2.0 (Baevski et, 2020). Since these methods learn directly from raw audio without the need for human labels, they can productively use large datasets of un-labeled speech and have been quickly scaled up to 1,000,000 hours of training data (Zhang et, 2021), far more than the 1,000 or so hours typical of an academic supervised dataset.

When fine-tuned on standard benchmarks, this approach has improved the state of the art, especially in a low-data setting. These pre-trained audio encoders learn high-quality repre-sentations of speech, but because they are purely unsuper-vised they lack an equivalently performant decoder mapping those representations to usable outputs, necessitating a fine-tuning stage in order to actually perform a task such as speech recognition 1. This unfortunately limits their use-fulness and impact as fine-tuning can still be a complex process requiring a skilled practitioner. There is an addi-tional risk with requiring fine-tuning. Machine learning > * Equal contribution 1OpenAI, San Francisco, CA 94110, USA.

Correspondence to: Alec Radford com >, Jong Wook Kim com >. > 1 Baevski et al. (2021) is an exciting exception - having devel-oped a fully unsupervised speech recognition system methods are exceedingly adept at finding patterns within a training dataset which boost performance on held-out data from the same dataset. However, some of these patterns are brittle and spurious and don’t generalize to other datasets and distributions. In a particularly disturbing example, Rad-ford et al. (2021) documented a 9.2% increase in object classification accuracy when fine-tuning a computer vision model on the ImageNet dataset (Russakovsky et

, 2015) without observing any improvement in average accuracy when classifying the same objects on seven other natural image datasets. A model that achieves “superhuman” per-formance when trained on a dataset can still make many basic errors when evaluated on another, possibly precisely because it is exploiting those dataset-specific quirks that humans are oblivious to (Geirhos et, 2020). This suggests that while unsupervised pre-training has im-proved the quality of audio encoders dramatically, the lack of an equivalently high-quality pre-trained decoder, com-bined with a recommended protocol of dataset-specific fine-tuning, is a crucial weakness which limits their usefulness and robustness.

The goal of a speech recognition system should be to work reliably “out of the box” in a broad range of environments without requiring supervised fine-tuning of a decoder for every deployment distribution. As demonstrated by Narayanan et al. (2018), Likhomanenko et al. (2020), and Chan et al. (2021) speech recognition sys-tems that are pre-trained in a supervised fashion across many datasets/domains exhibit higher robustness and generalize much more effectively to held-out datasets than models trained on a single source. These works achieve this by combining as many existing high-quality speech recogni-tion datasets as possible.

However, there is still only a moderate amount of this data easily available. SpeechStew (Chan et, 2021) mixes together 7 pre-existing datasets totalling 5,140 hours of supervision. While not insignifi-cant, this is still tiny compared to the previously mentioned 1,000,000 hours of unlabeled speech data utilized in Zhang et al. (2021). Recognizing the limiting size of existing high-quality super-vised datasets, recent efforts have created larger datasets for speech recognition. By relaxing the requirement of gold-standard human-validated transcripts, Chen et al. (2021) and Galvez et al. (2021) make use of sophisticated automated > arXiv:2212.04356v1 AS] 6 Dec 2022 Robust Speech Recognition via Large-Scale Weak Supervision 2pipelines to scale weakly supervised speech recognition to 10,000 and

30,000 hours of noisier training data. This trade-off between quality and quantity is often the right call. Although understudied so far for speech recognition, recent work in computer vision has demonstrated that mov-ing beyond gold-standard crowdsourced datasets such as ImageNet (Russakovsky et, 2015) to much larger but weakly supervised datasets significantly improves the ro-bustness and generalization of models (Mahajan et, 2018; Kolesnikov et, 2020). Yet these new datasets are only a few times larger than the sum of existing high-quality datasets and still much smaller than prior unsupervised work. In this work we close that gap, scaling weakly supervised speech recognition the next order of magnitude to 680,000 hours of labeled audio data.

We call our approach Whisper 2. We demonstrate models trained at this scale transfer well to existing datasets zero-shot, removing the need for any dataset-specific fine-tuning to achieve high-quality results. In addition to scale, our work also focuses on broaden-ing the scope of weakly supervised pre-training beyond English-only speech recognition to be both multilingual and multitask. Of those 680,000 hours of audio, 117,000 hours cover 96 other languages. The dataset also includes 125,000 hours of X→en translation data. We find that for sufficiently large models there is no drawback and even benefits to joint multilingual and multitask training.

Our work suggests that simple scaling of weakly supervised pre-training has been underappreciated so far for speech recognition. We achieve these results without the need for the self-supervision or self-training techniques that have been a mainstay of recent large-scale speech recognition work. To serve as a foundation for further research on robust speech recognition, we release inference code and models at the following URL: com/openai/ whisper. 2. Approach 2.1. Data Processing Following the trend of recent work leveraging web-scale text from the internet for training machine learning systems, we take a minimalist approach to data pre-processing.

In contrast to a lot of work on speech recognition, we train Whisper models to predict the raw text of transcripts without any significant standardization, relying on the expressive-ness of sequence-to-sequence models to learn to map be-tween utterances and their transcribed form. This simplifies > 2If an acronym or basis for the name is desired, WSPSR stand-ing for Web-scale Supervised Pretraining for Speech Recognition can be used. the speech recognition pipeline since it removes the need for a separate inverse text normalization step in order to produce naturalistic transcriptions. We construct the dataset from audio that is paired with tran-scripts on the Internet.

This results in a very diverse dataset covering a broad distribution of audio from many different environments, recording setups, speakers, and languages. While diversity in audio quality can help train a model to be robust, diversity in transcript quality is not similarly bene-ficial. Initial inspection showed a large amount of subpar transcripts in the raw dataset. To address this, we developed several automated filtering methods to improve transcript quality. Many transcripts on the internet are not actually human-generated but the output of existing ASR systems. Recent research has shown that training on datasets of mixed human and machine-generated data can significantly impair the per-formance of translation systems (Ghorbani et

, 2021). In order to avoid learning “transcript-ese”, we developed many heuristics to detect and remove machine-generated tran-scripts from the training dataset. Many existing ASR sys-tems output only a limited subset of written language which removes or normalizes away aspects that are difficult to pre-dict from only audio signals such as complex punctuation (exclamation points, commas, and question marks), format-ting whitespace such as paragraphs, or stylistic aspects such as capitalization. An all-uppercase or all-lowercase tran-script is very unlikely to be human generated. While many ASR systems include some level of inverse text normaliza-tion, it is often simple or rule-based and still detectable from other unhandled aspects such as never including commas.

We also use an audio language detector, which was created by fine-tuning a prototype model trained on a prototype ver-sion of the dataset on VoxLingua107 (Valk & Alum ¨ae, 2021) to ensure that the spoken language matches the language of the transcript according to CLD2. If the two do not match, we don’t include the (audio, transcript) pair as a speech recognition training example in the dataset. We make an exception if the transcript language is English and add these pairs to the dataset as X→en speech translation training examples instead. We use fuzzy de-duping of transcript texts to reduce the amount of duplication and automatically generated content in the training dataset.

We break audio files into 30-second segments paired with the subset of the transcript that occurs within that time segment. We train on all audio, including segments where there is no speech (though with sub-sampled probability) and use these segments as training data for voice activity detection. For an additional filtering pass, after training an initial model we aggregated information about its error rate on training Robust Speech Recognition via Large-Scale Weak Supervision 3data sources and performed manual inspection of these data sources sorting by a combination of both high error rate and data source size in order to identify and remove low-quality ones efficiently.

This inspection showed a large amount of only partially transcribed or poorly aligned/misaligned tran-scripts as well as remaining low-quality machine-generated captions that filtering heuristics did not detect. To avoid contamination, we perform de-duplication at a tran-script level between the training dataset and the evaluation datasets we thought were at higher risk of overlap, namely TED-LIUM 3 (Hernandez et, 2018). 2.2. Model Since the focus of our work is on studying the capabilities of large-scale supervised pre-training for speech recogni-tion, we use an off-the-shelf architecture to avoid confound-ing our findings with model improvements. We chose an encoder-decoder Transformer (Vaswani et

, 2017) as this architecture has been well validated to scale reliably. All audio is re-sampled to 16,000 Hz, and an 80-channel log-magnitude Mel spectrogram representation is computed on 25-millisecond windows with a stride of 10 milliseconds. For feature normalization, we globally scale the input to be between -1 and 1 with approximately zero mean across the pre-training dataset. The encoder processes this input representation with a small stem consisting of two convolu-tion layers with a filter width of 3 and the GELU activation function (Hendrycks & Gimpel, 2016) where the second convolution layer has a stride of two.

Sinusoidal position embeddings are then added to the output of the stem after which the encoder Transformer blocks are applied. The transformer uses pre-activation residual blocks (Child et, 2019), and a final layer normalization is applied to the en-coder output. The decoder uses learned position embeddings and tied input-output token representations (Press & Wolf, 2017). The encoder and decoder have the same width and number of transformer blocks. Figure 1 summarizes the model architecture. We use the same byte-level BPE text tokenizer used in GPT-2 (Sennrich et, 2015; Radford et, 2019) for the English-only models and refit the vocabulary (but keep the same size) for the multilingual models to avoid excessive fragmenta-tion on other languages since the GPT-2 BPE vocabulary is English only.

2.3. Multitask Format Although predicting which words were spoken in a given audio snippet is a core part of the full speech recognition problem and extensively studied in research, it is not the only part. A fully featured speech recognition system can involve many additional components such as voice activ-ity detection, speaker diarization, and inverse text normal-ization. These components are often handled separately, resulting in a relatively complex system around the core speech recognition model. To reduce this complexity, we would like to have a single model perform the entire speech processing pipeline, not just the core recognition part.

An important consideration here is the interface for the model. There are many different tasks that can be performed on the same input audio signal: transcription, translation, voice activity detection, alignment, and language identification are some examples. For this kind of one-to-many mapping to work with a single model, some form of task specification is necessary. We use a simple format to specify all tasks and conditioning infor-mation as a sequence of input tokens to the decoder. Since our decoder is an audio-conditional language model, we also train it to condition on the history of text of the transcript in the hope that it will learn to use longer-range text context to resolve ambiguous audio.

Specifically, with some proba-bility we add the transcript text preceding the current audio segment to the decoder’s context. We indicate the beginning of prediction with a <|startoftranscript|> token. First, we predict the language being spoken which is repre-sented by a unique token for each language in our training set (99 total). These language targets are sourced from the aforementioned VoxLingua107 model. In the case where there is no speech in an audio segment, the model is trained to predict a <|nospeech|> token indicating this. The next token specifies the task (either transcription or trans-lation) with an <|transcribe|> or <|translate|> token.

After this, we specify whether to predict timestamps or not by including a <|notimestamps|> token for that case. At this point, the task and desired format is fully specified, and the output begins. For timestamp predic-tion, we predict time relative to the current audio segment, quantizing all times to the nearest 20 milliseconds which matches the native time resolution of Whisper models, and add additional tokens to our vocabulary for each of these. We interleave their prediction with the caption tokens: the start time token is predicted before each caption’s text, and the end time token is predicted after.

When a final tran-script segment is only partially included in the current 30-second audio chunk, we predict only its start time token for the segment when in timestamp mode, to indicate that the subsequent decoding should be performed on an au-dio window aligned with that time, otherwise we truncate the audio to not include the segment. Lastly, we add a <|endoftranscript|> token. We only mask out the training loss over the previous context text, and train the model to predict all other tokens. Please see Figure 1 for an overview of our format and training setup.

Robust Speech Recognition via Large-Scale Weak Supervision 4⋯ ⋯ > 2× Conv1D + GELU ⋮ > cross attention > Log-Mel Spectrogram # ~ > SOT EN TRANS- > CRIBE 0.0 The quick > Tokens in Multitask Training Format > Transformer > Encoder Blocks Transformer > Decoder Blocks > EN 0.0 The quick brown ⋮ ⋮ > next-token > prediction > Sinusoidal > Positional > Encoding > Learned > Positional > Encoding Multitask training data (680k hours) Sequence-to-sequence learning Multitask training format English transcription Any-to-English speech translation Non-English transcription No speech 🗣 “Ask not what your country can do for ⋯” 📝 Ask not what your country can do for ⋯ 🗣 “El rápido zorro marrón salta sobre ⋯” 📝 The quick brown fox jumps over ⋯ 🗣 “언덕 위에 올라 내려다보면 너무나 넓고 넓은 ⋯” 📝 언덕 위에 올라 내려다보면 너무나 넓고 넓은 ⋯ 🔊 (background music playing) 📝 ∅ > PREV > special > tokens > text > tokens > timestamp > tokens > START OF > TRANSCRIPT > LANGUAGE > TAG > NO > SPEECH > EOT > TRANSCRIBE > TRANSLATE > begin > time > NO > TIMESTAMPS ⋯ > end > time > text tokens begin > time > end > time > text tokens > text tokens > Voice activity > detection > (VAD) > Custom vocabulary / > prompting > Time-aligned transcription > Text-only transcription > (allows dataset-specific fine-tuning) > X→ English > Translation > previous > text tokens > X→ X > Transcription > Language > identification > MLP > self attention > MLP > self attention > MLP > self attention > MLP > cross attention > self attention > MLP > cross attention > self attention > MLP > cross attention > self attention > TRANS- > CRIBE Figure 1.

Overview of our approach. A sequence-to-sequence Transformer model is trained on many different speech processing tasks, including multilingual speech recognition, speech translation, spoken language identification, and voice activity detection. All of these tasks are jointly represented as a sequence of tokens to be predicted by the decoder, allowing for a single model to replace many different stages of a traditional speech processing pipeline. The multitask training format uses a set of special tokens that serve as task specifiers or classification targets, as further explained in Section 2.3. 2.4. Training Details We train a suite of models of various sizes in order to study the scaling properties of Whisper.

Please see Table 1 for an overview. We train with data parallelism across accelerators using FP16 with dynamic loss scaling and activation check-pointing (Griewank & Walther, 2000; Chen et, 2016). Models were trained with AdamW (Loshchilov & Hutter, 2017) and gradient norm clipping (Pascanu et, 2013) with a linear learning rate decay to zero after a warmup over the first 2048 updates. A batch size of 256 segments was used, and the models are trained for 2 20 updates which is between two and three passes over the dataset. Due to only training for a few epochs, over-fitting is not a large concern, and we do not use any data augmentation or regularization and instead rely on the diversity contained within such a large dataset to encourage generalization and robustness.

Please see Appendix F for full training hyperparameters. 3 During early development and evaluation we observed that Whisper models had a tendency to transcribe plausible but almost always incorrect guesses for the names of speakers. This happens because many transcripts in the pre-training dataset include the name of the person who is speaking, encouraging the model to try to predict them, but this infor-mation is only rarely inferable from only the most recent 30 > 3 After the original release of Whisper, we trained an additional Large model (denoted V2) for 2.5X more epochs while adding SpecAugment (Park et

, 2019), Stochastic Depth (Huang et, 2016), and BPE Dropout (Provilkov et, 2019) for regularization. Reported results have been updated to this improved model unless otherwise specified. Robust Speech Recognition via Large-Scale Weak Supervision 5 > Model Layers Width Heads Parameters Tiny 4384 639M Base 6512 874M Small 12 768 12 244M Medium 24 1024 16 769M Large 32 1280 20 1550M > Table 1. Architecture details of the Whisper model family. seconds of audio context. To avoid this, we fine-tune Whis-per models briefly on the subset of transcripts that do not include speaker annotations which removes this behavior.

3. Experiments 3.1. Zero-shot Evaluation The goal of Whisper is to develop a single robust speech processing system that works reliably without the need for dataset specific fine-tuning to achieve high-quality results on specific distributions. To study this capability, we re-use a wide set of existing speech processing datasets to check whether Whisper is able to generalize well across domains, tasks, and languages. Instead of using the standard evaluation protocol for these datasets, which include both a train and test split, we evaluate Whisper in a zero-shot setting without using any of the training data for each of these datasets so that we are measuring broad generalization.

3.2. Evaluation Metrics Speech recognition research typically evaluates and com-pares systems based on the word error rate (WER) metric. However, WER, which is based on string edit distance, pe-nalizes all differences between the model’s output and the reference transcript including innocuous differences in tran-script style. As a result, systems that output transcripts that would be judged as correct by humans can still have a large WER due to minor formatting differences. While this poses a problem for all transcribers, it is particularly acute for zero-shot models like Whisper, which do not observe any examples of specific datasets transcript formats.

This is not a novel observation; the development of evalua-tion metrics that better correlate with human judgement is an active area of research, and while there are some promising methods, none have seen widespread adoption for speech recognition yet. We opt to address this problem with ex-tensive standardization of text before the WER calculation to minimize penalization of non-semantic differences. Our text normalizer was developed through iterative manual in-spection to identify common patterns where naive WER penalized Whisper models for an innocuous difference. Ap-pendix C includes full details. For several datasets, we observe WER drops of up to 50 percent usually due to a quirk such as a dataset’s reference transcripts seperating contractions from words with whitespace.

We caution this development procedure comes at a risk of overfitting to the transcription style of Whisper models which we investigate in Section 4.4. We are releasing the code for our text nor-malizer to allow for easy comparison and to help others study the performance of speech recognition systems in out-of-distribution settings. 3.3. English Speech Recognition In 2015, Deep Speech 2 (Amodei et, 2015) reported a speech recognition system matched human-level perfor-mance when transcribing the LibriSpeech test-clean split. As part of their analysis they concluded: “Given this result, we suspect that there is little room for a generic speech sys-tem to further improve on clean read speech without further domain

” Yet seven years later the SOTA WER on LibriSpeech test-clean has dropped another 73% from their 5.3% to 1.4% (Zhang et, 2021), far below their re-ported human-level error rate of 5.8%. Despite this massive and unanticipated further improvement in performance on held-out but in-distribution data, speech recognition mod-els trained on LibriSpeech remain far above human error rates when used in other settings. What explains this gap between reportedly superhuman performance in-distribution and subhuman performance out-of-distribution? We suspect a large part of this gap between human and machine behavior is due to conflating different capabilities being measured by human and machine performance on a test set.

This claim may seem confusing at first; if both humans and machines are taking the same test, how can it be that different skills are being tested? The difference arises not in the testing but in how they trained for it. Humans are often asked to perform a task given little to no supervision on the specific data distribution being studied. Thus human performance is a measure of out-of-distribution generaliza-tion. But machine learning models are usually evaluated after training on a large amount of supervision from the evaluation distribution, meaning that machine performance is instead a measure of in-distribution generalization.

While both humans and machines are being evaluated on the same test data, two quite different abilities are being measured due to a difference in train data. Whisper models, which are trained on a broad and diverse distribution of audio and evaluated in a zero-shot setting, could potentially match human behavior much better than existing systems. To study whether this is the case (or whether the difference between machine and human per-formance is due to yet-to-be-understood factors) we can compare Whisper models with both human performance and standard fine-tuned machine learning models and check which they more closely match.

Robust Speech Recognition via Large-Scale Weak Supervision 60 1 2 3 4 5 6 7 8 > WER on LibriSpeech dev-clean (%) > 0 > 10 > 20 > 30 > 40 > 50 > Average WER on [Common Voice, CHiME-6, TED-LIUM] (%) > Supervised LibriSpeech models > Zero-shot Whisper models > Zero-shot Human (Alec) > Ideal robustness (y = x) Figure 2. Zero-shot Whisper models close the gap to human robustness. Despite matching or outperforming a human on Lib-riSpeech dev-clean, supervised LibriSpeech models make roughly twice as many errors as a human on other datasets demonstrating their brittleness and lack of robustness.

The estimated robustness frontier of zero-shot Whisper models, however, includes the 95% confidence interval for this particular human. To quantify this difference, we examine both overall ro-bustness, that is average performance across many distribu-tions/datasets, and effective robustness, introduced by Taori et al. (2020), which measures the difference in expected performance between a reference dataset, which is usually in-distribution, and one or more out-of-distribution datasets. A model with high effective robustness does better than expected on out-of-distribution datasets as a function of its performance on the reference dataset and approaches the ideal of equal performance on all datasets.

For our analy-sis, we use LibriSpeech as the reference dataset due to its central role in modern speech recognition research and the availability of many released models trained on it, which allows for characterizing robustness behaviors. We use a suite of 12 other academic speech recognition datasets to study out-of-distribution behaviors. Full details about these datasets can be found in Appendix A. Our main findings are summarized in Figure 2 and Table 2. Although the best zero-shot Whisper model has a relatively unremarkable LibriSpeech clean-test WER of 2.5, which is roughly the performance of modern supervised baseline or the mid-2019 state of the art, zero-shot Whisper models have very different robustness properties than supervised LibriSpeech models and out-perform all

benchmarked Lib-riSpeech models by large amounts on other datasets. Even wav2vec 2.0 Whisper RER Dataset Large (no LM) Large V2 (%) LibriSpeech Clean 2.7 2.7 0.0 Artie 24.5 6.2 74.7 Common Voice 29.9 9.0 69.9 Fleurs En 14.6 4.4 69.9 Tedlium 10.5 4.0 61.9 CHiME6 65.8 25.5 61.2 VoxPopuli En 17.9 7.3 59.2 CORAAL 35.6 16.2 54.5 AMI IHM 37.0 16.9 54.3 Switchboard 28.3 13.8 51.2 CallHome 34.8 17.6 49.4 WSJ 7.7 3.9 49.4 AMI SDM1 67.6 36.4 46.2 LibriSpeech Other 6.2 5.2 16.1 Average 29.3 12.8 55.2 Table 2. Detailed comparison of effective robustness across various datasets.

Although both models perform within 0.1% of each other on LibriSpeech, a zero-shot Whisper model performs much better on other datasets than expected for its LibriSpeech performance and makes 55.2% less errors on average. Results reported in word error rate (WER) for both models after applying our text normalizer. the smallest zero-shot Whisper model, which has only 39 million parameters and a 6.7 WER on LibriSpeech test-clean is roughly competitive with the best supervised LibriSpeech model when evaluated on other datasets. When compared to a human in Figure 2, the best zero-shot Whisper models roughly match their accuracy and robustness.

For a detailed breakdown of this large improvement in robustness, Table 2 compares the performance of the best zero-shot Whisper model with a supervised LibriSpeech model that has the closest performance to it on LibriSpeech test-clean. Despite their very close performance on the reference distribution, the zero-shot Whisper model achieves an average relative error reduction of 55.2% when evaluated on other speech recognition datasets. This finding suggests emphasizing zero-shot and out-of-distribution evaluations of models, particularly when at-tempting to compare to human performance, to avoid over-stating the capabilities of machine learning systems due to misleading comparisons.

3.4. Multi-lingual Speech Recognition In order to compare to prior work on multilingual speech recognition, we report results on two low-data benchmarks: Multilingual LibriSpeech (MLS) (Pratap et, 2020b) and VoxPopuli (Wang et, 2021) in Table 3. Whisper performs well on Multilingual LibriSpeech, out-performing XLS-R (Babu et, 2021), mSLAM (Bapna Robust Speech Recognition via Large-Scale Weak Supervision 70.1 1 10 100 1K 10K 100K 1M > Hours of transcribed audio > 2.5 > 5 > 10 > 20 > 40 > 80 > 160 > Word Error Rate (WER) > r2= 0.83 > SW > PT > JA > FI > ML > FR > RO > GL > KO > UK > NE > LO > AZ > MK > LT > NL > MS > GU > IS > MY > CA > TE > TR > CS > NB > AR > AF > HR > UZ > DE > VI > LV > ID > PL > SV > TA > FA > HY > TH > BN > KM > EN > HU > UR > BS > KA > ZH > SL > SK > CY > RU > BG > FIL > EL > HI > KN > MT > BE > HE > IT > MR > PA > DA > ES > KK > TG > ET > SR Figure 3.

Correlation of pre-training supervision amount with downstream speech recognition performance. The amount of pre-training speech recognition data for a given language is very predictive of zero-shot performance on that language in Fleurs. Model MLS VoxPopuli VP-10K + FT - 15.3 XLS-R (1B) 10.9 10.6 mSLAM-CTC (2B) 9.7 9.1 Maestro - 8.1 Zero-Shot Whisper 7.3 13.6 Table 3. Multilingual speech recognition performance. Zero-shot Whisper improves performance on Multilingual LibriSpeech (MLS) but is still significantly behind both Maestro, XLS-R, and mSLAM on VoxPopuli. et, 2022), and Maestro (Chen et, 2022b) in a zero-shot setting. We caution that we do use a simple text standardizer for this result which prevents direct comparison or claims of SOTA performance.

On VoxPopuli, however, Whisper significantly underperforms prior work and only beats the VP-10K+FT baseline from the original paper. We suspect the underperformance of Whisper models on VoxPopuli could be due to other models including this distribution as a major source for their unsupervised pre-training data and the dataset having significantly more supervised data, which benefits fine-tuning. While MLS has 10 hours of training data per language, the average amount of training data per language is roughly 10 × higher for VoxPopuli. These two benchmarks are somewhat narrow since they only include 15 unique languages, almost all of which are in 1 10 100 1K 10K 100K > Hours of translated audio > 0 > 5 > 10 > 15 > 20 > 25 > 30 > 35 > 40 > BLEU > r2= 0.24 > HR > AM > NL > MY > SW > EL > NE TH > KN > PA > DA > AR > MI > BG > ML > MR > TE > SV > IT > FIL > GL > RO > UK > FA > UZ > BE > KM > TG > AS > ET > OC > CA > IS > KK > HE > FR AF > VI > HA > MT > LO > BN > PT > HU > FI > KO > SD > ID > UR > LN > LV > AZ > YO > LB > CY > HY > PL > LT > DE > KA > RU > MK MS > SR > ES > ZH > JA > NB BS > MN SN > TR > PS > SK > SO > CS > SL > HI > GU > TA Figure 4.

Correlation of pre-training supervision amount with downstream translation performance. The amount of pre-training translation data for a given language is only moderately predictive of Whisper’s zero-shot performance on that language in Fleurs. the Indo-European language family and many of which are high-resource languages. These benchmarks only provide limited coverage and room to study Whisper models multi-lingual capabilities which include training data for speech recognition in 75 languages. To study the performance of Whisper more broadly we also report performance on the Fleurs dataset (Conneau et, 2022). In particular, we were interested in studying the relationship between the amount of training data we have for a given language and the result-ing downstream zero-shot performance for that language.

We visualize this relation in Figure 3. We find a strong squared correlation coefficient of 0.83 between the log of the word error rate and the log of the amount of training data per language. Checking the regression coefficient for a linear fit to these log-log values results in an estimate that WER halves for every 16 × increase in training data. We also observed that many of the largest outliers in terms of worse than expected performance according to this trend are languages that have unique scripts and are more distantly related to the Indo-European languages making up the ma-jority of the training dataset such as Hebrew (HE), Telugu (TE), Chinese (ZH), and Korean (KO).

These differences could be due to a lack of transfer due to linguistic distance, our byte level BPE tokenizer being a poor match for these languages, or variations in data quality. Robust Speech Recognition via Large-Scale Weak Supervision 8X → English High Mid Low All XMEF-X 34.2 20.2 5.9 14.7 XLS-R (2B) 36.1 27.7 15.1 22.1 mSLAM-CTC (2B) 37.8 29.6 18.5 24.8 Maestro 38.2 31.3 18.4 25.2 Zero-Shot Whisper 36.2 32.6 25.2 29.1 Table 4. X→en Speech translation performance. Zero-shot Whisper outperforms existing models on CoVoST2 in the overall, medium, and low resource settings but still moderately under-performs on high-resource languages compared to prior directly supervised work.

Language ID Fleurs w2v-bert-51 (0.6B) 71.4 mSLAM-CTC (2B) 77.7 Zero-shot Whisper 64.5 Table 5. Language identification performance. Zero-shot Whis-per’s accuracy at language identification is not competitive with prior supervised results on Fleurs. This is partially due to Whisper being heavily penalized for having no training data for 20 of Fleurs languages. 3.5. Translation We study the translation capabilities of Whisper models by measuring their performance on the X→en subset of CoVoST2 (Wang et, 2020b). We compare with Maestro, mSLAM, and XLS-R, the highest-performing prior work. We achieve a new state of the art of 29.1 BLEU zero-shot without using any of the CoVoST2 training data.

We at-tribute this to the 68,000 hours of X→en translation data for these languages in our pre-training dataset which, al-though noisy, is vastly larger than the 861 hours of training data for X→en translation in CoVoST2. Since Whisper eval-uation is zero-shot, it does particularly well on the lowest resource grouping of CoVoST2, improving over mSLAM by 6.7 BLEU. Conversely, the best Whisper model does not actually improve over Maestro and mSLAM on average for the highest resource languages. For an additional analysis on an even wider set of languages, we also re-purpose Fleurs, which is a speech recognition dataset, as a translation dataset.

Since the same sentences are transcribed for every language we use the English tran-scripts as reference translations. In Figure 4 we visualize the correlation between the amount of translation training data per language and the resulting zero-shot BLEU score on Fleurs. While there is a clear trend of improvement with increasing training data, the squared correlation coefficient is much lower than the 0.83 observed for speech recognition 40 30 20 10 0 -10 > signal-to-noise ratio (dB) > 1 > 2 > 5 > 10 > 20 > 50 > 100 > WER on LibriSpeech test-clean (%) white noise > 40 30 20 10 0-10 > signal-to-noise ratio (dB) pub noise > unispeech-sat-base-100h-libri-ft > wav2vec2-base-100h > wav2vec2-base-960h > wav2vec2-large-960h > wav2vec2-large-robust-ft-libri-960h > wav2vec2-large-960h-lv60-self > asr-crdnn-rnnlm-librispeech > asr-transformer-transformerlm-librispeech > hubert-large-ls960-ft > hubert-xlarge-ls960-ft > s2t-medium-librispeech-asr > s2t-large-librispeech-asr > stt_en_conformer_ctc_large > stt_en_conformer_transducer_xlarge > Whisper Figure 5.

WER on LibriSpeech test-clean as a function of SNR under additive white noise (left) and pub noise (right). The accuracy of LibriSpeech-trained models degrade faster than the best Whisper model (F). NVIDIA STT models (•) perform best under low noise but are outperformed by Whisper under high noise (SNR < 10 dB). The second-best model under low noise (H) is fine-tuned on LibriSpeech only and degrades even more quickly. and only 0.24. We suspect this is partly caused by the noisier training data due to errors in audio language identification. As an example, Welsh (CY) is an outlier with much worse than expected performance at only 13 BLEU despite sup-posedly having 9,000 hours of translation data.

This large amount of Welsh translation data is surprising, ranking 4th overall for translation data and ahead of some of the most spoken languages in the world like French, Spanish, and Russian. Inspection shows the majority of supposedly Welsh translation data is actually English audio with English cap-tions where the English audio was mis-classified as Welsh by the language identification system, resulting in it being included as translation training data rather transcription data according to our dataset creation rules. 3.6. Language Identification To evaluate language identification, we use the Fleurs dataset (Conneau et, 2022).

The zero-shot performance of Whisper is not competitive with prior supervised work here and underperforms the supervised SOTA by 13.6%. However, Whisper is heavily disadvantaged for language identification on Fleurs, since the Whisper dataset contains no training data for 20 of the 102 languages in Fleurs, upper-bounding accuracy at 80.4%. On the 82 overlapping lan-guages the best Whisper model achieves 80.3% accuracy. Robust Speech Recognition via Large-Scale Weak Supervision 9 3.7. Robustness to Additive Noise We tested the noise robustness of Whisper models and 14 LibriSpeech-trained models by measuring the WER when either white noise or pub noise from the Audio Degrada-tion Toolbox (Mauch & Ewert, 2013) was added to the audio.

The pub noise represents a more natural noisy envi-ronment with ambient noise and indistinct chatter typical in a crowded restaurant or a pub. Among the 14 models, twelve are pre-trained and/or fine-tuned on LibriSpeech, and the other two are NVIDIA STT models trained on a mixture dataset similar to prior work like SpeechStew that includes LibriSpeech. The level of additive noise corresponding to a given signal-to-noise ratio (SNR) is calculated based on the signal power of individual examples. Figure 5 shows how the ASR performance degrades as the additive noise becomes more intensive. There are many models that out-perform our zero-shot performance under low noise (40 dB SNR), which is unsurprising given those models are trained primarily on LibriSpeech, but all models quickly degrade as the noise becomes more intensive, performing worse than the Whisper model under additive pub noise of SNR below 10 dB.

This showcases Whisper’s robustness to noise, es-pecially under more natural distribution shifts like the pub noise. 3.8. Long-form Transcription Whisper models are trained on 30-second audio chunks and cannot consume longer audio inputs at once. This is not a problem with most academic datasets comprised of short utterances but presents challenges in real-world applications which often require transcribing minutes- or hours-long au-dio. We developed a strategy to perform buffered transcrip-tion of long audio by consecutively transcribing 30-second segments of audio and shifting the window according to the timestamps predicted by the model. We observed that it is crucial to have beam search and temperature scheduling based on the repetitiveness and the log probability of the model predictions in order to reliably transcribe long audio.

The full procedure is described in Section 4.5. We evaluate the long-form transcription performance on seven datasets consisting of speech recordings of various lengths and recording conditions, to cover as diverse a data distribution as possible. These include a long-form adapta-tion of TED-LIUM3 (Hernandez et, 2018) concatenated so that each example is a full-length TED talk, a collection of jargon-laden segments taken from The Late Show with Stephen Colbert (Meanwhile), sets of videos/podcasts that has been used as ASR benchmarks in online blogs (Rev16 and Kincaid46), recordings of earnings calls (Del Rio et, 2021), and the full-length interviews from the Corpus of Regional African American Language (CORAAL) (Gunter et

, 2021). Full details about the long-form datasets can be found in Appendix A. We compare the performance with open-source models as well as 4 commercial ASR services. The results are sum-marized in Figure 6, showing the distribution of word error rates from Whisper and the 4 commercial ASR services, TED-LIUM3 Meanwhile Kincaid46 Rev16 Earnings-21 Earnings-22 CORAAL > 0 > 5 > 10 > 15 > 20 > 25 > 30 > 35 > 40 > Word Error Rate (%) > Whisper Company A Company B Company C Company D NVIDIA STT (CTC large) > Figure 6.

Whisper is competitive with state-of-the-art commercial and open-source ASR systems in long-form transcription. The distribution of word error rates from six ASR systems on seven long-form datasets are compared, where the input lengths range from a few minutes to a few hours. The boxes show the quartiles of per-example WERs, and the per-dataset aggregate WERs are annotated on each box. Our model outperforms the best open source model (NVIDIA STT) on all datasets, and in most cases, commercial ASR systems as well. Robust Speech Recognition via Large-Scale Weak Supervision 10 as well as the NVIDIA STT Conformer-CTC Large model from the NeMo toolkit (Kuchaiev et

, 2019) which per-formed the best among the open-source models. All com-mercial ASR services are queried using their default English transcription settings as of September 1st, 2022, and for the NVIDIA STT model we used their buffered inference implementation in the FrameBatchASR class to enable long-form transcription. The results show that Whisper per-forms better than the compared models on most datasets, especially on the Meanwhile dataset which is heavy with uncommon words. Additionally, we note the possibility that some of the commercial ASR systems have been trained on some of these publicly available datasets, and therefore these results may not be accurately reflecting the relative robustness of the systems.

3.9. Comparison with Human Performance Because of ambiguous or indistinct speech as well as la-beling errors, there are different levels of irreducible error in each dataset, and with WER metrics from ASR systems alone it is difficult to make sense of how much room for improvement exists in each dataset. To quantify how close Whisper’s performance is to the human performance, we se-lected 25 recordings from the Kincaid46 dataset and used 5 services to obtain transcripts produced by professional tran-scribers, among which one provides computer-assisted tran-scription and the other four are entirely human-transcribed. The audio selection covers various recording conditions such as scripted and unscripted broadcast, telephone and VoIP calls, and meetings.

Figure 7 shows the distribution of per-example WERs and aggregate WER across the 25 recordings, where the computer-assisted service has the lowest aggregate WER that is 1.15% point better than Whis-per’s, and the pure-human performance is only a fraction of a percentage point better than Whisper’s. These results indicate that Whisper’s English ASR performance is not perfect but very close to human-level accuracy. 4. Analysis and Ablations 4.1. Model Scaling A large amount of the promise in weakly supervised train-ing approaches is their potential to use datasets much larger than those in traditional supervised learning.

However, this comes with the cost of using data that is possibly much noisier and lower quality than gold-standard supervision. A concern with this approach is that although it may look promising to begin with, the performance of models trained on this kind of data may saturate at the inherent quality level of the dataset, which could be far below human level. A re-lated concern is that as capacity and compute spent training on the dataset increases, models may learn to exploit the Whisper A B C D E F G H I > ASR human transcription > computer-assisted > 0 > 5 > 10 > 15 > 20 > 25 > 30 > Word Error Rate (%) > Figure 7.

Whisper’s performance is close to that of profes-sional human transcribers. This plot shows the WER distri-butions of 25 recordings from the Kincaid46 dataset transcribed by Whisper, the same 4 commercial ASR systems from Figure 6 (A-D), one computer-assisted human transcription service (E) and 4 human transcription services (F-I). The box plot is superimposed with dots indicating the WERs on individual recordings, and the aggregate WER over the 25 recordings are annotated on each box. idiosyncrasies of the dataset, and their ability to generalize robustly to out-of-distribution data could even degrade. To check whether this is the case, we study the zero-shot generalization of Whisper models as a function of the model size.

Our analysis is summarized in Figure 8. With the exception of English speech recognition, performance con-tinues to increase with model size across multilingual speech recognition, speech translation, and language identification. The diminishing returns for English speech recognition could be due to saturation effects from approaching human-level performance as analysis in Section 3.9 suggests. 4.2. Dataset Scaling At 680,000 hours of labeled audio, the Whisper dataset is one of the largest ever created in supervised speech recog-nition. Exactly how important is the raw dataset size to Whisper’s performance? To study this, we trained a series of medium-sized models on subsampled versions of the dataset which are 0.5%, 1%, 2%, 4%, and 8% of the full dataset size and compared their performance with the same medium-sized model trained on the whole dataset.

Early stopping based on the validation loss was used to select model checkpoints for each dataset size. Evaluation was performed on an exponential moving average estimate of the parameters (Polyak & Juditsky, 1992) using a smooth-ing rate of 0.9999 to help reduce the effect of the learning rate not fully decaying to zero for the models trained on the subsampled datasets due to early stopping. Performance on English and multilingual speech recognition and X→en translation is reported in Table 6. Robust Speech Recognition via Large-Scale Weak Supervision 11 38M 73M 244M 768M 1549M 1549M > Model parameters > 0.0 > 2.5 > 5.0 > 7.5 > 10.0 > 12.5 > 15.0 > 17.5 > 20.0 > WER on 12 datasets (%) > English Speech Recognition > Average > Large V2 > 38M 73M 244M 768M 1549M 1549M > Model parameters > 0 > 20 > 40 > 60 > 80 > 100 > WER on 67 languages (%) > Multilingual Speech Recognition (Fleurs) > Average > Large V2 > 38M 73M 244M 768M 1549M 1549M > Model parameters > 0 > 10 > 20 > 30 > 40 > 50 > BLEU on 21 languages > X->En Translation (CoVoST2) > Average > Large V2 > 38M 73M 244M 768M 1549M 1549M > Model parameters > 30 > 40 > 50 > 60 > 70 > 80 > Accuracy on 102 languages (%) > Language Identification (Fleurs) > Average > Large V2 Figure 8.

Zero-shot Whisper performance scales reliably across tasks and languages with increasing model size. Lightly shaded lines represent individual datasets or languages, showing that performance is more varied than the smooth trends in aggregate performance. Large V2 distinguished with a dashed orange line since it includes several changes that are not present for the smaller models in this analysis. Dataset English Multilingual X→En size WER (↓) WER (↓) BLEU (↑)3405 30.5 92.4 0.2 6811 19.6 72.7 1.7 13621 14.4 56.6 7.9 27243 12.3 45.0 13.9 54486 10.9 36.4 19.2 681070 9.9 29.2 24.8 Table 6. Performance improves with increasing dataset size.

English speech recognition performance refers to an average over 12 datasets while the Multilingual speech recognition reports per-formance on the overlapping subset of languages in Fleurs and X→en translation reports average BLEU on CoVoST2. Dataset size reported in hours. All increases in the dataset size result in improved perfor-mance on all tasks, although we see significant variability in improvement rates across tasks and sizes. Performance improves rapidly on English speech recognition from 3,000 to 13,000 hours and then slows down noticeably between 13,000 and 54,000 hours. Using the full dataset, which cor-responds to another 12.5 × increase in size results in only a further 1 point drop in WER.

This mirrors the diminishing returns observed with model size scaling for English speech recognition and could similarly be explained by saturation effects when approaching human-level performance. Improvements in WER follow a power-law trend for mul-tilingual speech recognition till 54,000 hours and then de-viate from this trend, improving only a further 7 points when increasing to the full dataset size. For X→en transla-tion, performance is practically zero when training on 7,000 hours of audio or less, and then follows a roughly log-linear improvement trend till 54,000 hours before also showing diminishing returns when further scaling to the full dataset size.

The general trend across tasks of diminishing returns when moving from 54,000 hours to our full dataset size of 680,000 hours could suggest that the current best Whisper models are under-trained relative to dataset size and performance could be further improved by a combination of longer training and larger models. It could also suggest that we are nearing the end of performance improvements from dataset size scaling for speech recognition. Further analysis is needed to characterize “scaling laws” for speech recognition in order to decided between these explanations. 4.3. Multitask and Multilingual Transfer A potential concern with jointly training a single model on many tasks and languages is the possibility of negative transfer where interference between the learning of several tasks results in performance worse than would be achieved by training on only a single task or language.

To investigate whether this is occurring, we compared the performance of models trained on just English speech recognition with our standard multitask and multilingual training setup and measured their average performance across our suite of zero-shot English speech recognition