whisper-test_skill

Overview

This skill transcribes WAV audio files using OpenAI Whisper to assess speech intelligibility and basic audio quality. It reports transcription text, audio metrics (duration, RMS, peak, silence), and can compute word error rate (WER) when expected text is provided. It is designed to run without ffmpeg by loading WAV data via scipy and supports model selection for different accuracy and resource trade-offs.

How this skill works

The skill loads one or more WAV files with scipy, extracts audio statistics (duration, RMS, peak, percent silence) and runs Whisper to produce a transcription. When an expected transcript is supplied, it computes WER to quantify correctness. Models range from tiny/base/small/medium/large-v3; large-v3 is recommended for reliable TTS verification. Output can be printed human-readably or emitted as JSON for automation.

When to use it

• Verify whether synthesized TTS outputs are intelligible and match expected text.
• Run quick smoke tests on voice-cloned or generated audio to spot major regressions.
• Batch-compare multiple TTS variants against a target transcript.
• Gate quality in CI for TTS systems where accuracy matters (use large-v3).
• Detect silent, extremely quiet, or noise-only files before downstream processing.

Best practices

• Use large-v3 for TTS evaluation; smaller models hallucinate or miss words.
• Provide an expected transcript to get WER and a clearer pass/fail signal.
• Prefer WAV inputs; the script loads WAV directly to avoid ffmpeg dependency.
• Interpret audio metrics alongside transcription (e.g., low RMS or high silence implies unintelligible audio).
• Run batch tests to compare variants and flag regressions automatically.

Example use cases

• Quality gate for TTS model updates: run large-v3 and block deploys if WER exceeds a threshold.
• Compare multiple voice-cloned variants to pick the clearest output for production.
• Sanity-check recorded audio from an ingestion pipeline to reject silent or noisy files.
• Automate nightly regression tests for synthesized speech to detect degradations early.