Speech Capture App — Technical Design

Two broad families of speech-recognition models exist:

Android’s built-in SpeechRecognizer is neither of these directly — it’s an OS-level API that hands audio off to a Google-provided recognizer (cloud or on-device depending on settings) and gives you back discrete "utterance" results. You don’t control the underlying model.

Already covered in the earlier discussion, restated for the design record: it’s built around short discrete utterances, not continuous long-form dictation. Its restart-on-silence loop drops a word or two at each restart boundary — the same class of failure as Gboard’s cutoff problem, just less frequent. It’s also a black box: no control over chunking, no offline guarantee across all devices, no ability to bias it toward your own vocabulary.

Whisper is OpenAI’s open-weight speech-recognition model, trained on a very large multilingual audio/text dataset. It comes in sizes from tiny (~75MB) to large (~3GB), trading size for accuracy. whisper.cpp is a C/C reimplementation that runs the quantized model on CPU without needing Python or a GPU — which is what makes it viable on a phone. There's no official mobile SDK for it; every Android integration is a hand-built JNI bridge over the C library. That’s more integration work than a "just add a dependency" library, but it’s a well-trodden path — plenty of open-source Android apps have done exactly this.

This is worth flagging explicitly because it changes the engineering difficulty a lot. Whisper’s architecture was not designed for live streaming: real-world reports of people running whisper.cpp in a "re-transcribe on every 1-2 second buffer" live-captioning loop on Android have measured it running roughly 5x slower than real-time — the repeated re-evaluation of overlapping audio is expensive on a mobile CPU.

The implemented MVP does not show a live transcript. It records continuously, writing a 25-second WAV chunk at a time, but it does not run Whisper while the user is listening. After Stop, it finalizes the current partial chunk, then passes every saved chunk to Whisper in capture order and joins the resulting text.

There is currently no chunk overlap and no boundary de-duplication. That keeps the first pass simple and ensures the entire recording exists before expensive CPU work begins, but it makes boundary accuracy a benchmark item: a word split across two chunks must be assessed on real recordings before this is treated as production-ready.

The active MVP model is Whisper small.en (about 488 MB). It is downloaded once into app-private storage and then runs locally. medium.en is catalogued as a later option, but is not selected until its accuracy benefit justifies its substantially larger storage, memory, thermal, and processing cost on real devices. base.en was only an integration proof and is no longer the active model.

Current decision: whisper.cpp, post-stop chunked transcription with small.en.

The short version: model weights get compressed (quantized, usually to 4-bit) and the parameter count is kept small (1-4 billion, versus 100B+ for something like GPT-4-class models). That combination is small enough to fit in a phone’s RAM and run on its CPU/GPU/NPU in a few seconds per response. The job here is narrow — reformatting a rough transcript into structured text — not open-ended reasoning, so a small model is genuinely enough; you’re not leaving quality on the table by avoiding a bigger model.

This is Google’s current production on-device inference framework — the successor to the older MediaPipe LLM Inference API, which Google’s own docs now list as maintenance-only. LiteRT-LM is genuinely cross-platform: a single C++ core with official Kotlin (Android), Swift (iOS/macOS), Python, and JavaScript bindings, running on Android, iOS, Web, and Desktop. It supports Gemma, Llama, Phi-4, and Qwen model families via a .litertlm model-bundle format, with GPU acceleration via OpenCL on Android, Metal on iOS, and WebGPU on the web. Given your stated iOS/desktop future plans, this cross-platform reach matters — it’s the strongest alignment of any option here with where you say this is headed.

The existing prototype can invoke a Gemma 4 E2B .litertlm bundle when present, but cleanup is deferred as a product capability until Whisper quality and timing are accepted. The next cleanup pass needs an explicit output contract, prompt, download/storage behaviour, and raw-text fallback rather than simply enabling a model file.

Recommendation: LiteRT-LM with Gemma 4 (E2B).

Declaring the API correctly isn’t the whole story. Aggressive battery managers on MIUI (Xiaomi), OneUI (Samsung), ColorOS (Oppo), and similar OEM skins are known to kill foreground services anyway unless the user separately grants battery-optimization exemptions and, on some OEMs, "autostart" permission. This is a well-documented pain point across the Android ecosystem (the community site dontkillmyapp.com catalogs per-OEM workarounds). Given you’re testing primarily on Indian-market devices, budget real device testing here — this is exactly the kind of thing that looks fine on a Pixel and silently fails on a Redmi.