Skip to content
sysout.dev

In My Head - Implementation Plan

Current checkpoint

Android has a working local-first record → transcribe app. It is not a production-quality voice-note app yet; this document records what is actually implemented and the gates before building the Gemma cleanup experience or broader product UI.

  • The app records PCM audio in an Android foreground service and preserves the resulting WAV files in app-private storage.

  • It downloads and uses Whisper small.en locally. The active model is about 488 MB; medium.en is catalogued but is not selected or benchmarked.

  • It transcribes only after Stop, stores the raw transcript locally, keeps audio when transcription fails, and lets the user retry without rerecording.

  • The transcript list and detail screens show a failure explanation and the recorded transcription duration.

  • The optional LiteRT-LM/Gemma integration remains intentionally behind the Whisper quality work; it is not the current product flow to validate.

Completed foundation

Phase 1 - Theme and app shell

Completed:

  • Material 3 light/dark theme from the visual design document.

  • Record, transcript list, and transcript detail screens.

  • Shared domain models and ViewModels.

Phase 2 - Local persistence

Completed:

  • SQLDelight recordings and transcripts storage.

  • A versioned database migration for transcription failure text and processing duration.

  • One Android repository provider shared by the Activity and foreground service, so status changes reach the UI immediately.

Phase 3 - Android capture pipeline

Completed first pass:

  • AudioRecord capture in a microphone foreground service.

  • Automatic 25-second WAV chunks plus a valid final partial chunk after Stop.

  • Recording files remain available after a failed transcription for retry.

  • Capture stays available while a durable first-in, first-out queue processes one earlier recording at a time.

Still to prove:

  • Background, screen-off, Bluetooth, and OEM battery-management behaviour on a representative set of physical devices.

  • Long-session reliability (15-20 minutes or more).

Phase 4 - whisper.cpp integration and performance baseline

Completed first pass:

  • Standalone :whisperNative Android library with CMake/JNI bridge.

  • First-run atomic model download into app-private storage.

  • small.en selected as the current English-only model; the app does not treat the previous base.en download as the active model.

  • Native debug builds now compile Whisper and its GGML compute libraries with -O3, so device testing does not benchmark unoptimized native code.

Verified on the current Motorola Edge 50 Fusion test device:

  • A 6.0-second recording transcribed with small.en in 23.7 seconds, roughly 4x real time. This is a useful baseline, not a quality or thermal benchmark.

Current native decision:

  • GGML OpenMP is disabled. Earlier OpenMP builds required libomp.so, which was not safely packaged in the app. The current non-OpenMP GGML thread pool uses the device’s four performance cores. Re-enabling OpenMP is an isolated experiment only after its runtime is packaged and launch-tested.

Next: Whisper benchmark and quality gate

Before adding Gemma cleanup or redesigning screens, create a repeatable device benchmark record for small.en:

  • Measure 30-second, 1-minute, and multi-chunk recordings; record audio length, transcription duration, device temperature/charging state, and battery impact.

  • Compare cold-model and warm-model runs.

  • Test English with Indian and US accents, normal room noise, and sequential speakers in a meeting-room setting. Speaker labels/diarization are not part of this pass; the output is one ordered transcript.

  • Confirm the transcript preserves chunk boundaries naturally enough. There is no overlap or boundary de-duplication today.

  • Test 2/4/6/8 thread counts only with a controlled saved-WAV benchmark.

  • Run a separate OpenMP packaging and benchmark experiment. Keep it only if it launches reliably and improves the measured result.

  • Decide whether small.en is sufficient. Do not move to medium.en without measuring its quality gain, processing time, storage, memory, and thermal cost on the same device.

The exit criterion is not a particular benchmark number alone: Whisper must be fast enough and accurate enough that a user will wait for the result, while recording failure never loses their audio.

Follow-on product work

UI/UX pass

The 0.1 decisions are recorded in product/0.1-release.adoc. Implement them as focused, separately validated changes:

  • Separate model download/retry from recording. The record action is disabled until Whisper is ready, and downloading does not request microphone access.

  • Add a static Settings destination with on-device Whisper/Gemma status and private-storage information; it must not imply a missing Gemma download feature.

  • Expose existing recording/transcription diagnostics, copy/selectable text, generated recording titles, and sort/grouping without adding sessions, streaming transcription, or an audio player.

  • Apply Material 3 semantic colour roles, typography, standard shapes, and accessible labels. The selective record action remains primary/red; supporting actions and errors use their semantic roles.

Gemma cleanup pass

Deferred until the Whisper quality gate passes:

  • Define what cleanup should produce, rather than treating it as generic note formatting.

  • Choose the Gemma prompt, model bundle, download/storage behaviour, and failure semantics.

  • Preserve and show raw Whisper text whenever cleanup is unavailable or fails.

  • Benchmark cleanup separately from transcription.

Output and retention

Pending after the core record/transcribe/cleanup flow:

  • Plain-text export.

  • User-controlled deletion and storage management for recordings and models.

  • Cross-app model sharing, if a second Sysout app has a concrete requirement.

Validation and commit workflow

Each focused change follows this loop:

  1. Implement one bounded change.

  2. Compile the relevant shared/native/app targets and run applicable tests.

  3. Install on the physical Android device when behaviour or native code changes.

  4. User tests the behaviour.

  5. Commit only the intended source and documentation files after that test.