Model runtime

Munin talks to the on-device AI engine through one interface: GemmaModelService. Controllers do not depend on the concrete engine package.

Service contract

GemmaModelService defines the runtime surface:

• isBundleInstalled: checks whether required model assets exist and are valid.
• installBundle: installs the model bundle and streams aggregate progress.
• deleteBundle: removes installed model files.
• loadAssistant: loads the assistant model into memory.
• generate: streams assistant output for a chat turn.
• generateUtility: runs short deterministic utility calls.
• applyPerformance: adjusts context and backend settings.
• unload: releases loaded model resources.

The interface is intentionally small. All higher-level orchestration lives in controllers and service helpers.

Production implementation

FlutterGemmaService is the production implementation. It registers:

• LiteRT-LM inference through flutter_gemma_litertlm.
• LiteRT embeddings through flutter_gemma_embeddings.
• SQLite vector storage through flutter_gemma_rag_sqlite.
• Stateless skill executors from flutter_gemma_agent.

The service passes the system identity as a session systemInstruction. It does not concatenate the system prompt into raw user text.

Model bundle

ModelConstants.requiredBundle defines the required assets:

• FunctionGemma 270M: utility and routing model.
• Gemma 4 E2B: primary assistant model in LiteRT-LM format.
• Gecko 512: embedding model for memory.

The order matters. The assistant is installed after the utility model so it remains the active model for chat.

Download flow

ModelDownloadController drives model installation:

1. It reads saved progress from StorageKeys.downloadProgressBytes.
2. It subscribes to GemmaModelService.installBundle.
3. It updates DownloadState with received bytes, total bytes, current asset, speed, and status.
4. It persists progress sparingly, roughly every 2 percent or on forced pause.
5. It removes saved progress when installation completes.

Pause cancels the progress subscription but preserves received bytes. Retry restarts the subscription from the last saved progress. Cancel resets progress and deletes bundle files.

Background transfer

The app configures background_downloader notifications in main.dart. The real engine uses foreground download behavior so large model transfers can continue when the app is backgrounded. The controller reflects progress and persists recovery state.

Utility generation

generateUtility is used for short tasks:

• Search decision checks.
• Query planning.
• Conversation title generation.
• Plan drafting and review in workspaces.
• Memory fact extraction.

Utility calls prefer the small function model. If it is unavailable, the service falls back to the assistant model. The service serializes native generation because the LiteRT-LM C API is not treated as reentrant across sessions.

Performance profiles

Performance mode is stored under StorageKeys.performanceMode and applied by GemmaModelService.applyPerformance.

Modes:

• battery: smaller context, capped output, CPU backend.
• max: full context and GPU preference.
• auto: chooses based on reported device RAM.

Changing the profile unloads the model. The new configuration applies on the next load.

Failure behavior

Model initialization and installation failures are handled defensively:

• Bootstrap failures are logged in debug builds.
• Download errors put the controller in failed state with a message.
• Utility model load failures disable the utility model path and fall back.
• Chat generation errors settle the assistant placeholder with a retryable failure message.