Optional on-device LLM
Run the LLM enhancement pass locally via llama.cpp. No cloud, no key, no per-token cost. Pick a model to match your GPU memory.
By default the LLM pass uses a cloud provider (or is Off). Optionally, Dimmy can run an on-device LLM via llama.cpp. The default model is Phi-4 Mini Q4 (2.5 GB, 3.8B params), a compact instruction-following model. The Settings → Output model picker lists all available models so you can trade speed for quality.
Enable it
- Open Settings → Output → Local LLM model.
- Pick a model from the list and click Download.
- Pick On-device LLM as the active LLM in Settings → Output → LLM provider.
- Run a dictation. The pill shows the style icon while the model processes the text.
In the picker, models already on disk show a green check next to their name, exactly like the Voice model picker. The ones you have not downloaded yet show their size instead.
Available models
Performance
- Apple Silicon: Metal acceleration, around 50 tok/s on M2.
- NVIDIA GPU: CUDA, around 80-120 tok/s on RTX 3060 and up.
- AMD/Intel GPU: Vulkan, around 30-50 tok/s.
- CPU only: works but slow (5-10 tok/s).
Trade-offs vs cloud
Local LLM is fully private and free, but less capable than cloud Claude or GPT-4o. For Correct and Summarize styles, the smaller models are fine. For complex Prompt or Elaborate, cloud is noticeably better. The 12B model narrows that gap but needs more VRAM and is slower.