Building Voice-Controlled AI Android Apps Without Server

Why Offline Voice AI Matters for Mobile Users

Six months ago, I was building a productivity app for a client in Southeast Asia where network connectivity is spotty at best. The requirement was simple: users should control the app via voice commands—add tasks, set reminders, search notes—without sending audio to a cloud service. That's when I realized the power of building an AI Android app with on-device processing.

Voice-controlled AI Android apps are no longer a luxury—they're becoming table stakes. Users expect instant responses, privacy guarantees, and offline functionality. Cloud-dependent voice solutions mean 500ms+ latency, server costs scaling with usage, and privacy concerns that make enterprises uncomfortable. On-device AI eliminates all three.

"On-device AI isn't just faster—it's the only choice when privacy and latency matter. Building with local LLMs changed how I architect mobile AI entirely."

In this post, I'll walk through how I built a voice-command system using quantized LLMs directly on Android, the architecture decisions that matter, and the gotchas I hit along the way.

Choosing an LLM for Your AI Android App

Not every LLM works on mobile. When you're targeting phones with 4–8GB RAM and limited CPU, you need models that are quantized, compact, and purpose-built for inference.

The Model Candidates

• Phi-2 (2.7B) – Excellent reasoning for its size; ~3GB quantized. My go-to for voice command understanding.
• Mistral-7B – More capable, but needs 6–8GB RAM. Good for devices with headroom.
• GGUF quantization – Reduces model size by 75% with minimal accuracy loss. 7B models become ~2.4GB in Q4_K_M format.
• TinyLlama-1.1B – Ultra-lightweight for basic intent recognition; fits in 800MB.
• Specialized models – Consider task-specific models: intent classifiers trained on command patterns, not general-purpose LLMs.

I tested Phi-2 and Mistral-7B side-by-side. Phi-2 won for voice commands—faster, smaller, and accurate enough for 98% of user intents. Mistral handles edge cases better but bloats the APK and kills battery on older devices.

Architecture Design for On-Device AI Integration

Building machine learning mobile apps requires clean separation between inference, voice processing, and app logic. Here's the architecture I've settled on:

Layered Design

• Voice Input Layer – Capture audio using Android's AudioRecord or ML Kit Speech Recognition.
• Inference Engine – Isolated LLM wrapper (local repository pattern) using ONNX Runtime or llama.cpp JNI bindings.
• Intent Parser – Post-process LLM output to extract command intent and parameters.
• Action Executor – Repository pattern that maps intents to app actions (without knowing about the LLM).
• Response Handler – Text-to-speech feedback (also on-device with TTS engines).

This separation means: testing each layer independently, swapping out the LLM later without refactoring business logic, and handling model updates cleanly.

Threading Model

LLM inference is CPU-bound and can block the main thread for 2–5 seconds. I use Kotlin Coroutines with dispatchers to manage this:

// Inference layer wrapped in a coroutine-friendly interface
interface LLMInferenceEngine {
    suspend fun processCommand(input: String): String
}

class OnnxLLMEngine(
    private val session: OrtSession,
    private val tokenizer: Tokenizer
) : LLMInferenceEngine {
    override suspend fun processCommand(input: String): String =
        withContext(Dispatchers.Default) {
            val tokens = tokenizer.encode(input)
            val output = session.run(tokens)
            tokenizer.decode(output)
        }
}

// Usage in ViewModel
class VoiceCommandViewModel(
    private val inferenceEngine: LLMInferenceEngine,
    private val commandRepository: CommandRepository
) : ViewModel() {
    fun handleVoiceInput(audioText: String) {
        viewModelScope.launch {
            val result = inferenceEngine.processCommand(audioText)
            val intent = parseIntent(result)
            commandRepository.executeCommand(intent)
        }
    }
}

By isolating inference in Dispatchers.Default, the UI stays responsive even during heavy model computation.

Implementation: Voice to Action in 4 Steps

Step 1: Integrate ONNX Runtime or Ollama

I prefer ONNX Runtime for Android because of stable Java bindings and excellent quantization support. Add to your build.gradle:

dependencies {
    implementation 'com.microsoft.onnxruntime:onnxruntime-android:1.16.0'
}

Download your quantized model (e.g., Phi-2 in ONNX format) and bundle it in assets/models/.

Step 2: Build the Inference Wrapper

Abstract the model details behind a clean interface:

class OnnxModelManager(
    context: Context,
    modelFileName: String = "phi-2-quantized.onnx"
) {
    private val env = OrtEnvironment.getEnvironment()
    private val session: OrtSession

    init {
        val modelBytes = context.assets.open(modelFileName).readBytes()
        session = env.createSession(modelBytes)
    }

    fun infer(prompt: String, maxTokens: Int = 256): String {
        val inputIds = encodePrompt(prompt)
        val tensorInput = OrtUtil.createTensorFromFloatBuffer(
            floatArrayOf(*inputIds.map { it.toFloat() }.toFloatArray()),
            longArrayOf(1, inputIds.size.toLong())
        )
        val outputs = session.run(mapOf("input_ids" to tensorInput))
        return decodeOutput(outputs)
    }

    private fun encodePrompt(text: String): IntArray {
        // Tokenization logic (use huggingface tokenizers library)
        return IntArray(0) // Placeholder
    }

    private fun decodeOutput(outputs: Map<String, OrtValue>): String {
        // Extract tokens from output tensor and decode
        return "" // Placeholder
    }
}

Step 3: Capture Voice Input

Use ML Kit Speech Recognition for robust audio-to-text conversion on-device:

val recognizer = SpeechRecognition.getClient(context)
val intent = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH).apply {
    putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, 
             RecognizerIntent.LANGUAGE_MODEL_FREE_FORM)
    putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault())
}
recognizer.startListening(intent, object : RecognitionListener {
    override fun onResults(results: Bundle) {
        val matches = results.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION)
        matches?.firstOrNull()?.let { userVoiceInput ->
            // Pass to LLM inference
        }
    }
    // Handle other callbacks...
})

Step 4: Map Intent Output to Commands

Parse the LLM output and route to business logic:

data class CommandIntent(
    val action: String,
    val parameters: Map<String, String>
)

class CommandIntentParser {
    fun parse(llmOutput: String): CommandIntent? {
        return when {
            llmOutput.contains("add task", ignoreCase = true) ->
                CommandIntent("add_task", mapOf("title" to extractTitle(llmOutput)))
            llmOutput.contains("set reminder", ignoreCase = true) ->
                CommandIntent("set_reminder", mapOf("text" to extractText(llmOutput)))
            else -> null
        }
    }
}

Optimization & Common Pitfalls

Model Loading Performance

Loading a 2–3GB ONNX model into memory takes 2–5 seconds. I cache it at app startup using Hilt singleton scope:

@Singleton
class ModelProviderModule {
    @Provides
    fun provideOnnxManager(context: Context): OnnxModelManager {
        return OnnxModelManager(context) // Loaded once, reused forever
    }
}

Memory Pressure

Quantized LLMs still consume 2–4GB RAM during inference. Monitor with Runtime.getRuntime() and show a loading indicator. Some devices with low free memory will crash—graceful degradation is critical.

Tokenization Overhead

Tokenization can be as slow as inference if done naively. Use pre-compiled tokenizer libraries (HuggingFace's Rust tokenizers compiled to Android) rather than pure Kotlin implementations.

Battery Drain

CPU-intensive inference burns battery fast. Batch commands when possible, limit context length, and consider using TinyLlama for frequent, lightweight inferences.

Key Takeaways

• On-device AI eliminates latency and privacy risks – No network round-trips, no cloud storage of voice data. Users get instant feedback and trust your app with sensitive information.
• Quantization is non-negotiable – A 7B LLM becomes mobile-friendly only when quantized to Q4_K_M or lower. This cuts model size by 75% with negligible accuracy loss.
• Architecture matters more than model choice – Isolate inference behind repositories and use Coroutines for threading. Swapping models becomes trivial when business logic isn't coupled to the AI layer.
• Test early with real devices – ONNX inference timing varies wildly between Snapdragon 888 and budget Helio chips. Profile on actual target hardware, not emulators.
• User experience wins with transparency – Show loading states, handle timeouts gracefully, and fallback to cloud-based inference if on-device fails. Users don't care how it works, only that it works reliably.