0ms Latency Dreams vs. NPU Reality: Engineering a Production-Ready AI Search with Gemini Nano
"Hello World" demos make on-device AI look effortless; shipping it in a production travel app shows a different reality. Traditional search filters are predictable, but they miss the nuance of what people actually mean.
This session goes behind the scenes of replacing rigid UI filters with an AI-driven search parser built on the ML Kit Prompt API: a candid case study of optimising natural-language entity extraction, and how inference time fell from a sluggish 18.3 seconds to under one second by mastering the art of prompting.
Key takeaways
The prompt evolution
The "polite conversation" approach (18 s) against the "structured constraint" approach (5 s).
Zero-inference latency
Prefix caching to skip static instructions, with a dynamic TODAY_DATE injected for relative dates like "tomorrow".
The logic bridge
Chaining calls in Kotlin and mapping Gemini Nano's JSON output straight to booking modules.
Streaming for speed
Kotlin Flows and generateContentStream for a UI that feels instant.
A production reality check
Fragmented availability, inconsistent behaviour, and the hybrid fallback built from user feedback.