All talks
Talk · Case study · ML Kit Prompt API

0ms Latency Dreams vs. NPU Reality: Engineering a Production-Ready AI Search with Gemini Nano

"Hello World" demos make on-device AI look effortless; shipping it in a production travel app shows a different reality. Traditional search filters are predictable, but they miss the nuance of what people actually mean.

This session goes behind the scenes of replacing rigid UI filters with an AI-driven search parser built on the ML Kit Prompt API: a candid case study of optimising natural-language entity extraction, and how inference time fell from a sluggish 18.3 seconds to under one second by mastering the art of prompting.

Case studyML Kit Prompt APISpeaker: Dinoy Raj

Key takeaways

The prompt evolution

The "polite conversation" approach (18 s) against the "structured constraint" approach (5 s).

Zero-inference latency

Prefix caching to skip static instructions, with a dynamic TODAY_DATE injected for relative dates like "tomorrow".

The logic bridge

Chaining calls in Kotlin and mapping Gemini Nano's JSON output straight to booking modules.

Streaming for speed

Kotlin Flows and generateContentStream for a UI that feels instant.

A production reality check

Fragmented availability, inconsistent behaviour, and the hybrid fallback built from user feedback.