Build Your Pocket Brain: Open On-Device LLMs via MediaPipe on Android
MediaPipe's LLM Inference API leads the way for Android developers. Beyond running pre-trained models such as Gemma on the device, this session dives into customisation with Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning (PEFT) technique.
It walks the end-to-end workflow for a specialised on-device LLM: fine-tuning a base model like Gemma or Phi-2 on your own dataset with the PEFT library in Python, converting the base model and LoRA weights to MediaPipe's FlatBuffer format, and integrating the result in an Android app by configuring LlmInferenceOptions in Kotlin. The payoff is personalised AI that is fast, works offline and stays private.
Key takeaways
Choosing the right tool
When to use MediaPipe LLM Inference and when Gemini Nano for on-device generative AI.
Setup and configuration
Configuring MediaPipe LLM Inference, its options and multimodal prompting.
LoRA fine-tuning
Adapting Gemma-2B or Phi-2 to a specific use case, cost-effectively.
Deployment and ethics
Deployment workflows, GPU-accelerated LoRA inference and responsible AI practice.