Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

Introduction to Tiny Models [0:00]

• The talk focuses on the state-of-the-art of tiny AI models and their applications.
• It covers background, small models, the need for even smaller models, and production deployment.
• The speaker works on Google's AI Edge team, developing open-source projects for edge AI deployment.

  We are going to need tiny models.

Why Edge AI? [3:00]

• Edge AI offers benefits like low latency, consistent speed, privacy, offline use, and cost savings compared to cloud processing.
• Challenges include DRAM cost, a wide pool of target devices, and less research focus on smaller LLMs.
• DRAM cost is a significant constraint, impacting device design and model choices.

  DRAM cost is really, really significant.

Small Models: Capabilities and Limitations [5:30]

• Small models, typically 1-4 billion parameters, are found in OSs and high-end devices.
• They require significant DRAM (4-8 GB), limiting their use to devices like laptops and mobile phones.
• Techniques like quantization and prompting are used to minimize their footprint.

  So then kind of restricts these models to, you know, things like laptops, mobile phones, or kind of higher end electronics.

Gemma and Optimized Small Models [8:00]

• Gemma models, specifically the 2 billion parameter version, are used as an example.
• Optimization techniques like mixed-precision quantization (2-bit, 4-bit, 8-bit) reduce memory footprint.
• A 2B parameter model can require around 2 GB of active RAM, leading to a 4 GB+ rule of thumb for deployment.

  We do other tricks like per-layer embeddings.

Performance of Small Models on Edge Devices [10:30]

• Performance metrics are shown for running a 2B parameter model on Raspberry Pi (7.6 tokens/sec) and Jetson Nano (24 tokens/sec).
• Porting to a Qualcomm IoT board shows high pre-fill and decode speeds, enabling near real-time applications.
• These speeds are sufficient for many compelling applications if the hardware cost is acceptable.

  If you go to a higher, a more capable device like a Jetson or in Nano, we can get up to maybe 24 tokens per second decode.

AI Edge Gallery and Open Source Tools [13:00]

• An app is available on iOS and Android to test small models on phones.
• The app and runtime are open source, allowing users to learn how to build similar applications.
• This provides an easy way to get started with testing small models.

  So, it's available on AI Edge Gallery.

OpenDoc Mini V2 Robot Example [14:00]

• An example of the OpenDoc Mini V2 robot demonstrates AI capabilities on hobbyist hardware.
• Robots using Jetson Nano and Raspberry Pi can read signs, react to stimuli, and take voice/image input.
• The Jetson Nano version offers good real-time interaction, while the Raspberry Pi version is slower.

  And you'll see that the robot is able to, it's able to kind of like read signs like and react to things and kind of nod its head.

The Need for Tiny Models [15:30]

• Small models can still be too large for older laptops or consumer edge devices, and may not meet responsiveness needs.
• Tiny models are needed for background tasks or when the model is not the primary feature.
• Tiny models, typically 50 million to 500 million parameters, require less RAM and can run very fast.

  So we also need a smaller model for system health as a common pattern.

Tiny Models: Off-the-Shelf and Fine-Tuning [17:00]

• The playbook for tiny models involves using off-the-shelf models or fine-tuning.
• Off-the-shelf examples include ASR, vision (e.g., Apple fast VLM), and embedding models.
• Fine-tuning allows for highly specialized and reliable performance on specific tasks.

  Or else if that doesn't work, you're going to be left in a world of kind of fine-tuning a model to achieve a given outcome.

Fine-Tuning for Mobile Actions [19:00]

• Fine-tuning models like Function Gemma (270 million parameters) significantly improves performance (45 tokens/sec).
• A "mobile actions" model can reliably perform function calling for common mobile tasks (scheduling, settings).
• Combining ASR with fine-tuned models enables robust voice-to-function calling, crucial for IoT devices.

  This model knows about 10 different output functions and can call them at over 86% reliability.

Synthetic Data for Fine-Tuning [21:30]

• The process involves picking a base model, checking performance, and synthetically generating data for fine-tuning.
• Datasets like "Mobile Actions" can be used, and 10,000 to 10 million samples are often sufficient.
• Fine-tuning allows achieving high quality with much smaller models for tasks like summarization or proofreading.

  Depending on the model, like, there's a data set we've open sourced here called Mobile Actions that's available on Hugging Face.

Production Example: Offline Voice Dictation [23:00]

• An app developed for offline voice dictation uses fine-tuned Tiny Gemma models.
• It performs dictation, cleans up speech disfluencies, and offers personalization.
• This transforms a subscription-based server feature into an offline, high-quality app.

  And both of these are fine-tuned versions of Tiny Gemma models.

Key Takeaways and Future Ambitions [26:00]

• Small LLMs are fast and easy to use on NPUs for consumer devices and robotics.
• Tiny models enable reach to a much larger pool of devices.
• Voice-to-function calling can be made robust with tiny models and appropriate synthetic data.
• Future ambitions include generalizing voice-to-function calling and improving visual input models.

  Tiny models will enable reach for much, much larger pool of devices.