New LLMs Are Unlocking Robot-Use Agents

The Rise of Robot Use Agents [00:00:00]

• Recent years have seen LLMs generalize across domains, now extending to robot control.
• Frontier researchers suggest we are entering an era of "robot use agents" where general-purpose models enhance robot capabilities.
• Companies like Waddle Labs and RoboCurve are developing startups focused on making robots more capable using LLMs.

  One of the big surprises the last few years has been the generalizability of coding agents across different domains and now frontier researchers are showing that this includes controlling robots.

Early Research: LLMs and Robot Control [00:02:14]

• Early successful approaches used pre-trained language models to control robots, exemplified by the RT2 paper.
• RT2 fine-tuned a language model to output robot actions (end-effector pose) instead of text.
• This approach is similar to current LLM capabilities but relied on fine-tuning, whereas newer models perform tasks more directly.

  One of the earliest successful approaches of using AI on robots is the RT2 paper where they use a pre-trained language model on web text and images and use that to control robots.

The "Bitter Lesson" and Data Modalities [00:05:30]

• The "bitter lesson" suggests that giving AI models more autonomy and resources allows them to achieve tasks previously requiring fine-tuning.
• Vision-Language-Action (VLA) models are built on language models and can potentially reason and write code.
• The bottleneck in VLA progress may be the data, not necessarily the architecture, highlighting the importance of diverse data modalities.

  I feel like at the end of the day, it's a lot about the bitter lesson, right? If you give the agents or you give the AI model more autonomy and you, if you unshackle it a bit more and give it more resources, it can actually do a lot of the things that we fine tune it to do.

Evolution from Fine-Tuning to Direct Control [00:07:20]

• Models have improved significantly since RT2, particularly in tool use and writing complex policies as code.
• In-context learning allows models to adapt and improve with examples without explicit fine-tuning.
• The key difference between VLAs and general LLMs is often the approach to training and the benefit derived from diverse data types.

  There's a few things that models have got much better at since R2. Um, like one, I mean, they're much better tool use for example. So, you know, one, one kind of data is now these models can, uh, well, I mean, they write code much better.

Code as Policies: Empowering Agents [00:10:07]

• "Code as policies" research explores using LLMs to generate code that directly controls robots.
• Early work like Voyager demonstrated how coding agents could use tool use and on-the-fly tool creation.
• This approach leverages the LLM's ability to understand and generate code, treating it as a set of executable instructions for robot actions.

  I probably rewind back to Voyager where like Voyager was like, the one of the first, what is, what does a coding agent mean? Coding agent requires good tool use and on the fly tool creation. That's called code.

Early Code Policy Capabilities [00:13:00]

• Early "code as policies" papers, particularly from Google DeepMind, created functions for robot actions (e.g., pick up, move).
• Coding agents could write code using these functions to control robots for complex tasks, often in a one-shot manner.
• This ability to work with code without extensive robot-specific data was a key motivator for further research.

  Those CODIS policy papers, especially ones for Google DeepMind team were incredible because what they did is they created these kind of functions, like pick up an object, lift up, or like move to certain posts.

In-Context Learning vs. Weight Updates [00:15:30]

• In-context learning (ICL) allows LLMs to adapt to new tasks by appending examples to the prompt.
• ICL is efficient and requires no gradient updates but has limitations: non-monotonic improvement, quick saturation, and constraint by context window length.
• More permanent learning can be achieved through methods like LoRa or full Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL).

  I've done this experiment where like, you take an LLM that we've trained on and I've held out a task, let's just say GSMAK for simplicity. And then I ICL it and I want to measure on the vowel set how much it improves on a per sample basis.

Harnesses and Domain Specificity [00:19:00]

• A "harness" can be seen as a form of domain specificity, consolidating learned skills into reusable programs or tools.
• This process distills past experience for future agents, potentially allowing for smaller, faster models wrapped within a larger system.
• The relationship to meta-learning is explored, where a larger model programs or specializes smaller models for specific tasks.

  Yeah, the first thing is, we've been sort of thinking about this idea of a harness almost as a form of like domain specificity. Like when you, for example, deploy a robot in a new environment, maybe it's in a wet lab, and it just needs to do a lot of tests you're picking up.

Astra and Spatial Intelligence [00:23:30]

• Astra demonstrates impressive control over robot arms, picking up and placing objects.
• While direct LLM control can have latency issues, advancements are rapidly reducing this, with potential for real-time control soon.
• The "Platonic Representation Hypothesis" suggests that LLMs trained on vast data converge to consistent world representations, making them effective for new tasks like robotics.

  Yes. So it does look at all the cameras. It has all the camera feeds. Instead of code, it's more like a tool call. So it sends commands to the robot to control and put the block into the bowl.

The Role of Diverse Data [00:27:00]

• Astra's success in spatial tasks is attributed to extensive pre-training on computer use data (e.g., CAD, Blender) and egocentric videos.
• This data teaches models about spatial reasoning, object manipulation, and physical world concepts, bridging the gap between language and robotics.
• Integrating diverse data types like coding, computer use, and egocentric videos into a single model is key to creating capable robot use agents.

  I think what Astra does incredibly well is its, like, vision capabilities. It was probably pre-trained on way more computer use data than ever before. It's probably pre-trained on so much, like, CAD data. And it's like all of these kinds of data probably teach the model, like, similar, uh, like, understanding of, like, the physical world as, like, a lot of robot data might.

Future of Robotics and Society [00:31:00]

• Consensus in frontier labs suggests general-purpose robots will be common within two years, a capability society may be unprepared for.
• Challenges like latency are being addressed by consolidating LLM-generated skills into faster, more efficient policies.
• The future may involve a cycle of LLM-driven exploration, skill consolidation, and weight updates, akin to biological sleep and memory compression.

  There's some consensus within the frontier labs and also in the robotics foundation, uh, models companies that we will have general purpose robots within the next two years or even earlier.

Managing Skills and Context [00:34:00]

• Managing a growing context of skills and deployment data is crucial for developing advanced robotics.
• Refactoring traces and skills into more compact representations is an ongoing area of research.
• The development of robotics is progressing rapidly, with significant advancements expected in the coming years.

  I think with that, I think this was an excellent discussion. Thank you so much, Vincent, Hanming, and Jay for being here. I'm very excited for all the incredible robotics advancements.