Chelsea Finn: This is the State of the Art in Robotics

The State of Physical Intelligence [0:00]

• Physical Intelligence aims to develop robots capable of performing any task in the real world.
• Recent progress includes complex tasks like folding laundry, washing greasy pans, and making grilled cheese sandwiches.
• The focus is shifting from "cool videos" to developing general-purpose robots that are truly useful.

  "And what I'd really like to focus on today isn't cool videos of robots doing lots of different things, but what it actually takes to get robots to be useful in the real world."

Evolution of AI in Production [1:31]

• Early AI applications focused on product recommendations and ad ranking, initially using machine learning and later deep learning.
• The launch of ChatGPT in 2022 marked a significant shift, demonstrating a general-purpose model used widely by the public.
• Subsequent advances include useful coding agents like GitHub Copilot.

  "And this was the first time where we saw a general purpose model truly being used by many different people in the real world."

Generalist Models vs. Physical AI [3:05]

• Current AI applications often involve customers making decisions based on AI recommendations, allowing for a tolerance for errors.
• Physical AI and robotics, however, require direct decision-making in the physical world, demanding higher autonomy and fewer mistakes.
• Waymo's success with autonomous rides demonstrates the possibility of trustworthy, autonomous operation in the physical world.

  "And this means that if the system makes a mistake, that's okay, because usually the person can kind of recognize that or decide what to do even despite that mistake."

The Challenge of Long-Term Autonomy [4:26]

• To be useful, robots need to operate autonomously for extended periods, not just provide predictions for humans to act upon.
• Making espresso is used as an example of a complex task requiring precise control, smooth handling of liquids, and accurate timing.
• The goal is to achieve over 90% reliability for such tasks.

  "If we want it to actually be useful for us, we need it to make espresso reliably so that we don't have to babysit the robot very frequently in order for it to help serve drinks."

Iterative Improvement in Machine Learning [5:21]

• Traditional ML involves collecting data, training, and evaluating, often requiring manual iteration to improve reliability.
• This manual tuning by humans can be tiring and limits the achievable reliability.
• An ideal system would automatically identify areas needing more data or supervision for self-improvement.

  "And while this generally improves the reliability of the model, people eventually get tired, and it's hard to get really, really high reliability with a person that's manually tuning this."

Reinforcement Learning for Robotics [6:34]

• The approach resembles reinforcement learning, where the AI attempts tasks, learns from failures, and improves autonomously.
• Scaling reinforcement learning algorithms like PPO and GRPO, successful in language models, presents challenges in robotics due to the high cost of real-world attempts.
• A single one-minute robot task requiring a million trajectories could equate to 700 robot days.

  "And if we were to translate this very, very approximately to robotics, say we had maybe not millions or tens of millions, but just one million trajectories of a one-minute robot task, this is even shorter than the espresso task that I talked about."

Improving Reinforcement Learning Efficiency [7:48]

• Inefficiencies in RL include spending time on "dead end trajectories" where the robot pursues an unhelpful path.
• Human intervention, like teleoperation, can guide the robot to recover from errors and efficiently use data.
• Terminating episodes early or showing the robot how to recover prevents wasted time and improves learning.

  "And so instead of spending a lot of time trying to do that task, what we'll do is we'll actually have a human intervene and show the robot what to do and how to recover from that situation."

Amortizing Costs in RL [9:31]

• Standard RL algorithms make many attempts per prompt to estimate good vs. bad responses.
• This cost can be amortized by learning a general value function that estimates what is good or bad across different scenarios.
• A general-purpose value function, predicting time to success, can significantly reduce the number of attempts needed.

  "But we can actually amortize this cost rather than trying to collect a lot of attempts for a single prompt."

A Scalable Recipe for High Reliability [10:39]

• The improved approach trains a foundation model on diverse data, collects experience with human intervention, and uses a general-purpose value function.
• This allows for fine-tuning to higher degrees of performance.
• Examples include making a latte, constructing and stacking cardboard boxes, and folding/closing items in new environments.

  "And with this sort of improvement, we're able to kind of fine tune a foundation model to higher degrees of performance."

Quantitative Measures of Reliability and Throughput [12:48]

• Throughput, combining success rate and speed, is measured to evaluate performance.
• RL post-training stages show a significant increase in success rate and speed, nearly doubling throughput.
• The espresso task achieved over 90% success rate.

  "And specifically around a 2x throughput just from the RL stage itself, showing how we can get much greater reliability from reinforcement learning."

The Critical Ingredient: Memory [13:58]

• Most state-of-the-art foundation models for robotics lack memory or context, operating only on current sensor observations.
• Memory is critical for long-horizon tasks that involve multiple sequential steps.
• Naively feeding video context into models is computationally prohibitive due to the large number of tokens required.

  "They're just operating on the current sensor observations, the current camera readings and predicting actions based off of that."

Multi-Timescale Memory Solutions [14:55]

• A solution involves short-term video memory (approx. 10 seconds) computed efficiently.
• For longer memory (minutes to hours), a compressed textual summary of past events is used.
• This multi-timescale memory enables robots to perform tasks for 10-15 minutes autonomously, such as cleaning a kitchen.

  "And then for longer memory, for memory that spans multiple minutes or multiple hours, we don't necessarily need video of exactly what happened in that past history."

The Evolution of Generalist AI [16:06]

• Milestones include deep learning outperforming specialized methods (ImageNet, 2012), the rise of pre-trained models (BERT), and out-of-the-box generalist models (GPT-2).
• Compositional generalization, seen in models like Dolly (2021), signifies an ability to combine concepts.
• Robotics has lagged, often requiring bespoke datasets and training from scratch.

  "And then from there, I think the next big phase and the next big transition in generalist AI models wasn't using pre-trained models, but kind of moving from a pre-training fine tuning regime to a regime where we're just using generalist models out of the box."

Towards a Single General-Purpose Robot Model [17:59]

• The goal is an out-of-the-box model that doesn't require fine-tuning for each task, analogous to GPT.
• The second goal is compositional generalization, enabling robots to combine skills and understand concepts.
• Achieving this requires a large, diverse dataset and a model with sufficient capacity, using detailed prompting.

  "And so our first goal is to move towards a single general purpose model that can actually do all of the things that you want it to do."

Training a General-Purpose Foundation Model [19:49]

• The approach uses diverse robot demonstration data, policy rollout data, human videos, and web data.
• Detailed prompting includes memory, instructions, subtask construction, metadata, and optional sub-goal images.
• This detailed prompting allows the model to effectively utilize heterogeneous data.

  "And specifically what this looks like is we're going to train a foundation model that takes as input the memory like that I mentioned before, an instruction of what to do."

PI07: A Single General-Purpose Model [21:31]

• The PI07 model demonstrates capabilities like folding shirts, precise assembly, and replacing trash bags, all from a single model.
• It matches or outperforms specialized, fine-tuned models in tasks like coffee making and box building.
• This signifies a move towards a GPT-like era for robotics.

  "And on left, you can see it doing things like folding a collared shirt. On the top right, it's doing a really precise assembly step where it needs to insert a screw and drill that screw into a robot arm."

Compositional Generalization in Robotics [23:07]

• The model demonstrates compositional generalization by interacting with an appliance (air fryer) not explicitly in the training data.
• It also shows generalization across robot platforms, folding clothes on a different robot arm without specific training data for that platform.
• This indicates strong signs of combining skills and adapting to new scenarios.

  "And specifically, when you have compositional generalization, when you can basically bridge the concept like of an avocado and a chair and show that you kind of can combine those two, it means that the model has at least some kind of conceptual understanding of what an avocado is and what a chair is to the point that it can combine them into something that exhibits both concepts at the same time."

Importance of Diverse Data and Prompting [25:43]

• Experiments show that diverse data is crucial for generalizing to new tasks.
• Metadata prompting significantly improves performance, especially when incorporating lower-quality data.
• This detailed prompting allows the model to extract more value from diverse and even imperfect data.

  "Whereas with the metadata prompting, the performance actually increases when you add that low quality data."

Current Status and Future Outlook [27:03]

• Physical intelligence is now firmly in a GPT-like era for robotics.
• Models are being deployed in real-world applications by companies like Ultra and Weave.
• These models are adaptable to various robot embodiments, including drones and surgical robots.

  "And I think that we've kind of went there in just a few years. And lastly, we also have these models that are actually deployed in real-world circumstances."

The "ChatGPT Moment" for Robotics [28:29]

• A direct parallel to ChatGPT's rapid user adoption is unlikely due to the need for physical robots.
• However, the capabilities of robotics models are rapidly approaching usefulness in the real world.
• The next few years are expected to bring significant advancements in robotic AI capabilities.

  "I think that the distribution channel for physical models is going to be slower, unfortunately, because you actually need a physical robot there."

Transitioning to Generalist Policies [29:52]

• Starting with and fine-tuning generalist policies is often effective, even for small teams.
• Open-source generalist policies are readily available and powerful.
• The primary scenario where generalist models might not be suitable is in highly constrained environments with limited resources.

  "I think that just starting with a generalist policy and then fine tuning it even like right off the bat can be really effective."

PhD vs. Industry in Robotics [31:05]

• A PhD offers opportunities to learn about handling uncertainty, picking problems, and conducting research with freedom.
• Industry offers immediate impact, engineering roles, and opportunities to contribute to ML infrastructure and research.
• The decision is personal, with both paths providing valuable experiences.

  "And I think that basically like in research, you don't, no one kind of even gives you the problem to work on. You have to pick the problem."

The Robotics Equivalent of Internet-Scale Data [33:59]

• The robotics equivalent of internet-scale data is robot experience operating in real-world circumstances.
• This includes teleoperation data and, increasingly, autonomous experience.
• While human videos are useful, direct robot experience is critical for effective learning.

  "And so I think the equivalent, uh, in robotics is data of robots operating in real world circumstances."

Democratization of Robotics Models [35:42]

• The cost of embodied data and hardware might concentrate the best models in well-resourced labs.
• However, the trend of open-sourcing large datasets and powerful models is expected to continue, fostering an open-source community.
• The exact path of democratization remains uncertain but optimism exists.

  "I think that it's harder to get data even to like distill a model, for example, just like readily on the internet."

Model Output: Joint Positions vs. Motor Commands [37:23]

• Current models output target joint positions, with a controller then achieving those angles.
• Models can also predict target gripper positions in 3D space.
• Directly outputting motor torques or voltages is an option with pros and cons.

  "The model, all the models that I showed were outputting, um, target joint positions. So like, what is the angle of this joint? What is the angle of this joint and so forth that you want to hit?"

The Role of "Imagination" in Robotics [38:41]

• Models with the ability to predict future images (imagination) show improvement, as seen in a short folding example.
• However, models can perform surprisingly well even without this capability.
• Predicting the future seems like a relevant objective for learning.

  "The pilot seven model that I showed has something like this, where it can kind of imagine what a future image should look like and then try to accomplish that."

Improving Robot Speed [40:00]

• Speed improvements have been observed from reinforcement learning, with some policies exceeding human teleoperation speed.
• Bottlenecks include the slow pace of human teleoperation for teaching robots.
• Further projects are underway to develop faster policies.

  "I think one of the bottlenecks is that when you teleoperate robots to do things, which is the easiest way to teach a robot to do something. People are kind of slow at teleoperating the robot."

Most Surprising Robot Task [41:11]

• A robot demonstrated an emergent capability of transferring skills between its left and right hands, assembling a pinwheel despite never seeing data of that specific hand configuration.
• This suggests an learned equivariance between the robot's grippers.
• The speaker is excited by emergent capabilities and pushing the boundaries of long-term reliability and new capabilities.

  "The robot essentially had learned this sort of equivariance between his left hand and his right hand so that it could actually transfer, uh, behaviors from one hand to another. Despite the fact that that was never in the data."

Breaking into Robotics from Software Engineering [43:11]

• Join a robotics company as a software engineer.
• Experiment with open-source models and hardware, as demonstrated by an individual who successfully transitioned into the field.
• Getting hands-on experience and showcasing passion is key.

  "And so someone, um, who actually now works at physical intelligence, her name is Jenny. She, uh, worked in trading algorithmic trading for a while."