Why Robotics Still Isn't Solved - But Could Be Soon | YC Paper Club

The Promise and Reality of 2026 Robotics [00:00:00]

• The speaker recalls past predictions of robotics being "solved" in the near future, citing AlphaGo and Mujoco as examples.
• Despite advancements like the Aloha system, enabling tasks like watering plants and making coffee, widespread robot adoption has not materialized.
• The year 2026 was specifically promised as the "year of the robots," with confidence in VLA and diffusion policy advancements.

  "Clearly, next year, robotics is going to be solved."

The "Year of the Demos" [00:01:30]

• Halfway through 2026, consumers cannot yet purchase general-purpose robots, though pre-orders exist.
• While success is seen in work cells, the "Rosie the robot" ideal remains elusive.
• The current year is definitively "the year of the demos" due to the difficulty of data collection.

  "So the reason why is because, you know, raise your hand if you've ever done teleops collect data collection."

Challenges in Scaling Robotics [00:02:15]

• Teleoperation for data collection is described as "remarkably hard" and "very, very finicky."
• Scaling this data collection is a major bottleneck for advancing robotics.
• Four key areas present walls to scaling: physical real-world modeling, deformable objects, action space representation, and the sensory-motor issue.

  "And so if we're relying on this type of data, as and we need to scale that up like crazy, then it's we're kind of doomed."

The Sim-to-Real Gap [00:03:00]

• Video models deployed in the real world often fail to respect physics, unlike in simulations where errors might be magically resolved.
• The "sim-to-real gap" remains a significant unsolved problem.
• This gap is particularly evident when simulating complex scenarios like driving.

  "In the real world, it really doesn't respect physics."

Deformable Objects and Action Space [00:04:30]

• Determining the transition function for deformable objects is even more challenging than rigid body dynamics.
• Action conditioning requires substantial data, and estimating dynamics functions is complex.
• Representing the action space effectively is an "unsolved thing" for rapid learning.

  "Deformable objects, even worse is, that's just determining the transition function from ST to ST plus one."

The Sensory-Motor Issue [00:05:30]

• Humans possess rich sensory feedback (normal force, friction, temperature) that robots lack.
• Robots typically have limited sensors, like a single force-torque sensor on a fingertip.
• This sensory deficit prevents robots from performing tasks that humans do intuitively, like finding an object by feel.

  "We have these nerve endings that can do so much, we can detect the normal force, we can detect the tangent force, we can detect moisture, temperature, vibration, we can estimate the coefficient of friction."

Embodiment Drift [00:07:00]

• "Embodiment drift" refers to the degradation of actuators over time due to dust, corrosion, or wear.
• This drift causes unpredictable variations in robot actions, requiring retraining of policies.
• The variability in a car's acceleration or battery power over time exemplifies this issue.

  "And so in this state, if I take this action, this is how much force is actually going to be applied."

MEM: Multi-Scale Embodied Memory [00:09:30]

• Marcel presents MEM (Multi-Scale Embodied Memory), a system designed to address long-horizon tasks.
• Current policies often lack memory, leading to repetitive or erroneous actions (e.g., washing dishes forever, burning grilled cheese).
• Adding memory to policies faces effectiveness (performance degradation) and efficiency (resource intensiveness) challenges.

  "But my claim is that in order to obtain all of these insights, we actually need to add memory into our policies."

MEM Architecture and Short-Term Memory [00:11:30]

• MEM decomposes policies into high-level and low-level components, with memory split into short and long contexts.
• Short-term memory uses a compressed visual representation from dense frames for dexterous manipulation.
• A novel encoder based on ViT with temporal attention is used for efficient compression.

  "Our idea here was to design a new encoder that is based on the VIT. But instead of just taking a single frame, we're actually also going to add some attention temporal layers."

MEM Capabilities [00:13:00]

• MEM enables tasks like making a grilled cheese without burning it, unloading groceries by remembering item locations, and cleaning a window efficiently.
• These tasks, seemingly simple, require memory to track progress and avoid repetitive actions.

  "Here we have our main flagship tasks that we did when I when I was at Physical Intelligence, and it was like making this grilled cheese."

MEM Long-Term Memory [00:14:00]

• Long-term memory uses a compressed language representation of recent events for the high-level policy.
• This textual memory is more compressed than images, aiding training and inference.
• It allows for tasks lasting tens of minutes, like preparing ingredients for a recipe.

  "This actually, like much more, much more compressed representation than images, since text uses way less stock tokens and images."

In-Context Adaptation with Memory [00:15:30]

• Memory enables "in-context adaptation," allowing robots to react to and correct mistakes.
• Policies without memory repeat errors, while those with memory can adjust their actions (e.g., picking up a dropped chopstick, switching sides to open a fridge).
• This ability to adapt is crucial for real-world robotic performance.

  "But actually, now when we add memory, we're going to see that the robot policies are going to make a mistake in the first time, right?"

Discussion on Textual Memory [00:17:30]

• Questions arise about how to ensure the right information is encoded in textual memory and how it's used during inference.
• The current approach uses annotated data, but future work could involve reinforcement learning on the memory space.
• Textual memory is stored in RAM and fed as normal text tokens to the VLA.

  "Yeah, that's an awesome question. So right here, we train our high-level policies with SFD. So we had to annotate all of our data."

Self-Supervised Bootstrapping of Embodied Reasoning [00:20:00]

• Milan discusses moving towards robots that teach themselves to reason, focusing on Vision-Action Models (VLAs).
• VLAs are trained on scarce robotics data, augmented by reasoning signals like "chain of thought."
• The challenge is identifying what to reason about and where the "oracle source" for this reasoning data comes from.

  "The idea of similar to chain of thought for the LLM land, where you go from question to answer by, um, explicitly providing some sort of logical steps."

R&B Encore: Embodiment-Specific Reasoning [00:23:00]

• R&B Encore (Refine and Bootstrap Embodiment-Specific Chain of Thought Reasoning) is a self-improving pre-training cycle.
• It treats reasoning as an unobserved latent variable, using variational inference to propose and validate reasoning traces.
• The system scores reasoning based on concision, non-triviality, and action predictiveness.

  "So one is a reasoning proposer. You can think of this as a annotator model, um, which proposes for a given context for a given demonstration, um, various types of proposed reasoning."

R&B Encore Applications and Findings [00:25:00]

• Tested on manipulation, R&B Encore found move and gripper position reasoning useful, while perceptual reasoning was less so.
• For legged locomotion, structural affordances and movements were more important than terrain reasoning.
• In self-driving, meta-action and visible object reasoning proved useful, pruning out hallucinated experiences.

  "We find that move reasoning and gripper or move and gripper position and, uh, position type of reasoning is very useful."

SimToolReal: Dexterous Manipulation in Simulation [00:28:30]

• Tyler presents SimToolReal, a policy trained entirely in simulation for dexterous manipulation.
• The policy learns to control a 22-DOF hand and 7-DOF arm, performing tasks zero-shot with novel tools.
• It unifies dexterous tool manipulation as goal reaching, using a single policy for multiple tasks.

  "So rather than learning the policy from teleoperation, we train it entirely in simulation using sim-to-real reinforcement learning."

SimToolReal's Goal-Conditioned Approach [00:30:00]

• The key insight is to treat manipulation as goal reaching, where the policy moves objects to desired poses.
• At inference, human video demonstrations provide the sequence of goal poses.
• The policy tracks these goals at 60 Hz, demonstrating impressive dexterity without retraining.

  "Our key insight is that we can unify dexterous tool manipulation as goal reaching."

SimToolReal Performance and Limitations [00:32:00]

• The policy generalizes to 12 unseen tools and tasks, performing well on tools with long handles.
• Performance is weaker on heavier tools and smaller objects due to pose tracking failures.
• Recovery behaviors, like re-grasping a dropped hammer, are impressive, partly due to simulated random forces during training.

  "We evaluate the same frozen policy across 12 unseen tools and target behaviors, and achieve substantial task progress across every tool family."

Play to Perfect: From Play to Assembly [00:35:00]

• Follow-up work, Play to Perfect, extends SimToolReal towards precise, contact-rich assembly.
• It uses a pre-training phase of task-agnostic play, followed by fine-tuning on assembly tasks.
• This enables diverse assembly behaviors, including tight insertions and multi-part assembly.

  "So we argue that before we can learn the hard problem of precise assembly, we must first learn the easier problem of playing with objects in free space."

Rerun: Data Infrastructure for Physical AI [00:37:30]

• Nico discusses Rerun, a company building data infrastructure for physical AI.
• Rerun provides tools for data collection, visualization, querying, and loading for robotics.
• Their platform aims to simplify working with multimodal, multirate, and episodic physical data.

  "At Rerun, we are building this sort of unified data layer for physical AI."

Robotics Application Companies [00:39:00]

• Nico identifies a growing pattern of "robotics application companies" that own full business problems end-to-end.
• These companies focus on operations, deploy with minimal custom hardware, and often start with teleoperation.
• This model is seen as the "new SaaS" for the physical world, transforming the economy.

  "So basically that new category is what might go just robotics application companies."

Iterative Development in Robotics [00:41:00]

• The recommended approach is to start with a single customer problem, solve it with teleoperation and off-the-shelf hardware, and then iterate.
• The physical world is "brutal," requiring rapid learning from real-world deployments.
• Learning from teleoperation provides a foundation for training models.

  "The physical world is brutal, right? Everything that you do is going to break."

Data Collection and Evaluation [00:44:00]

• Key steps include replicating customer environments, establishing repeatable success metrics, and collecting effective training data.
• Continuous training, evaluation, and debugging of data are crucial.
• Data infrastructure must handle the unique nature of physical data (multimodal, multirate, episodic).

  "First thing you need to do is to have a replica of the customer's environment in your own office."

World Action Models (WAMs) [00:48:30]

• Bill and Guan Ming from General Instinct discuss World Action Models (WAMs) as an alternative to VLAs.
• WAMs use diffusion transformers to predict future frames and kinematics simultaneously.
• While powerful, WAMs like NVIDIA's Dreamer are computationally expensive and not scalable for robotics.

  "What a word action model is, however, is you have a central diffusion transformer that's trying to predict what the future looks like and future kinematics at the same time."

Optimizing WAMs for Robotics [00:50:30]

• General Instinct focuses on optimizing WAMs for speed and efficiency.
• They employ techniques like distillation, splitting transformers, and reducing autoregressive steps.
• Their optimized WAM can run significantly faster, making it more viable for real-time robotics.

  "We also did distillation on the auto-regressive flow matching sampling. Previously, it might take 50 steps or like 100 steps to do the flow matching decoding."

Future of Robotics Infrastructure [00:54:00]

• The discussion highlights the need for specialized infrastructure for physical AI models, analogous to VLLM for LLMs.
• General Instinct aims to provide this infrastructure for WAMs and VLAs.
• The focus is on enabling these models to run efficiently on real-world robots.

  "For physical AI models like word action models and VLAs, you would have us General Instinct."

Key Takeaways on Robotics Development [00:56:30]

• Selective reasoning is more important than exhaustive reasoning.
• Self-supervised bootstrapping addresses the "chicken and egg" problem of model and data sources.
• The approach generalizes across embodiments and VLA sizes.
• Embodied reasoning is a resource to be discovered and budgeted, not a fixed template.

  "Selective reasoning is way more important than exhaustive reasoning."

Challenges and Opportunities in Robotics Applications [00:58:30]

• Robotics faces a "death by a thousand cuts" due to numerous failure modes.
• Early success is often seen in areas amenable to teleoperation, like data centers, warehouses, and small-scale manufacturing.
• Companies are supply-constrained due to the difficulty of building reliable robots.

  "Why haven't there been a bunch of these robot application companies yet that have been, you know, at billions of revenue?"

Estimating Data Needs for Robotics [01:01:00]

• Estimating the data scale needed for deployable models is challenging.
• The strategy of starting with teleoperation allows for learning and plotting scaling curves.
• Autonomy is often viewed as a scaling factor rather than a strict requirement from the outset.

  "I think that's really one of the core ideas between if you can tell up first, you it's not obvious that all many businesses work without full autonomy."