Jeff Dean: The 1% Rule for Building in AI

AI's Agentic Future [0:49]

• Models are now capable of complex coding tasks, aligning with the prediction of AI reaching junior engineer level.
• The ability of AI to handle more complex tasks and excel in non-coding domains has grown faster than anticipated.
• By 2027, expect significant automation of ML systems themselves, with AI improving capabilities through automated experimentation and problem decomposition.

  "I think you will see a lot more automation of ML systems themselves, basically getting ML systems to improve their capabilities by running lots of experiments, breaking things down into sub-problems, you know, running those sub-problems in a tight, automatic experimentation loop, putting the results together, and being able to then, you know, get some improved system out from that sort of fully automated problem decomposition and automated experimentation."

The "Fits in Memory" Moment for Inference [1:37]

• Recalling Google Search's 2001 shift to RAM, Dean identifies the current parallel in the need for high-performance, low-energy inference hardware.
• Latency is critical for making agent-based systems accessible, and specialized hardware offers greater energy efficiency and lower latency than general-purpose devices.
• A 50x improvement in latency could unlock significant new capabilities.

  "Because I think everyone is now realizing that inference is the key to making, you know, these agent-based systems be available to more and more people. And that latency is really important. and that specialization of the hardware is a really key way you can make things that are more energy efficient and lower latency than more general purpose computational devices like, say, GPUs or TPUs."

Challenging AI Assumptions [2:34]

• A common false assumption is underestimating the potential for agent-based systems to run for days or weeks on complex tasks.
• These long-running agents can implement entirely new software versions in different programming languages, focusing on properties like safety or performance.
• The ability for AI agents to tackle multi-week, complex tasks is a significant development not yet fully internalized by everyone.

  "I mean, I think probably one thing is people don't quite realize how possible it is to have, you know, agent-based systems that can run not just for an hour or two hours on a problem you care about, but for some problem domains and with highly capable models underlying them, you can get them to run for days or weeks and do really, really complicated tasks."

The Origin of TPUs: Napkin Math and Energy Efficiency [3:40]

• A 2013 calculation showed that widespread speech recognition use would require doubling Google's server fleet, prompting the development of specialized hardware.
• This led to the creation of TPUs, optimized for low-precision, dense linear algebra, which are highly efficient for ML inference.
• TPUs offered significant energy efficiency (30-80x) and lower latency (20-30x) compared to CPUs and GPUs of the time.

  "And so we came up with TPUs, which are sort of very specialized for essentially low-precision, dense linear algebra, which is at the heart of nearly all of the modern machine learning algorithms we use today."

Latency Numbers and Energy as the Core Metric [7:10]

• Modern AI systems require understanding key metrics like memory bandwidth, energy per operation, and inter-chip communication bandwidth.
• Energy consumption is the primary measurement unit, with data movement (IO) being orders of magnitude more energy-intensive than computation.
• This energy disparity necessitates techniques like batching to amortize data movement costs, impacting real-time applications.

  "You want to know how much energy does it take to do a single multiplier operation? You know, what is the interconnect bandwidth between chips and how much does that, how many chips can you connect with that bandwidth?"

Context Engineering and Startup Opportunities [11:34]

• AI progress now involves more than just model size; retrieval, memory, and agent tools (context engineering) are crucial.
• Context engineering is accessible to smaller teams, unlike the resource-intensive model training, leveraging APIs and custom setups.
• Startups can win by focusing on niche domains or personal data organization where general models may not have an advantage, but must consider the rapid improvement of general models.

  "But for context engineering everyone here could do it you have you need the API to something like Gemini and then work on your own setup for your own retrieval your own tool calls and et cetera et cetera so what are some tips for everyone here how does everyone get better at and become exceptional at context engineering"

The Power of Clear Specifications and Taste [19:08]

• Managing fleets of agents requires clear design docs and specifications, akin to translating software between languages.
• The most critical skill for founders is "taste"—knowing which problems are worth solving and have the potential for significant positive impact.
• Developing taste comes from experience, evaluating past predictions, and engaging in thought experiments that challenge fundamental assumptions.

  "I think it's really having incredibly good taste in what you ask your agents to work on right that is the crux of you know from my background a research problem you know a researcher can have all the tools and all the techniques but often most of the battle is what problem are you going to spend your time on and if you pick the problem well and you that's way better than if you delightfully execute a research investigation into a rather boring problem"

Self-Improving Systems and Scientific Method [23:02]

• AI is enabling automated scientific methods, where systems propose, run, and evaluate experiments at scale.
• This accelerated loop can tackle complex problems in ML, science, and engineering, like designing chips or discovering new molecules.
• Faster validation models, potentially learned approximations, can dramatically speed up research cycles.

  "And I think there are more and more problems that are now possible to implement where that whole loop of running, you know, not just a few experiments but running many many experiments because you're able to automate that loop and make the latency of that loop extremely low is going to be really important."

Perseverance and Impact [26:55]

• The influential distillation paper, enabling smaller, efficient models, was initially rejected, highlighting the need for perseverance.
• Dean emphasizes that even with rejections, continuing to work on impactful problems is crucial.
• He advises young engineers to focus on working on problems they care about, with people they enjoy, and where their work can make a positive difference in the world.

  "I mean, yeah, I mean, I think I don't fault the program committee because, you know, a lot of times a paper gets three reviews and someone will look at one of the reviewers will look at it and in this case they said, oh, it's unlikely to have significant impact unlikely to have significant impact but, you know, I think, you know, when we wrote the paper we actually saw this was a super important problem..."

Future Frontiers and Collaboration [30:31]

• Exciting future problems lie in new hardware approaches, more efficient ML algorithms, continual learning, and multi-agent interactions.
• Dean encourages building consequential systems by focusing on problems that matter, collaborating with talented individuals, and continuously expanding one's skill set.
• The goal should be to make a positive impact on the world, whether through niche applications or broad technological advancements.

  "I think there's a lot of interesting problems in the world and I'll just rattle off a few this is not exhaustive because the world is a very big place and full of problems you know I'm particularly excited about new approaches to hardware you know we thought experiment there was kind of, you know, a, you know, indication of that or much more efficient inference hardware you know I think there are radically different kinds of algorithms for machine learning that might be much more data efficient than the approaches we're using today"