Why The Harness Matters More Than The Model | YC Paper Club

The Genesis and Evolution of Harnesses [00:00:00]

• The YC Harness Club event introduces the concept of AI harnesses, initially seen as mere prompt engineering wrappers.
• Early dismissals of harnesses as "subpar research" are contrasted with their proven impact, such as an 18% performance increase and enabling critical functions like RKGI.

• "Here's another great one. Context engineering is not a research problem."

Static vs. Self-Improving Harnesses [00:01:45]

• The progression of harnesses is categorized into a "Static Harness Era" and the more recent "self-improving Harnesses."
• The latter six months have focused on harnesses that can improve themselves over time.

• "And then this latest, maybe the last six months, has all been on the self-improving Harnesses."

Harnesses and Adaptability [00:03:00]

• Harnesses are crucial for enabling AI agents to adapt quickly to new problems and data distributions, a key aspect of Arc AGI.
• This adaptability is contrasted with traditional methods that struggle to learn from "batch size one" or saturate after limited in-context learning.

• "That's kind of the main point of Arc AGI is how quickly it adapts to a new problem, a new distribution, and does well in it."

Dramatic Performance Gains [00:04:00]

• The effectiveness of harnesses is demonstrated with Claude Opus, where performance on a private benchmark jumped from 30% to 95% with harness integration.
• AVO from NVIDIA achieved 100% performance on the same task, showcasing the power of these systems.

• "Really, Claude Opus was one of the first that was actually verified on the holdout on the private that no one else has access to other than Greg and Jolet."

Accidental Harness Creation [00:05:15]

• The speaker recounts building a harness by accident while trying to create a user interface for Andrej Karpathy's auto-researcher.
• The goal was to see diffusion LLMs ensemble to beat AR LLMs by sharding them.

• "I forked it and I was playing around with it. And all I wanted to do was make, like, a little user interface to kind of see what's happening and track it."

The Auto-Researcher Harness Architecture [00:06:00]

• The harness specifies a purpose (e.g., ensembling diffusion LLMs), seed ideas, and a validation metric (e.g., GSM8K).
• It involves a suite of agents: scoping, PI, research, council, author, and a cockpit for monitoring progress.

• "And then I specify a valmetric. And maybe I want to do GSM8K and, like, GPT-2 setup or something like that."

Historical Context of Harnesses [00:10:00]

• A brief history traces harnesses from GPT-2 (2019) as a simple end-of-sequence loop to more complex systems.
• Early "V0 harness" lacked tool calling or skills, relying on basic prompt-response mechanisms.

• "The initial harness, the GPT to February, 2019 is a while not end of end of sequence loop."

Innovations in Static Harnesses [00:11:00]

• Context innovations like few-shot learning (July 2020) and chain-of-thought (CoT) improved performance by providing examples or smearing logic over more tokens.
• Action space innovations introduced tool calling (Toolformer, WebGPT) and memory manipulation (MemGPT).

• "And so that was the chain of thought idea. Very cool. And that was all context innovations, output space innovations, action space innovations."

Chaining Tools and Learning Skills [00:13:00]

• Voyager demonstrated chaining tools to achieve tasks and learning/distilling these skills back into the system prompt, notably in Minecraft.
• Intercode allowed for on-the-fly tool or skill generation by outputting code.

• "And then Voyager said, okay, well, we have these tools, but like, what if I want to chain together these tools to achieve a task and then I learn it?"

Self-Improvement and Multi-Agent Systems [00:14:30]

• React, Self-Refine, and Reflection introduced multi-agent collaboration for self-improvement, where agents critique and refine outputs.
• RLM extended this to recursive agent calls, allowing a main orchestra agent to manage sub-agents.

• "But this idea, if I have multiple, um, agents that, that have different roles and they can help self-improve self-improve on the context, then I can, um, get smart and smarter."

Harness V1: The Static Harness Era [00:16:00]

• Harness V1 is characterized by a static system prompt and loop structure, managing agent specs, system prompts, turns, tool calls, and skills.
• It can be spawned via prompts or run on a cron schedule.

• "And that's largely what the V1, um, is."

The Rise of Self-Improving Harnesses [00:17:30]

• The most exciting advancements involve harnesses that learn, either by updating the system prompt or modifying the harness code itself.
• DSPY uses genetic programming to learn optimal system prompts.

• "Which is very trippy. And so one of the famous ones that I actually wanted them to talk, but they're actually running a 150 person DSPY meetup tonight in, in, in, in San Francisco, they couldn't make it."

Darwin Godel Machines and Meta Harnesses [00:19:00]

• Darwin Godel Machines go further by allowing agents to modify their own harness code.
• This creates a meta-harness that produces new harnesses, leading to progressively better agents over time.

• "And then Darwin Godel machines actually go a step further. Not only are you allowed to change the system prompt, but you're allowed to change the harness itself, the harness code that is actually running."

Continual Harness and Test-Time Training [00:21:00]

• Continual Harness adds memory classes like history and enables dagger-style online learning (test-time training) on the LLM weights.
• This allows for updating the model based on small amounts of newly learned examples.

• "But the coolest part I think is for the classic RL people, I see Robert back there. He definitely would enjoy this dagger style online learning where you can actually update the weight file itself."

Prime Agent: A Self-Improving RLM Harness [00:25:00]

• Seth presents Prime Agent, a self-improving RLM harness designed with a first-principles approach.
• It emphasizes expressibility, providing features like persistent state, tools, compute, and messaging between sub-agents.

• "The harness itself is the layer between the LLM and the world that adds things like this persistent state tools and compute."

Prime Agent's Architecture and Capabilities [00:27:00]

• Prime Agent offers an agent's view with an overview of parallel sessions and a root session as the project orchestrator.
• It leverages sub-agents programmatically and runs within an IPython shell, managing tools, memories, and sub-agents.
• Features include CRUD operations on harness components and a cache-like context management system (L1, L2, L3).

• "This is so that, you know, when you close, uh, your laptop or you, you control C out of the session, it's still running in the background."

Prime Agent's Performance and RKGI Results [00:35:00]

• Prime Agent demonstrates strong performance on RKGI, achieving 99.9% initially (with a borrowed prompt) and 78% with GPT-SOL after proper sandboxing.
• It shows parity or superiority to other harnesses like MS Agent and Cloud Code in various benchmarks, with a better cost-to-performance ratio.

• "And I was like, oh my God, the first run that I got, it hit 99.9%."

Open Jarvis: On-Device Personal AI [00:42:00]

• John Sadfalcone introduces Open Jarvis, aiming to build the core of a personal AI stack entirely on-device.
• It addresses the cost, privacy, and energy concerns of cloud-bound AI assistants.

• "Personal AI is everywhere, but it's mostly cloud bound today."

Open Jarvis Primitives and Optimization [00:44:00]

• Open Jarvis is built on simple primitives: user interfaces, agentic logic, intelligence (LLMs), inference engines, tools/memory, and learning mechanisms.
• It can be optimized using cloud LLMs to improve local stacks, achieving significant cost and latency reductions.

• "We wanted to create the simplest set of primitives for which you define any sort of harness or personal AI stack."

QM: YC's Open-Source Agent Harness [00:50:00]

• Josh and Regan present QM, YC's open-source agent harness designed for work, available in Slack or via web UI.
• It provides customizable, personal contexts with sandboxes, files, and crons, and supports multiplayer interactions.

• "QM is one system that, uh, gives every employee at YC an open claw-like assistant, uh, that's like fully customizable and available in Slack or via web UI."

QM's Development and Architecture [00:52:00]

• QM evolved from earlier internal agent projects, moving towards giving agents more capabilities and autonomy.
• Key architectural shifts include centralizing context in Postgres and treating sandboxes as resources rather than fixed homes for agents.

• "So with, uh, you know, with Hermes and with OpenClaw, you effectively have, uh, the agent has its own computer, which is super powerful, but it's also trapped inside that computer."

QM's Capabilities and Challenges [00:55:00]

• QM integrates broadly with company resources via APIs and manages credentials securely.
• Challenges include agents giving up too early (addressed by setting budgets) and difficulty understanding social contexts and nuanced information sharing.

• "And so having like local affordances for this has been something that's been, uh, that's been really important."

Future of Harnesses and Agentic AI [00:58:00]

• The discussion concludes by emphasizing the growing importance of agentic context management, swarms, RLMs, and standardized evaluations.
• The trend points towards a future where a majority of inference calls will be on local devices, driven by better accelerators and LLMs.

• "I think that you should think about agentic context management, you should think about swarms and looking into further depth, RLMs and trying to run standardized evals."