OpenAI Just Confused Everyone... Again

The video clarifies the distinction between "agents" as defined by some in the industry and OpenAI's new "agent builder." While industry standard terms like "workflows" describe systems where LLMs and tools are orchestrated through predefined code paths or declarative graphs, OpenAI's "agent builder" is presented as a tool for designing these very workflows. The speaker contrasts this with a more autonomous definition of agents, which can independently plan, act, and adapt their plans based on feedback and environmental observations.

This distinction is crucial as the speaker unpacks OpenAI's historical perspective from a 2023 guide, which critiqued declarative, graph-based workflow builders for becoming cumbersome with complexity and potentially requiring specialized languages. OpenAI's new agent builder, despite its potentially useful applications and smooth integrations, is seen by some as building on these workflow concepts rather than true autonomous agents. The speaker suggests that while the agent builder might not kill existing workflow tools like Zapier or Make, its effectiveness, especially regarding vendor lock-in to OpenAI models, remains to be seen.

The core of the discussion revolves around the definition of an agent. Industry definitions, like Anthropic's, highlight LLMs dynamically directing their processes and tool usage. In contrast, OpenAI's definition is described as "extremely vague," stating an agent is a system that "can do work independently on behalf of the user." The video argues that what OpenAI is currently calling "agent builder" is essentially a visual tool for constructing workflows, a concept different from agents that exhibit genuine autonomy in planning and action.

• The industry often refers to systems with chained LLM calls and conditional logic as "workflows," while OpenAI is now using "agent builder" for this concept.
• A more robust definition of an agent involves the AI independently planning, taking actions, and modifying its plan based on feedback and outcomes.
• OpenAI's previous guide critiqued declarative, graph-based workflow builders for becoming complex and cumbersome for dynamic tasks.
• While OpenAI's agent builder offers integrations and features like evals, it's argued to be primarily for building workflows, not autonomous agents, and may lead to vendor lock-in.
• The speaker suggests that the effectiveness of the agent builder and its potential to integrate other models through MCPS still needs to be tested over time.

Related Video Summary

• NEW Agent Builder: Did OpenAI just kill n8n?
• Why 99% of AI Products Fail: A CTO's Hard-Won Lessons
• The Latest in AI: Job Loss, Elon & Sam Altman Chip Race & the "AI Bubble" w/ Brian (Blitzy) & Emad

Defining AI Agents and Workflows [0:05]

• The term "agent" can be confusing, particularly how OpenAI defines it.
• What the industry calls "workflows" involve multiple LLM calls chained together through conditional logic gates.
• OpenAI is now labeling this "agent builder."
• OpenAI's new definition of an AI agent is a "system that can do work independently on behalf of the user," which is described as "extremely vague" and possibly intentional.
• Anthropic defines agents as systems where LLMs dynamically direct their own processes and tool usage, maintaining control over task accomplishment.

This section introduces the ambiguity surrounding the definition of AI agents, contrasting industry-standard "workflows" with OpenAI's broad new definition. It highlights the speaker's view that the vagueness in OpenAI's definition may be deliberate.

Here is the definition from Enthropic. So they say agents are systems where LLMs dynamically direct their own processes and tool usage maintaining control over how they can accomplish tasks.

Declarative vs. Code-First Approaches to AI Orchestration [0:57]

• A "declarative graph" requires the user or developer to define every step and available tools beforehand; the flow is predefined.
• This is what Anthropic and the industry refer to as "workflows"—systems where LLMs and tools are orchestrated through predefined code paths.
• OpenAI's blog post and terminology are seen as confusing, for example, "design workflows with agent builder."
• A 2023 OpenAI guide distinguished between declarative and non-declarative graphs.
• Declarative frameworks require developers to explicitly define every branch, loop, and conditional upfront through graphs, which can become cumbersome for dynamic and complex workflows.

This part of the discussion breaks down the concept of declarative graphs and contrasts it with other approaches, pointing out the potential limitations of highly structured, predefined workflows.

While beneficial for visual clarity, this approach can quickly become cumbersome and challenging as workflows grow more dynamic and complex, often necessitating the learning of specialized domain specific languages.

OpenAI's Agent Builder and the Agent SDK [2:48]

• OpenAI's new "agent builder" appears to be focused on creating declarative workflows.
• The "agent SDK," in contrast, adopts a more flexible "code-first approach."
• Developers using the SDK can directly express workflow logic using familiar programming constructs without needing to predefine the entire graph upfront, enabling more dynamic orchestration.
• The agent SDK can generate code and uses this technology in the backend, but the speaker reiterates that this is for defining workflows, not necessarily building agents in a more autonomous sense.

This section delves into OpenAI's Agent SDK, presenting it as a more flexible, code-driven alternative to the visual "agent builder," which is still perceived as workflow-centric.

So with this you can actually go and generate code and is using the agent SDK in the back end.

Understanding Workflows and Autonomous Agents [3:20]

• A workflow can be translated into a declarative decision graph, with an LLM helping to express actions.
• The LLM can make decisions at different points based on available tools, but the user defines the actions and conditions.
• For most business applications, this user-defined workflow approach is often what's desired.
• Agents, on the other hand, have the ability to plan, take actions, and modify their plans based on feedback from the environment or the results of those actions.
• This involves a cycle where a model selects a tool, takes action, generates results, and then uses those observations to update its plan and take more actions, indicating greater flexibility and autonomy.

This part of the transcript aims to solidify the distinction between the two concepts: workflows are user-guided processes, while agents are systems capable of self-directed planning and adaptation.

So there is a lot more flexibility in autonomous agents and we have seen very interesting response to OpenAI's workflow builder or agent builder.

Critiques of Visual Workflow Builders and the Future of Agent Tools [5:07]

• Visual workflow builders are criticized for not being low-barrier to entry, even for mass audiences, and remaining difficult for average non-technical users.
• Complex tasks quickly become too complicated to manage within visual builders.
• There's a speculation that OpenAI's agent builder might impact existing visual builders like Zapier or Make.
• However, the speaker believes this is unlikely, as these platforms have support for integrations that can be brought into OpenAI's agent kit via MCPS.
• The new agent kit is advertised to developers but might not be specifically for them, as developers often prefer more control than visual builders offer.

This section addresses criticisms of visual builders and discusses the potential impact of OpenAI's new tool on the existing market, suggesting that integration rather than replacement is more probable.

First of all they are not low barrier to entry. So here argue that despite being built for mass audience, it is still not easy for average non-technical user to use them.

Vendor Lock-in and the Potential of OpenAI's Agent Kit [7:10]

• A significant concern with OpenAI's agent kit is strong vendor lock-in, limiting users to OpenAI models.
• However, support for MCPS theoretically allows for the integration of other models.
• It's suggested that a workflow could be built and tested within the OpenAI kit and then potentially have its models replaced with others, though this needs testing.
• The speaker asks viewers for their thoughts on agent kit and the difference between workflows and autonomous agents.

The concluding part of the video raises the critical issue of vendor lock-in with OpenAI's platform and explores potential workarounds, while also inviting community engagement on the topic.

However, keep in mind if you're using OpenAI agent kit, it has very strong vendor lock in.