Stanford Global Alumni Webinar | August 2025 | AI Agent Simulation of Human Behavior

The Challenge of Incomplete Information in Decision-Making [00:36]

• Human decision-making often relies on incomplete information about how others will react, leading to "bad bets" or incorrect outcomes.
• This challenge spans various domains, from organizational strategy and product design to personal management and policy-making.
• The inability to predict reactions is not due to a lack of intelligence but the inherent difficulty and limited ability to gather feedback before acting.
• Historical examples, like sociologist Robert Merton's observations from 1906, illustrate the enduring difficulty in designing for predictable group behavior.

"We make these decisions based on the best data we can. But that data is often incomplete."

The Concept of a "What-If Machine" and AI Simulation [02:40]

• The idea of a "what-if machine" is introduced, a tool that could predict potential future scenarios and reactions before implementation.
• This machine would allow individuals and organizations to explore hypothetical outcomes of decisions, such as customer reactions to a new strategy or potential failure pathways.
• The claim is that such a tool could significantly improve decision-making accuracy and frequency.
• The core research question is whether simulated AI agents can be created to replicate human behavior, thereby enabling these "what-if" scenarios.

"What if you had a whatif machine? What would you use that for?"

Historical Context of Simulation and Agent-Based Models [04:01]

• Simulation is not a new concept, with its roots tracing back to agent-based models developed by Thomas Schelling in 1978, recognized with a Nobel Prize.
• These models are still used today, for instance, in pandemic spread simulations run on supercomputers.
• Simulations are also prevalent in entertainment, such as the popular video game "The Sims," which simulates people's lives.
• The increasing deployment of AI into human-interacting environments necessitates agents that can reason about human responses.

"Simulation isn't a new idea either, right? It goes all the way back to 1906."

Limitations of Previous Simulation Models [05:39]

• Historically, simulation models have been rigid, offering limited ways to represent human behavior.
• One approach was to define humans by a small number of parameters (e.g., five), which is too sparse to capture the richness of human behavior.
• Another approach, seen in games like "The Sims," involved writing explicit scripts, which are limited by the creators' imagination and inherently incomplete.
• Academic literature concluded that these models were "highly stylized and have had minimal impact" in practical applications.

"The basic conclusion is... the models have been highly stylized and have had minimal impact."

The Emergence of Generative Agents and Large Language Models [06:49]

• Recent advancements in AI, specifically Large Language Models (LLMs) like ChatGPT and Claude, have provided a new foundation for simulating human behavior.
• LLMs have been trained on vast amounts of human behavior research and social media data, exposing them to the complexities of human interaction.
• These models can be prompted to adopt the perspectives of individuals with different backgrounds, experiences, and traits.
• By creating multiple such prompts, an entire crowd of simulated individuals can be generated.

"What we recognized was that you can prompt these large language models to take on the perspectives of a bunch of different people with different backgrounds, experiences, and traits."

Smallville: A Simulated Town of Generative Agents [08:07]

• A notable demonstration involved creating a "little town called Smallville" populated by 25 autonomous generative agents.
• Each agent was designed to replicate a different person in the town, with the simulation visualizing their daily activities.
• This setup allows for learning about people and interventions in a simulated environment.
• The work gained significant attention, with some viewing it as the "next generation of market research tools."

"We call them generative agents and we created a little town called Smallville."

Core Architecture of Generative Agents: Memory, Reflection, and Planning [18:50]

• Memory: Agents are equipped with a "memory stream" that logs their observations. Retrieval Augmented Generation (RAG) is used to prioritize recent, important, and relevant memories for the agent's current context, preventing LLMs from getting distracted by long contexts.
• Reflection: Agents are prompted to "reflect" on their memories to develop higher-level self-perceptions, dispositions, interests, and goals, moving beyond simple episodic logs. This process involves grouping memories and reinserting reflections to build a more consistent self-identity.
• Planning: Agents plan their day in a hierarchical manner (full day, hour-by-hour, minute-by-minute). They can adapt their plans based on new observations in the environment, replanning as necessary to react to stimuli.

"The first thing we need to do to create an agent like this is we need to give them the ability to remember."

Measuring the Accuracy of Simulated Human Behavior [24:25]

• A critical question is whether these agents are not just believable but also accurate in replicating human behavior.
• Traditional methods like demographic or persona agents can lead to simplified and stereotyped behaviors.
• A more robust method involves conducting 2-hour interviews with a representative sample (e.g., 1,000 Americans) to create "digital twins" of individuals.
• These agents, with interview transcripts as their memory, then take the same surveys and experiments as the real individuals to measure replication accuracy.

"Everything I've showed so far is... shows believability, but Disney characters are believable. Cartoons are believable, but they may not be accurate."

Quantitative Results on Agent Accuracy [30:14]

• Generative agents created from rich interview data can accurately replicate attitudes and behaviors.
• Accuracy is measured by comparing agent responses to real people's responses, normalizing for the fact that individuals themselves are not perfectly consistent over time (a baseline of 1.0 means agents replicate as well as people replicate themselves two weeks later).
• Random guessing achieves a baseline of approximately 30% replication. Persona/demographic agents achieve about 70% accuracy.
• Agents derived from full interviews achieve approximately 85% accuracy on the General Social Survey, and similar high levels on other measures like the Big Five personality index.

"What we find is that these agents do accurately replicate attitudes and behavior."

Mitigating Risks and Understanding the "Ladder of Trust" [37:00]

• Risks exist, as some studies have failed to replicate, and simulations can produce significant quantitative errors (e.g., a 13% vs. 1.2% accuracy difference).
• A "ladder of trust" is proposed, moving from possibility (what might happen, no probability) to qualitative outcomes (attitudes, safer) to quantitative outcomes (histograms, where precision matters) and finally to multi-agent simulations (requiring the most trust).
• For the "possibility" rung, plausible chains of events are key. For "qualitative," accurate estimation of individual attitudes is needed. For "quantitative," actual measurement of accuracy is crucial.
• Multi-agent simulations are considered less ready for decision-making due to the difficulty in ensuring individual agent accuracy and emergent outcomes.

"The way I think about this today is think about it as a ladder where you're as you climb up the ladder, you take on more risk of issues. But it also gets more ambitious."

Applications and Future Directions [42:13]

• Look Before You Launch: These tools can help online platforms anticipate and fix policy backfires before deployment, preventing "dumpster fires."
• Training Soft Skills: Generative agents can act as "sparring partners" for practicing negotiation, conflict resolution, and other soft skills in a simulated environment, improving real-world performance.
• Business Applications: Significant opportunities exist in market research, with companies like Simile emerging from this research.
• The ultimate goal is to leverage the "what-if machine" for more informed decision-making.

"There are frontiers of this of this space... The set of tools around look before you launch."