Databricks CEO: AGI Is Already Here, Why Companies Are Behind, and the Superintelligence Debate
All-In Podcast
85,019 views • 2 days ago Save 53 min 9 min read
Video Summary
Databricks CEO Ali Ghodsi argues that while AI has reached a level of "artificial general intelligence" capable of transforming the planet, its widespread adoption in enterprises is hindered by four key challenges: context, security, cost, and vendor lock-in. He demonstrates how Databricks' "Genie" platform addresses these issues by building an "ontology" of a company's data, people, and processes, enabling AI agents to operate effectively and securely within an organization.
Ghodsi dismisses fears of imminent superintelligence and existential risk from AI, framing recursive self-improvement (RSI) as unlikely given current limitations in training speed, resource efficiency, and generalized accuracy. Instead, he highlights immediate risks like cybersecurity threats and the slow pace of integrating AI into legacy systems, emphasizing that true transformation will take decades, not months. Databricks aims to help organizations navigate these complexities, turning AI's potential into tangible business value and competitive advantage.
Short Highlights
- Databricks CEO Ali Ghodsi believes we've achieved "artificial general intelligence" (AGI) capable of transforming the planet.
- Enterprise AI adoption is slow due to four main challenges: context, security, cost, and vendor lock-in.
- Databricks' "Genie" platform uses an "ontology" to create a company-specific knowledge graph for effective AI integration.
- Ghodsi downplays fears of superintelligence and existential AI risk, citing a lack of evidence for recursive self-improvement (RSI).
- The primary immediate risks are cybersecurity threats and the slow integration of AI into legacy systems.
- True enterprise-wide AI transformation is a long-term endeavor, likely taking decades.
- Databricks aims to help organizations overcome these hurdles to unlock AI's business value.
Key Details
The Genesis of Databricks [00:00:00]
- Databricks was founded out of "despair" when the founders, from UC Berkeley's Amplab, found zero industry interest in applying machine learning to enterprise data.
- At the time, "AI" was largely associated with robotics, and "machine learning" was the prevailing term.
"So it was kind of out of despair that we started Databricks because we thought, if we don't start a company around this, no one else is listening to it."
Early Vision and Market Misconceptions [00:01:00]
- The initial use case for Databricks' technology was a recommendation algorithm for Netflix.
- Founders believed AI would be a much larger market than just ad tech, a view met with skepticism.
- The realization that organizations were bogged down by decades-old legacy software and complex processes became a key insight.
"And we thought, no, no, no, this is going to be a bigger market."
The Cloud as an Accelerant [00:03:00]
- The shift from on-premises data centers to cloud computing was crucial, separating storage and compute.
- Cloud enabled the illusion of infinite storage and on-demand compute, paving the way for big data and machine learning.
- Databricks bet heavily on the cloud, AI use cases, and open source, which were contrarian at the time.
"What the cloud enabled was to just completely decentralize this and separate compute and storage so that you could just imagine you have infinite storage."
The Netflix Prize and AMP Lab's Role [00:05:00]
- The Netflix Prize in 2008-2009 inspired many computer scientists and was an early example of leveraging data for prediction.
- AMP Lab (Algorithms, Machine Learning, and People) at Berkeley included human feedback in AI systems, recognizing AI's limitations.
"So we had people there as well. Putting the people in there makes sense because Berkeley is a little hippy-dippy."
Redefining Artificial General Intelligence (AGI) [00:07:00]
- Ghodsi argues that AGI has already been achieved, based on the consensus within the Amplab in 2009-2010.
- He believes humanity possesses sufficient intelligence to transform the planet if progress stopped today.
"I actually think we already have artificial general intelligence. I've been saying it since last year..."
Distinguishing AGI from Superintelligence [00:08:00]
- Superintelligence, as defined by Bostrom, refers to an AI capable of performing millennia of thought in minutes, a vastly different concept from AGI.
- Labs are focused on achieving this superintelligence, often through complex mathematical problems.
"And there he's talking about something completely different. He's talking about something that can do a millennia worth of thought."
The Enterprise AI Adoption Gap [00:10:00]
- Despite AI's capabilities, enterprises use it in rudimentary ways, like glorified Google searches or basic coding assistance.
- Ghodsi questions why organizations lag behind if AI can solve complex math problems.
"And when I go talk to organizations out there, enterprises, I see that they're just using AI in very rudimentary ways."
The Four Challenges to Enterprise AI [00:11:00]
- Context: AI needs access to all organizational data, including meetings and internal communications, which many companies resist recording due to liability.
- Security: Granting AI access to vast amounts of data creates significant cybersecurity risks and an expanded attack surface.
- Cost: Exponentially rising token costs for AI must be matched by exponentially rising revenue, a rare occurrence.
- Data Sovereignty/Lock-in: Companies risk having their proprietary data incorporated into AI models, leading to a loss of IP.
"So basically, what was missing from our research at UC Berkeley was that we didn't think about the security, the governance."
Databricks' Solution: The Ontology [00:15:00]
- Databricks' "Genie" platform builds an "ontology," a constantly evolving graph of a company's people, processes, and data.
- This ontology allows AI agents to understand context, make decisions, and operate effectively within the organization.
"The key thing here is that, so we have a product called Genie, but the key thing that Genie has, because, you know, there's so many chatbots out there."
Genie in Action: Unlocking Insights [00:17:00]
- Ghodsi demonstrates Genie analyzing employee survey data, revealing correlations and causal factors for low "action" scores.
- Genie identified specific departmental issues, like product management's dislike of decision-making complexity and engineering's issues with tools.
"And it immediately pulled up. Okay, this is the correlation between these two. And this is how it's looking."
Understanding Organizational Dynamics [00:20:00]
- Genie can analyze organizational charts, identify spans of control, and even assess the culture within different departments.
- This allows leaders to understand deviations from core company principles and identify potential issues early.
"It immediately told me, like, hey, there is an org inside Databricks, which is very deep."
Developer Productivity and Cost Efficiency [00:23:00]
- An analysis of Databricks' engineers showed an "L" shape in code commits, with top performers committing significantly more.
- This led to identifying that some engineers were using automated factories, and importantly, not always at a higher cost, suggesting efficient token usage is possible.
"Look at the middle here. It's maybe per week, roughly. But it has this L shape, you know, and you look at this L shape, you know, the top people are actually committing way more."
Reducing Human Blockers and the Blame Game [00:27:00]
- AI can help identify and reduce "human blockers" – delays caused by waiting for approvals or inter-departmental agreements.
- It also provides objective data, reducing the "blame game" by clearly showing what has or hasn't been resolved.
"So we measure that throughout the organization and we just see, you know..."
Management "Moneyball" and Identifying All-Stars [00:30:00]
- AI enables a "management moneyball" approach, identifying key contributors and decision-makers within an organization.
- This helps founders and CEOs identify their "all-stars" based on objective data rather than subjective assessment.
"We're actually getting into like management moneyball, aren't we?"
The Debate on Going Public [00:33:00]
- Ghodsi discusses the pros and cons of staying private versus going public, acknowledging the vibrant secondary market.
- Advantages of going public include access to capital, marketing, and predictable liquidity for employees, though it involves significant organizational effort.
"You get access to capital. You get much, much better marketing. You know, it's like everybody knows about you."
Competition and Peripheral Product Moves [00:37:00]
- Ghodsi acknowledges competition from companies like Snowflake and the rapid pace of AI feature development from others.
- He emphasizes the need for peripheral product moves to stay competitive as AI capabilities grow.
"Everybody seems to be looking in their peripheral vision and because you can build software fast..."
Dismissing Superintelligence Fears [00:40:00]
- Ghodsi believes the risk of superintelligence leading to existential threats (like the "paperclip maximizer" scenario) is close to zero.
- He argues that the four conditions for recursive self-improvement (RSI) – faster training, less resource use, improved accuracy, and continuous loops – are not being met.
"The idea that there's a 10% chance that 7 billion people die because of AI seems so farcical and out there..."
Real Risks: Cybersecurity and Legacy Systems [00:43:00]
- The primary immediate risks are cybersecurity threats and the slow pace of integrating AI into legacy systems.
- He contrasts the asymmetric risk of cyberattacks (attackers need one success, defenders need 100% vigilance) with the manageable nature of these threats.
"I think the cyber risk is real and really bad things can happen with that."
AI's Role in Drug Discovery and Social Good [00:46:00]
- Databricks is involved in accelerating drug discovery through models like TEDDY (Transformer Enabled Drug Discovery).
- It also powers applications for social good, such as Crisis Text Line detecting self-harm and Virtue Foundation for drone deliveries.
"Transformer enabled drug discovery is a model transformer model but instead of, you know, the next token being English it's the gene regulatory network GRN..."
The Slow Pace of Economic Transformation [00:49:00]
- Ghodsi believes that while AI's potential is immense, its impact on GDP will be slow, taking decades due to organizational inertia and the difficulty of recording meetings.
- He compares the current AI transition to the early internet, where widespread adoption and understanding took time.
"I actually think GDP will move very slowly and it's going to take decades for us to actually get through it."
The Morris Worm and Human Vulnerabilities [00:51:00]
- The Morris Worm, which took down 10% of the internet, is cited as an example of past cyber risks.
- Ultimately, Ghodsi suggests that human error and vulnerabilities remain a significant factor, even with advanced AI.
"There was a kid called Robert Morris who took down 10% of the internet with the worm it's called the Morris Worm."
Conclusion and Future Outlook [00:53:00]
- Ghodsi reiterates that Databricks is focused on solving the context gap in enterprises and believes this will take 10-15 years.
- He emphasizes that AI will make the world a better place, while acknowledging and addressing the associated risks.
"So we see these use cases all the time with AI and it is going to make the world a better place."