How Googlebot Crawls the Web

Google Search Central

5,793 views • 1 year ago Save 25 min 7 min read

Video Summary

Crawlers, fundamental to search engines since their inception, are essentially HTTP clients that fetch data from the network. Early crawlers, akin to simple browser functions or command-line tools like curl and wget, could recursively follow links to discover and download web pages. While the internet was much smaller in the 1990s, enabling full crawls from popular pages, modern crawling faces challenges due to the web's vast growth and automatic traffic. Responsible crawling now involves respecting robots.txt, monitoring host health, and adjusting crawl rates to avoid overwhelming servers.

As Google expanded beyond its core search engine, new products like Adwords and AdSense, and even features within Gmail, required fetching data from the web. Initially, all these requests might have used a generic Googlebot user agent, leading to confusion about the origin of the traffic. To manage this complexity and maintain consistent behavior across different services, a unified system was developed, allowing various crawlers to fetch data while specifying their own user agent strings. This ensures a more organized and predictable approach to interacting with websites.

The evolution of HTTP protocols, from HTTP/1.1 to HTTP/2, has improved crawling efficiency through features like streaming. However, the core concept of crawling remains, with well-behaved crawlers generally adhering to robots.txt and respecting server load signals. The future of crawling is likely to see increased automatic traffic from AI agents, but the web's architecture is designed to handle such loads, with ongoing technological advancements addressing potential congestion issues. Projects like Common Crawl offer alternative data sets, allowing users to consume crawled information without needing to perform the crawling themselves, further supporting efficient data access.

Short Highlights

Crawlers are essentially HTTP clients that request and fetch data from the network.
Early crawling methods involved simple scripts or tools that recursively followed links.
Modern crawling requires respecting robots.txt, managing host load, and adjusting crawl rates.
Google introduced multiple crawlers for different products, eventually consolidating them under a unified system with distinct user agent strings.
Advancements in HTTP protocols like HTTP/2 enable more efficient crawling through features like streaming.

Key Details

The Nature of Crawlers [1:51]

Crawlers have been around as long as search engines and are essential for requesting data from the network.
Technically, crawlers are HTTP clients, similar to a browser but often simpler, designed primarily for fetching data.
Tools like curl and wget can function as basic crawlers, capable of fetching content and even recursively exploring URLs with depth limits.
Early internet pioneers, like Sergey Brin, theorized that following links from popular pages could theoretically lead to crawling the entire internet.

Technically crawlers are just HTTP clients, right? Like much like your browser, which is also more advanced I guess HTTP client because it can do more things than just fetching data from over the network.

The fundamental purpose of a crawler is to request and retrieve data from the internet, a function as old as search engines themselves. While basic, these tools are capable of exploring vast networks of information by following links, a process that has evolved significantly with the internet's growth.

Early Internet and Crawling Challenges [4:36]

In the mid to late 1990s, the internet was significantly smaller and easier to crawl.
Following links from a popular page was a viable strategy for discovering and indexing web content.
The simplicity of early websites meant that fetching large amounts of data was less resource-intensive.
However, bandwidth was considerably more expensive, requiring systems to manage and monitor its usage.
Early search engines like "Worldwide web warm" (dubdubdub) in 1994 and "WebCroller" indexed millions of pages, demonstrating the early efforts in large-scale data ingestion.

If you take a very popular page again, this was like mid '90s or end of '90s or something like that and if you take a very popular page like the homepage of CNN or Wall Street Journal, Fox Fox News or whatever and you can just follow the links where follow means that once you found the a URL on a page, you will just fetch it again.

The early web, though simpler, presented unique challenges regarding bandwidth costs and managing server load, necessitating careful consideration of how crawlers interacted with websites.

Google's Crawler Evolution and Product Integration [13:45]

Google's initial crawler, used for its search engine, likely started with simple fetching mechanisms, potentially named "Googlebot" around 1999.
Support for robots.txt was present from the early days of Google, allowing site owners to opt out of crawling.
With the introduction of new products like Adwords (early 2000s) and AdSense (2003), and features like Gmail (2005-2006) which proxied image fetches, more services began needing to ingest data from the web.
For a period, all fetching requests might have been attributed to "Googlebot," making it difficult to distinguish the origin of specific fetches.
This lack of distinction led to the introduction of new, distinct crawlers for various products to better manage and identify traffic.

But then we started having multiple or new products, right? Um like we had Adwords coming out in early 2000s. Um and then AdSense 2003 I thinkish and then you also have some kind of fetching in Gmail which is a 2005 2006 thing.

The expansion of Google's product ecosystem necessitated a more sophisticated approach to web fetching, moving from a single crawler to a system of specialized agents.

Managing Crawl Traffic and User Experience [16:54]

The proliferation of independent crawlers written by different engineers presented maintenance challenges and potential inconsistencies in behavior regarding robots.txt and host load.
To address this, a unified system was implemented where crawlers could fetch from the internet but were required to specify their own user agent string.
This consolidated approach ensures that all crawlers behave consistently, respecting web standards and site owner preferences.
However, a distinction is made for user-initiated actions, such as fetching data for a spreadsheet, where ignoring robots.txt might be appropriate as it's not a robotic fetch.
User-initiated fetches aim for immediate retrieval, contrasting with the batched processing of crawler requests, which can sometimes lead to significant waiting times.

So we introduce new crawlers. Um but that would also mean that with all the engineers, software engineers that we have and computer scientists, every now and then someone would came up with a brilliant idea that oh I will just write my own crawler because I need my own user agent.

The development of distinct crawlers and a unified system reflects an effort to balance efficient data ingestion with responsible web interaction and a positive user experience.

The Future of Crawling and Web Congestion [23:22]

The evolution of HTTP protocols, such as HTTP/2, has enhanced crawling efficiency by enabling features like streaming and multiplexing connections.
While technical protocols change, the core crawling process remains largely the same.
Crawler policies can vary significantly between different companies, with some publishers reporting issues with specific crawlers.
Well-behaved crawlers generally honor robots.txt, respect server load signals, and adjust their behavior accordingly.
Adversarial crawlers, like malware scanners, may require different policies as their purpose is to detect, not necessarily be hidden from, potentially malicious content.

But technically crawling doesn't actually change. Okay. Um and then how different companies policy their or set policies for their crawlers that of course differs greatly.

The future of crawling involves adapting to new technologies and managing the increasing volume of automated traffic, with a focus on maintaining a healthy and accessible web for all.