Why Netflix, Instagram, and Twitter Pick Different Databases

Netflix's High-Volume Write Throughput with Cassandra [00:56]

• Netflix generates a massive volume of data from user interactions, leading to over 3 million writes per second.
• They chose Apache Cassandra primarily for its ability to handle this extreme write throughput at horizontal scale.
• Cassandra functions as a distributed hashmap, where writes are hashed by a partition key and routed to responsible nodes.
• Writes go to an in-memory structure (memtable) and receive an acknowledgment without immediate coordination with other nodes, enabling rapid write processing.
• The trade-off for this write performance is limited read flexibility; Cassandra lacks joins and ad hoc query capabilities.
• Netflix compensates by modeling data around queries, creating multiple tables for the same data organized differently to support specific lookup needs.
• Aggressive denormalization is a strategy in Cassandra to ensure queries are simple partition key lookups, contrasting with relational databases where data duplication is avoided.

Cassandra is built for right throughput at horizontal scale.

Instagram's Relational Scaling with PostgreSQL [03:31]

• Instagram's core functionalities, such as feeds, profiles, and comments, involve highly relational queries.
• They successfully ran their entire backend on PostgreSQL for years, supporting over a billion users.
• This choice was made because their workload is read-heavy, and PostgreSQL excels at complex queries like joins, aggregations, and sorted pagination.
• Using Cassandra would have required extensive pre-computation and denormalization for every potential query pattern, a less efficient approach for their access needs.
• To scale PostgreSQL, Instagram employed connection pooling (with PG bouncer), read replicas, partitioning, and smart indexing.
• This demonstrates that robust engineering can extend a database's capabilities beyond its intended limits.

You can stretch any database far beyond its intended limits with good engineering.

Twitter's Instant Timeline Delivery with Redis [06:09]

• Twitter's primary challenge is delivering timelines instantly, requiring single-digit millisecond response times.
• They use Redis as a cache, not a primary database, serving timelines at over 300,000 requests per second per node.
• Timelines are pre-computed using a "fan out on write" strategy: when a user tweets, it's immediately pushed into the Redis timeline cache of all followers.
• Redis's in-memory operation, lacking disk I/O, query parsing, or index lookups for this purpose, allows for microsecond read times.
• Crucially, Redis is never treated as the source of truth; actual tweets are stored in a durable database, with data replicated to Redis.
• This hybrid approach of writing to a durable store first and then replicating to a cache is a common pattern to prevent data loss.
• For high-follower accounts (celebrities), Twitter uses a hybrid "fan out on read" to manage the immense cost of fanning out a single tweet to millions of followers.

Twitter never treats Redis as the source of truth. Redis is a cache.

The Framework for Database Selection [08:54]

• Choosing the right database hinges on three core questions: access pattern, willingness to sacrifice, and necessity.
• Access patterns dictate the choice: flexible queries favor relational databases (PostgreSQL), massive write throughput favors NoSQL (Cassandra), and ultra-low latency reads favor caching layers (Redis).
• Every database involves trade-offs: PostgreSQL offers flexibility but struggles with write throughput; Cassandra scales writes but lacks query flexibility; Redis offers speed but sacrifices durability and capacity.
• The most critical question is necessity; most systems can perform well with a well-indexed PostgreSQL, and complex databases should only be adopted when the access pattern and scale genuinely demand it.

It comes down to three questions. First, what's your primary access pattern? Second, what are you willing to sacrifice? Third, do you actually need it?