CPSC 416 W25T2 Distributed Systems

Understanding System Failures [01:10]

• Key Insight: The fundamental challenge in building distributed systems lies not in making them work under ideal conditions, but in ensuring they remain functional and consistent when failures occur.
• Key Insight: When a node fails, its state becomes unknown, making it difficult to return to a known consistent state, which is a recurring problem in distributed systems development. "> Because we don't build distributed systems for the case where everything's working, right? That's for applications developers. This is the hard part of systems. This is how do I make sure things work even after everything's gone wrong."

Capstone Project and Project Ideas [03:08]

• Key Insight: Students are encouraged to reach out to TAs or the instructor for capstone project ideas, with options ranging from simple implementations to more complex systems like a sharded key-value store using Paxos for replication.
• Key Insight: Paxos and Raft are discussed as consensus protocols, with Raft being presented as an easier-to-understand alternative to the more complex Paxos. "> If you run out of imagination, you just go, I'm too tired to actually think about this. I just want something to do that's easy. Well, um easy is a relative term."

Failure Models: Crash Failures [05:27]

• Key Insight: Failure models include fail-stop or crash failures, where a node ceases to function permanently. This is a simplified model, as real-world systems often recover.
• Key Insight: When a computer crashes, the typical response is to reboot and attempt to return it to a consistent state, highlighting the need for recovery mechanisms for lost volatile state. "> So in a fail stop, once a node fails, it's gone forever. It never comes back."

Real-World Failures and Software Bugs [08:05]

• Key Insight: A real-world bug example involved a system crashing on machines with more than 64 logical processors due to a cache with only 64 entries, illustrating how underlying system changes can break older software.
• Key Insight: Failures can occur when components a system relies on are changed, altering their contract or API without immediate detection, especially if the issue only manifests under specific, less common conditions. "> And someone had put a cache in there that only had 64 entries in it because that was the maximum number of logical processors that were available at the time."

Types of Failures and Recoverability [12:19]

• Key Insight: Omission failures involve lost messages, timing failures occur when replies are delayed, and Byzantine failures involve nodes providing incorrect information, either deliberately or inadvertently.
• Key Insight: Crash failures result in the loss of volatile state (in memory). Recovering from these requires restarting programs and handling potential inconsistencies in persistent state updates. "> Byzantine failure is a really interesting case and it's one that we will visit a couple of times. That's where something on the network is either deliberately or inadvertently giving us false information."

The Importance of Persistent State and Atomicity [16:40]

• Key Insight: Systems lose volatile state upon crashing, making it crucial to manage persistent state and ensure that operations on it are atomic, meaning they either complete fully or not at all.
• Key Insight: Even simple operations like updating multiple files can be complex to make atomic. A "recoverability layer" is needed to handle roll-forward or roll-backward operations across discrete steps. "> And if you're building a product that's sitting on top of a database, then you don't worry about it because the database worries about it for you."

Safety, Invariants, and Consistent States [19:48]

• Key Insight: In distributed systems, "safety" refers to operations working as expected, maintaining consistency, and upholding defined invariants.
• Key Insight: Invariants, like those established by assert statements, define conditions that must always be true. Violating an invariant signals a problem that requires stopping or fixing. "> An assert is an invariant. You say this must be true or this must be false. And that establishes an invariant."

ATM Withdrawal Scenario: Failures and Invariants [21:52]

• Key Insight: The ATM withdrawal scenario is used to illustrate potential failures and the invariants that should hold, such as ensuring that money is dispensed only after the account is debited and that the system can recover to a consistent state.
• Key Insight: The complexity of transactions, especially multi-party ones, drives the need for robust protocols to handle failures and ensure recovery. "> So what can possibly go wrong? So take five minutes alone together. It's fine with me. Just think about what what can leave the system in a bad state and what invariance should hold."

Multi-Party Transactions and Recovery Challenges [29:30]

• Key Insight: Multi-party transactions, where multiple actors are involved (like an ATM and a bank), significantly complicate recovery because the sequencing of operations between these actors must be managed.
• Key Insight: When a failure occurs during a multi-party transaction, recovering to a consistent state becomes more challenging, necessitating protocols like two-phase commit. "> And when it's a multi-party transaction, which is all the interesting cases here, when it's a multi-node transaction, that's when recovery becomes more interesting."

Write-Ahead Logging for Recovery [40:48]

• Key Insight: A traditional technique for ensuring recoverability is the write-ahead log (WAL), where intended operations are recorded in a log before being applied to the actual data.
• Key Insight: WAL can introduce overhead due to "write amplification" (writing to both the log and the data), but sequential writes to a log can sometimes be faster than random writes to storage. "> So, what you learn and I've built these kinds of logs. What you learn is you can actually make them run faster but you have to think about it."

Log as a Source of Truth and Testing Challenges [46:00]

• Key Insight: The log serves as a sequential, controlled source of truth that survives crashes, enabling recovery by redoing or undoing operations based on its entries.
• Key Insight: Testing recovery mechanisms is challenging because tests must simulate failure scenarios, which are less common in typical operation but critical for system robustness. "> The log ends up being a source of truth. It ends up being something we get to control that we can say, like I said, it's it's it's sequential, right?"

Distributed Lock Servers and Split-Brain Scenarios [01:06:01]

• Key Insight: Distributed lock servers serialize operations across independent nodes. Failures in these systems can lead to "split-brain" scenarios where multiple nodes believe they own the same lock, negating the lock's purpose.
• Key Insight: The failure of a primary lock server before its state is replicated to a backup can cause inconsistency, where the new primary does not know about locks granted by the old primary. "> And now what you have is you have split brain. You have client A who thinks A on the lock and client B that thinks A on the lock. And we have the classic race condition."

Turtles All The Way Down: Recursion in Systems [01:11:26]

• Key Insight: The "turtles all the way down" analogy from Terry Pratchett's work illustrates recursion, where solutions at one level (e.g., logging for a single node) are applied to subsequent levels (e.g., logging for multi-node systems).
• Key Insight: Designing robust distributed systems involves layers of solutions, each addressing failures and ensuring consistency, much like a recursive structure. "> So it's just turtles all the way down. And that's kind of the idea of recursion here, right? Which is we we did logging for a single node and now we're going to have multi-node and what we're going to do is we're going to do logging there and we're going to have logging underneath that."