Distributed Key-Value Store with Raft

Why build Raft from scratch

Consensus is the part of a distributed system that’s simple to state and notoriously hard to get right: get a cluster of unreliable machines to agree on an ordered log, even as nodes crash and the network drops, delays, and reorders messages. Raft is the algorithm that made this tractable to understand — it was designed explicitly for understandability over the more opaque Paxos — so it’s the natural one to implement end to end rather than just read about.

This project is a from-scratch Raft implementation in Go, with a replicated key-value store built on top of it.

Architecture

The system is layered: clients talk to a thin client library, which talks to a key-value state machine, which sits on top of the Raft module that does the actual replication.

Clients → Clerk (retry + request IDs)
              │ RPC
        KV Server (state machine) ←── apply channel ─── Raft module
                                                         │  ├─ Election (terms, votes, timeouts)
                                                         │  ├─ Log replication (AppendEntries)
                                                         │  └─ Persistence (term / vote / log to disk)
                                                    Peers ↔ RPC

Raft exposes a small surface: start a command, get committed entries back on an apply channel, and keeps every hard invariant behind that boundary.

The core: election, replication, persistence

Three subsystems make up the consensus core. Leader election uses randomized timeouts and term numbers so that exactly one leader emerges per term and a new one is chosen when the old one fails. Log replication propagates entries via AppendEntries with a consistency check that detects and rolls back conflicting entries, advancing the commit index only once an entry is safely replicated on a majority. Persistence writes the term, vote, and log to stable storage so a node can crash and rejoin without violating safety.

The key is to follow the spec pretty much line-by-line and word-by-word, since the overwhelming majority of consensus bugs are minor deviations from it rather than novel problems (which are especially hard to debug).

Testing: why “it passed” doesn’t count

In distributed systems a test that passes once has told you almost nothing, the interesting bugs surface 1-in-many runs under adversarial timing. The core is validated against the MIT 6.5840 fault-injection test suite, which injects partitions, crashes, and RPC delays and drops, run in a loop rather than once so that rare failures actually get a chance to show up.

Two invariants are asserted throughout: at most one leader per term, and committed entries are never lost. Structured logging with node IDs, terms, and log indices makes the interleaved-log debugging that consensus work demands actually feasible, reading interwoven per-node logs is most of the work in getting Raft right.

What’s next

Work in progress, in order:

• A linearizable KV layer — puts/gets/appends with client-side retry and duplicate-request suppression, so operations survive a leader crash mid-request without being applied twice.
• Measured optimizations, each benchmarked before/after against a correctness-only baseline: request batching and pipelining for throughput, and leader leases for local reads without a quorum round-trip.

The goal for the optimization layer is a writeup of its own: what each production Raft optimization is individually worth, measured — numbers to come once the KV layer is solid.

Status

Consensus core (election, replication, persistence) implemented and validated against the fault-injection suite under looped runs. KV layer and optimization benchmarks in progress.