A Kafka log may already contain the latest value for every key. Yet whenever we simply want to ask, “give me the value for user-42,” we often end up copying that data somewhere else designed for fast reads.
Marcos Sánchez decided this duplication annoyed him enough to turn it into software.
His project is called rpkv. It rests on an almost obvious idea: don’t copy the value. Keep only the location where it lives.12
In its Pebble index, a key points to a (partition, offset) pair. When a client requests a value, rpkv checks this small index, then fetches the exact record from Redpanda through the Kafka protocol. The log remains the storage layer. The index is only a map.2
This architecture is less interesting as a new key-value database than as an experiment with a trade-off that has become almost invisible: we duplicate huge amounts of data to buy read speed.
The log knows how to preserve history, not answer like a dictionary
Kafka is built around an ordered log. Producers append records to partitions, and consumers read through them by offset. A compacted topic adds a very useful property: for a given key, Kafka can remove older values and retain the latest one.4
That already looks a lot like a key-value table.
Except the access model is different.
A log is very good at saying, “read from this offset.” It is not designed as a query engine that immediately turns user-42 into “partition 3, offset 918272.” To get that property, systems generally build derived state.
Kafka Streams does this explicitly. Its documentation recommends a persistent KeyValueStore based on RocksDB for most use cases. This state lives locally on disk and can be rebuilt from a compacted changelog.3
That choice is perfectly reasonable. A local copy can answer requests without asking the broker for every value. It can also support richer operations and isolate application reads from the network path.
But it creates an amusing situation: the log contains the value, then the state store contains the value again, because the first representation is not convenient enough for the second job.
Sánchez puts the problem more personally: his homelab already has the disks it has, and he does not want to buy more just to store the same bytes a second time.1
rpkv takes the opposite direction.
Twelve bytes of pointer, then the real bill
The index’s logical format is tiny: a 32-bit partition and a 64-bit offset. Naturally, the key, Pebble’s storage, its metadata, log files and internal structures add to that.2
In the benchmark published with the project, a closed index containing one million keys takes an average of 14.2 bytes per key on disk in that run. The footprint includes Pebble’s SSTables, WAL and manifests. The document specifies that this is a single run and that Pebble’s compaction state can change the result.2
So this figure is not a physical constant of rpkv. It gives a sense of the scale of the idea: keeping pointers costs far less than duplicating large values.
Imagine objects measuring a few hundred kilobytes, metadata blobs, documents or reasonably heavy serialized state. The larger the value, the more interesting the gap becomes. A pointer does not grow with it.
But the bill has not disappeared. It has moved to another column.
Every read must now go back to the broker.
The author’s benchmark, run against a local Dockerized Redpanda instance with 5000 random reads among 100,000 keys containing 256-byte values, records 11.1 ms at p50 and 16.9 ms at p99.2
Eleven milliseconds may be irrelevant for an administration API and disastrous for a path called tens of thousands of times per second. The README does not try to hide the problem either: small values and very hot reads are explicitly described as poor use cases.2
That is what makes the project worth examining. It does not promise to eliminate a cost. It chooses which one to pay.
A disposable index changes how you think about failure
The second interesting idea is that the index is not treated as a new source of truth.
Its directory can be deleted and rebuilt from the log. The project reports roughly 996,000 keys per second in its rebuild benchmark.2
Again, that number belongs to the author’s test setup, not to a universal law. The result will change with another broker, another disk, different message sizes or remote storage.
The conceptual property matters more: the index is a disposable projection.
That removes an entire class of questions. There is no value-retention policy to synchronize between the log and a second database. There is no replication mechanism specific to rpkv. Each instance consumes the topics, maintains its private Pebble store and can be rebuilt.2
This simplicity immediately creates another limitation. Two replicas may be at different checkpoints. The project therefore guarantees neither read-your-writes nor monotonic reads when a client moves from one instance to another. Freshness depends on each replica’s ingestion lag, exposed as a metric.2
You save on specialized replication by accepting more modest consistency.
That is a constant throughout the project: every piece of infrastructure removed reappears as an explicit trade-off.
Compaction makes the pointer more subtle than it looks
There is a trap in the idea of “key → offset.”
Kafka can compact the log. An older value for a key disappears while the newer one remains at its original logical offset. Kafka’s documentation describes compaction as a policy that retains the latest value for each key.4
Now suppose the rpkv index still points to the old version at the exact moment the broker has already compacted it.
A fetch at that offset must not be interpreted as “this is necessarily my value.” The protocol may return the next available record instead. rpkv therefore checks both the offset and the key of what comes back.2
If the pointer is stale, the server waits for its ingestion checkpoint to pass the relevant offset, resolves the key again, then retries the read within a total budget of two seconds. If the situation is not resolved, it returns 503 rather than silently serving the wrong value.2
Retention creates a different case. If the pointed-to record is genuinely gone because the delete policy has moved the start of the log beyond that offset, rpkv returns 410 Gone.2
This detail matters because it breaks the project’s prettier version. The index is not just twelve bytes and a GET. Much of the work lies precisely in proving that the pointer remains interpretable while the underlying storage changes.
For that reason, the repository includes an integration suite that generates thousands of overwrites on an aggressively compacted topic, forces several compactions and restarts rpkv on the same index before checking values and tombstones through its HTTP API.2
This is still far from a production track record. But at least the prototype tackles its most unpleasant problem instead of putting it in a “future work” section.
Why materialization is still often the right answer
It would be easy to turn this story into a small moral against RocksDB, caches and “an industry that wastes storage.”
That would miss the point.
Materialization exists because it buys useful properties. A local copy avoids the round trip to the broker on the read path. It shields that path from some of the Kafka cluster’s load and incidents. It can provide different indexes, support range scans, maintain aggregates or serve queries that the raw log does not offer.3
Storage is not waste when it pays for a feature you actually need.
The problem appears when the decision becomes automatic.
You see a log, so you add a consumer. The consumer writes to a database. Then you add a cache in front of the database. Then another copy for analytics. Each layer is locally defensible, but no one always goes back to ask whether the value really needed to be materialized there.
rpkv is interesting because it asks the question in reverse: what is the smallest additional structure that makes the existing storage queryable enough?
In this case, the answer is an address index and acceptance of a network fetch.
In another system, it might be a Bloom filter, a metadata table, a secondary index or simply nothing at all because the read frequency does not justify another database.
Hardware costs put old architectures back on the table
Sánchez’s post starts with a very down-to-earth constraint: he does not want to buy more disk.1
That constraint can seem almost petty amid architecture discussions that casually assume storage is cheap. But interesting systems often begin this way. A resource stops feeling free to a developer, and choices that had faded into the background become visible again.
Redpanda adds another dimension with tiered storage: the log can keep some data remotely while continuing to expose offsets through the Kafka interface.5 rpkv has even added a benchmark in which local segments are evicted and the data goes through object storage. The author reports roughly 10.6 ms p50 when cold in his local MinIO environment, a result counterintuitive enough that it mainly deserves to be reproduced elsewhere before anyone draws a conclusion from it.2
But the idea raises a broader question: if the reference system already knows how to move its bytes across several storage tiers, how many application-level copies are we still adding simply because our read interface is not the one we want?
The right unit may not be the terabyte saved
rpkv has only just come out. To our knowledge, it does not yet have the operational track record needed to say that this architecture is a good production idea. Its figures come from the author, in his scenarios, and its ideal domain is narrow: large values, modest read volumes and an environment where disk genuinely matters.2
That is still enough to make the experiment useful.
Because its real unit of work is not storage. It is duplication.
Every time a system transforms one representation into another, it pays something: space, latency, compute, operational complexity, consistency or availability. Materialization shifts the slider toward space. rpkv pushes it toward read latency and dependence on the broker.
There is no free version.
But there is a difference between paying consciously and stacking copies because that has become the default recipe.
For now, Sánchez’s project comes down to a rather healthy gesture: before buying another disk, check whether you really needed to copy what was already on it.