Tailscale had used SQLite as the primary database behind its coordination-server shards since 2022. It was supposed to be boring technology in the best sense: known, local, and accessed by one Go process per shard.1
Then databases started corrupting.
Across six months, Tailscale says it dealt with 19 separate corruption incidents. Repairs could take a shard’s control plane offline. The databases did not contain private encryption keys or network traffic, but the outages were real and repeatedly damaged reliability.1
The interesting part is not that old software contained a bug. It is how Tailscale and SQLite’s maintainers ended up building the instruments required to observe a race condition that refused to appear on demand.
The bug had no useful pattern
Incidents did not correlate with a shard, customer, feature, time of day, or load. Sometimes they arrived hours apart. At one point nothing happened for six weeks.1
Tailscale reviewed its low-level SQLite code, added diagnostics, and bought professional support from SQLite’s developers. The teams systematically tested theories around POSIX locks, memory ownership, and threading.1
Meanwhile the service still had to run. Shards were changed to stop immediately after detecting corruption. Backups were continuously checked with PRAGMA integrity_check, and recovery procedures were tightened.1
The breakthrough came from a tool initially built for recovery rather than diagnosis.
Transaction replay became a microscope
Tailscale began logging every SQL statement that modified a database. Because each shard had a single writer, that history was linear enough to replay against the latest good backup.1
It reduced the risk of losing recent changes. It also exposed something stranger: in two incidents, data committed by one transaction was missing from later transactions.1
Attention shifted to WAL checkpointing.
In write-ahead logging mode, SQLite writes changed pages into a WAL file before checkpoints copy them back into the main database. Tailscale manually controlled checkpointing to make fast, consistent backups.1
SQLite now documents the failure as the WAL-reset bug. It can occur when a database in WAL mode has multiple connections in separate threads or processes and a write collides with a checkpoint in a very narrow timing window.2
Tailscale’s unusual, aggressive checkpointing made that narrow path easier to hit.
SQLite itself needed new instrumentation
SQLite’s developers created a virtual-filesystem tracing shim called tmstmpvfs. Tailscale deployed it in production and waited for the next corruption.1
The trace finally exposed a race between a checkpoint and a write transaction. In the rare sequence, the checkpoint could believe pages had already moved from the WAL into the main file when they had not. Those pages could be lost and the database left inconsistent.1
SQLite says the bug was probably present from 3.7.0, released in July 2010, through 3.51.2. It was fixed in 3.51.3 on March 13, 2026.23
Nearly sixteen years in SQLite, largely invisible because ordinary workloads almost never entered the required timing window.
Even the first fix produced a false ending
Tailscale initially rolled out SQLite 3.52.0 with the fix. Its backup monitor then reported corruption in thirteen databases.1
Those files were not actually corrupt. A separate behavior change had made expression indexes stale and caused false integrity warnings. SQLite withdrew 3.52.0 and shipped 3.51.3 with the WAL-reset fix.13
The official release history confirms both the withdrawal and the 3.51.3 correction.3
The lesson is not “don’t use SQLite”
SQLite explicitly describes the WAL-reset bug as rare. It requires WAL mode, multiple connections to the same file, and a very specific collision between writing and checkpointing. The maintainers had to add test logic that deliberately forced the circumstances to reproduce it reliably.2
Tailscale’s more useful conclusion is that boring technology becomes less boring when you leave the well-travelled operating path, even if the configuration is documented and supported.1
What is worth copying is the escalation of instrumentation.
Normal logs were not enough, so they added integrity monitoring. Backups did not explain the corruption, so they built transaction replay. Replay exposed an impossible state, so SQLite’s maintainers instrumented the VFS.
At each stage, a recovery mechanism became an observation mechanism.
When a failure refuses to reproduce, the next move may not be another synthetic test. It may be building the apparatus that can catch the real failure next time it passes through.