Tailscale engineers spent months chasing database corruption that knocked out shards and forced manual recoveries. Their persistence led to the identification of a rare data race in SQLite’s Write-Ahead Logging mechanism. The issue had lingered undetected since 2010.
Customers noticed the effects through status-page alerts and occasional lost configuration changes. Metadata only, never keys or traffic. Recovery times dropped from over an hour to under one as automation improved. Still, repeated incidents tested trust in a service built for reliability.
The control plane runs SQLite per shard in single-writer mode with WAL enabled for performance. Backups snapshot the full file every few minutes to S3. Integrity checks on those backups first flagged corruption last August. Nineteen cases followed over six months. Engineers ruled out obvious suspects through telemetry and expert consultation. No code changes explained the pattern. Load, time of day, and shard location showed no correlation. They added transaction logging to replay changes against clean backups. That pipeline revealed writes vanishing without error in two incidents.
SQLite developers joined the effort under a support contract. They built a custom virtual filesystem shim called tmstmpvfs to trace checkpoint activity. Deployment caught the next corruption event quickly. Logs showed the checkpoint process miscounting pages after a concurrent write reset the WAL. Pages never reached the main database file. Referenced structures like indexes pointed to missing data. Corruption followed.
The root cause traced to a narrow timing window between checkpoint and transaction commit. The fix adds a single check in the checkpoint routine to detect WAL resets. It landed in SQLite 3.51.3 on March 13, 2026. Backports cover select earlier releases. The bug affected versions back to 3.7.0. Official details appear in the SQLite WAL documentation.
Broader analysis followed. Canonical’s June 2026 post models the race with TLA+ to confirm dqlite avoids the same flaw. Discussions on Hacker News and Lobsters highlighted how single-writer setups and manual checkpointing made the condition visible at Tailscale’s scale. Most deployments never trigger it. The episode shows even mature, widely deployed components can harbor edge-case defects that surface only under sustained production load. Tailscale has applied the update and restored steady uptime. Their detailed account credits direct collaboration with SQLite maintainers for the breakthrough.