GitLab Deleted Its Own Database. All 5 Backups Failed

January 31, 2017, 11 PM UTC. A GitLab engineer, about to sign off for the night, runs a delete command meant for a lagging replica (db2). The terminal in focus was db1: production. • He stopped it within seconds. Of ~300 GB, only ~4.5 GB was left. • GitLab had five backup/replication methods. In its own words, "none are working reliably or set up in the first place." • The nightly pg_dump used version 9.2 against a PostgreSQL 9.6 database, so it failed every night and the S3 bucket was empty. The failure emails were rejected (DMARC). Disk snapshots were never enabled for the database servers. • What saved them: an LVM snapshot he had taken by hand ~6 hours earlier, to test load balancing on staging. • Staging's disks were throttled to ~60 Mbps, so copying it back took ~18 hours, streamed live on YouTube with up to ~5,000 viewers. • Lost: ~5,000 projects, ~5,000 comments and ~700 new accounts. Git repositories and wikis survived. The lesson isn't "have backups." It's "test your restores." Sources • GitLab, GitLab.com database incident (Feb 1, 2017): https://about.gitlab.com/blog/gitlab-dot-com-database-incident/ • GitLab, Postmortem of database outage of January 31 (Feb 10, 2017): https://about.gitlab.com/blog/postmortem-of-database-outage-of-january-31/ • @gitlabstatus on X: https://x.com/gitlabstatus/status/826591961444384768 When did you last restore one of your backups? 👇 ⚡ Wiredeep — the tech you use every day, explained in under a minute. 👉 Follow for more! #gitlab #devops #programming