During the day I am an ops sys-admin. During the night I am a husband, father of two, and a CEO of a bootstrapped start-up. After launch, my first project was to schedule regular backups of user data and archive off-site. My goal was to create backups but never need them. Boy was I lucky ...
Yes, leave it to me to inadvertently delete the VPS root disk. One of the major cloud providers places the "rename" and "remove" disk buttons right next to each other and I learned a nasty habit of clearing pop-ups without reading them (thanks Windows).
"Honey! I just deleted LinkPeek.com"
The horror... My stomach felt like I took a tumble in a roller-coaster. Instantly I tossed off my developer hat and put on my operations hat. I checked the off-site backups. I had nightly dumps of MongoDB and weekly tar backups of the /etc partition. The user data was in MongoDB and most of the system configuration information was in the tar. I used the tar to recover 2 upstart scripts, 2 supervisord scripts, 2 complex nginx confs, an ssl cert, and the pyramid production.ini.
I set out to stand up a new server, re-install the needed packages, recover the user data, and restore the service. After 1.5 hours of feverish typing, https://linkpeek.com/ was back online.
What I learned and my plan going forward
If you are a small team or a start-up, you must have somebody dedicated to operations. Without backups I would not have been able to gracefully recover. Most likely I would have reimbursed the existing members and shuttered the doors.
This experience was eye-opening. In my next couple of posts I will explain how I create and maintain backups and my next project will implement a configuration management and provisioning system.
This system will allow me to:
- take out the human element of recovery
- significantly reduce the time-to-recover from a catastrophic failure
- test disaster recovery procedures before needing them
- provision development and production environments without effort
- have a reproducible blueprint of "how to build a LinkPeek server"