Database backups, data loss, and steps we've made to lessen future impacts
Forum rules
Only admins and moderators can post announcements. Members can read and reply.
Only admins and moderators can post announcements. Members can read and reply.
- Shurijo
- Admin
- Posts: 13979
- Title: Scourge of the Jedi
- User Class: Sith Lord
Database backups, data loss, and steps we've made to lessen future impacts
Hey all,
As many of you know, our server's host's database server crashed on Monday. This affected all of their customers who used that database server. After the crash, they restored their latest server backup which was 7 days old. Eventually, they restored the daily backups, which we ended up using the daily backup taken Monday morning around 6am (ET).
After we recovered using the morning's database, we ended up losing about 14-16 hours worth of posts, topics, etc. Those hours just happen to include all daytime posts (which just happens to be our busy/peak hours of the day).
On Monday, we were relying on 2 different daily backups: the server host (which usually ran around 6am ET, retains 30 backups) and a forums plugin (which usually ran around 6-7pm ET, retains 10 backups). I had planned for webserver failures, with backups of source code, files, etc. but a database server failure is much, much rarer occurrence in the IT world.
So the site's admin & moderator team has implemented our own 3rd backup routine: hourly database backups. The backups are scheduled to run on the hour and we retain the last 48 (2 days) worth of hour backups on the server (in addition to the daily backups retained). We also upload the hourly backups to a DropBox account, in case of Armageddon, Ragnarok, T-virus escapes Raccoon City, or whatever.
I just wanted to inform everyone of what happened and tell you what steps we've taken to prevent another disaster like we experienced on Wormhole Monday.
thanks,
Shurijo
As many of you know, our server's host's database server crashed on Monday. This affected all of their customers who used that database server. After the crash, they restored their latest server backup which was 7 days old. Eventually, they restored the daily backups, which we ended up using the daily backup taken Monday morning around 6am (ET).
After we recovered using the morning's database, we ended up losing about 14-16 hours worth of posts, topics, etc. Those hours just happen to include all daytime posts (which just happens to be our busy/peak hours of the day).
On Monday, we were relying on 2 different daily backups: the server host (which usually ran around 6am ET, retains 30 backups) and a forums plugin (which usually ran around 6-7pm ET, retains 10 backups). I had planned for webserver failures, with backups of source code, files, etc. but a database server failure is much, much rarer occurrence in the IT world.
So the site's admin & moderator team has implemented our own 3rd backup routine: hourly database backups. The backups are scheduled to run on the hour and we retain the last 48 (2 days) worth of hour backups on the server (in addition to the daily backups retained). We also upload the hourly backups to a DropBox account, in case of Armageddon, Ragnarok, T-virus escapes Raccoon City, or whatever.
I just wanted to inform everyone of what happened and tell you what steps we've taken to prevent another disaster like we experienced on Wormhole Monday.
thanks,
Shurijo
- Olligarchy
- Level 13 Elite
- Posts: 2056
- Location: The Cold North
- User Class: Noble
Database backups, data loss, and steps we've made to lessen future impacts
Excellent!
Perhaps not the place to ask this, but does this cause extra costs to the admin and host team, and if so... how would we contribute to ease up on said issue?
Perhaps not the place to ask this, but does this cause extra costs to the admin and host team, and if so... how would we contribute to ease up on said issue?
Reznak * Artys ** Henry ** Orin
- Shurijo
- Admin
- Posts: 13979
- Title: Scourge of the Jedi
- User Class: Sith Lord
Re: Database backups, data loss, and steps we've made to lessen future impacts
No extra cost. The DropBox account is free and we're fine with disk storage right now.Olligarchy wrote:Excellent!
Perhaps not the place to ask this, but does this cause extra costs to the admin and host team, and if so... how would we contribute to ease up on said issue?
thanks,
- Scout89
- Level 13 Elite
- Posts: 2078
- Location: Brazil
- User Class: Gadgeteer
Re: Database backups, data loss, and steps we've made to lessen future impacts
Awesome news! Thanks, Shurijo!
A.K.A. Tvrod.
- WalterPeck
- Level 12
- Posts: 1642
Re: Database backups, data loss, and steps we've made to lessen future impacts
Thanks for all the hard work.
Too bad that post where Tet gave me 50 xp got lost. <_< >_> yaaaah 50 xp...
Too bad that post where Tet gave me 50 xp got lost. <_< >_> yaaaah 50 xp...
Asad Uller - Blood for Gold, Brayden Weathers - Shattered Kingdoms, Marshal Buckler - War without End, Ragnar - Dawn of Days, Henry Mallister - Shattered Kingdoms
Italics = internal thoughts.
Put your weapon on happy and chamber a smile.
Italics = internal thoughts.
Put your weapon on happy and chamber a smile.
- Aegor
- Level 10
- Posts: 824
- Location: North Tower
- Title: Subculture of One
- User Class: Scum
Re: Database backups, data loss, and steps we've made to lessen future impacts
Thank you Shurijo!
There is a crack, a crack in everything
That's how the light gets in.
That's how the light gets in.
- Shurijo
- Admin
- Posts: 13979
- Title: Scourge of the Jedi
- User Class: Sith Lord
Re: Database backups, data loss, and steps we've made to lessen future impacts
Thanks all.
I just wanted to let everyone know that the staff are active in trying to fix problems. A lot of times, things are happening in the background, so I like to keep everyone informed that we feel your pain and we're trying our best to address issues (both technical/server and forum improvements).
I just wanted to let everyone know that the staff are active in trying to fix problems. A lot of times, things are happening in the background, so I like to keep everyone informed that we feel your pain and we're trying our best to address issues (both technical/server and forum improvements).
- Muskrat
- Level 21 Elite
- Posts: 10518
- Location: The Secret Places Beneath the Earth
- Title: Keeper of the Hidden Word
- User Class: Scholar
Re: Database backups, data loss, and steps we've made to lessen future impacts
Thanks for your hard work Shurijo and that of all the other admins who do the dirty work to keep this place going.
- Hermit
- Level 12
- Posts: 1190
- User Class: Trader
Re: Database backups, data loss, and steps we've made to lessen future impacts
A tip of the hat to you Shurijo and the others for all the effort you've put into this site and ensuring its maintenance.
''At the center of your being you have the answer; you know who you are and you know what you want.'' - Lao-Tzu
- Vardaen
- Admin
- Posts: 66394
- Location: Miskatonic University
- Title: Great Old One
- User Class: Unshackled AI
Re: Database backups, data loss, and steps we've made to lessen future impacts
Don't let Shurijo fool you, he does all the hard work. We just pile on with things we want to change, and get to run all the games! He deserves all the praise for any technical issues. We would not be here without him.
"He that breaks a thing to find out what it is has left the path of wisdom." - Gandalf
J.R.R. Tolkien, Council of Elrond, The Fellowship of the Ring
J.R.R. Tolkien, Council of Elrond, The Fellowship of the Ring