Database backups, data loss, and steps we've made to lessen future impacts

News, information, and announcements are posted here.
Forum rules
Only admins and moderators can post announcements. Members can read and reply.
Post Reply
User avatar
Shurijo
Admin
Admin
Posts: 13979
Title: Scourge of the Jedi
User Class: Sith Lord

Database backups, data loss, and steps we've made to lessen future impacts

Post by Shurijo » Wed Aug 05, 2015 2:09 pm

Hey all,

As many of you know, our server's host's database server crashed on Monday. This affected all of their customers who used that database server. After the crash, they restored their latest server backup which was 7 days old. Eventually, they restored the daily backups, which we ended up using the daily backup taken Monday morning around 6am (ET).

After we recovered using the morning's database, we ended up losing about 14-16 hours worth of posts, topics, etc. Those hours just happen to include all daytime posts (which just happens to be our busy/peak hours of the day).

On Monday, we were relying on 2 different daily backups: the server host (which usually ran around 6am ET, retains 30 backups) and a forums plugin (which usually ran around 6-7pm ET, retains 10 backups). I had planned for webserver failures, with backups of source code, files, etc. but a database server failure is much, much rarer occurrence in the IT world.

So the site's admin & moderator team has implemented our own 3rd backup routine: hourly database backups. The backups are scheduled to run on the hour and we retain the last 48 (2 days) worth of hour backups on the server (in addition to the daily backups retained). We also upload the hourly backups to a DropBox account, in case of Armageddon, Ragnarok, T-virus escapes Raccoon City, or whatever.

I just wanted to inform everyone of what happened and tell you what steps we've taken to prevent another disaster like we experienced on Wormhole Monday.

thanks,
Shurijo

User avatar
Olligarchy
Level 13 Elite
Level 13 Elite
Posts: 2056
Location: The Cold North
User Class: Noble

Database backups, data loss, and steps we've made to lessen future impacts

Post by Olligarchy » Wed Aug 05, 2015 2:13 pm

Excellent!

Perhaps not the place to ask this, but does this cause extra costs to the admin and host team, and if so... how would we contribute to ease up on said issue?
Reznak * Artys ** Henry ** Orin
Image

User avatar
Shurijo
Admin
Admin
Posts: 13979
Title: Scourge of the Jedi
User Class: Sith Lord

Re: Database backups, data loss, and steps we've made to lessen future impacts

Post by Shurijo » Wed Aug 05, 2015 2:15 pm

Olligarchy wrote:Excellent!

Perhaps not the place to ask this, but does this cause extra costs to the admin and host team, and if so... how would we contribute to ease up on said issue?
No extra cost. The DropBox account is free and we're fine with disk storage right now.

thanks,

User avatar
Scout89
Level 13 Elite
Level 13 Elite
Posts: 2078
Location: Brazil
User Class: Gadgeteer

Re: Database backups, data loss, and steps we've made to lessen future impacts

Post by Scout89 » Wed Aug 05, 2015 3:07 pm

Awesome news! Thanks, Shurijo!
A.K.A. Tvrod.

User avatar
WalterPeck
Level 12
Level 12
Posts: 1642

Re: Database backups, data loss, and steps we've made to lessen future impacts

Post by WalterPeck » Wed Aug 05, 2015 3:21 pm

Thanks for all the hard work.

Too bad that post where Tet gave me 50 xp got lost. <_< >_> yaaaah 50 xp...
Asad Uller - Blood for Gold, Brayden Weathers - Shattered Kingdoms, Marshal Buckler - War without End, Ragnar - Dawn of Days, Henry Mallister - Shattered Kingdoms

Italics = internal thoughts.

Put your weapon on happy and chamber a smile.

User avatar
Aegor
Level 10
Level 10
Posts: 824
Location: North Tower
Title: Subculture of One
User Class: Scum

Re: Database backups, data loss, and steps we've made to lessen future impacts

Post by Aegor » Wed Aug 05, 2015 3:34 pm

Thank you Shurijo!
There is a crack, a crack in everything
That's how the light gets in.

User avatar
Shurijo
Admin
Admin
Posts: 13979
Title: Scourge of the Jedi
User Class: Sith Lord

Re: Database backups, data loss, and steps we've made to lessen future impacts

Post by Shurijo » Wed Aug 05, 2015 3:34 pm

Thanks all.

I just wanted to let everyone know that the staff are active in trying to fix problems. A lot of times, things are happening in the background, so I like to keep everyone informed that we feel your pain and we're trying our best to address issues (both technical/server and forum improvements).

User avatar
Muskrat
Level 21 Elite
Level 21 Elite
Posts: 10513
Location: The Secret Places Beneath the Earth
Title: Keeper of the Hidden Word
User Class: Scholar

Re: Database backups, data loss, and steps we've made to lessen future impacts

Post by Muskrat » Wed Aug 05, 2015 4:38 pm

Thanks for your hard work Shurijo and that of all the other admins who do the dirty work to keep this place going.

User avatar
Hermit
Level 12
Level 12
Posts: 1190
User Class: Trader

Re: Database backups, data loss, and steps we've made to lessen future impacts

Post by Hermit » Wed Aug 05, 2015 5:06 pm

A tip of the hat to you Shurijo and the others for all the effort you've put into this site and ensuring its maintenance.
''At the center of your being you have the answer; you know who you are and you know what you want.'' - Lao-Tzu

User avatar
Vardaen
Admin
Admin
Posts: 66394
Location: Miskatonic University
Title: Great Old One
User Class: Unshackled AI

Re: Database backups, data loss, and steps we've made to lessen future impacts

Post by Vardaen » Wed Aug 05, 2015 5:16 pm

Don't let Shurijo fool you, he does all the hard work. We just pile on with things we want to change, and get to run all the games! He deserves all the praise for any technical issues. We would not be here without him.
"He that breaks a thing to find out what it is has left the path of wisdom." - Gandalf
J.R.R. Tolkien, Council of Elrond, The Fellowship of the Ring

Post Reply