Fellowship of the Dice

Hey all,

As many of you know, our server's host's database server crashed on Monday. This affected all of their customers who used that database server. After the crash, they restored their latest server backup which was 7 days old. Eventually, they restored the daily backups, which we ended up using the daily backup taken Monday morning around 6am (ET).

After we recovered using the morning's database, we ended up losing about 14-16 hours worth of posts, topics, etc. Those hours just happen to include all daytime posts (which just happens to be our busy/peak hours of the day).

On Monday, we were relying on 2 different daily backups: the server host (which usually ran around 6am ET, retains 30 backups) and a forums plugin (which usually ran around 6-7pm ET, retains 10 backups). I had planned for webserver failures, with backups of source code, files, etc. but a database server failure is much, much rarer occurrence in the IT world.

So the site's admin & moderator team has implemented our own 3rd backup routine: hourly database backups. The backups are scheduled to run on the hour and we retain the last 48 (2 days) worth of hour backups on the server (in addition to the daily backups retained). We also upload the hourly backups to a DropBox account, in case of Armageddon, Ragnarok, T-virus escapes Raccoon City, or whatever.

I just wanted to inform everyone of what happened and tell you what steps we've taken to prevent another disaster like we experienced on Wormhole Monday.

thanks,
Shurijo

Excellent!

Perhaps not the place to ask this, but does this cause extra costs to the admin and host team, and if so... how would we contribute to ease up on said issue?

Olligarchy wrote:Excellent!

Perhaps not the place to ask this, but does this cause extra costs to the admin and host team, and if so... how would we contribute to ease up on said issue?

No extra cost. The DropBox account is free and we're fine with disk storage right now.

thanks,

Awesome news! Thanks, Shurijo!

Thanks for all the hard work.

Too bad that post where Tet gave me 50 xp got lost. <_< >_> yaaaah 50 xp...

Thank you Shurijo!

Thanks all.

I just wanted to let everyone know that the staff are active in trying to fix problems. A lot of times, things are happening in the background, so I like to keep everyone informed that we feel your pain and we're trying our best to address issues (both technical/server and forum improvements).

Thanks for your hard work Shurijo and that of all the other admins who do the dirty work to keep this place going.

A tip of the hat to you Shurijo and the others for all the effort you've put into this site and ensuring its maintenance.

Don't let Shurijo fool you, he does all the hard work. We just pile on with things we want to change, and get to run all the games! He deserves all the praise for any technical issues. We would not be here without him.