2/14-2/15 outage

This forum is for descriptions of new site features and information about the site in general. This is also the place to discuss new site ideas.
Locked
User avatar
Brennor
Emeritus Admin
Emeritus Admin
Posts: 9161

2/14-2/15 outage

Post by Brennor » Tue Feb 17, 2015 11:43 pm

Hey guys,

Investigated the outage and figured out what happened. The issue was caused by database log files filling up their partition, mostly because of a big attack on the site. When the nightly log rotation happened, there was no free space on the logging partition, so the DB shut down safely so as to not lose any data.

This was fixed on 2/15 by the log rotations when they gzipped the old (big) log file freeing up a bunch of space.

I've doubled the quota of the log location and have added archiving so that older log files are moved to a storage location and won't count against the log file quota.
[url=telnet://brennor.dyndns.org:1896]Akarian Dawn MUSH[/url]
SRGC v0.22: SR2+++ SR3++ h+ b+++ UB+ IE+ RN W+ dk+ ri+ ad+ m+ gm+ M- P+

User avatar
TetNak
Emeritus Admin
Emeritus Admin
Posts: 46375
Location: Seagard
User Class: Brewmaster

Re: 2/14-2/15 outage

Post by TetNak » Tue Feb 17, 2015 11:46 pm

Thank you B!
"Kings have no friends, only subjects and enemies."

- King Stannis Baratheon, First of His Name

User avatar
Brennor
Emeritus Admin
Emeritus Admin
Posts: 9161

Re: 2/14-2/15 outage

Post by Brennor » Tue Feb 17, 2015 11:58 pm

I should probably also add as an aside that the big attack ended up failing to break in, though it did cause a denial of service because of the DB shutdown.
[url=telnet://brennor.dyndns.org:1896]Akarian Dawn MUSH[/url]
SRGC v0.22: SR2+++ SR3++ h+ b+++ UB+ IE+ RN W+ dk+ ri+ ad+ m+ gm+ M- P+

User avatar
Trogdor
Emeritus Admin
Emeritus Admin
Posts: 27260
Title: The Burninator
User Class: Jedi Master

Re: 2/14-2/15 outage

Post by Trogdor » Wed Feb 18, 2015 12:00 am

Good job!

User avatar
Vardaen
Admin
Admin
Posts: 66394
Location: Miskatonic University
Title: Great Old One
User Class: Unshackled AI

Re: 2/14-2/15 outage

Post by Vardaen » Wed Feb 18, 2015 12:43 am

Thanks!

Diskspace limits, the bane of my life at work too!
"He that breaks a thing to find out what it is has left the path of wisdom." - Gandalf
J.R.R. Tolkien, Council of Elrond, The Fellowship of the Ring

User avatar
Brennor
Emeritus Admin
Emeritus Admin
Posts: 9161

Re: 2/14-2/15 outage

Post by Brennor » Wed Feb 18, 2015 2:53 am

Yes, though limits on logging space prevent things from snowballing and allow safe shutdown and prevention of data mangling... so a tradeoff I'm willing to make.
[url=telnet://brennor.dyndns.org:1896]Akarian Dawn MUSH[/url]
SRGC v0.22: SR2+++ SR3++ h+ b+++ UB+ IE+ RN W+ dk+ ri+ ad+ m+ gm+ M- P+

User avatar
Muskrat
Level 21 Elite
Level 21 Elite
Posts: 10513
Location: The Secret Places Beneath the Earth
Title: Keeper of the Hidden Word
User Class: Scholar

Re: 2/14-2/15 outage

Post by Muskrat » Wed Feb 18, 2015 5:36 am

Thanks for getting everything up and running again.

I would certainly say it's better to have a temporary shut down than to lose all our data. I mean, the loss of all the recent posts would shoot everything to hell.

But why on earth would anyone attack our site? We're pretty innocuous, unless the attackers are Jack Chic-types who think gaming is a form of devil-worship.

User avatar
Hayabusa
Level 25 Mythic
Level 25 Mythic
Posts: 15431
Location: Ireland
Title: Advocatus Diaboli
User Class: Alliance Officer

Re: 2/14-2/15 outage

Post by Hayabusa » Wed Feb 18, 2015 11:07 am

We do NOT worhsip the devil? I'm out of here. ;)

In any case, although I'm not as savvy with computers (though I certainly understand a DoS attack) I am glad to hear that it was a fruitless thing from whoever did it. Agree with Muskrat. Better down for a bit than put back and losing the data.
Image
"All we have to decide is what to do with the time that is given us."

User avatar
Brennor
Emeritus Admin
Emeritus Admin
Posts: 9161

Re: 2/14-2/15 outage

Post by Brennor » Wed Feb 18, 2015 8:05 pm

Countless attacks are launched at web servers every day, just people looking for more hosts to join their botnets. Some days we fend off more attacks than get legitimate traffic.
[url=telnet://brennor.dyndns.org:1896]Akarian Dawn MUSH[/url]
SRGC v0.22: SR2+++ SR3++ h+ b+++ UB+ IE+ RN W+ dk+ ri+ ad+ m+ gm+ M- P+

Locked