2/14-2/15 outage
- Brennor
- Emeritus Admin
- Posts: 9161
2/14-2/15 outage
Hey guys,
Investigated the outage and figured out what happened. The issue was caused by database log files filling up their partition, mostly because of a big attack on the site. When the nightly log rotation happened, there was no free space on the logging partition, so the DB shut down safely so as to not lose any data.
This was fixed on 2/15 by the log rotations when they gzipped the old (big) log file freeing up a bunch of space.
I've doubled the quota of the log location and have added archiving so that older log files are moved to a storage location and won't count against the log file quota.
Investigated the outage and figured out what happened. The issue was caused by database log files filling up their partition, mostly because of a big attack on the site. When the nightly log rotation happened, there was no free space on the logging partition, so the DB shut down safely so as to not lose any data.
This was fixed on 2/15 by the log rotations when they gzipped the old (big) log file freeing up a bunch of space.
I've doubled the quota of the log location and have added archiving so that older log files are moved to a storage location and won't count against the log file quota.
[url=telnet://brennor.dyndns.org:1896]Akarian Dawn MUSH[/url]
SRGC v0.22: SR2+++ SR3++ h+ b+++ UB+ IE+ RN W+ dk+ ri+ ad+ m+ gm+ M- P+
SRGC v0.22: SR2+++ SR3++ h+ b+++ UB+ IE+ RN W+ dk+ ri+ ad+ m+ gm+ M- P+
- TetNak
- Emeritus Admin
- Posts: 46375
- Location: Seagard
- User Class: Brewmaster
Re: 2/14-2/15 outage
Thank you B!
"Kings have no friends, only subjects and enemies."
- King Stannis Baratheon, First of His Name
- King Stannis Baratheon, First of His Name
- Brennor
- Emeritus Admin
- Posts: 9161
Re: 2/14-2/15 outage
I should probably also add as an aside that the big attack ended up failing to break in, though it did cause a denial of service because of the DB shutdown.
[url=telnet://brennor.dyndns.org:1896]Akarian Dawn MUSH[/url]
SRGC v0.22: SR2+++ SR3++ h+ b+++ UB+ IE+ RN W+ dk+ ri+ ad+ m+ gm+ M- P+
SRGC v0.22: SR2+++ SR3++ h+ b+++ UB+ IE+ RN W+ dk+ ri+ ad+ m+ gm+ M- P+
- Trogdor
- Emeritus Admin
- Posts: 27260
- Title: The Burninator
- User Class: Jedi Master
Re: 2/14-2/15 outage
Good job!
- Vardaen
- Admin
- Posts: 66394
- Location: Miskatonic University
- Title: Great Old One
- User Class: Unshackled AI
Re: 2/14-2/15 outage
Thanks!
Diskspace limits, the bane of my life at work too!
Diskspace limits, the bane of my life at work too!
"He that breaks a thing to find out what it is has left the path of wisdom." - Gandalf
J.R.R. Tolkien, Council of Elrond, The Fellowship of the Ring
J.R.R. Tolkien, Council of Elrond, The Fellowship of the Ring
- Brennor
- Emeritus Admin
- Posts: 9161
Re: 2/14-2/15 outage
Yes, though limits on logging space prevent things from snowballing and allow safe shutdown and prevention of data mangling... so a tradeoff I'm willing to make.
[url=telnet://brennor.dyndns.org:1896]Akarian Dawn MUSH[/url]
SRGC v0.22: SR2+++ SR3++ h+ b+++ UB+ IE+ RN W+ dk+ ri+ ad+ m+ gm+ M- P+
SRGC v0.22: SR2+++ SR3++ h+ b+++ UB+ IE+ RN W+ dk+ ri+ ad+ m+ gm+ M- P+
- Muskrat
- Level 21 Elite
- Posts: 10524
- Location: The Secret Places Beneath the Earth
- Title: Keeper of the Hidden Word
- User Class: Scholar
Re: 2/14-2/15 outage
Thanks for getting everything up and running again.
I would certainly say it's better to have a temporary shut down than to lose all our data. I mean, the loss of all the recent posts would shoot everything to hell.
But why on earth would anyone attack our site? We're pretty innocuous, unless the attackers are Jack Chic-types who think gaming is a form of devil-worship.
I would certainly say it's better to have a temporary shut down than to lose all our data. I mean, the loss of all the recent posts would shoot everything to hell.
But why on earth would anyone attack our site? We're pretty innocuous, unless the attackers are Jack Chic-types who think gaming is a form of devil-worship.
- Hayabusa
- Level 25 Mythic
- Posts: 15431
- Location: Ireland
- Title: Advocatus Diaboli
- User Class: Alliance Officer
Re: 2/14-2/15 outage
We do NOT worhsip the devil? I'm out of here.
In any case, although I'm not as savvy with computers (though I certainly understand a DoS attack) I am glad to hear that it was a fruitless thing from whoever did it. Agree with Muskrat. Better down for a bit than put back and losing the data.
In any case, although I'm not as savvy with computers (though I certainly understand a DoS attack) I am glad to hear that it was a fruitless thing from whoever did it. Agree with Muskrat. Better down for a bit than put back and losing the data.
"All we have to decide is what to do with the time that is given us."
- Brennor
- Emeritus Admin
- Posts: 9161
Re: 2/14-2/15 outage
Countless attacks are launched at web servers every day, just people looking for more hosts to join their botnets. Some days we fend off more attacks than get legitimate traffic.
[url=telnet://brennor.dyndns.org:1896]Akarian Dawn MUSH[/url]
SRGC v0.22: SR2+++ SR3++ h+ b+++ UB+ IE+ RN W+ dk+ ri+ ad+ m+ gm+ M- P+
SRGC v0.22: SR2+++ SR3++ h+ b+++ UB+ IE+ RN W+ dk+ ri+ ad+ m+ gm+ M- P+