Mood: Tired and I have a sore throat. So mildly annoyed.
Music: The continual sound of rebooting mail servers.
Game: World of Warcraft, CoD2, EVE Online
Book: The Hacker Crackdown, Bruce Sterling.
Muffin: None. Wish I had one, though.
Last night, while my new IT guy (and friend) here at work was attempting to configure the UPS, the UPS unceremoniously decided to shut itself down, taking down all the servers at the same time, equally unceremoniously, by the simple expedient of taking all power away from them.
Curiously, most computers don't LIKE it when you just say "NO POWER FOR YOU!" and sometimes get headaches from such occurrences. The fileserver (thank god) came back with not even a "What'd you do that for?" That new server, of which I've already sung praises, is built bulletproof. The webserver was a bit more recalcitrant, requiring coaxing to actually start UP the webserver processes, but they came up in minutes. The mailserver...ah, the mailserver.
Mailservers are finicky beasts, as well they should be. Even a company the size of the one I work at, which is 50 people, gets a TON of mail a day, incoming and outgoing. The mail comes at all times of the day and night, and the fearless mailserver is constantly routing, sorting, blocking, accepting, rejecting, filtering all this fabulous mail (and spam, which likely takes up a good half to 3/4 of all mail....) Anyway, when the mailserver came back up, the directory journals were corrupted. Not good.
The LDAP server is quirked...won't allow connections to it. Hardware is fine, because we can see the box. Everything SEEMS to be running the way it's supposed to. But obviously, since our LDAP server won't allow logins, and our mailserver is BASED on LDAP logins, things aren't quite so peachy.
I left the machine running overnight to do a consistency scan...and the drives themselves are fine. Good thing, in a way...insofar as the hardware seems to be good...which isn't always the case. But it looks like we have corrupt files...and that means rebuilding the LDAP server. I HOPE the data is fine. Re-entering all the data for every employee would kinda suck.
In any case, I'm here at 7am, trying to get this thing working before folks really start working at 9am, and I get constant questions, usually intelligent, like "Is there something wrong with email?"
Back to the server room. Updates later, maybe.
-----
UPDATE: Servers are all online. LDAP server got fragged. Once it was rebuilt, it all came back just fine.
Posted by Glenn at November 9, 2005 07:36 AM