Scary sysadmin Halloween stories

Gremlin recently ran a small twitter hashtag challenge called “#talesfromtheNOC” where people were invited to share their scary sysadmin stories. Reading through of the entries I was reminded of one slightly less than welcoming environment that led to a lot of learning, frustration and trepidation. I’ve captured my posts here.

Opening Volley

I was hired as the new sysadmin at a financial services company and had no hand over as my predecessor had apparently ‘left on short notice’. On my first day the CTO and I took a trip to the colo to check the asset register against the running machines.

After unlocking the back of the first rack I gave the door a little tug and it didn’t budge. “It sticks sometimes. Give it a decent pull.” said the CTO helpfully, so I put a little more force behind it and yanked the door open. Luckily the company never skimped on cable tie quality so I managed to pop out every individually tied to the rack door(!) power cable in one swing. It took a moment for us to realise what had happened as dozens of fans were suddenly silenced…

I later learned more about the meaning of ‘left on short notice’ and found a number of other little presents over my first few months in that role.

The second incursion

Same place as the previous story: I plugged the KVM into the management machine and was greeted by a full X-windows session. Unusual but not the first time I’d seen it. I logged in, decided to see if the previous admin still had access and ran visudo. The process seemed to hang and I paused for a few moments to consider what was going on.

Shuddering slowly to life on a heavily underspecced machine was a full install and process of Eclipse with /etc/sudoers open. $EDITOR was set to eclipse. What kind of person does that? A MONSTER!

Saboteur Strikes

To finish the ‘trilogy of twattery’ the same person used to leave a sendmail m4 file on the machine that DID NOT generate the running .cf file. So if you didn’t move it out before the machines rebooted the daemon regenerated its config and you came up with broken mail servers.

That was made slightly more annoying coupled with the booby trapped power cabling previously mentioned.

It was a very long first day.