The Rules of Releases

You’ve gathered the requirements, written the code, debugged it, received the new requirements, rewritten the code, got more change requests, reached a ‘compromise’ with QA (and hidden the bodies) and now you want to have the sysadmins do the release.

Don’t be like everyone else - when it comes to releases too many people fail at the last mile and make obvious mistakes. In an attempt to save myself some pain (and have something to point co-workers at) here are some of the software release principles that I hold dear.

Out of hours releases will have adequate support

Or as I like to think of it - “out of hours releases will hurt you as much as they will me. And a little bit more” If the release is important enough to require me in the office late at night or over the weekend then it’s important enough to have development support and a manager present Just In Case. It’ll also force people to be a little more considerate of my time and availability.

No live release will happen after 4pm (at the latest)

There’s nothing quite as frustrating as getting two third of the way through a live release, hitting a problem or needing clarification of something that the staging environment didn’t pick up (yes, I know it should have. Let’s fix it for next time) and discovering it’s 6pm and everyone else is already on the tube or in the pub.

You then have the pleasure of either backing the release out (if you actually can) and explaining why you killed the scheduled release or hanging around with half an upgraded system waiting for someone to get your voice mails and call you back. Which is even less likely to happen if you ignore…

No release will happen on the day before a non-work day.

Or the day the lead developer goes on holiday.

“We’re nearly done. Can you get $dev to have a look at this line in the application error log please?” “Actually he’s in Peru for the next three weeks. I’ll get someone else who’s never seen the system before to confirm that everything’s fine.” Apart from the obvious sign this is a made up conversation (application error logs that contain information - HAH!) I’ve been bitten by this a number of times. It always seems to end with some other poor developer with a postage stamp of hand over notes looking sheepishly at me while explaining that the log line could never happen.

You'll provide me with a list of what's changed

When you’re developing you should maintain decent change logging above and beyond ‘commit -m’. I’d like the world to agree that commit messages are for developers and release notes are for sysadmins; let’s pretend I’m not paranoid enough to read the commits list anyway.

If you’re using one of agile methodologies that uses stories for everything then feel free to put the story number in the notes to provide some background. However I still expect a one sentence summary of each change.

If you don’t have a decent, and comprehensive, list of changes expect me to get… inquisitive about undocumented changes. I will diff the packages (better version of this coming soon) or source code (and if I ever get the time to look at SQL::Translator I’ll be aggressively double checking your schemas). If you don’t mention it I can’t prepare (and add monitoring) for it, QA can’t test all the new paths and I’ll make a point of in the release retrospective meeting.

It should be possible to stagger releases across machines

I’m a fan of the one, few, many approach to software releases. I want the ability to role out the new system in chunks. I should be able to break off a couple of web servers so I have a warm standby just in case something goes wrong. I know this gets difficult to do once you involve databases but it’s still a goal that should be considered - especially with read only copies of the data, replication slaves and data snapshots in the tool belt.

So in closing, I’m a demanding bugger. However, just like my Cron Commandments post, it’s nice to have this list somewhere online to point people at.