Tales from the Script

A number of roles ago the operations and developer folk were blessed with a relatively inexperienced quality assurance department that were, to put it kindly, focused on manual exploratory testing. They were, to a person, incapable of writing any kind of automated testing, running third party tools or doing anything in a reproducible way. While we’ve all worked with people lacking in certain skills what makes this story one of my favourites is that none of us knew they couldn’t handle the work.

The manager of the QA team, someone I’d never have credited with the sheer audacity to pull off this long con, always came to our meetings with an earnest face and excuses about the failure of “The Script”. We, being insanely busy modern technical people, took this at face value; how would you run all the regression tests without a script? “There was a problem running the script”, “the newest changes to the script had caused regressions” and similar were always on the tip of their tongue and because the developers were under a lot of time pressure no deep investigations were done. Everyone was assumed to be doing their best and what a great QA manager they were in protecting their people from any fallout from the failures. On it went, all testing was done via “the script” and everything was again good. Or so we assumed.

In one of our recurring nightmare incident reviews, this one after something we’d previously covered had come back for the third time, a few of us began to get suspicious. We decided to build our own little response team and do some digging for the sake of every ones sanity. Now, this was before the days of GitHub and everyone being in one team of sharing and mutual bonding. We knew we’d have to go rooting around other departments infrastructure to see what was going on. Over the course of the next few days the group targeted one of the more helpless QA engineers and began to help him with everything technical he needed. He had the most amazing, fully hand held, on-boarding the department had ever seen and we, in little bits and pieces, began to pierce the veil of secrecy that was the QA teams process.

One day, just before lunch, one of the senior developers involved in our investigation hit the mother-load. The QA engineer had paired with them on adding testing to “the script” for a new feature the developer had written and suddenly he had a full understanding of the script and its architecture.

It was an Excel spreadsheet.

It was a massive, colour coded, interlinked Excel spreadsheet. Each row was a separate test path or page journey. Some rows were 40 fields of references to other rows to form one complete journey. Every time we did a release to staging they’d load up the Excel document from the network share and arrow key their way through row upon row of explicit instructions. Seeing it in use was like watching an insane cross between snake and minesweeper. Some of the cells were links to screen grabs of expected outputs and page layouts. Some of them had a red background to show steps that had recently failed. It was a horrific moment of shared confusion. A team of nearly forty testers had ended up building this monstrosity and running it for months. It was like opening up a word doc and having Cthulhu glare back at you. So we did the only thing we could think of, went to lunch and mostly sat in stunned silence.

And I almost forgot the best part of the story, the Excel spreadsheet? It was named “The_Script.old.xls”