Speed = Risk

No turtles were harmed in the making of this post.

As a QA professional, one of your biggest fears will always be allowing something big to get past you. You miss something that ends up with your company directors being dragged out of bed in the middle of the night, and requires diversion to a shunt page or honest-to-goodness downtime for your site to get fixed. That will inevitably end up with all those awkward questions being asked about why this issue was missed in testing.

I don’t have an immediate answer to those questions, and your mileage may vary depending on the scale of your issue and the reason behind it, but I’d strongly advise against saying ‘Hey, it’s not like QA put the bug in there in the first place!’ and throwing your developers under the bus unless you want to completely alienate yourself within your technology team. There are no winners when people start playing the blame game.

In short, if you’ve missed a P1 shipstopper issue, step up and own it. Investigate properly, find out what was missed and how it was missed, work with your developers on the solution, and perhaps most importantly, make sure you don’t make the same mistake twice.

But how can you make sure this never happens to you?

You can’t. And that’s okay.

We’ve all heard the trite saying along the lines of the one thing that you don’t test will be the one thing that causes problems. But to me, that particular pearl of wisdom can be lumped into the same category as your keys always being in the last place you look. By definition, edge case issues occur in areas that you’d never dream of testing, or involve steps that you’d never believe that anyone would actually take.

So even if, for some insane reason, you’ve opted to attempt 100% test coverage, and decided to test every user situation that you can think of, you cannot and will not cover everything. Taking zero issues to production is an utterly impractical goal that will only lead to frustration and disappointment when it is inevitably not met. You might cover everything you can think of, but you’ll never be able to cover everything that everyone ever thought of – not to mention the things that nobody thought of.

Most folks in modern technology organisations understand that shorter SDLCs and releasing at speed comes with risks. In these environments, the QA team’s function is not to catch everything, but to keep the risk of issues making it to production to a minimum. Sadly, not all business stakeholders see things that way, and it’s down to technology directors to make them understand that going faster comes at a price. As per Brooks’ Law, throwing more people at QA just muddies the waters, so it’s important that QA thinks carefully about their priorities when testing at speed.

Of course, it is vital that you ensure that business requirements have been met and that your software does what it has been designed to do (according to the precise needs of the business – not your interpretation or the development team’s interpretation of those needs). If your sprint planning sessions are done right and involve the right people, you’ve already nailed this. But that isn’t the be-all and end-all of your testing scope, as just testing the happy path is a recipe for disaster.

As a QA professional, you know what the software is supposed to do in detail. But your end users don’t have the requirements documentation or the user stories, and they inevitably wander off and do stuff they’re not supposed to (Incidentally – who says they’re not supposed to?). Once you’ve established the software’s fitness for purpose, exploratory testing is necessary to ensure your technology director doesn’t get that 3am phone call from the CEO telling him that his latest release is causing satellites to drop out of the sky. But how far down the exploratory testing rabbit hole do you go?

There’s an old joke that goes something like this:

 A QA Engineer walks into a bar.

He orders a beer.

He orders 3 beers.

He orders 2976412836 beers.

He orders 0 beers.

He orders -1 beer.

He orders q beers.

He orders nothing.

Él ordena una cerveza.

He orders a deer.

He tries to leave without paying.

He starts ordering a beer, then throws himself through the window half way through.

He orders a beer, gets his receipt, then tries to go back to the point before he’d ordered.

He orders a beer, goes to the door of the bar, throws a handful of cookies into the street, then goes back to the bar to see if the barmaid still recognises him.

He orders a beer, and uses a device to very carefully watch while the barmaid puts his order into the till to make sure nothing in his request got lost along the way.

He starts ordering a beer, and tries to talk the barmaid into handing over her personal details.

He orders a beer, sneaks into the back, turns off the power to the till, and waits to see how the barmaid reacts, and what she says to him.

He orders a beer while calling in thousands of robots to order a beer at exactly the same time.

With a bit more time than I’m willing to put into it, that list could go on and on, and in the same way, you could easily overdo exploratory testing. So you must properly identify the areas that your software touches on and flows through, and the situations which are most likely to occur on a regular basis to reduce the time the software spends in test, and to help in speeding up your development cycle.

And that’s the thinking behind risk-based testing. In a world where short SDLCs and being first to production, or at least the fastest of followers, is business critical, this is the only feasible approach for QA.

The idea is that you pinpoint the essential function of the change, determine the scope for testing in and around that change, analyse the associated risks, then plan and prioritise your testing accordingly.

Start by making sure the change is fit for purpose and, just as importantly, does no harm. This is non-negotiable, and you can’t provide your part of the team sign off without giving this assurance. That should be followed by smart, correctly prioritised exploratory testing of the areas which are next most likely to cause problems, as determined by your intimate knowledge of the system under test.

You’d be forgiven for missing an edge case that only happens at 23:15 on one day of the year on the Kenyan version of your software when it is accessed via Konqueror. But a study of your application’s user metrics will allow you to properly assess which browsers, devices, geographical regions etc., are most popular with your users, and that will allow you to properly prioritise which access points you use while testing.

The real key to using risk-based testing properly is a strong knowledge of your systems and architecture, and a good understanding of the paths users will take. As a QA Engineer, you should be familiar with all the touch points around the functions you are responsible for testing, how they interact with each another as well as other applications in your overall architecture, and all the paths the user will take into, through, and out of your area. This will allow you to easily identify and isolate the important areas for any given change, and allow you to focus testing around those areas.

Do you cover everything? No, absolutely not. Like I said before, that’s an unrealistic expectation. But properly planned and executed risk-based testing will allow you to go faster AND stop phones from ringing in the middle of the night. Unless, of course, you’re testing software to automate alarm calls. Then that’s WAD.

Exploratory Testing ≠ Random. Exploratory Testing = Chaos.

If you’re describing exploratory testing as anything like ‘Randomly poking about in the corners of the system’, you’re using the wrong language.

I first published this article on my LinkedIn profile about a year ago, but I’m so proud of it, I thought it was worth republishing on my own site.

Many times, I’ve heard exploratory testing described using terms alluding to randomness. I’ve almost certainly been guilty of it myself. But on reflection, this is a stance that I wholeheartedly disagree with.

I’ve mentioned on a few occasions that I feel exploratory testing is a highly specialised skill. It should be the largest and most important part of a manual testing role, and it dovetails with standardised automated checks of the system under test to provide a true assurance of quality. So if you’re describing exploratory testing as anything akin to ‘Randomly poking about in the corners of the system’, you’re using the wrong language. You’re giving the impression that you’re only going to find issues by dumb luck, and you’re doing both yourself and your profession a great disservice.

And really, any randomness in testing renders that test unreliable. There’s nothing worse than finding a big, nasty bug that seriously compromises the quality of your system and not knowing exactly how you got there. If you can’t recreate what you did, then you can’t fully prove the issue exists, you can’t systematically check the area around the bug to see if it occurs in just the way you’ve discovered or if there are other ways to trigger it, and worst of all, you can’t get anything done about it. Having a bug returned with a ‘Cannot Reproduce’ status is highly frustrating.

Software development, and especially exploratory testing, includes many elements of chaos in the mathematical sense. But it is very important not to incorrectly use chaos as a synonym for random. Edward Lorenz described chaos as: “When the present determines the future, but the approximate present does not approximately determine the future.”

In our industry’s terms, this means that the user journey through a single path of the system may be a complicated one with a large number of variables along the way. But it is still a deterministic practice where the pattern for each variable can only evolve within a limited scope, making it possible to predict the end state of the system, rather than a stochastic practice where outcomes are truly random. Therefore, with good understanding of the system under test, well written acceptance criteria, and proper insight into the path the developers will take to meet those criteria, a tester should be able to predict how the system will be affected by a change, precisely determine the automated checks they’ll need to run, and the areas they will need to focus on during exploratory testing.

One of the most often used examples of chaos theory is that of a butterfly flapping its wings in Brazil causing tornadoes in Texas. This is very poetic, and is almost entirely incorrect.

Chaos theory actually states that a butterfly flapping its wings can change the course, accelerate, decelerate, or otherwise greatly affect the outcome of a tornado already in effect. That man Lorenz again: “A very small change in initial conditions had created a significantly different outcome.” The butterfly does not create or power the tornado, but it causes a tiny change to the initial state of the system being examined, which cascades to incrementally bigger changes in subsequent events. It’s the theoretical flap that makes the difference between a breeze blowing itself out over the Gulf of Mexico, and an F5 tornado levelling Dallas.

It is therefore vitally important to understand that we work with the actual definition of chaos rather than the wider perception. The butterfly flap of a change to the search mechanics of a retail site won’t directly cause tornadoes in the payment handling system (Unless the two systems are somehow intrinsically linked – but I bet you a penny that they’re not), so there’d be no point in exploring the payment system during testing of the change to the search mechanics.

But, for example, if that change is not explored properly and issues are not pinpointed, a quirk in the requests and responses made as a result of the search could go unnoticed, leading to the service returning a search results page with incomplete or incorrect information, which will in turn lead to a downturn in traffic going to the payment handling system and a loss of revenue. A tornado, which could have been predicted and prevented, has hit.

So with chaos theory stating that even a small change to an initial state can yield catastrophic results down the line, it makes sense to explore the area where the cause of any issues will originate first, and move on from there. Examine the movement of butterfly’s wings to determine the actual path and end state of the event, rather than finding yourself picking through the rubble that used to be Cowboys Stadium, cursing all Lepidoptera.

What IS a QA Evangelist, anyway?

Short Answer: I am.
Longer Answer: Read on…

Short Answer: I am.

Longer Answer: Well, perhaps it’s better to start with the things that being a QA Evangelist DOESN’T mean.

Despite the name, my current position and title have no religious connotations. I don’t deliver my presentations from a pulpit, I don’t wear tech-branded vestments, I don’t swing incense about the place, and I don’t go door-to-door preaching the gospel according to software testing… at least, not in a way that makes people turn off the telly, hide behind the sofa, and pretend they’re not at home.

Certainly one of the best parts of having an unusual job title is that it’s a great ice-breaker and conversation starter when I go out and meet people. Believe me, when you tell anyone in the technology world that you’re a QA Evangelist, they immediately want to know more. So here’s the story of how my role came about, and exactly what it entails.

I first conceived the role about 18 months ago, as I felt it was something that should have existed in our company. A role that would fit my skills, make me more useful and valuable to the business, and would help to raise the profile of our company in the wider technology community. As a company, we knew we were (and indeed, still are) very good at what we did, we know we were at the cutting edge, and we knew were innovators. It was high time other people knew it too.

My idea involved taking the QA Manager role I was currently filling, and reinventing it. I didn’t want to just be responsible for the day-to-day management of software testing, as the folks we take on generally know what they’re doing, and don’t need someone standing over them the whole time. But of course, I have a great deal of knowledge around both our profession and our company, so I still wanted to be a point of reference and guidance, and I still wanted to be first line support for the QA team. I also wanted to retain my responsibilities for recruitment, mentoring new hires, and all the other good stuff that the traditional QA Manager role entails. It was, and still is, very important to me to keep an eye on the day-to-day stuff so I can make the most appropriate recommendations and offer the most relevant advice possible – but it’s no longer my single raison d’être.

I also wanted to take responsibility for showing our QA team that there’s a world outside their individual delivery team. I thought it was important to regularly share blog posts and industry articles to get the team thinking more about what they do and how they do it, and to encourage them to create and share their own work in this way. I wanted my teams to look at interesting conferences and webinars, drive engagement with other teams, other areas of the business, and even other companies to share ideas, strategies, and seek solutions to issues they’re encountering. And I wanted to encourage teams to get out and do this stuff themselves – to present at meetups and conferences, to trumpet their successes to the wider testing world, and to really build themselves testing communities that they can easily reach out to. All of which would also benefit the company by raising our profile in the wider technology world.

I, personally, also wanted to act as an advocate to other areas of the business, and to other companies – to tell the rest of our business about the value of proper testing and QA in software development, as well as getting guest speakers in for our teams, putting myself out there to speak at other companies, meetups and conferences. So in that respect, I guess I do sort of go door-to-door preaching the gospel of software testing.

I spent the best part of a year pitching the concept of the role to everyone who might be affected, refining it and tailoring it for our business, and doing the groundwork to establish myself in that position. All the while, I was nervously waiting to see if the Technology leadership understood the role and its value, if the new organisational structure created to accommodate the role would successfully navigate any red tape thrown up by our (excellent) HR department.

Finally, in March this year, I was promoted into the role, exactly as I’d conceived and pitched it, with very little in the way of enforced change from Tech Leadership and HR. And it’s no exaggeration to say that the conception and implementation of the QA Evangelist role, and my promotion into that role, is my proudest professional achievement.

Since taking on the role officially, I’ve attended a few conferences and delivered my learning back to my teams. I’ve launched an internal-only podcast, which I’m planning to put out to the big wide world once I’ve got the formula nailed, and I’m getting out there and doing more speaking about our profession. I’m busier, but more satisfied in my work than I’ve ever been.

I feel it’s a natural evolution of the QA Manager role in an industry that is moving towards expertise being prevalent at all levels, and towards iterative and / or continuous delivery as the norm. Being able to not just lead, but to influence and guide my teams, to get them to really communicate about what they do and how they do it, to demonstrate the value of our role in the delivery of quality products, and to share the stories of our success to the widest possible internal and external audiences is, to me, the wave of the future for software testing.

I’m proud to have nailed my colours to the mast in the first place, to have driven this change in the way we think about and execute our testing, and to have put our company in the position to be on the crest of that wave.

EvangelistPunk SPEAKS!

I’ll be delivering my talk ‘Bug Hunting – A Masterclass: The mindset for analysis, identification and classification of problems in software” at the SDET & Automation Network Meetup in London on Thursday 13th October.

Event details are here if you’d like to attend.

We don’t break software!

‘Well, our software was working fine until the testers broke it!’

There’s a whacking great big chunk of hypocrisy in the tagline of my blog: The bit where it says ‘Vandalising Software’. It’s a good line, one that I’m very happy with, but I do feel I need to address it for my own peace of mind.

It’s hypocritical because I always resist saying anything like ‘I break software for a living’. It’s a fun thing to say, and I’m certain we’ve all used something similar to describe our job to people outside the industry. But I make a point of correcting people I hear using that sort of language, because it’s inaccurate and detrimental to our profession. I’ve even seen it turned around and used against testers before – ‘Well, our software was working fine until the testers broke it!’

The truth is that we don’t break software, because the software is already broken when it comes to us. We break unwarranted assumptions. We break illusions. We test, investigate, explore and experiment to highlight areas where software is broken. We give as much information as we can to our stakeholders as quickly, clearly and efficiently as possible. And we collaborate on fixes and solutions to ensure deadlines are met and releases aren’t delayed.

If we were really in the business of breaking software, we’d be Developers.

A Word on Certification

If you, or your company, paid for an ISTQB Certification, you got robbed. And here’s why.

I am vehemently opposed to industry-wide testing standards and certification schemes. And I am of the opinion that if you, or your company, paid for an ISTQB Certification, you got robbed. And here’s why.

ISTQB teaches you what it calls the industry standard of terminology. But you’ll find that as soon as you get into a company, or even when you move from one project team to another within the same company, the playing field is a little different. To me, as long as all the folks involved with a project have common ground, and a mutual understanding of the terms and requirements in use, that’s absolutely fine.

I know it’s the done thing to put ‘ISTQB Certified’ on a CV, or to add it as a requirement in a Job Specification when recruiting. I know some companies even filter CVs based on whether or not the ISTQB keyword is present in a CV. So hey, maybe put ‘I am not ISTQB Certified, and I’m happy to explain why.’ on your CV to get around the hiring company’s automatic filter and get into their ‘Accept’ pile.

I couldn’t possibly advocate just putting the qualification on your CV whether you have it or not – but it’s probably worth me telling you that, as a hiring manager, I have never once checked whether or not a candidate actually IS certified when they’ve said they are. During the recruitment process, I focus on a number of things, but whether or not the prescribed ISTQB terminology is being used is not one of them.

Frankly, I don’t care if you communicate by the standards set out in ISTQB, CSTE, ISCB, Pig Latin, Semaphore or Khoisan – As long as you can get the abstracts and concepts I’m looking for across to me, that’s perfectly okay. You certainly don’t need a piece of paper telling me that you know how to refer to things in a certain way.