Presentation Reference Materials

Bug Hunting: A Masterclass


  • Bug (n.)
    Anything that threatens the value of the product to someone that matters
  • Issue (n.)
    Anything that threatens the value of testing, the project or the business
  • Inconsistency (n.)
    A bug based on a requirement, or a requirements document

Questions for Testers (and their clients)

  • How should we test this product?
  • Where should we look for bugs?
  • Are there bugs here?
  • Do those bugs matter? Who decides? Whose values matter?
  • What might slow or prevent testing?
  • Are we ready to release?
  • How might we be fooling ourselves?

FEW HICCUPS Oracles Mnemonic

When a product is inconsistent with any of the following criteria, we can reasonably surmise that there may be a problem with it:

  • Familiarity
  • Explainability
  • World
  • History
  • Image
  • Comparable Products
  • Claims
  • User Desires*
  • Purpose
  • Statutes & Standards

*On a basic level, User Desires can be expressed as Quality Criteria:

  • Capability – The functionality of the the product itself
  • Scalability – Can this thing be expanded as the user base or the load it takes on grows?
  • Reliability – Is it going to break? Will it require lots of maintenance?
  • Compatibility – Will it work well with our existing technologies?
  • Usability – Is it easy to use? Would it pass the ‘Granny Test?’
  • Performance – Does it work efficiently on our test and production architecture? Does it affect the speed and performance of our product?
  • Charisma – Does it engage people? Do people want to use it?
  • Installability – Does it install cleanly and easily? Even the best app in the world is of no use if it can’t be installed.
  • Security – Of paramount importance. Is the product secure? Does it inspire confidence?

Many test approaches focus on testing for Capability (functionality) and underemphasise the other criteria. Yet inconsistency amongst any of these criteria may represent diminished value to a person who matters.

As a subset of users, a development team has their own set of Quality Criteria:

  • Supportability – It it simple to fix problems, or change things based on feedback?
  • Maintainability – How easy is it to carry out maintenance and keep things running smoothly?
  • Portability – Can it be transferred between platforms? Between Desktop and Mobile App, between different flavours of mobile?
  • Localisability – Can it be easily translated for different languages? Different markets?
  • Testability – Do we have the opportunity to observe and control the product? Does it create log files? Does it have a scriptable interface? Reduced testability gives bugs more time to entrench, and more opportunities to hide.

The Language of Testing

The Phoenix Checklist

The Phoenix Checklist is a method developed by the CIA for deep-diving into a challenge, and considering both the challenge and your approach to it from numerous angles.
Method for use:
  • Write your challenge.  Isolate the challenge you want to think about and commit yourself to an answer by a certain date.
  • Ask questions.  Use the checklist to dissect the challenge into a variety of areas.
  • Record your answers.
The Problem Phase:
These questions are used to fully define the problem and gather as much information about is as possible
  1. Why is it necessary to solve the problem?
  2. What benefits will you gain by solving the problem?
  3. What is the unknown?
  4. What is it you don’t yet understand?
  5. What is the information you have?
  6. What isn’t the problem?
  7. Is the information sufficient? Or is it insufficient? Or redundant? Or contradictory?
  8. Should you draw a diagram of the problem? A figure?
  9. Where are the boundaries of the problem?
  10. Can you separate the various parts of the problem? Can you write them down? What are the relationships of the parts of the problem?
  11. What are the constants (things that can’t be changed) of the problem?
  12. Have you seen this problem before?
  13. Have you seen this problem in slightly different form?
  14. Do you know a related problem?
  15. Try to think of a familiar problem having the same or a similar unknown.
  16. Suppose you find a problem related to yours that has already been solved. Can you use it? Can you use its method?
  17. Can you restate your problem? How many different ways can you restate it? More general? More specific? Can the rules be changed?
  18. What are the best, worst, and most probable outcomes you can imagine?
The Plan Phase:
These questions are used in conjunction with the information gathered in the Problem phase to create a plan to address the challenge
  1. Can you solve the whole problem? Part of the problem?
  2. What would you like the resolution to be? Can you picture it?
  3. How much of the unknown can you determine?
  4. Can you derive something useful from the information you have?
  5. Have you used all the information?
  6. Have you taken into account all essential notions in the problem?
  7. Can you separate the steps in the problem-solving process? Can you determine the correctness of each step?
  8. What creative thinking techniques can you use to generate ideas? How many different techniques?
  9. Can you see the result? How many different kinds of results can you see?
  10. How many different ways have you tried to solve the problem?
  11. What have others done?
  12. Can you intuit the solution? Can you check the result?
  13. What should be done? How should it be done?
  14. Where should it be done?
  15. When should it be done?
  16. Who should do it?
  17. What do you need to do at this time?
  18. Who will be responsible for what?
  19. Can you use this problem to solve some other problem?
  20. What is the unique set of qualities that makes this problem what it is and none other?
  21. What milestones can best mark your progress?
  22. How will you know when you are successful?