So you’ve decided to hire testers. Good for you. I know there are some organizations that may have misconceptions about the role of testers in software engineering, and some individuals even make the case that they don’t need testers. Joel Spolsky from Joel on Software previously articulated and debunked some of the more common of these misconceptions in his post Top Five (Wrong) Reasons You Don’t Have Testers. One of the interesting parts of this article, other than the responses to the misconceptions, which are dead on, is when Joel addresses “Anybody qualified to be a good tester doesn’t want to work as a tester.” He mentions that it is difficult to find good testers and that “like programmers, the best ones are an order of magnitude better than the average ones.” Anyone who has worked with testers before knows this to be true. So how do you determine which testers are a magnitude better and measure for this in an interview? This post will describe some of the types of questions I have used to specifically measure this in interviews.
Before I go into the answer, I’d like to review some of the aspects of good testers that you want to measure in an interview. If you ask the average developer what makes a good tester, he or she might say “finding a lot of bugs” or something to that effect. This demonstrates one difficulty in finding the right testers for an organization, especially an organization that doesn’t have any testers currently. The understanding of what aspects make a tester good is not entirely realized by members of a team that aren’t involved in the day to day activities of a test team. This is not to say that they cannot evaluate the performance of a tester that they have worked with, they just don’t spend a lot of time evaluating what aspects are driving some testers to be better than others, or seeing the cause and effect relationship of these aspects leading to the outward display of test excellence. The understanding of these aspects as well as the ability to construct questions that accurately measure them is the most important part of assessing testers in the short time you have in a round of interviews.
To directly address why “finding a lot of bugs” is not a good measure of a tester’s productivity, let’s talk about bugs in general and the philosophy that I implicitly believe good testers share with respect to them. First, it is not hard to find bugs. They are everywhere, even in good software. Just because a tester finds a lot of bugs doesn’t make them even a good tester. Conversely, if that tester logs a bunch of low priority bugs while overlooking those that are really important, he or she could even be worse for quality than not having a tester at all, as all of the bugs logged need to be reviewed and deciphered by someone on the team who could be fixing more important bugs, working on a new feature, or contributing in another way that is more beneficial. Being a good tester has to do with finding the right bugs.
Being a good tester is about developing a strategy to maximize how many of the right bugs can be found, working with other team members to make the scope of your testing and associated tradeoffs between time and test coverage explicit so that everyone can contribute to talks about quality, and enabling as much of the team as possible to participate in quality in a sort of mutually-symbiotic way that makes everyone more efficient. Yes, there is the act of executing tests too, but if you are good at developing test strategies and technically competent (which is easier to measure), actually doing the testing should be a matter of executing a series of simple, well thought out steps.
Now that we have that out of the way, let’s talk about a few aspects of the type of tester that I have described and how to structure questions to measure these. I will not mention standard interview questions about work ethic, culture fit, or other topics of this nature. These are definitely important, but there are many people that have already talked about how to measure these who are more qualified to give you advice on this topic than me. Using the analogy of interviewing as a college curriculum, the work ethic / culture fit questions would be prerequisites; whereas, I am going to focus solely on the electives.
In my opinion, here are the main aspects that are shared by great testers:
- Collaboration Skills
- Problem Solving Approach
- Application of Technical Skills
Collaboration Skills
Collaboration skills, especially with those beyond the test team, are among the most valuable qualities a tester could possibly possess. If you are working in an organization where the test team is small compared to the development team (basically anywhere except for Microsoft), these skills are absolutely necessary as there will not be enough time for a single tester to cover all of the areas that need to be covered. To further complicate this, even in organizations where the test team is large, there will be situations where unexpected bugs are found that jeopardize the timeline of your project. In these situations, testers need to play a very active role understanding the intricate details of complications that can occur as a result of changing code so late in the release cycle and coordinate with developers to make sure sufficient steps are taken to mitigate risk.
When a bug occurs late in the release cycle, a good tester will discuss the nature of the bug with developers, review the code that is being changed for the fix, ask the right questions, and get everyone to help out in testing it. Good testers act as a force multiplier and use, in some cases, unconventional means to uncover issues sometimes without doing testing in the typical sense. I can think of several times where I was working on a team and one of these bugs cropped up in a timeframe where the most optimal way to mitigate the risk was to do a detailed code review and write some extremely crude unit tests targeting specific concerns from the code review itself (which many times have uncovered issues). Using this tactic, even without doing any testing personally, the tester enables everyone to have quality on their mind as they are making decisions in writing code, writing tests, and reviewing details with the test team. Experienced developers know this to be in their best interest as with the right person coordinating this testing, there is less of a chance of their day being disrupted by a high priority live site issue that they need to hotfix (which I can only assume is the most frustrating and counter-productive use of developer time). Therefore, someone who possesses the ability to do this or is naturally inclined to work in this fashion is the type of person that you want in your test organization.
Sample Questions Measuring Collaboration Skills:
- Can you think of an example a time in the past where a bug that required a complicated change was discovered late in the release cycle, and how you approached it?
- Explain a time when you worked closely with the development team to define a testing strategy.
- What are some ways in the past that you have engaged members outside of the test team in the ownership of quality?
Problem Solving Approach
If ever there was a better way to determine how someone would go about testing a complicated system, it is to give them that exact question. You will be very surprised at how revealing the actions one takes when given such an open-ended question will be. The most effective way to ask this question is to draw a diagram of a simple but multi-tiered system on a whiteboard while explaining the purpose of the system and how the parts of the system work together, then ask the interviewee to write the basics of a test plan for it on the whiteboard. Don’ t focus too much on telling all of the details of the implementation of this mock system, as an important part of the question is to see what type of clarifying questions they ask as they are developing their response. If they don’t ask any questions and go full-steam ahead making a series of assumptions without explicitly stating them, this is an indication that the interviewee may act the same way in a real situation, which is definitely negative.
The important thing to note about this question is there is no right answer, it is a simple exercise to see what the candidate’s approach to testing is, which will be useful in determining whether they fit well in your organization. A senior tester will likely start circling sub-systems in the diagram and making an outline for each of the subsystems, then detail out what types of testing will be done on each sub-system and methodically work through each of the systems asking a lot of questions. Conversely, junior testers will usually be inclined to start at a user-facing element of the system and writing out individual test cases before organizing it into a cohesive approach. There is nothing wrong with this approach, and if your organization has a well-defined area that needs testing and just needs someone to ploy through writing tests (as opposed to someone to define and drive the larger direction or quality) this may acceptable (assuming that when going deep on one aspect of the system the interviewee applies all of the correct test techniques). If you already have a senior tester, it may also be a good growth experience for a tester of this nature. From the responses of the candidate, you should be able to determine the candidate’s general problem solving approach, and this will give you a good idea as to whether this is the type of tester you need for your team. The question is more-so a framework for discussion about test strategies that you can continue to revise over the course of the candidate working through it to measure many different things about the candidate’s approach to testing and ability to apply the correct techniques.
Things to Look for in a Solution:
- Did they approach the question in an organized fashion?
- Did they ask the clarifying questions when developing their approach?
- Did they explicitly state the scope of their testing / things that are out of scope?
- Did they plan for the correct types of testing on the appropriate areas of the system (functional, performance, load, security, etc.)
- Did they make an attempt to prioritize tests?
Useful Sub-Questions
- Ask them if they were given a shorter amount of time than the scope defined, how would they change their approach (they may mention cutting scope, utilizing non-test members of the team for some classes of testing, or something of this nature)
- Make a small adjustment to the architecture and see ask how they would change their test plan (introduce an asynchronous element or define an additional locale)
Application of Technical Skills
I intentionally distinguish between coding skills and application of those skills because I believe the ability to apply technical skills in a test setting is actually more important than raw coding skills for most testers. I am definitely not saying that raw coding skills are not important as they will be needed to be able to find problems during code reviews, understand code enough to write targeted test plans, and to write useful and scalable test tools. That being said, the implications of this are that if you only ask general coding questions (not specific to testing) to someone who is going to be a tester, you are overlooking a key component of their competence. Since most of the code written by a tester will be used in a test setting, it makes sense that you would want to base your question around this specifically.
An effective approach to measuring the application of technical skills in a test setting is a coding challenge prior to the in-person interview. Give the applicant a written problem that they must submit a solution for within a couple days (which is much more true-to-life than writing code on whiteboard). You can ask another coding question in person on a whiteboard to verify their skills are consistent with those they demonstrated in the solution they submitted to you (and didn’t have a friend write it for them), but the coding they do for the code screen will be more accurate in assessing their skills than whiteboard code due to the natural setting in which they are able to solve the problem. The important thing in the coding challenge is making sure they have to come up with test cases in some algorithmic way. Questions of a combinatorial nature or dealing with state-machines are good as most experience testers have encountered a situation dealing with these concepts, but I will leave it up to your creativity to determine what type of system you ask them to test (you probably want the system to be similar in some respect to what they will actually be working on).
In addition, ask them to submit a short document with answers to questions about their solution: why they chose their approach, what conditions their solution covers, how they would define the coverage of there test cases, how they would improve their solution if given more time, etc. Having the answers to these questions will allow you to understand their thought process and put their coding in perspective. It will also help you gauge the candidate’s ability to quantify the effectiveness of his or her own testing and give you some indication of the candidate’s written communication skills.
Summary
There are many factors to determine whether someone will be a good employee that should be accounted for in questions asked in interviews. In addition to those questions, if you are hiring a tester, you will want to determine the candidate’s ability to drive quality through collaboration across disciplines, come up with effective approaches for testing complicated systems, and utilize their technical skills in a way that allows them to test in creative and efficient ways. From my experience, these are the areas that distinguish great testers, and with some simple questions, you should be able to determine whether or not your candidate is a good tester and a good fit for your team.
JeffMW - The Test Ranger Blogs 
Leave a Reply