If you’re doing any significant work at all, you should also be doing a lot of consumer research. Software development? You should be prototyping fast and early and running experience sessions. Branding? You should spend most of your time collecting information, then working some stuff out, then collecting again. Marketing? Try everything and AB test like crazy. You get the picture – nothing good will emerge in isolation, so you must have a solid information pipeline between you and your target.
It baffles me how often careful, critical, analytical professionals miss the opportunity to design their own procedures better. It’s like we have an unspoken rule that our skills are only to be used directly with whatever we’re building, not on trying to question the efficacy and improve our tools.
Now, often people focus too much on what they want to know and forget to think about how important it is to think about how they’re going to know that. Consumers are humans (I hope), and a consumer interview is a social encounter, often among complete strangers and with a lot of underlying expectations. Would you go unprepared for a blind date? Then you cannot just hope for the best here – you need to carefully plan your research sessions, for how they will be run will have a crucial effect on the quality of information you’ll get.
Most research is about asking things, so for now I’ll tackle that. When it comes to questioning users, we seem to use closed questions more often than necessary, and this kills off a lot of potential feedback, besides providing us with data which is at best meaningless and at worst misleading.
Closed-ended questions and when they work
A closed-ended question is one that has a limited set of possible answers – like “Yes” or “No” or choosing between options A, B or C. By contrast, an open-ended question is one whose answer can be given in free-form – that is, the respondent can write or say whatever he wants.
Closed questions are really great for quantitative research, given that they’re easy to analyze statistically. The thing is, most consumer research is qualitative, hence closed-ended questions are not a good match.
Ideally, don’t ever mention numbers or percentages unless they’re statistically significant. People want easy data and easy conclusions – give then a number, and they’ll just repeat it endlessly instead of actually analyzing the data.
Say you’re running an usability test on your new mobile app; it’s a research session with six users. It’s pretty meaningless to say that 50% percent of users answered “yes” to a question with a sample size that small. Without statistical significance, statistical insights are meaningless; and for statistical significance you will need a big sample size, controls, and a carefully designed research procedure.
Not only that, but closed-ended questions stop the conversation and can be really, really coercive. When you ask such questions, you often limit the possible answers to the things that you believe are true, or at very least to the things that you expect – this closes the doors to insights outside what you already think will happen during that experience. The coercive effect also cannot be understated: consumers will always favor the answers that are easier to give and seem to fit what you expect, and it’s easy to see the kind of answer you want to get with a closed-ended question.
The magic of open-ended questions
Does that mean that all non-qualitative research is meaningless? Surely not. Get then to talk in depth about their experience with the signup process instead of asking “are you satisfied with the signup process? (Y/N)”, and you may get some very valuable data here. That should be your primary concern – getting great qualitative insights by making people talk freely.
That’s where open-ended questions really shine – they allow you to get a lot of awesome information that you wouldn’t otherwise get in any other way. When you get people to explain their behavior, they often reveal their mental models, feelings, expectations, concerns, etc. This all is very, very valuable. Let’s compare that a few closed-ended questions with open equivalents:
Did you find what you were looking for? (before) Can you tell me how you would find that?
Can you tell me when you find it?
(after) Where was the item?
What else did you find there?
Did this work as you expected? What would you expect to happen when you do this?
Are you satisfied with this? What did you find annoying or confusing here?
What worked well for you?
Did you find this information useful? What do you expect to be here?
What things you think would be useful to have here?
Much better, right? Those questions will actually have people talking about things instead of replying with just a polite Yes or No.
On being open-ended
- start questions with “how” and the “w-words” – what, when, where, who, which.
- …but do not ask “why”. The problem with “Why” is that people will quickly rationalize explanations when asked directly instead of telling a more emotional, descriptive narrative. Use “tell me about it”, “what happened?”, etc instead.
- prefer collecting short stories instead of one-sentence answers.
- if you ask a closed question, you can open it up by quickly following up with an open-ended question, like “what else did you notice?”, “in which ways did you find this interesting?” and such.
- practice so that respondents don’t find your questions pushy – this will often happen if they feel they’re being pressured or manipulated, and will severely compromise the quality of given answers. Learn how to cultivate a feeling that they can talk freely about the topic.