So far we've been looking at questions and we've been doing some really fun work looking at how do we generally frame questions. Now we're going to dive into specific types of questions and for this module we're going to talk specifically about open ended questions. We're going to learn basically about two different types of open-ended questions and how to construct them. And then we're going to learn more about how to write compelling open-ended questions. How do we actually write open-ended questions that are going to matter to your respondents? And get the kind of information that you want from your survey? Open-ended questions are kind of a bread and butter of the survey industry. And especially for UX researchers, if feels like we often depend on open-ended questions. To really get into what our respondents and our participants are thinking about, they provide a lot of flexibility. But unfortunately if there's a lot of open-ended questions, or they're vague, respondents skip over them more frequently in surveys. So we have to really think about how do we construct open-ended questions? And what are we going to need to do once people actually do respond to them? Because part of one of the issues with open-ended questions is that when we just leave it up to the respondent. What they're going to put into a text box, they can put in anything. And sometimes we need to clean that up and then code it and analyze it. So open-ended questions are easy to write to some extent, though they're trickier than you may think. And they belong in your survey, but they can often take more work on the back end. So here's three typical open-ended questions that you might see, and these are three types of open-ended. I think when you think of an open-ended question in your mind, you're probably thinking of the first one. Which we would call a descriptive open-ended ask. So this is a type of question where we have a prompt, in this case, what was your experience like using this application? And then we provide a large text box and hope that the respondent puts in lots of great information for us to use later on. There is of course also a really common one that you use but it's hard to think of as open-ended. And that's a numerical response, this asks for account of some activity. So in this case, how many years have you been using the Internet, we want a specific count of years that the person has. Then a third open-ended question that's pretty common in use is a list of items. And this is basically where we want people to list a set of experiences or things based off a prompt that we give to them. So, in all of these, we have some commonalities. We have some common things that we want to make sure that we write into our open-ended questions. The first one is that we want to specify the response desired in the question stem, right? So this is where we get into really very detailed information about how we write the question. One of the common mistakes I see is that just putting why, right? As the only prompt for an open-ended question, and I think you want to go a little more detailed than that. So here's a great example of a numerical response we got. This is from a questionnaire that I developed a few years ago when we were doing some research on Facebook. And it turns out that it's actually really hard to figure out how often a person uses Facebook. What's the right way to ask how much you use Facebook on a general basis? So the question we started off with was, how often do you check Facebook every day? Now respondents when we did pre testing on this had a really hard time answering this question. One common response was, too much, or, none of your business, we got sometimes. A lot of times we got two or three times, or, hard to say. We got a lot of data that wasn't particularly useful for us in analysis later on when we were doing the analysis for the survey. So what could we do, how do we reframe this question to be a little bit more specific? So another way to reframe this question and to ask basically the same question in a different way. About how many minutes per day do you spend actively using Facebook? Now, this is better than the previous question because it provides a common frame of reference, minutes per day. And we also have specified, do you spend actively using? But we found out that respondents were still really fuzzy about, what does using Facebook mean? Does it mean looking at the news feeds or does it mean actually posting? So it's really hard to write a really evocative open-ended question like this. So here's another example for a different type of question, what are your most important criteria in trusting an online store? This question seems pretty typical, but you can see a couple of problems immediately when you look at this. The first one is that your respondents may not understand what criteria means. Not necessarily that they don't understand what the word means, but they might not know what you're after. It's also really hard to think about a direction for this. How much information do you want? What kinds of criteria are you looking for? What exact response are you looking to get? So here's another way of framing that that's a little more useful. What are three things that are most important to you for trusting an online store? Now you can see here, we've replaced the word criteria with most important to you. And that we also now have three things to try to prompt a specific set of responses. Now using a number like this, asking for a list basically within an open question. Is great because it really does give the respondent a sense of, they want to know a list of things, right? It helps to really frame the response that you're going to get. There's a couple of dangers with a question like this, the first one might be that they're making up the third thing. That really there's only one thing. And it's hard for you to tell when you're analyzing this data whether things one and two are as important as thing three. And the other thing it might be that you are missing five, six and seven, right?. That there might be actually more than three things that they consider and you're getting a lot of false positives. Or missing data because you've asked for a specific number of criteria from the respondent. So you want to be careful with this, again, it's a matter of pre testing and thinking through, what data do I want on the back end? And really thinking about, what's the respondent experience going to be like when they see a question like this? When you're writing open-ended questions, it's also really important to provide motivation to respond. As I said a few minutes ago, it's really easy for respondents to skip open-ended questions or to, what we call, satisfice them. Satisficing basically means that they're going to give you a yes or no answer. They're going to try to answer the question with as little effort to themselves as they possibly can, right? As the interviewer, you don't really want that, you want actually as much data as you can get from the respondent. So there's this negotiation going on between you and your respondent. They want the process to be easy for themselves, you want to get as much data as you can. So how do you provide motivation so that they come more to your side of that equation? One tip is to use open-ended questions sparingly. You should not have an open-ended question after every closed-ended question, and we've all seen that, right? A series of closed-ended questions with a why open-ended after every one of them. Yes, those are easy to write, yes, you get lucky with one or two respondents who fill in a ton of information. But it's better to be targeted, you should have just as clear a purpose for every open-ended descriptive question that you have. For every other question in your survey. You also want to elaborate on why people should respond to your survey. Your prompt should not just be, why? Or it should not be, could you say more about that? You do want to provide an explanation of how you're going to use that open-ended data. So that they see that it's worth their effort to actually provide that information to you. So for instance, you can put in a prompt like, in order to improve our product for other people. We'd like to ask why you answered the way you did above, right? This is more wordy, it's a little more verbose than maybe you'd want to do. The important part is this first clause, in order to improve our product for other people. That provides, again, that social exchange we talked about several modules ago, to help incentivize people to respond to your question. Another way to get people to be more responsive in their open-ended questions is to use a series of non-biasing probes. Probes are basically follow-up questions, especially when you're doing phone interviews. They're very commonly used to get people to expand on answers that they've given to open-ended questions. There's been a lot of work done on probes in survey methodology. And back when I ran a large survey, a nationally based survey, we had a lot of open-ended questions. And we trained our interviewers to use a very specific, very limited set of probes. You noticed I say the work non-biasing here, that's because you don't want to lead a person into giving a specific type of answer. You really want their honest response, so probes like, tell me more about that, anything else, could you be more specific? Have been designed to be as neutral as possible and to unpack the information that people give in their answers. In online surveys, it can be tougher to build in probes. Now, it's not impossible, especially if some tools allow you to notice that there's very few characters. Or to look for certain keywords in an open-ended question, in those cases, you can add probes to the open-ended question. It's also possible to use programming and skip logic to be able to pull answers from previous questions. And ask more about them in an open-ended question, if you can tailor it like this. If you can use JavaScript or whatever program that you're basing your survey into ad specific probes like this. You're going to get more elaboration on the questions that you're asking and better data overall. So, with open-ended questions, it's really a fairly wonderful set of opportunities that you get as a survey researcher. You get tons of great data, open-ended questions allow you to ask about anything. Especially where you don't necessarily have a preset idea of what responses are going to be. There's three major types, the descriptive open-ended, numerical responses, and list of items. You want to make sure that you promote responses to your open-ended questions. They're great for getting surprising data, they're great for getting data you're not going to be able to guess ahead of time. But they're also super easy for your respondent to skip. So make clear why you're asking the question, probe whenever possible. And really use the open-ended question as a way to get into the why your respondent is thinking certain things. Which is almost impossible to do through close-ended questions, but really a wonderful way to get data through surveys. In the next module, we're going to talk more about closed-ended surveys and how we write those more generally. And then dive into different types of closed-ended surveys.
