In this module, we're going to be talking about designing nominal closed-ended questions. We're going to be learning some general guidelines and tips for writing these types of questions. As a reminder, nominal closed-ended questions are questions where responses are on ranked categories. Yes/no, pick among this list of items, and things like that. These are in a position to ordinal ranked questions, which are things like satisfaction scales, likert scales, those questions that you would imagine. So, tip number one when writing nominal questions, is to ask respondents to rank only a few items at once rather than a long list. Now, this gets into basically a respondent burden again. How many questions you can ask is of course dependent a little bit on your mode. If you're doing a phone survey, you can ask fewer items because people can only hold so many items in their head. If you have written down your survey either on the web or on paper, then you can have a longer list, but people may find it to be more work to still respond to that many items. So, a really simple of course response categories two options, Yes/no. Are you worried about being harassed on social media? Yes or no. The binary choice here is cognitively easier for respondents to answer. They understand the question hopefully and they are able to respond quickly. Now there's two ways that you can introduce burden to this set of response categories. One is to really complicate the response categories. The other is to add a lot of different response categories to the option set. So, what would it look like if we complicated the response categories? So, which of the following best expresses your feelings about harassment in social media? Then you can see for fairly complicated response options that people have here. Now, this set of possible answers is much more complex and it is going to require more work on the part of respondents to understand what I'm asking about, to understand what's the differences between these different response categories, and to really be able to give a good answer if they care enough about giving a good answer. It's very possible that a respondent would look at this large set of questions with its complications and just move on to the next question or quit the survey. So, how do you balance really getting good detailed information out of your respondents through closed-ended questions like this? Well, incentivizing them to continue and to reduce their cognitive burden for responding. In other way, you can add cognitive load to your respondents is to have a lot of different categories for them to respond to. So, here's an example, which of the following social media sites do you think has the most trouble managing harassment? Then I have eight social media sites that they can pick from. Now, in this list, I'm asking them to do a rank order. So, what they would do, is they would actually move the social media site up or down, but I'm really depending on a couple of key insights about respondents that may or may not be true. One would be that they know what all of these social media sites are and have experienced enough with them to say. Most people aren't going to know for instance, the differences between Snapchat and Instagram well enough to be able to rank order them. Ello is fairly obscure social media site and many respondents may not be able to place that in an appropriate rank order. So, one type of cognitive load I've added to this question is, how do I actually answer this when I don't know all of the sites? The other is, there's a lot of these, and what's the difference between a four and a five in this case? When you have a large list like this, a lot of respondents again will satisfies. They only move a few of them in order to change the rank order, but they might think that this rank looks good enough and only change a couple of things, move Twitter up, or Facebook down, to the bottom or whatever it happens to be. You're generally going to want to randomize this order to prevent that satisfies and we'll talk more about order effects in a couple of slides. Another cognitive load I've added here is that they actually have to manipulate the interface. There's lots of different ways I could put the response categories in for this question. So, let's look at a different mode for the respondent to work on. Again here, what they would do is they would pick one of these social media sites that they would use the arrows on either side of this text box to move that social media site up or down the list. Now, this is the same exact question. Does it reduce or increase the cognitive load for the respondent? It maybe a little bit easier for them to actually understand once they see this what to do, but at the same time, it's a lot of clicking the arrows and clicking back on the social media site and arranging things. So, to me, it feels like a wash, but it's just a different format. The third format altogether is for them to put a number next to each social media site to do ranking. Now, this is a really cognitively expensive thing for them to do. They have to actually add the numbers and type them in, a lot of clicking, as well as a lot of thinking about what the differences are between these different social media sites. This would of course be an appropriate way to do this for a paper survey where you're not going to have the flexibility to actually use the interface to move options, but in a web survey, this is going to add a lot of different cognitive burden to the respondent. The nice thing that it does is it forces them to assign a number to every social media site, which will get us out of satisficing that we saw in the first response category. So, picking one of these response categories, you can see is not an easy choice. There's no clear right way to pick a format for your response category. You're really just trying to again get into that respondent mindset, think about how they're going to experience these response categories, and do the best you can to balance out the needs you have to get your concept answered, and the needs that the respondent has to reduce their cognitive load. The second tip for writing nominal closed-ended questions is to avoid unequal comparisons in your response categories because they can reduce bias. So, what's an example of this look like? Which of the following do you think is most responsible for teen bullying on social media? Irresponsible parents, lax school policies, lenient social media site enforcement, bad social media literacy. Now in each of these cases, you can see that there's been an adjective attitude added to them. In general, be very careful having adjectives in your response categories. Now, you might think, ''Well, I'm interested in irresponsible parents not responsible parents, or I'm interested in school policies that are lax versus those that are strong around bullying.'' But that then is adding double-barreled questions and really complicating the response categories that you're offering in this particular question. You want to really drill down to what is the concept you're interested in. So, for instance, if you're interested in just this as a nominal question, then you reduce those adjectives and just say parents, schools, social media sites, literacy. That will get you a much clearer less biased answer than in the previous question. Now, if you're interested in a slightly different construct and you're interested in more of a ranking of these things, then you would convert these to a series of ordinal questions. So, instead of asking this question, you might ask, how responsible do you think parents are for teen bullying on social media and have an ordinal ranking from various possible to not at all responsible? That would lead to an entirely different outcome of data, but it might be better data given what your research goals are. In third tip for writing nominal closed-ended questions is to randomize response options if you're worried about introducing bias through the order of your response options. So, let's look at our example from a couple of questions ago. Which of the following social media sites do you think has the most trouble managing harassment? Then we have a whole passel of options that people can pick from. Now, there's two quirks of human cognition that you need to be aware of when you look at a list like this. One, is about primacy and recency. So, when we look at something in this case Facebook as the first option, Facebook is going to be the prime or the most recent thing we think about. If we read LinkedIn, that'll be the most recent, whereas Facebook is the primal thing that we're thinking about. That shapes and add signposts basically to how we think about all of these other options. Anchoring is another cognitive cog where an early option becomes the comparison for other options. In this case, what would happen is that Facebook would become the comparison point for all of the other social media sites that we have listed here. In good survey design, what you would do is just randomize these for every respondent. So, every respondent is going to see a different ordering of this list so that you're basically taking out order effects as an explanation for results that you're getting. Again, most software applications that you would use allow you to very easily set up this kind of randomization of your response categories. So, a fourth tip for writing nominal closed-ended question is to use force-choice questions instead of check-all-that-apply questions. What does that look like in practice? So, the really common type of question that you might ask respondents is, which of the following items do you have? Please check all that apply, because we're interested in for instance in how many computing devices respondent has in their home, or how familiar they are with technology as proxied by their ownership of these kinds of technologies? So, this is great. This will provide you with some data, people will select desktop computer or e-reader and you'll know something about them. This type of format is great for saving space. You can write this question, you just put in your response options and the respondent hopefully picks the right set of things. Unfortunately, these turn out to be cognitively tough for people. If you look at the research literature on this, what happens is that people look at this list and recency effects kick in again. What they're going to think about instead of the question itself is, what was the most recent device that I used? If it was my laptop computer, I might pick that, but forget about my tablet that I use three days ago. Remember that human memory is pretty bad. So, these recency effects can really shape and bias the results that you get. A better way to ask these is to force a choice. So, here's the same basic question with a force-choice set of responses, and where each choice has to be made explicitly by the respondent. So, do you have a desktop computer? Yes or no. A laptop computer? Yes or no. What this is going to do is trigger them to think about each of these separately without considering the whole set of these options as they get started. You're going to get better data, but of course, it's going to take up more space in your survey, and it's going to take a little more time for the respondents to answer. It's cognitively easier for them to answer this type of question, but it still is going to be a little clunkier than the previous mode. So, in summary. Nominal closed-ended questions are really common in writing survey questionnaires. A few things you want to keep in mind when you're writing your nominal questions. Having more options for selecting and ranking can be confusing for people. You could have anywhere from 0-100 different options for people. What's the right balance? That is a tension between how much data you want to get, and how cognitively expensive it is for respondents to respond to your survey. You want to avoid unequal response options. You want to make sure that all of the responses that you have are not biasing the results that you get. You want to be very careful about constructing your response categories to remove your biases or any potential biases based off the wording of those response categories from your results. You want to randomize your response orders where things like primacy, recency, and anchoring are in play. If you have a large list of options, it's almost always best to randomize those options because otherwise people are just going to pick the first one and you're going to bias your results by the order, not just by the wording. Another tip for writing nominal questions is that you get more responses by forcing a selection. Mark all that apply types of questions are easy to write and layout wise they look great, but they can be cognitively expensive for respondents and lead to satisficing that cause people to skip over really important options. Forcing an option is great for respondents to really carefully consider each of the options that you want them to think about. So, nominal questions in survey design are wonderful and can lead to lots of great data. In the next module, we're going to talk about ordinal questions, or likert scale questions, which are another way of getting at concepts that are important to us when we're doing UX research.
