This course will cover the basic elements of designing and evaluating questionnaires. We will review the process of responding to questions, challenges and options for asking questions about behavioral frequencies, practical techniques for evaluating questions, mode specific questionnaire characteristics, and review methods of standardized and conversational interviewing.
Response Process: Comprehension
Loading...
Del curso dictado por University of Michigan
Questionnaire Design for Social Surveys
236 calificaciones
University of Michigan
236 calificaciones
Curso 3 de 7 en SpecializationSurvey Data Collection and Analytics
De la lección
Unit 2: Response Process
Comprehension; Retrieval; Judgment; Response
Conoce a los instructores
Frederick Conrad, Ph.D.Research Professor, Survey Methodology
Institute for Social Research
Frauke Kreuter, Ph.D.Professor, Joint Program in Survey Methodology
Adjunct Research Professor, Institute for Social Research
[BLANK_AUDIO]
This is the first of four segments on the survey response process.
Today our focus is on comprehension.
If you think about it, a respondent needs to go
through at least these stages in order to answer a question.
The respondent needs to understand what it is they're
being asked to do, so there's a comprehension stage.
They need to retrieve relevant information whether
the question is about behaviors or opinions.
They need to supplement what they can recall
with a series of judgment and estimation processes.
And when the result of those first three stages or
sets of processes don't correspond to response options that are offered
they need to select a response option by mapping what's
in their heads to the options that have been offered.
In a model like this
the stages are presented sequentially
but respondents can back track. So for example, if a respondent is in
the process of retrieving, let's say, doctor's visits
and it's becoming increasingly difficult as the
doctor visits are going back further in
time, respondent might think, "There's no way
they intended for me to actually
recall or count up all my doctor visits over this long time period."
"Let me go back and reconsider the
question, reconsider what I'm being asked to do."
So, there's the potential to backtrack through these stages.
Ideally, a respondent will perform each of these stages
for that problem, but much of what we're focused on
is the kinds of problems they encounter.
So they might misunderstand the question in
the comprehension stage and actually try to answer
a question other than the one that the
authors of the questionnaire intended them to answer.
They, when it comes to retrieval, there might be nothing to retrieve.
They might not have actually encoded or recorded.
The relevant events in their memories.
And then there's probably nothing that can be done to bring them to mind.
They might've recorded the events in their memories, but have forgotten them.
So, forgetting is a problem when it comes to retrieval.
They might take shortcuts.
So, one class of shortcuts is called
satisficing, which is the tendency to give
acceptable but not optimal answers.
For example, if there's a long series of response options,
respondents might not give full attention to all of them.
They might, instead, give more attention to the earlier response options.
Or they might engage in what's called acquiescence,
which is the tendency to give positive answers,
so to say "I agree"
in agree/disagree questions, when they might not fully agree.
And they might intentionally misreport answers.
Giving answers that are more socially desirable to
questions on topics that are considered to be sensitive.
Our focus today is on the comprehension process,
so the first of those stages in the model we just looked at.
There are problems in understanding survey questions of at least three sorts.
Some have to do with the words and
what they mean and these are called lexical problems.
Others arise when the individual words are combined
into an overall sentence or, in our case, question.
And this is what's known as the literal meaning of the question which is
often contrasted with the meaning that respondents
come up with in doing pragmatic processing.
So, pragmatics have to do with going beyond the literal meaning
and inferring what the speaker really intended the listener to understand.
So, a set of inferences about what was really intended
is what we mean when we're afraid of pragmatic processes.
Here's an example of what we mean by lexical processes
and the kinds of problems that arise when doing lexical processing.
So the ideas that words may have many meanings even to the same person.
In the study by Suessbrick and her
colleagues, they asked respondents about their tobacco use,
administering a set of questions from
the current population survey's tobacco use supplement.
And they followed the interview with a post test,
to get at how respondents had interpreted the questions.
This was a multiple choice
post test,
which they presented possible interpretations of the questions.
So the graphic here presents the proportion of respondents who endorsed
each of three possible interpretations they'd been offered for the phrase "smoke a cigarette"
in the question "Have you smoked at least 100 cigarettes in your entire life?"
So, as you can see, while more than half
the respondents endorse one interpretation, even just one puff,
the rest of the respondents, the other half
roughly, are evenly divided between the other two interpretations.
So the point is, the same phrase can be interpreted
by respondents and people in general in quite different ways.
This has implications for the quality of the answers that respondents provide.
So, Suessbrick and her colleagues,
after conducting the interview and administering the post
test on interpretation, they readministered the original questions
and gave half of the respondents a definition.
What they were interested in was whether giving them a definition led to
more change between the first time they
answered the question and the second time
then when no definition was given.
And as you can see in the figure, this was the case, giving respondents definitions,
the two bars on the right, led to more change than not giving them a definition.
And the implication is that respondents initially interpreted the
questions in a way other than what was intended
so that they changed their answers when they were given a definition.
In fact, this can lead to problems
beyond the particular question where misinterpretation occurs.
They found that 10% of respondents changed their answers on filter questions.
Filter questions determine what the following
sequence of questions is for a respondent.
And the overall point is that variation in how respondent interpret individual words
can affect the answers they give, can
affect the accuracy of the answers they give.
When interpreting words in a question, respondents are
affected by the context in which those words occur.
So one type of context is question context or questionnaire context.
More specifically, the information that respondents use
to answer prior questions in the same questionnaire
can affect the way they interpret the question they're currently being asked.
Here's an example from a study by Schwarz and his colleagues.
They found that if they asked respondents about their overall life satisfaction
after they asked them a more specific question about their marital satisfaction,
that the answers to the two questions were highly correlated, as if the respondents
continued to think about marital satisfaction when
answering the more general question about life satisfaction.
If they reversed the order of the question and asked
the more general question first, this correlation dropped substantially to .32.
The idea being that when you're asked a question about
overall life satisfaction you can be thinking about any number of
dimensions of life satisfaction
and so you're less likely to be thinking
specifically about marital satisfaction when that question follows.
Another type of context is visual.
So, for example, this is particularly an issue
in web surveys where it's common to use images.
Cooper and his colleagues showed that
visual context can affect respondents answers
in particular, the way they evaluated their own health.
So they presented, along with a question asking
respondents to rate their health, images of either a
person who was clearly healthy and fit or
a person who was sick and not very healthy.
So here's an example of the question "How would you
rate your health?" accompanied by an image of a healthy, fit
jogger.
It is also appears to the left side of the question.
Here is the same question but presented with
an image of a person in a hospital bed.
She's clearly not healthy.
What they found was that, if the
image of the healthy person accompanied the question,
respondents rated their health lower than if the
image of the sick person accompanied the question.
And the idea is that respondents were comparing their
health to the health depicted in the image, so
that if the image depicted a very healthy person
they might say, "Well, I'm not quite that healthy."
So they'd rate their health lower.
But then, if the image depicted somebody who was clearly
not healthy, and say, "I'm doing better than that, I'll rate my health higher."
So we see this pattern of results when
the image appeared to the left of the question
and when it appeared on the prior screen.
Actually this pattern appears to reverse but it really goes away
when the image is presented in the banner, in the middle of and the top of the page.
This is referred to as banner blindness and
the idea is that respondents and web users in
general really don't fully process the information when
it's presented in the center and at the top.
But when the images are on the prior page or immediately to the left of the
question, respondents take this into account in
interpreting, in this case, what's meant by health.
Semantic processes, involve assembling the
meanings of individual words into an overall question or sentence meaning.
Here's an example of the kind of problem that can come up
in a semantic process in which respondents don't quite know how to
map their circumstances or how their circumstances correspond
to the concepts being asked about in the question.
"Have you purchased or had expenses for household furniture?"
A respondent answers: "I purchased a floor lamp.
Does that count as household furniture?"
So the idea is that respondents are doing
a fine job in interpreting the individual words.
What they don't know is whether the concept should
include this particular idea of purchasing a floor lamp.
In a standardized interview the interviewer would not
be able to clarify this for respondents.
In a conversational interview, the interviewer
could communicate to the respondent that in this
survey, "The following definition applies."
Here are data that we've seen previously concerning response
accuracy for this type of question and set of
circumstances, which are called
complicated mappings, when interviewers either
can provide clarification, conversational
interviews, or cannot, standardized interviews.
And as you can see, accuracy increases substantially when
interviewers are able to clarify what's meant by, in
this case household furniture, whether a floor lamp should
or should not be included, versus when they can't.
The third set of processes we'll discuss are called
pragmatic processes or pragmatic inferences.
And they come mostly from or our understanding of them
is mostly due to what Grice called the cooperative principle, which is
the way that listeners can make sense of what speakers are saying
by inferring what they really meant.
Grice talked about the cooperative principle in terms
of four sub-principles or what he called maxims.
The maxim of quantity: Say as much, but no more than necessary.
The maxim of quality: Do not say what you believe to be false.
The maxim of relation: Be relevant.
And the maxim of manner: Avoid obscurity and ambiguity.
So the idea is that by keeping this principle in mind, one can
make sense of what a speaker says when it might not make sense literally.
So if a speaker were to ask me, "Can you pass the salt?"
I would probably not think of that as
a question about my ability to pass the salt
even though that's literally what it is.
Instead I would probably make the almost automatic
inference that she's asking me to pass the salt
as a request to pass the salt
and I'd presumably comply.
Related to this is the idea that the speaker would
not say something if she didn't intend to inform me.
And so I would take that into account in interpreting the rest of what she says.
So how might pragmatic inferences play a role in answering survey questions.
This study by Schwarz and colleagues is a good example.
They asked respondents, "How successful would you say you have been in life?"
And asked them to answer on a scale, not at all successful to extremely successful.
For half the respondents
the verbal label "not at all successful" was acompanied by a minus five
and "extremely sucessful" by a plus five. For the other half of the respondents the
label "not at all sucessful" was acompanied by a zero and "extremely sucessful"
by a ten. How do you think this might have effected the way respondents answered?
Well, 34 percent of the 0 to 10 group responded in the lower half, that is the 0-5
part of the scale, the less successful part of the scale, but
only 13% of respondents in the -5 to +5 group
responded in the lower half of the scale. So respondents were less willing
to position themselves in the unsuccessful part of the scale, when there was
a negative number attached to the verbal label
"not at all successful."
The author suggests that when there is a zero associated with
the label, respond and interpret that as the absence of success,
zero success.
But when there's a negative number associated with "not at
all successful," they interpret that as the presence of failure.
So this is a case where respondents are going beyond the literal
meaning to make sense of what's, presented in the question.
So they're using these numbers to interpret the verbal labels
when this was probably not the intention of the questionnaire designer.
Okay, so what are some of the implications of the
kinds of processes and problems we've been discussing for questionnaire design?
Here's some possibilities.
With respect to lexical issues, try to use
terms that most people interpret the same way.
This may be very difficult and
will require pretesting to confirm your intuition
that the terms you've chosen are interpreted in a uniform way.
But trying to find terms like this is certainly a worthy design goal.
With respect to semantic issues, providing definitions or either
that respondents can click to view in a web questionnaire
or by training interviewers to provide those definitions
as needed should help address the kinds of ambiguities
we talked about before when attempting to map
one's own circumstances to the concepts in the surveys.
And with respect to pragmatic issues, a good goal would be to
try to block respondents unintended inferences,
for example by avoiding gratuitous design features.
That's a brief overview of comprehension issues in the, response process.
When we resume we'll talk about memory and recall issues
and how they can relate to error in answering survey questions.
Explora nuestro catálogo
Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.
Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.
© 2018 Coursera Inc. Todos los derechos reservados.
