4.3_A Review of Validity

So we're going to go back to where we started that first week and

discuss how you can be a critical consumer of the research and

the media's interpretation of that research.

In your reading this week, I asked you to review a research article,

Eavesdropping on Happiness, and then a popular press article reviewing it.

We'll use this example to talk about how to be a critical consumer of the research

and the media.

Let's talk a bit about the study first.

The authors in this Eavesdropping on Happiness study

were interested in whether conversation type was related to well-being.

They observed 79 students naturally without direct interference.

They did this by using a digital audio recorder which allowed for

unobtrusive tracking of 300 waking recordings per student.

And they did this by 30 second recordings every 12.5 minutes over four days.

Then they coded conversations and

differentiated between different types of conversation.

So they looked at alone time, talking to others,

small talk and substantive conversations.

Then they assessed students' well-being using self reports, informant reports and

the satisfaction with life scale.

Students also took the big five inventory twice

to capture information on personality.

Scores were averaged with informants ratings of students on the same inventory.

The results of this study demonstrated that higher well-being was associated with

more conversations and less alone time.

Higher well-being was also associated with more substantive conversations and

less small talk.

These results remained consistent even when controlling for

personality type and whether data was collected on weekdays or weekends.

So the happiest participants were alone 25% less of the time.

And spent about 75% more time talking.

So how should we go about assessing this research study, and

the media's summary of it?

Well, remember the concept of validity, which means truth?

As John Keats puts it, beauty is truth, truth beauty.

That is all you know on earth and all you need to know.

We've talked about three types of validity which are critical to evaluating research.

Internal validity, external validity, measurement validity.

And today we'll introduce a fourth type, consequential validity.

First, internal validity.

Internal validity, as you'll recall, captures whether the study

can demonstrate that the independent variable causes the dependent variable.

And there are two main questions to consider when looking at

internal validity.

First, the direction of the effect.

So can you demonstrate that the independent variable

leads to the dependent variable, or could it be the other way around?

And second, could there be other variables that could explain the outcome that

you didn't actually measure?

So in the case of the Eavesdropping on Happiness article, did having more

conversations and less alone time actually lead to higher well-being?

Or was it the other way around?

Did having higher well-being actually lead you to

have more substantive conversations?

Or was there another variable, altogether,

that the authors didn't measure that led to well-being.

Now, the authors did control for some other personality variables captured

through the big five but, certainly not all of them.

So there could have been another variable.

I'm sure you all could think of many that could lead to increased well-being

that this study didn't assess.

And the authors can't account for other factors or

the direction of the effect because there wasn't random assignment to groups.

So this means we should be interpreting these results as correlational not causal.

So how well did the media article do?

Well, in the opening lines of this popular press article the author states that,

new research suggests that less small talk and

more substantive conversation causes increased happiness.

Later he does go on to state that correlation doesn't necessarily prove

causation.

Which was an important observation, but he does use misleading language throughout.

So he uses language like conversation may galvanize a previously unexamined life.

Implying causation.

Now this is one of the most important things to look for

in a popular press article.

As the media will often overstate findings as causal

when the study was only correlational.

Why you might ask?

Well because it certainly makes for a flashier article and better headlines.

The second type of validity to consider is external validity.

External validity captures whether the results will translate to other

populations.

So, for example, we need to consider if and

how results will translate to different settings, people, places, and times.

The Eavesdropping on Happiness article only looked at undergraduates,

which means there are limitations in external validity.

So the question remains, could this even be applied to non-undergraduates?

Or is there something about the types of conversations you have in college

that are associated with your well-being that wouldn't apply outside of college?

Now this is something that the media article doesn't address at all and

is oftentimes another one of the limitations that you'll find in these

types of media articles.

The media like to make it seem like the findings of an article are universal and

would apply in any setting, when often they are quite context specific.

The third type of validity we've discussed is measurement validity.

How well does the study capture its intended constructs?

The study uses validated measures with high reliability like the satisfaction of

life scale.

The big five inventory and

measures of well-being, which is the positive aspect of the study.

They also use multiple measures which increases the convergent validity.

Remember, no one measure is perfect, but using multiple measures is a solution to

get at that imperfection and reduce the error of any one measure alone.

However, the measures are still self report and informant measures,

which as we talked about in week two, have several limitations.

Perhaps there was social desirability bias and

students answered questions in a certain way.

Or perhaps there was reference bias in this group of undergraduates

meaning that they had a different reference point

about what well-being meant that wouldn't apply in other settings.

Now these limitations weren't touched on at all in the popular media article.

So as a critical consumer of the research it's something you need to look for.

And the final type of validity is consequential validity,

which refers to the positive or negative consequences of a measure or study.

We need to consider how measures should and

shouldn't be used based on their limitations.

So while this study was interesting and

useful, it's not clear how much we should make of the results.

We can't demonstrate the direction of the effect, the study was only conducted with

undergraduates, and there are limitations in self report and informant measures.

So we shouldn't go as far as to the popular media article suggestion and

conclude that real conversation

might be the necessary stimulant to galvanize a previously unexamined life.

It might be, but we'd need more research to demonstrate this was the case.

So this bring us to the end of the final video

introducing key concepts for this week.

There are a few other components for this week.

Again, researcher videos featuring Angela Duckworth applying these concepts to

her work.

Practitioner videos featuring practitioners applying these concepts

to their work.

A short quiz on key concepts, and two optional extension activities.

A peer review activity where you discuss what kind of research design

you would use to test your research question and hypothesis.

And explore the strengths and weaknesses associated with that design.

And an advanced activity where you read the popular media article on true grit and

discuss how effectively you think the author captured the key findings from

the study.

Positive Psychology: Character, Grit and Research Methods

Revisiones

Impartido por:

Claire Robertson-Kraft, Ph.D.

Angela Duckworth, Ph.D.

Explora nuestro catálogo

Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.