Finding Persistence: Regression to the Mean (People Analytics)

Loading...

Del curso dictado por University of Pennsylvania

Business Analytics Capstone

357 calificaciones

Explore Related Videos

At Coursera, you will find the best lectures in the world. Here are some of our personalized recommendations for you

University of Pennsylvania

Business Analytics Capstone

357 calificaciones

Curso 5 de 5 en SpecializationBusiness Analytics

The Business Analytics Capstone Project gives you the opportunity to apply what you've learned about how to make data-driven decisions to a real business challenge faced by global technology companies like Yahoo, Google, and Facebook. At the end of this Capstone, you'll be able to ask the right questions of the data, and know how to use data effectively to address business challenges of your own. You’ll understand how cutting-edge businesses use data to optimize marketing, maximize revenue, make operations efficient, and make hiring and management decisions so that you can apply these strategies to your own company or business. Designed with Yahoo to give you invaluable experience in evaluating and creating data-driven decisions, the Business Analytics Capstone Project provides the chance for you to devise a plan of action for optimizing data itself to provide key insights and analysis, and to describe the interaction between key financial and non-financial indicators. Once you complete your analysis, you'll be better prepared to make better data-driven business decisions of your own.

De la lección

Module 3: Your Strategy

In Module 3, you will focus on creating your recommended strategy for GYF to address adblockers. Your strategy does not have to be lengthy, but it must be clear, and it must address the problem. (Hint: if you have a clearly defined problem, your strategy is much more likely to be clearly defined as well). You'll be submitting your strategy for peer review, and then also reviewing the work of at least 3 of your peers. It's OK if reviewing the strategies of other learners in this course gives you further ideas for revising your own strategy. One of the primary benefits of peer review is to expand the range of feedback you can get, and we designed this Module around peer review so that you can get as much feedback as possible before moving on to the next phase of the project. You may find the resources and lectures below helpful in formulating your strategy and considering how data can be leveraged and appropriately understood. You are strongly encouraged to complete and include your response to Application Exercise 2 as part of your Strategy.

Performance Evaluation: the Challenge of Noisy Data (People Analytics)6:46

Finding Persistence: Regression to the Mean (People Analytics)11:56

Extrapolating from Small Samples (People Analytics)5:53

The Wisdom of Crowds: Signal Independence (People Analytics)5:56

Process vs. Outcome (People Analytics)7:47

Hiring 1 (People Analytics)8:50

Hiring 2 (People Analytics)9:07

Conoce a los instructores

Wharton Teaching Staff
Educators
The Wharton School

In that extended example,

we saw the value of doing good analytics with these performance measures.

What first appeared to be significant differences in skill turned out to be

purely chance.

That really emphasizes is importance of persistence.

We need to find ways of testing for persistence.

In that case we just looked at split sample,

how performance varied in one year and then how it varied in the next year.

Finding ways to do that,

is one of the most fundamental ways you can parse signal from noise.

We're going to focus on four additional issues for

the rest of the module, regression to the mean, sample size, signal independence and

process versus outcome.

These are all important concepts to have in mind as you dig into your data, and

they're also issues that we tend to have if we only reason about data intuitively.

They're issues that data can improve, analytics can improve, but

analytics aren't a panacea, you can still make these mistakes even with data.

Let's drop into them.

The first is regression to the mean.

And I want to start that with a very Simple Model of performance

where you can think of performance in terms of Real Tendency + Luck.

And we've been talking about this a little bit, we can formalize it and

don't get too put off by the baby math here, but in formal terms you

can think of performance y as a function of x true ability and

e some error, some randomly distributed error around 0.

Now, what does that mean for when we sample on extreme performance?

What underlies extreme success and failure?

If that's the model of the world, and everything we've been saying so

far says it is, that there's some noise in these performance measures,

what does it mean when we sample on extreme performance?

Well, it means that extreme success

suggests that the person might in fact have superior ability or

tried very hard, but also that they got lucky, that error was positive.

And conversely, extreme failure perhaps means inferior ability or

that they did not try very hard but also negative error or that they got unlucky.

We can be sure that as we sample very extremely on performance measure,

a noisy performance measure, and they're all noisy,

we can be sure that when we go to the extremes, we get extreme error as well.

What are the consequences of that?

There's one very important consequence, and that is in subsequent periods,

error won't be negative again, it will regress to the mean.

You'd expect it to be zero.

Error is, by definition, zero.

And if you got very positive error in one period,

you would expect less error in the following period.

This is a notion called regression to the mean, and

it's one of the most important notions in performance evaluation.

An example.

There was a study a few years ago of mutual fund performance, in the 1990s.

The study divided the 1990s into two halves.

1990-94, and then 1995-1999.

And they looked at the top 10 performing funds from the first half of the decade.

And here, I'll show them to you.

We anonymized them.

This is just supposed funds A through J, and their performance in the early 1990s.

There were 283 funds in this study.

These were only the top 10 performing funds.

Then they did two things.

They go and ask how do these funds perform in subsequent years?

And they did an interesting thing in between.

They ask people, what do they predict happened in the next few years?

What do they think performance would be realized in the second half of the decade?

Here are the predictions, the estimations from the people that they ask.

They thought the top performing,

they didn't think the top performing firm A would again be the top performing firm,

but they thought maybe tenth and so on down the list.

E which is the fifth performing firm they thought, well, maybe 44th and so

you can see that they didn't expected the firms to be as good, but

they expected some regression to the mean.

Then they looked to what actually happened.

What actually happened?

It ranged from 129th, 21st, 54th.

The interesting this is that on average, the firms performed.

Their rank was 142.5.

What is the significance of 142.5?

It's half of the total number of firms in the study.

In other words, the average performance of the top 10 firms, in the second period,

the second half of the 90s, was perfectly average for this sample.

They've regressed entirely.

The top 10 mutual funds in the top half the 90s and

of early 90s regressed entirely to the mean in the second half of the 90s.

If that's the case ,what does that say about how much skill versus luck

was involved with how those firms did in the first half of the 90s?

If they regress all the way to the mean in the second period,

it suggests that there was no skill.

That the differences that we saw, and

there are huge consequences to those differences because we know that new

funds flow to successful funds, were in fact, entirely based on luck.

There are many other examples.

Danny Conoman, Nobel Prize winner Danny Conoman gives a famous example of being

an officer in the Israeli Air Force.

He was studying the officer in the Israeli Air Force,

this was early in Conoman's career.

And the officer told him, punishment is more effective than praise.

Whenever I punish a pilot after a really poor flight,

I see better performance the next time.

Whenever I praise a pilot after an excellent flight,

I see worse performance the next time.

Therefore, it must be that punishment is more effective than praise.

What's the more parsimonious explanation?

The more parsimonious explanation is that there's a little chance involved with

whether a pilot has a good performance, a good flight or a bad flight.

And after a good flight, if there's some chance involved there,

you would expect that the following flight wouldn't be as good and conversely.

After a bad flight, if there's some chance involved, you would expect,

you would predict that the next flight would, on average, be better.

This is exactly why we have to be so careful about regression to the mean.

We have the wrong model of the world if we don't appreciate regression to the mean.

We walk around like the Israeli Air Force officer who believed that it's all about

praise and punishment as opposed to merely statistical regression to the mean.

There's another example which comes from,

we're not going to pick on Israeli Air Force officers.

One comes from Tom Peters one of the original business book Peters and

Waterman were McKenzie consultants no less.

And they did a study and it begins as an internal study and

they've eventually published it as a hugely best selling book,

on what determines excellence in companies.

They selected 43 high performing firms and tried to learn what they could

about business practices from these top 43 firms.

But, subsequently a few folks evaluated the performance of those 43 firms,

and what do they find?

5 years later, there were still some Excellent Companies.

And there were some that were solid, but not exactly the top of their industries.

And then there were quite a few in weakened positions, and there were

even some from these supposed 43 excellent companies who were fully troubled.

Now, this is exactly what you'd expect from regression to the mean.

And that suggests that, that sample, that Peters and Waterman had grabbed,

as supposedly excellent firms, they grabbed them.

Perhaps they were on average, a little bit better.

But they hadn't necessarily been lucky.

To make it into that sample, to be called the most successful of 43 firms in

the world, essentially, they were necessarily lucky, and

in subsequent periods, they're not going to have luck break their direction.

This is something that you'll see anytime you sample based on extreme values that

if you sample on one attribute, any other attribute that's not perfectly related

will tend to be closer to the mean value.

We've been talking about performance at points in time.

If you sample on extreme performance at one time period, the subsequent time

period won't be as extreme whether you sample extremely good or extremely bad.

But it can also be attributes within an individual or within an organization.

If you sample, say, a person's running speed, and

then look at what their language ability is, these things are imperfectly related.

If you only looked at the fastest runners,

how would you expect them to perform on some language ability test?

They wouldn't be, they wouldn't be a sign.

The fastest runners will almost find definition

not necessarily be the people with the best language ability.

But that's not because their some inverse relationship between running and

language ability is that this two traits or simply imperfectly correlated.

And so when you sample on the extreme,

you have to expect regression to the mean on any other attribute.

We could spend a day on regression to the mean.

In fact, there aren't many concepts that are more important

in understanding the world than regression to the mean.

We could spend hours on this.

And, I would be very happy if you walked away from this course with only two or

three ideas, if this was one of them because it's going to help your reasoning

about the world.

Why is this so hard?

Why is this such a hard concept to stay, to live?

Well, there are a few things to get in the way.

Among others, we have this outcome bias.

I mentioned it with Hershey and Bear, and referenced Hershey and Bear, and

they're the ones that came up with the study originally.

We tend to believe that good things happen to people who worked hard,

bad things happened to people that work badly and

we draw too strong an inference based on this.

We tend to judge decisions and people by outcomes and not by process.

This is a real problem and it gets in the way of our understanding

this regression to the mean, framework for the world.

Two others, one is hindsight bias.

Once we've seen something occur we have a hard time

appreciating that we didn't anticipate it occurring.

We in fact often misbelieve

that we anticipated that's exactly what would happen.

We show hindsight bias.

And again, if that's the way we reason about what happens,

then we're not going to appreciate that what happens next just could possible be

just regression to the mean.

And then finally narrative seeking, we want to make sense of the world,

we want to connect the dots.

We tend to believe things better if we can tell a causal story in between

what took place at time one and what took place at time two.

And if we can tell a causal story, then

we actually have a great confidence in our ability to predict what happens next.

We seek these stories as opposed to what I've been telling you which is this dry

statistical reason for why things happen.

We seek stories.

And that again gets in the way of our understanding

the statistical processes that actually drive what's going on.

In short, we make sense of the past.

We are sense making animals, and we make sense of the past, and

there's not a lot of sense to be had for merely regression to the mean.

But it's going to it's going to get in the way of predicting what happens next.

We try to find stories that connects all the dots, and

we by doing that get a chance to smaller role and those stories.

There was an Internet meme that captures these well a year or two ago where,

if this is knowledge distributed in your experience in the past this is knowledge

you might have and with that knowledge perhaps you can add some experience and

start connecting the dots, drawing simple eyes,

create something from that knowledge.

That's good.

That's what we want experience to do.

Bu then sometimes we're inclined to do this,

which is get a little too creative and over fit those lines.

We turn what should be a pretty straight grid, pretty parsimonious

connections into something that is unlikely to replicate in the future.

It might be very satisfying, might be a very satisfying interpretation of

the past, but it's over fit, and an over fit interpretation of the past is going to

make very bad predictions about the future.

Explora nuestro catálogo

Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.

Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.

© 2019 Coursera Inc. Todos los derechos reservados.

Descargar en la App Store

Consíguelo en Google Play