Financial Engineering is a multidisciplinary field involving finance and economics, mathematics, statistics, engineering and computational methods. The emphasis of FE & RM Part II will be on the use of simple stochastic models to (i) solve portfolio optimization problems (ii) price derivative securities in various asset classes including equities and credit and (iii) consider some advanced applications of financial engineering including algorithmic trading and the pricing of real options. We will also consider the role that financial engineering played during the financial crisis. We hope that students who complete the course and the prerequisite course (FE & RM Part I) will have a good understanding of the "rocket science" behind financial engineering. But perhaps more importantly, we hope they will also understand the limitations of this theory in practice and why financial models should always be treated with a healthy degree of skepticism.
Statistical Biases in Performance Evaluation
Loading...
Del curso dictado por Columbia University
Financial Engineering and Risk Management Part II
430 calificaciones
De la lección
Practical Issues in Implementing Mean Variance
Problems with mean-variance analysis; ETFs and leveraged ETFs; VaR and CVaR for asset allocation; survivorship bias, performance evaluation and other statistical pitfalls.
Conoce a los instructores
Martin HaughCo-Director, Center for Financial Engineering
Industrial Engineering & Operations Research
Garud IyengarProfessor
Industrial Engineering and Operations Research Department
In this module we're going to discuss statistical biases in performance
evaluation. The goal here is not to build realistic
models but rather stylized models, so simpler models, but models that
nonetheless will help explain to us that biases do arise, and that biases can be
significant. When we are evaluating the performance of
fund managers. Let's get started.
Some fund managers claim to have special skills.
What do I mean by special skills? Well for example they might claim to be
very good at picking stocks or they might claim to be very good at timing the
market. And what do I mean by timing the market?
I mean the following. Maybe they know a good time to enter into
the market, in other words when to invest in the S&P 500 or the Euro stocks, or
whatever. And also they might claim that they know a
good time when to get out of the market, when is a good time to sell.
So, that's what timing the market means. This skill is often referred to
generically as alpha. And the term comes from the capital asset
pricing model. If you recall the cap end that you'ld seen
in the previous module. We know that the expected return on an
asset is equal to the risk free rate plus beta times the expected return on the
market minus the risk free rate. I've added in here this additional term
here alpha. So this can be viewed in the context of
the capital asset pricing model as the excess returner, the skill if you like,
that a manager can earn on an investment. I don't want you to think that alpha is
necessarily associated with account fund. It's a generic term used to evaluate
skill, whereby a manager can earn excess. Risk adjusted returns.
Why is this important? Well, a fund manager who has alpha, or who
has skill, can often charge substantial management fees.
And so what we want to know, is whether or not the skill is real.
In general, it's very hard to tell, but you can still do some interesting
analysis, and that's what we're going to do in this module.
We're actually going to look at this question from three different
perspectives. The first two perspectives, will require
the binomial distribution, and so we're going to review a little bit about the
binomial distribution now. If you recall, we say that x is a binomial
distribution. Or we write x tilde Bin n p, if the
probability that x equals r equals n choose r, times p to the r, times one
minus p to the n minus r. And so x might represent for example, the
number of heads in n independent coin tosses, where p is equal to the
probability of a head. The mean and variance of the binomial
distribution are given by these quantities here.
The expected value of x is n times p, and the variance of x is n p times 1 minus p.
You can also see from one that the probability that x is greater than or
equal to r is equal to the sum of the probabilities that x equals r, x equals r
plus 1 up to x equals n. And that's given to us by this summation
here on the right hand side. So, to see our first perspective on this
problem, consider the following situation. Suppose a fund manager has a track record
of ten years and that this fund manager has outperformed the market in nine of the
ten years. The fund manager claims to have great
skill and that his fees should reflect this.
What we want to know is, well how can we assess his claims?
Does he really have great skill? So first analysis might assume the
following, in a given year he outperforms the probability p and under performs with
probability 1 minus p. And by the way, when I say outperform, I
mean outperform relative to the risk the manager is taking on.
So I'm talking about risk adjusted returns here.
Outperformance or underperformance is assumed to be independent across years.
If the manager has skill, then p is greater than a half.
Otherwise p is less than or equal to a half, and the manager has no skill.
So the first question that now comes to mind is the following.
How likely is such a track record if the fund manager had no skill?
Well to answer this, let x be the number of outperforming years.
If the fund manager has no skill, then x is going to be binomial, with n equals 10,
and p equals a half. We said that the fund manager outperformed
in 9 of the 10 years. So, a track record as good as this would
corresponds to x being equal to 9 or indeed x equals 10.
So we want to compute the probability that x is greater than or equal to 9.
Assuming the manager had no skill. In other words, assuming p is equal to a
half. That is given to us by our binomial
probability we saw in the previous slide. We can evaluate this easily in Excel or
some other piece of software, and we find the answer of 0.0107.
So therefore, if the fund manager has no skill, then the probability of having a
track record as good as his, or better, is only 0.0107.
So at this point, you might think it's fair to conclude, that the fund manager
does indeed have skill. After all, you could think of this in a
statistic setting, for those of you who are familiar with the statistics, you'd be
aware of the concept of a p value. And usually a p value that's less than or
equal to 0.05 would be assumed to be significant.
So we've got a p value here if you like, of 0.0107, this seems significant.
It seems unlikely in this case that the fund manager is no skill.
Given the track record of 9 successful years out of 10.
So that's the first perspective we're going to take.
Here is a second perspective, suppose instead that there are M fund managers and
that the manager who claims to have skill has the best track record of these
managers. The questions that now arises is, does
this change anything in our analysis? Should it change something in our
analysis? To answer this suppose we start with the
hypothesis that none of the fund managers have skill and that track records of fund
managers are independent. There are two possible questions now that
arise. The first question is, how likely is the
third manager to have such a track record if all fund managers have no skill.
Well, in this case, if we've identified the third manager in advance, then the
first perspective gives us the answer. It would actually.
Actually be 0.0107, as we calculated beforehand.
The second question or possible question to, to ask, is how likely is the best
manager to have such a track record if all fund managers have no skill?
And now, what is the appropriate question here?
Which of these two questions is more appropriate?
This actually depends on our prior hypothesis.
What do I mean by that? Well let's come back to this slide here.
We've actually identified this manager here, this is the third manager.
And this is the best performing manager of the M managers.
And so what I mean by prior hypothesis is the following.
Did we identify this third manager in advance?
Maybe this third manager was our friend, or a cousin, or somebody else we specified
in advance before we saw any data. If that is the case, then this question is
the more appropriate question. We've seen the manager in advance, we're
interested in his performance, not because he was the best performing manager, but
because he was the third manager or some manager we've seen in advance.
However, if we're interested in the third manager because he was the best performing
manager, and that's why his track record has come to our attention, then the
appropriate question is actually the second one.
How likely is the best manager to have such a record, if all fund managers have
no skill? And this is because the reason we're
interested in the third manager is not because he was number three in the list.
But because he was the best performing out of all m managers.
And that's why he came to our attention. So, it really depends on our hypothesis.
Why are focusing on this manager? Is it because he was the third manager,
we'd pre-selected him in advance? Or is it because he was the best manager,
and that's why the manager's track record has come to our attention.
Depending on which hypothesis is correct, we get a very different answer.
With the first hypothesis, we see it's 0.0107.
With the second hypothesis, for our interest in the manager is because he's
the best performing manager out of M, the second hypothesis is the more appropriate.
So let's answer this second question. We're going to assume none of the managers
have skill. We're going to let z i be the event that
the ith manager out-performs in r years or more out of the n years.
We're going to let v be the event that the best manager outperforms in r years that
event, or more. Then what we're interested in is the
probability of v. The probability of v is equal to 1 minus
the probability of zed 1 bar up to zed m bar.
Now what is zed i bar? Zed i bar is the complement of zed i.
So zed i is the event that the ith manager performs in r years out of n, so zed i bar
is the event that the ith manager outperforms.
In less than r years out of n, so it's the complement of zed i.
So therefore just to make sense of this, if the best manager is to outperform in r
years, the event that the best manager doesn't outperform in r years and more.
Is equivalent to all managers. All m of them outperforming in less than r
years out of n. And that's this quantity over here.
Because we're assumption that these z, zed i's are independent, and IID.
I can write this as just the probability of zed 1 bar.
Times the probability of zed q bar, up to the probability of zed m bar.
Just keeping these separate. And these are all IIDs, so this is,
there's m of them. And so I get the probability of zed one
bar to the power of m. Now, recall, the probability of zed 1 bar.
Zed 1 bar is the event that's the compliment of zed 1.
So therefore, the probability of zed 1 bar is equal to 1 minus the probability of zed
1. And therefore, I can plug in the numbers.
I'm assuming, in this case, I believe, that m is equal to 20.
And I know that the probability of zed 1 is 0.0107.
Because I actually calculated this. Onto our first perspective.
So this is for a fixed single fund manager.
The probability of zed 1, we found to be 0.0107.
So we need the probability of zed 1 bar, here, which is 1 minus that.
Take it to the power of m, we get this number here, 0.1942.
And actually you can easily see what happens.
As m gets bigger, as we increase m, the number of fund managers in the
marketplace. Place then actually probability of v will
actually increase to one as m gets very large.
What are our conclusions. Well our conclusions are as follows.
If there are a lot of fund managers in the market, then the, the fact that one
particular fund manager has a fantastic record, does not, in and of itself
constitute evidence that, that fund manager has skill.
After all, we have shown here, that with just 20 fund managers we would expect the
best manager to have a record of outperforming in 9 years.
Or more out of 10 to recover probability 0.1942.
And that's not a very small number. If we take m equal to 30 or 40, this
number will get bigger. And so even though we've assumed all the
fund managers have no skill here. The fact that we're focusing on the best
manager's track record. Suggests that the best track record, will
be good. So we haven't established evidence to
suggest that a particular fund manager has skill just because he's got a great track
record. It all depends on the hypothesis.
How did we identify this manager? I should also point out that while we have
focused on the best manager here, we could also have focused on the second best
manager or the third best. Manager and so on.
If n gets sufficiently large, we're not going to just expect the best manager to
do well. The second best manager will also have a
very good track record. The third best manager will have a very
good track record. And so on.
As long as m gets sufficiently large. And this is true even if all of the fund
managers have no skill. Here's another perspective.
This is our third perspective. Suppose again that all fun, fund managers
have no skill. At the end of every year, fund managers
who have out performed the market that year, survive.
And fund managers who have under performed in the market that year, get fired.
Here's a question. After one year of this experiment, what
would be the average track record of fund managers in the market?
Well. It's got to be perfect.
Only the fund managers that have a good year survive, so they will have a perfect
track record. The fund managers that have an imperfect
track record, in other words they under performed that first year, well they've
been fired, so they're no longer available, they're no longer in the
marketplace so we won't see them. And so their performance does not enter
into our calculation of the average track record.
And indeed you can look at this after 2 years, or 3 years you'll only see perfect
fund managers in the market. So only perfect fund managers survive, and
they appear to have a perfect track record.
And yet it's clear that in fact, that these fund managers have no skill.
You could also generalize this in fact, as follows: you could imagine that fund
managers who've outperformed in a given year will stay in the market with some
probability p. And that fund managers who've
underperformed in the market will be fired with a different probability.
And we can assume that fund managers who've under-performed get fired more
frequently than fires, than managers who've over performed.
You could do that type of experiment, you could simulate that kind of system and
what you'll still see is that the average track record of fund managers in the
market would actually be greater than a half.
So actually they will appear to be a level of skill in the market, even though that
skill isn't really there. This is an example of what is called
survivorship bias. The reason being that some people have
survived, they're the people who have outperformed in the market, and they
actually make the market look better. They look like the, the collection of fund
managers look better than they really are. They've just survived because the poor and
unlucky fund managers have actually Under performed and lost their, and lost their
jobs. So this is an example called survivor bias
or survivorship bias. Its a very common phenomenon in finance
and beyond and we're actually going to see some more examples of survivor,
survivorship bias very soon. Now some final thoughts here.
I've been assuming up until now that the fund manager or the fund managers have no
skill. And you might ask the question, well, is
that fair? Is it fair to assume to begin with that
fund managers have no skill? After all, in practice there are some
managers with skill. And that's absolutely true.
There are some managers in practice with skill.
I don't there are too many of them and certainly lots of fund managers may think
they have the skill but the reality is quite different.
Nonetheless it is true that there are some fund managers out there with skill.
That having been said, I don't think we're being unfair in assuming to begin with
that fund managers have no skill. And that's because it is the
responsibility of the fund manager to convince us that they have skill.
After all, fund managers with skill try to charge fees to manage our money.
So it's up to them to convince us that they have skill.
It's not up to us to give them the benefit of the doubt and assume they have skill.
So it's perfectly reasonable to start off with the assumption that they've no skill,
and then to see where that takes us. And if we can find such a manager, is the
resulting out-performance sufficient to justify the management fees?
That is not clear as well. If the management fees are too large, then
those fees are going to dominate the out-performance.
That the manager provides and so in that situation we wouldn't want to invest with
the manager anyway.
Explora nuestro catálogo
Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.
Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.
© 2019 Coursera Inc. Todos los derechos reservados.
