So the second issue that I want to talk about is sample size, and I want to to begin with the psychology of sample size, essentially. And this motivates our use of data to increase samples and provide some caution on our use of small, intuitive samples. The first is an example, this is from a research study, classic research study where the participants were asked the following vignette. Your firm has two plants, one large and one small, which mass-produce a standard computer chip, other than the amount they produce, the two plants are identical in all essential regards. Both use the same technology to produce the same product. When properly functioning, this technology produces 1% defective items. Whenever the number of defective items from one day's production exceeds 2%, a special note is made in the quality control log to flag the problem. At the end of the quarter, which plant would you expect to have more flagged days in the quality control log? So, two identical plants big and small. How many fall outside the quality control acceptable limits? Everything else about the plant's the same, it's just that one is larger than the other. Does the small plant have more of these deviations? Does the large plant have more, or are they the same because after all they're identical in every other respect? So if you ask people this, about 22% say the smaller plant, 30% the larger plant, and then about half say the same, because in fact, all of the technology's the same and how should it be any different. What's the right answer? The right answer is the small plant, of course, because from small samples, you get greater variation. And what we're talking about here is variation, these are production outcomes that fall outside the quality control limits, those are variations from expectation. Small samples necessarily or more vary and people don't appreciate that and this is a major problem, for performance evaluation, because we are apt to draw too strong a conclusion from small samples. Let's look at this in a little more detail. Sample means converge to population means as the sample size increases. If you've had a statistics course, you might recognize this as the Central Limit Theorem. So, because this happens as we get larger and larger samples the converses exactly true. That the smaller the sample the more likely you see more extreme values. So, there is a couple example questions when you more like to see a 400 batting average in baseball? Which is a very good batting. Far higher than we typically see. Do you see it at the beginning of the season, around May 1, or towards the end of the season, September 1? Well of course, you're far more likely to see such an extreme batting average from a small sample, the kind of sample you would have at the beginning of the season then you would later. What about hospitals? Where are you likely to see a dramatically higher percentage of boys than girls or vice versa, born on any given day? A small community hospital, which might have four or five births a day? Or a large city hospital, which might have 100 births a day? Where are you more likely to see the population mean vary in that daily sample? The small hospital of course, we'll see large variations. You might have all girls one day, all boys another day and a small hospital there's no way you'll have that in a larger hospital. We emphasize this because it should give us great caution in what we say from small samples, but importantly it shows that we don't have that caution. We tend to believe that what we see in the small sample is representative of the underline population. We don't appreciate that a small sample can bounce around quite a bit and not in fact be representative of the underline population. This means that we should be very careful to not follow our intuition on what it means for will we have a small sample. This is a bias acknowledged in the literature, called in the literature the Law of Small Numbers. Which is silly of course the is no law of small numbers. It's not like the law of large numbers. The small numbers in fact, what makes them interesting is that there is no law, they bounce around. People believe small samples closely share the properties of the underlying population, but in fact they don't. This means they too readily infer population's properties, the average, from the samples. They neglect the role variability plays, chance plays, in these small samples. What are the implications? What are the implications in your organization? This is a fundamental issue in performance evaluation, because often, maybe even typically, performance evaluation is on small samples. The smaller the sample, the more we have to be cautious what kind of inference we draw from it. So, this is a place where data can make a big difference. This is a place where, by using more sources, more history, using data as a opposed to just relying on our intuition. We had the chance to collect larger samples, but I don't want to say data alone is sufficient. In fact, it's interesting to note that in academia, especially in psychology, but now it's expanding to other disciplines as well. We are discovering that we have been drawing too strong an inference from even our academic research. There's somewhat of a revolution going on in psychology now being picked up in other disciplines. That the main theme of which is, we've been drawing too strong an inference from the samples we've been using. So, this isn't just academia saying be careful out there you non-academics. It's even academics that have had to learn, that we too readily draw inferences from samples that aren't really sufficient. The lesson is almost always get more data, get bigger samples, be careful about the inferences you draw from these small numbers, and know that we have a bias toward jumping to quickly based on the small samples.