[MUSIC] The first reason that analytics is a leadership problem is that analytics requires managerial judgement. To see this, you have to understand that data science does not equal truth. There's a lot of room for judgment in the data science that drives analytics. You see analytics fundamentally speaks to management decisions, and management decisions are inherently political. There are different agendas and there are winners and there are losers. And because of that, before long you're going to be in a situation where team A reports back to you with result A backed up by data science team A. And team B will report back to you with result B backed up by data science team B. And guess what? Result A and result B will not be the same. And the problem is, you can't call your resident data scientist to help you figure out who's right, because there are already six data scientists in the room and they all disagree with each other. And so it falls on you to decide. You have to be able to make the judgement about who to believe. But that is not the only reason why analytics requires managerial judgement. I was recently invited to an executive retreat for a major company. One of the executives got up to present, and he told us that his company had recently been able to close the loop between online advertising and offline sales. Now, this is very hard to do in this industry and so there was a great deal of interest in the room about how online advertising affect sales. The executive then shows a graph like this and he says, what you see here is the probability that a customer buys a product in our industry during, I think a 90 day period, depending on what ad they saw on a search engine, for example Google or Yahoo. If consumers did not see any ads related to a product in our industry on search engines, he says. Their probability of buying in a 90 day period was 0.7%. This is what you see in the left column. If they had seen ads only from retailers, the probability of buying in a 90 day period was 3%. This is what you see in the second column from the left. If they had seen ads only from manufacturers, the probability of buying went up to 5%. And if they had seen ads from both retailers and manufacturers, their probability was 14%. That's what you see in the right column. And then the executive said, we learned two things from this. First, search engine advertising really works, and second, retailer and manufacturer advertising are compliments, not substitutes. This means if you do both in isolation you get a worse result than if you do them together. In other words, 3% plus 5% is less than 14%. Now there was an excited buzz in the room. I mean, these were new findings, and the executives in the room talked about how to use this information. This went on for about 15 minutes until somebody suggested perhaps, it would be a good idea to put out a press release about these findings. The conversation went on a little longer, and then somebody pointed at the left column where consumers had received no ads and asked, so why would somebody not see any product related ads on a search engine? And the answer, of course, is that these people did not search for the product on the search engine. And then pointing to the right column, he says, so why would somebody see ads from retailers and manufacturers in our industry? Well, the answer is that these consumers search for the product and perhaps a location, which then would have triggered keywords that both manufacturers and dealers bid for. And then this person says, so what we have shown is that if you are not interested in buying this product, and he points to the left column, you don't buy the product. And if you're very interested in buying this product, and he points to the right column, you do buy this product. And indeed, this graph says nothing about whether search engine advertising works and whether retailer and manufacturer advertising are compliments. And the reason is that we don't know whether the higher likelihood of purchase is due to the fact that the people who saw retailer and manufacturer advertising were inherently more interested in purchasing the product. Or whether the fact that they saw search advertising made them more interested in buying the product. It took the executives in the room 15 minutes and a prompt to find this problem. It should have taken them 15 seconds instead. But it is hard to find problems like these unless you have managerial judgement in analytics. And this matters because executives get bombarded with more and more evidence of this type. [MUSIC]
