Now, when we have benefit curve, let us think how it can be connected to model quality metrics. Well, statistical model quality metrics are other typical for binary classification problem. This one is called area under curve, AUC for short, and this metrics aggregates all possible values of true positive and false positive rates. Well, graphically, it can be represented as the ratio between two areas; the one is S1 and the second is the sum of S1 and S2, which are actually one. So mathematically, you can put it as an integral of TPR with respect to FPR. Well, let's derive some formulas so we can remind ourselves that true positive rate is the ratio of true positive and positive. So in its turn, true positive is the difference between the total number of positive clients and number of false negative ones. This gives us that our TPR is one minus ratio of false negative to positive clients. So finally, it's one minus false negative rate. We can put it into our integral and we see that model quality statistical metrics is negative to false positive and false negative model errors. It's really interesting because for the financial result, we have very same result. So having this, we can question ourselves, are statistical quality metrics and financial metrics connected? Let's try to answer that. We maximize our benefit with respect to see level subject to our ROC curve. Well, on the first plot, you can see ROC curve, and on the right, benefit curve. So now, let's do some math. Benefit is the formula on above, and we see that it is number of negative clients multiplied by error of false positive, multiplied by brackets, one minus false positive rate. Also, you have to subtract number of positive clients multiplied by error of a false negative mistake, multiplied by brackets, one minus two positive rate. So if you now put in equation for true positive rate from this formula, you will see that this is some constant plus the ratio of class balances and error cost types multiplied by false positive rate. Geometrically, you can put this line on the ROC plot, and as soon as we maximize our benefit, we want our constant make as large as possible. So the optimal acceptance rate in terms of benefit is reached when the isoline of benefit and ROC curve make a contact. So you have this point and the corresponding question that derivative of true positive rate with respect to false positive rate equals the ratio of class balances and the ratio of error costs. Well, when the difficult part of math is done, we can play with our plots. Suppose we have two models and the first one is statistically better than the second one, it means that AUC from the first model is well, greater than AUC of the second quantile. Geometrically, we see that the first row curve measurizes the second one. In this case, we will see the same picture with benefit curves. So the benefit curve for the first model will measurize the benefit curve for the second form. Well, is that always a case? Well, not. Because not only AUC value matters, but also the shape of a ROC curve. Here, you have an example where you have two models with same AUC metrics. So the areas under the curves are equal, but at the same time, we see that they have really different shapes. We see that ROC curve of the first model is not that accurate in the start and it's more like going to the horizontal line in the end. So if we have a certain class balance and ratio of error types costs, then we see that benefit isolines can touch or may contact with ROC curves at the points which are better for first model. It means that by applying the first model and putting an optimal acceptance rate, we will get better benefit with the model 1, despite the fact that their AUC metrics are equal. Well, two plots will look like that. On the left, we have the situation where our optimal threshold level C is on the red curve and the optimal acceptance rate level on the blue curve, and at the same time, benefit curve also intersect each other, pretty the same way as ROC curves do. Well, we see that the maximum benefit that is visible for this model will be better for the first model. Who. Now we can plot this in one graph. We have benefit curve and imagine that the first model, which is blue one, has AUC 80 percent and the second, which is not that good, has decreased AUC which is equal to 65 percent. We see that with the same acceptance rate, which is 0.3, we get more benefit with the model A. This situation can happen when our model is being implemented in the business processes and it is working there and some time, the inter-dependencies between our clients and default events slightly change, or maybe the inflow of clients has now different properties. Therefore, the quality of our model dance to decay. In this case, we can estimate in financial terms, the amount of money our bank or our institution loses due to that model decay. That would be difference by y-axis between two points between A and C. This is how useful benefit curve can be. Let's have another example. Suppose that there is some option to improve your model quality. For instance, you can buy some data source, some external data source that your organization does not possess, and use that data to improve your model accuracy. In that case, you can come up with a better model with a better statistical metrics, AUC, which is 90 percent in, with better benefit curve. In this case, this is the point D. Again, at the same acceptance rate C 0.3, we have a better benefit. This is really useful for business people who own business processes to understand, should they or should not invest into new data sources? Because when they do, they pay for that and they have to expect benefit increase of their decisions based on new models. This is the way how you can calculate this difference, by calculating benefit at point D and subtracting from that benefit from point A. As easy as that. Let's take an opposite direction and think of what can happen when our model is totally random. Remember those plots with ROC curves when there is your ROC curve and that random straight line which connects 0,0 point at 1,1 point. Yes, it's pretty familiar to everyone. We have the same here. But on the benefit curve, we have that random straight line that goes to the starting point and to the right end point. It is plotted with gray. You can calculate the angle of this line, its number of negative clients multiplied by error false positive costs, minus number of positive clients multiplied by error cost of false negative. Well, you see how this curve and this plots are dual to ROC curves. That's interesting. As far as the maximum benefit is concerned, where we have a situation when every client and every observation has the same error costs, false positive, and false negative, the maximum number is N multiplied by error of the false positive type. It means that you will pick all the best clients from your portfolio, from your inflow, which were predicted as good ones, and you will accept no bad clients. It means that everything you can earn is the total number of negative clients multiplied by credit margin in case of credit scoring, which is error false positive type cost. To wrap up, benefit curves are a powerful tool to assess expected financial results. In some sense they are dual to ROC curves. Well, if you come up with a model with a higher statistical quality, in general, you will have a better financial result, but the key thing to emphasize that in general, not always. For instance, there can be models with the same AUCs, but different financial results. Even an opposite relation can have place and you have always to put and draw your benefit curves for your historical data to check for those effects.