Fairness as an algorithm. Our goal in this lesson is to first explain how to use fairness as a modeling concept. What does it mean to actually implement fairness and optimized for it? Then we're going to define pareto efficiency and identify where they're used in machine learning. To start out, we get to this big question, how do we measure fairness in these machine learning models? We previously talked about fairness in the context of fairness versus accuracy. But how do you actually score fairness? Well, if we treat fairness just like an accuracy score, then we can come up with the same scoring methodology for fairness. We can run multiple models and then compare with the error score, and in doing so, we could find out which model makes the most sense to put into production, which has the best combination of error score and fairness score for our situation. It would look a little like this. We would have a plot here, and we'd start to put all these lines on the plot with the error rate and the fairness rate to figure out which ones scored best. If we started to map these out on a graph, we would see something like this start to form, a pattern would emerge. As we plot out these points, we would see that as the error rate goes up, the rejection rate would go down, and as the error rate goes down, the rejection rate would go up. If we categorize this as a fairness score, then we can start to see where a balance might occur. This line is called a pareto efficiency. The curve here is an optimal trade off between error rate and fairness score or rejection rate. What is important here is that a pareto efficiency is where resources cannot be re-allocated to make one piece better off without making at least one other piece worse off. It's optimal trade off here between fairness and accuracy. If you look at the other points, there will always be a better one to pick if it is on that curve and that is the pareto efficiency. Overall though, you'll notice there is no perfect score here, that gets the error rate as low as possible and the rejection rate as low as possible. We are still left to pick a point along this line. That is why it's important to note that models are always going to be imperfect. A decision will have to be made. Let's think about what happens when we start to scale these pareto efficiencies up to multiple groups. Let's say we run a model with attribute A versus B. We select the point on the curve that makes the most sense, the best optimized trade-off. Then we run models with attributes C and D and then select that trade-off that makes the most sense. Well, now we can run models with A plus C, A plus D, B plus C, B and D, and find the optimal models there. The algorithm at the end of the day would decide the best threshold that would score fairness versus accuracy for each of these. If we had just one model to run, you can see that we would start to run into a scenario where we'd have to think of a lot of different fairness scores versus accuracy. We now get to this point where if we start to scale up more and more subgroups that are protected, we start to get toward this notion of individual fairness. Let's say that we had 100 applicants and over 50 different subgroups protected, whether it's age, race, parents, salary, neighborhood, gross income, etc. If we protected all of these different groups, we might run into a scenario where as they enter into a model, they are individually protected, but they could also be individually harmed in this case, because accuracy will take a massive hit. We get back to this larger philosophical question. Is it actually more fair to have individual fairness? If you're deny alone from an individualized model and the model made a mistake, would it be helpful to know, "Hey, the model only made that mistake with you?" It probably would not be. Even though that singular harm might be less harm than caused by an entire group, it is still harm in its own right. It may be more fair, but it is still causing harm. Getting back to that point of that, we will always have imperfect models and it's up to us to decide where to draw the line. We'll talk about that next.
