In this video will briefly introduce some testing based procedures used in model selection, and we'll describe some of the issues that arise with this type of model selection method. So testing based procedures are really based on hypothesis tests like the F test and the T test that we've learned about. And there are a few different methods for using these types of tests to select models. So one testing based method is called backward elimination, and this is a really simple method for trying to select a model, but as we'll see, it's not really a super rigorous or statistically justified method. But backward elimination works something like this. So you start out with some full model, and this would include, all of the predictors that you would consider at all in a regression model. And then you would remove the predictor with the highest p-value greater than some threshold you might call it alpha not. And then you would refit the model with the remaining predictors, so all but the one that you left out that was large and greater than this alpha not threshold. And then you refit the model and look again at all of these T tests. And if you have another predictor that falls above that threshold, then you would remove the highest one above that threshold. And you would keep doing this,so this is an iterative process. You will continue to refit models with one fewer predictor until you had a model where all of the T tests were the p-values for those T tests were lower than Alpha not. Now forward selection is another test based selection method, and it works in sort of the reverse way of backward selection. So what you do in forward selection, is you start with all possible simple linear regression models. So if you have p possible predictors for your model, then you would, if your final model you would start out with p different simple linear regression models. So each one would have just one of your predictors as the only predictor. And then you would look through and for each of those models, you would look at the p-value for the T test and you would select the model that has the lowest p-value for that T test. So that would give you one simple linear regression model. And then you would in the next round of this forward selection procedure, you would add, you would construct p minus 1 regression models, each with two predictors. And so each one of these models has the original predictor that you selected, plus one additional predictor, and that would be different for each model. So then, among those p minus 1 models, you would select the one that has the lowest p-value for that additional predictor, and then you would iterate this process. You would keep going until you reach the model where you didn't have any additional, you had a new round where none of the additional predictors had a p-value that was below some threshold, like alpha not. So there are other variations on what you could call stepwise regression, these procedures where you iterate through and fit different models to try to select the best one. Another one might be called bidirectional stepwise regression, and in that one at each stage, so at each iteration, you allow for any variable to be added or removed. So there are different ways to do this, but it's a sort of a set of variations on either forward selection or backwards selection. Now, some of these procedures seem like they could be a lot of work and R can help us cut down on some of that work. So in R, there's a function called update and update can help you carry out some of these procedures. And typically, what you would do say, if you are doing backwards selection, you would fit the full model using the lm function, as we've done many times. And then you could use the update function you can plug in whatever you called your model fit for your full model. And then there's some syntax that you'll see in the slide here that would allow you to remove one of those variables and refit the model. And so you can use that update function to see the on the next iteration what the model looks like what the T test look like when you remove the additional predictor. Now we'll walk through these procedures a bit when we do a less than an R. But I think it's important to notice that these procedures are problematic in a few different ways. So these testing based procedures, they can be computationally inexpensive and they can be automated, which is nice. But, there's actually no great statistical justification for using these procedures. And so there are some warnings that we should give about them, and we should actually be pretty cautious in using them. There are better options, and the main reason that we're doing this lesson where we're learning about them is because, for better or worse, you'll see them in some literatures. And it's a good idea to understand what they are. And really, it's a good idea to understand how to avoid them. So one thing to notice is that because we're adding or removing variables one at a time, it's clearly possible to miss the optimal model. We're not really looking at every single possible permutation and combination of these predictors and different models, so it's possible that you step through the space of possible predictors in a way that misses the optimal model. Now, these procedures often lead to an overstating of the importance of the remaining predictors, the predictors that actually end up staying in your model. And this is true because the procedures rely on multiple tests, and we know that multiple tests can be problematic, right? They overstate the importance, and we see an increase in inflation in the rate of type 1 error. Now another thing that we can note about these procedures is that they're not really directly linked to the objectives that we have when we set out to do regression, namely either prediction or explanation. But there's nothing about this procedure that will pick out the causally relevant variables, right, so the variables that are related to explanation. And there's really not any great way in which these procedures help you even pick out ones that are are predictive for the response. So they seem to provide one way to select a model, but they're not really linked to the goals of regression in the first place, and that's a big reason why we should be cautious. Now another thing to keep in mind is that these procedures will sometimes tend to pick models that are a bit smaller than would be desirable for predictive purposes, so they would leave out certain predictors that could help make your model more predictably powerful. And so again, another reason why we should be cautious of these tests based sort of automated procedures. Again, they're not really linked to the goals of regression. So we will learn how to use at least one or two of these in our lesson on R but we'll move to better justified methods in future lessons.