One of the most common tasks performed by data scientists and data analysts are prediction and machine learning. This course will cover the basic components of building and applying prediction functions with an emphasis on practical applications. The course will provide basic grounding in concepts such as training and tests sets, overfitting, and error rates. The course will also introduce a range of model based and algorithmic machine learning methods including regression, classification trees, Naive Bayes, and random forests. The course will cover the complete process of building prediction functions including data collection, feature creation, algorithms, and evaluation.
Bagging
Loading...
Del curso dictado por Johns Hopkins University
Practical Machine Learning
2090 calificaciones
De la lección
Week 3: Predicting with trees, Random Forests, & Model Based Predictions
This week we introduce a number of machine learning algorithms you can use to complete your course project.
Conoce a los instructores
Jeff Leek, PhDAssociate Professor, Biostatistics
Bloomberg School of Public Health
Roger D. Peng, PhDAssociate Professor, Biostatistics
Bloomberg School of Public Health
Brian Caffo, PhDProfessor, Biostatistics
Bloomberg School of Public Health
This lecture is about bagging, which is short for bootstrap aggregating.
The basic idea is that when you fit complicated models,
sometimes if you average those models together, you get a
smoother model fit, that gives you a better balance between
potential bias in your fit and variance in your fit.
So bootstrap aggregating has a very simple idea.
The basic idea is take your data and take resamples of the data set.
So, this is the similar to the idea of bootstrapping, which you would have
learned about in the inference class that
is part of the data science specialization.
After you resample the cases with replacement, then
you recalculate your prediction function on that resampled data.
And then you either average the
predictions from all these repeated predictors that
you built or you majority vote or
something like that when you're doing classification.
The thing is that you get a similar bias that you would get from fitting any one
of those models individually, but a reduced variability
because you've averaged a bunch of different predictors together.
This is most useful for non-linear functions.
So, we'll show an example with smoothing, but it's
also very useful for things like predicting with trees.
So I'm going to go back to the ozone data, so it's in the ElemStatLearn package.
And I load the the ozone data set.
I then order for the purposes of showing you how this works,
I'm going to order the data set by the outcome, the ozone variable here.
And then I look at the data set and I
can see it has four variables, ozone, radiation, temperature, and wind.
So the idea is that I'm going to try
to predict temperature as a function of ozone.
[BLANK_AUDIO]
So the first thing that we can do is just show you an example of how this works.
So the basic idea is, I'm going to create a matrix
here and it's going to have 10 rows and a 155 columns.
Then what I'm going to do is, I'm going to resample the data set.
In for ten different times, so then a
loop over ten different samples of the data set.
Each time I'm going to sample width replacement from the entire data set.
Then I'm going to create a new data set, ozone0, which is
the resample data set for that particular element of the loop.
And that's just the subset of the data set corresponding to our random sample.
Then I'm going to reorder the data set every time by
the ozone variable, and you'll see why in just a minute.
Then I fit a loess curve each time, so a loess is
kind of a smooth curve that you can fit through the data.
It's very similar to the sublime model fits that we
saw in a previous example with modeling with linear regression.
And so the basic idea is we're fitting
a smooth curve relating temperature, to the ozone variables.
So temperature is the outcome, and ozone is the predictor, and each time I
use the resample data set as the data set I'm building that predictor on.
And I use a common span for each time, the
span being a measure of how smooth that fit will be.
I then predict for every single loess curve the outcome
for a new data set for the exact same values.
I always predict for ozone values 1 to 155.
So the ith row of this ll object is now the prediction from
the loess curve, from the ith resample of the date ozone.
So what I've done here?
I've resampled my data set ten different times, fit a smooth curve through it those
ten different times, and then what I'm
going to do is I'm going to average those values.
So, here's what it looks like in this plot.
So, here I've plotted ozone on the x
axis, these are the observed ozone values versus temperature
on the y axis, those are the observed
temperature values, and each black dot represents an observation.
Each gray line here represents the fit with one resampled data set.
So you can see the gray lines that have a lot of curviness to them.
They capture a lot of the variability in the data set.
But they also maybe over-capture some of the variability.
They're little bit too curvy.
Once I've averaged those lines together I get something that's a little
bit smoother and is closer to the middle of the data set.
That's the red line.
So the red line is the bagged loess curve.
It's basically the average of multiple fitted loess curves,
the same data set where I've resampled it every time.
There's a proof that shows that the bagging estimate will always have
lower variability but similar bias to the individual model fits that you do.
In the caret package there's some models that already perform bagging for you.
So if you're using the train function you
could set method to be bagEarth, treebag, or bagFDA.
And those are specific bagged models that the the
model that the caret package will fit for you.
Alternatively, you can actually build your own bagging function in caret.
This is a bit of an advanced use and so I recommend that you
read the documentation carefully if you're going to be trying to do that yourself.
The idea here though is you basically are going to
take your predictor variable and put it into one data frame.
So I'm going to make the predictors be a data frame that contains the ozone data.
Then you have your outcome variable.
Here's it's going to be just a temperature variable from the data set.
And I pass this to the bag function in caret package.
So I tell it, I want to use the predictors
from that data frame, this is my outcome, this
is the number of replications with the number of
sub samples I'd like to take from the data set.
And then bagControl tells me something about how I'm going to fit the model.
So fit is the function that's going to be applied to fit the model every time.
This could be a call to the train function in the caret package.
Predict is a the way that given a particular
model fit, that we'll be able to predict new values.
So this could be, for example, a call to the predict function from a trained model.
And then aggregate is the way that we'll put the var, the predictions together.
So for example it could average the
predictions across all the different replicated samples.
You can see that if you look at this
custom bag version of the conditional regression trees, you can
see that it gets some of the benefit that I
was showing you in the previous slide with bag loess.
So the idea here is I'm plotting ozone
again on the x-axis versus temperature on the y-axis.
The little grey dots represent actual observed values.
The red dots represent the fit from a single conditional regression tree.
And so you can see that for example, it capture, it doesn't capture the
trend that's going on down here very well, the red line is just flat.
Even though there appears to be a trend upward in the data points here.
But when I average over ten different bagged
model model fits with these conditional regression trees.
I see that there's an increase here in the values in
the blue fit, which is the fit from the bagged regression.
So we're going to look a little bit
at those different parts of the bagging function.
So in this particular case I'm using the ctreeBag function, which you
can look at in, if you've loaded the caret package in R.
So, for the fit part it takes the data frame
that we've passed and the predict, and the outcome that
we've passed, and it basically uses the ctree function to
train a tree, conditional regression tree on the data set.
This is the last command that's called the ctree command.
So it returns this model fit from the ctree function.
The prediction takes in the object.
So this is going to be an object from the ctree model fit.
And a new data set x, and it's going to get a new prediction.
So what you can see here is it basically calculates each time
the tree response or the outcome from the object and the new data.
It then calculates this probability matrix and
returns either the actually the observed levels that
it predicts or it actually re, just returns
the response, the predicted response from the variable.
The aggregation then takes those values and averages
them together or puts them together in some way.
So here what this is doing is
it's basically getting the prediction from every
single one of these model fits, so that' s across a large number of observations.
And then it binds them together into one data matrix by with
each row being equal to the prediction from one of the model predictions.
And then it takes the median at every value.
So in other words it takes the median prediction from
each of the different model fits across all the bootstrap samples.
So bagging is very useful for nonlinear models, and it's widely used.
It's often used with trees.
And you can think of an extension to this as
being random forest, which we'll talk about in a future lecture.
Several models use bagging and caret's main train
function, like I told you about in previous slide.
And you can also build your own specific bagging functions, for any
classification or prediction algorithm that you'd like to take a look at.
For further resources, I've linked to a couple
of different tutorials on bagging and boosting, as
well as the Elements of Statistical Learning which
has a lot more details about how bagging works.
But remember that the basic idea is to basically resample
your data, refit your nonlinear model, then average those model
fits together over resamples to get a smoother model fit,
than you would've got from any individual fit on its own
Explora nuestro catálogo
Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.
Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.
© 2018 Coursera Inc. Todos los derechos reservados.
