Learn how to analyze data using Python. This course will take you from the basics of Python to exploring many different types of data. You will learn how to prepare data for analysis, perform simple statistical analyses, create meaningful data visualizations, predict future trends from data, and more! Topics covered: 1) Importing Data sets 2) Cleaning the Data 3) Data frame manipulation 4) Summarizing the Data 5) Building machine learning Regression models 6) Building data pipelines Data Analysis with Python will be delivered through lecture, lab, and assignments. It includes following parts: Data Analysis libraries: will learn to use Pandas, Numpy and Scipy libraries to work with a sample dataset. We will introduce you to pandas, an open-source library, and we will use it to load, manipulate, analyze, and visualize cool datasets. Then we will introduce you to another open-source library, scikit-learn, and we will use some of its machine learning algorithms to build smart models and make cool predictions. If you choose to take this course and earn the Coursera course certificate, you will also earn an IBM digital badge. To find out more about IBM digital badges follow the link ibm.biz/badging.
Model Evaluation and Refinement
Loading...
Del curso dictado por IBM
Data Analysis with Python
82 calificaciones
De la lección
Model Evaluation
Conoce a los instructores
Joseph SantarcangeloPh.D., Data Scientist at IBM
Model evaluation tells us how our model performs in the real world.
In the previous module,
we talked about in-sample evaluation.
In-sample evaluation tells us how well our model fits the data already given to train it.
It does not give us an estimate of how well the train model can predict new data.
The solution is to split our data up,
use the in-sample data or training data to train the model.
The rest of the data, called Test Data,
is used as out-of-sample data.
This data is then used to approximate how the model performs in the real world.
Separating data into training and testing sets is an important part of model evaluation.
We use the test data to get an idea how our model will perform in the real world.
When we split a dataset,
usually the larger portion of data is used for
training and a smaller part is used for testing.
For example, we can use 70 percent of the data for training.
We then use 30 percent for testing.
We use training set to build a model and discover predictive relationships.
We then use a testing set to evaluate model performance.
When we have completed testing our model,
we should use all the data to train the model.
A popular function in the scikit-learn package for
splitting datasets is the train test split function.
This function randomly splits a dataset into training and testing subsets.
From the example code snippet,
this method is imported from sklearn cross-validation.
The input parameters y_data is the target variable.
In the car appraisal example,
it would be the price and x_data,
the list of predictive variables.
In this case, it would be all the other variables in
the car dataset that we are using to try to predict the price.
The output is an array.
x_train and y_train the subsets for training.
x_test and y_test the subsets for testing.
In this case, the test size is a percentage of the data for the testing set.
Here, it is 30 percent.
The random state is a random seed for random data set splitting.
Generalization error is a measure of how well
our data does at predicting previously unseen data.
The error we obtain using our testing data is an approximation of this error.
This figure shows the distribution of the actual values in
red compared to the predicted values from a linear regression in blue.
We see the distributions are somewhat similar.
If we generate the same plot using the test data,
we see the distributions are relatively different.
The difference is due to
a generalization error and represents what we see in the real world.
Using a lot of data for training gives us an accurate means
of determining how well our model will perform in the real world.
But the precision of the performance will be low.
Let's clarify this with an example.
The center of this bull's eye represents the correct generalization error.
Let's say we take a random sample of the data using
90 percent of the data for training and 10 percent for testing.
The first time we experiment,
we get a good estimate of the training data.
If we experiment again training the model with a different combination of samples,
we also get a good result,
but the results will be different relative to the first time we run the experiment.
Repeating the experiment again with
a different combination of training and testing samples,
the results are relatively close to the generalization error,
but distinct from each other.
Repeating the process, we get a good approximation of the generalization error,
but the precision is poor i.e.
all the results are extremely different from one another.
If we use fewer data points to train the model and more to test the model,
the accuracy of the generalization performance will be
less but the model will have good precision.
The figure above demonstrates this.
All our error estimates are relatively close together,
but they are further away from the true generalization performance.
To overcome this problem, we use cross-validation.
One of the most common out of sample evaluation metrics is cross-validation.
In this method, the dataset is split into K equal groups.
Each group is referred to as a fold.
For example, four folds.
Some of the folds can be used as a training set which we use
to train the model and the remaining parts are used as a test set,
which we use to test the model.
For example, we can use three folds for training,
then use one fold for testing.
This is repeated until each partition is used for both training and testing.
At the end, we use the average results as the estimate of out-of-sample error.
The evaluation metric depends on the model,
for example, the r squared.
The simplest way to apply cross-validation is to call the cross_val_score function,
which performs multiple out-of-sample evaluations.
This method is imported from sklearn's model selection package.
We then use the function cross_val_score.
The first input parameters,
the type of model we are using to do the cross-validation.
In this example, we initialize a linear regression model or object
lr which we passed the cross_val_score function.
The other parameters are x_data,
the predictive variable data,
and y_data, the target variable data.
We can manage the number of partitions with the cv parameter.
Here, cv equals three,
which means the data set is split into three equal partitions.
The function returns an array of scores,
one for each partition that was chosen as the testing set.
We can average the result together to estimate out of
sample r squared using the mean function NnumPi.
Let's see an animation,
let's see the result of the score array in the last slide.
First, we split the data into three folds.
We use two folds for training,
the remaining fold for testing.
The model will produce an output.
We will use the output to calculate a score.
In the case of the r squared i.e.
coefficient of determination, we will store that value in an array.
We will repeat the process using two folds for training and one fold for testing.
Save the score, then use
a different combination for training and the remaining fold for testing.
We store the final result.
The cross_val_score function returns
a score value to tell us the cross-validation result.
What if we want a little more information?
What if we want to know the actual predicted values
supplied by our model before the r squared values are calculated?
To do this, we use the cross_ val_predict function.
The input parameters are exactly the same as the cross_val_score function,
but the output is a prediction.
Let's illustrate the process.
First, we split the data into three folds.
We use two folds for training,
the remaining fold for testing.
The model will produce an output and we will store it in an array.
We will repeat the process using two folds for training one for testing.
The model produces an output again.
Finally, we use the last two folds for training,
then we use the testing data.
This final testing fold produces an output.
These predictions are stored in an array.
Explora nuestro catálogo
Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.
Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.
© 2018 Coursera Inc. Todos los derechos reservados.
