Running a k-Means Cluster Analysis in Python, pt. 1

Loading...

Del curso dictado por Wesleyan University

Machine Learning for Data Analysis

194 calificaciones

Wesleyan University

Machine Learning for Data Analysis

194 calificaciones

Curso 4 de 5 en SpecializationData Analysis and Interpretation

Are you interested in predicting future outcomes using your data? This course helps you do just that! Machine learning is the process of developing, testing, and applying predictive algorithms to achieve this goal. Make sure to familiarize yourself with course 3 of this specialization before diving into these machine learning concepts. Building on Course 3, which introduces students to integral supervised machine learning concepts, this course will provide an overview of many additional concepts, techniques, and algorithms in machine learning, from basic classification to decision trees and clustering. By completing this course, you will learn how to apply, test, and interpret machine learning algorithms as alternative methods for addressing your research questions.

De la lección

K-Means Cluster Analysis

Cluster analysis is an unsupervised machine learning method that partitions the observations in a data set into a smaller set of clusters where each observation belongs to only one cluster. The goal of cluster analysis is to group, or cluster, observations into subsets based on their similarity of responses on multiple variables. Clustering variables should be primarily quantitative variables, but binary variables may also be included. In this session, we will show you how to use k-means cluster analysis to identify clusters of observations in your data set. You will gain experience in interpreting cluster analysis results by using graphing methods to help you determine the number of clusters to interpret, and examining clustering variable means to evaluate the cluster profiles. Finally, you will get the opportunity to validate your cluster solution by examining differences between clusters on a variable not included in your cluster analysis. You can use the same variables that you have used in past weeks as clustering variables. If most or all of your previous explanatory variables are categorical, you should identify some additional quantitative clustering variables from your data set. Ideally, most of your clustering variables will be quantitative, although you may also include some binary variables. In addition, you will need to identify a quantitative or binary response variable from your data set that you will not include in your cluster analysis. You will use this variable to validate your clusters by evaluating whether your clusters differ significantly on this response variable using statistical methods, such as analysis of variance or chi-square analysis, which you learned about in Course 2 of the specialization (Data Analysis Tools). Note also that if you are working with a relatively small data set, you do not need to split your data into training and test data sets.

Running a k-Means Cluster Analysis in Python, pt. 18:14

Running a k-Means Cluster Analysis in Python, pt. 210:01

Conoce a los instructores

Jen Rose
Research Professor
Psychology
Lisa Dierker
Professor
Psychology

So let's try running a k-Means cluster analysis in Python.

First, we will call in the libraries that we will need.

In addition to the pandas, numpy, and matplotlib libraries we'll need

the train_test_split function from the sklearn.cross_validation library,

and the pre processing function from the sklearn library.

Finally, we will need the k-Means function from the sklearn.cluster library.

After calling in the data set using pd._read_csv

function we will do some data management.

Namely, we will create a new data set called data_clean in which we will delete

observations with missing data on any of the variables using the dropna function,

then we will create a data set called cluster

that includes only our clustering variables.

In cluster analysis variables with large values

contribute more to the distance calculations.

Variables measured on different scales should be standardized

prior to clustering, so

that the solution is not driven by variables measured on larger scales.

We use the following code to standardize the clustering variables to have a mean of

0, and a standard deviation of 1.

To standardize the clustering variables we will first create a copy of the cluster

data frame and name it cluster var, then we use the preprocessing.scale function to

transform the clustering variables to have a mean of 0 and a standard deviation of 1.

First, we list the name of the clustering variable, then an equal sign, and

preprocessing.scale, then in parentheses we type the name of our variable again and

add .astype, and then in another set of parentheses, float64 in quotes.

Astype float64 ensures that my clustering variables have a numeric format,

and we will do this for all the clustering variables,

then we will use the train test split function in the sklearn cross validation

library to randomly split the clustering variable data set into a training data set

consisting of 70% of the total observations, and

a test data set consisting of the other 30% of the observations.

First we'll type the name of the data set which we'll call clus_train followed

by the name of the test data set which we'll call clus_test,

then we type the function name,

train_test_split, and in parenthesis we type the name of the full

standardized cluster variable data set which we called clustevar.

The test_size option tells Python to randomly place 0.3,

that is 30% of the observations in the test data set that we named clus_test.

By default the other 70% of the observations are placed in

the clus_train training dataset.

The random_state option specifies a random number seat

to ensure that the data are randomly split the same way if I run the code again.

Now we are ready to run our cluster analysis, because

we don't know how many clusters actually exist in the population for

a range of values on the number of clusters before we begin we'll import

the cdist function from the scipy.spatial.distance library.

In this example we will use it to calculate the average distance of

the observations from the cluster centroids.

Later, we can plot this average distance measure to help us figure

out how many clusters may be optimal, then we will create an object called clusters

that will include numbers in the range between 1 and 10.

We will use this object when we specify the number of clusters we want to test,

which will give us the cluster solutions for k equals 1 to k equals 9 clusters.

In the next line of code we create an object called meandist

that will be used to store the average distance values that we will calculate for

the 1 to 9 cluster solutions.

The for k in clusters: code tells Python to run the cluster analysis code below for

each value of k in the cluster's object.

That is to run cluster analysis specifying 1 through 9 clusters,

then we will use the k-Means function From the sk learning cluster library

to run the cluster analyses.

In parentheses n_clusters indicates the number of clusters,

which in our example we substitute with k

to tell Python to run the cluster analysis for 1 through 9 clusters,

then we create an object called clusassign that will store for each observation

the cluster number to which it was assigned based on the cluster analysis.

After the equals sign model.predict asks that the results of the cluster analyses

stored in the model object be used to predict the closest cluster that each

observation belongs to, then we are going to calculate an average distance measure

for each cluster solution independent to our currently empty mean dist object.

The code following meandist.append computes the average

of the sum of the distances between each observation in the cluster centroids.

The formula first calculates the distance between each observation and the cluster

centroids where the number of centroids is equal to the number of clusters that were

specified in the cluster analysis in the first set of parentheses.

We use the cdist function from the scipy spacial distance library

to calculate the distance.

In parentheses we type the name of our clustering variable data set clus_train

followed by a comma, and model.cluster_centers_,

which is the name of the model results attribute that contains the cluster

centeroids, then we type another comma and the word Euclidean in single quotes.

This tells Python to use cdist to calculate the distance between each

observation in the clus_train data set in the cluster centroids using

Euclidean distance, then we use np.min function to determine the smallest or

minimum difference for each observation among the cluster centroids.

Axis equals 1 means that the minimum

should be determine by examining the distance between the observation and

each centroid taking the smallest distance as the value of the minimum,

then we use the sum function to sum the minimum distances across all observations.

Finally, the / clus_train.shape with 0 in brackets divides the sum of the distances

by the number of observations in the clus_train data set where the .shape with

0 in brackets code returns the number of observations in the clus_train data set.

Now that we have the average distance calculated for

each of the 1 to 9 cluster solutions we can plot the elbow curve

using the map plot lib plot function that we imported as plt.

In parenthesis clusters is the object that includes the values of 1 through 9 for

the range of clusters we specified and

meandist is the average distance value that we just calculated.

So what this plot shows is the decrease in the average minimum distance of

the observations from the cluster centroids for

each of the cluster solutions.

We can see that the average distance decreases

as the number of clusters increases.

Since the goal of cluster analysis is to minimize the distance between observations

and their assigned clusters we want to chose the fewest numbers of clusters

that provides a low average distance.

What we're looking for in this plot is a bend in the elbow that

kind of shows where the average distance value might be leveling off

such that adding more clusters doesn't decrease the average distance as much.

You can see how subjective this is though.

There appears to be a couple of bends at the line at two clusters and

at three clusters, but it's not very clear.

To help us figure out which of the solutions is best we should further

examine the cluster solutions for at least the two and three cluster solutions to see

whether they do not overlap, whether the patterns of means on the clustering

variables are unique and meaningful, and whether there are significant

differences between the clusters on our external validation variable GPA.

Explora nuestro catálogo

Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.

Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.

© 2018 Coursera Inc. Todos los derechos reservados.

Descargar en la App Store

Consíguelo en Google Play