Wide Transformations

Loading...

Del curso dictado por University of California San Diego

Hadoop Platform and Application Framework

calificación

Explore Related Videos

At Coursera, you will find the best lectures in the world. Here are some of our personalized recommendations for you

University of California San Diego

Hadoop Platform and Application Framework

calificación

This course is for novice programmers or business people who would like to understand the core tools used to wrangle and analyze big data. With no prior experience, you will have the opportunity to walk through hands-on examples with Hadoop and Spark frameworks, two of the most common in the industry. You will be comfortable explaining the specific components and basic processes of the Hadoop architecture, software stack, and execution environment. In the assignments you will be guided in how data scientists apply the important concepts and techniques such as Map-Reduce that are used to solve fundamental problems in big data. You'll feel empowered to have conversations about big data and the data analysis process.

De la lección

Spark

Welcome to module 5, Introduction to Spark, this week we will focus on the Apache Spark cluster computing framework, an important contender of Hadoop MapReduce in the Big Data Arena. Spark provides great performance advantages over Hadoop MapReduce,especially for iterative algorithms, thanks to in-memory caching. Also, gives Data Scientists an easier way to write their analysis pipeline in Python and Scala,even providing interactive shells to play live with data.

Resilient Distributed Datasets10:41

Spark Transformations10:18

Wide Transformations10:49

Conoce a los instructores

Natasha Balac
Director, Predictive Analytics Center of Excellence (PACE)
San Diego Supercomputer Center
Paul Rodriguez
Research Programmer
San Diego Supercomputer Center (SDSC)
Andrea Zonca
HPC Applications Specialist
San Diego Supercomputer Center (SDSC)

Okay, in the previous video we talked about transformations,

in particular narrow transformations,

which are transformations that don't imply transferring data through the network.

Now let's talk about wide transformation instead.

So, let's first start with an example.

So I took the same example we used in the video about RDDs.

So, we have our input text file which is a dataset of where

each element is a line of text then we run flatMap and

we provide the function split_words that splits each line into the words.

And finally,

we call the map transformation with the function create pair.

And the output of this is a key value pair where the key is the word and

the value is always one.

Because then we want to sum all those ones to get the final counts for each word.

So let's call collect, so

that we can get the data back to the driver for inspection.

And you see that the output here is the least of words,

and each of them has the key one.

So the operation I would like to show you is groupByKey.

So what groupByKey does is it takes, as an input, an RDD of key value pairs.

And, as an output, transfers all the data

that has the same key, to the same partition.

So let's see this example where on the top we have

the pair A1, and the bottom A1 to A5.

So they share the same key.

So the output we will have this key, with all of the three values.

In an iterable object.

In iterable object is an object can be the iterated through and

each time it will send back one of the value of our dataset.

So let's actually run this in practice on our VM,

and if we call groupBKey and

collect on our pairs_RDD, the results we have is unfortunately,

not very readable, because we have the keys that are our words, and

that's fine, but the values are these ResultIterable objects.

So we want to unpack those objects and the way of doing it is

we write a simple pi turn loop on the driver and

for which she's looping through each key value pair and

once we get the value we call the list functions that transforms this integrable

into a list and now the output is exactly what we were expecting.

So for each word we have 1.

The only exception is the word far, because there are two of them,

and so we get a pair of 1s as values.

So what actually happens across the network in our nodes.

Here for example you see that on the top and on the bottom the two worker nodes,

they share the same key which is A, and what happens is that the second.

A node needs to ship the data across the network to

the first node, and this operation is called a shuffle.

And it's extremely important because it's extremely resource intensive,

because you are because generally communication over the network

is one of the strongest bottlenecks so let's check side by side

in arrow against the Y confirmation so you see on the left that

A narrow transformation is local within each node,

so each node can run independently, and

the speed of this operation just depends on

the availability of local memory and on CPU speed.

Instead, in a groupbyKey operation on the right,

You have this transfer data through the network.

So interconnection speed between the nodes becomes an important factor.

So let's take a look at some other wide transformations, so

we know already groupByKey.

Another one is reduce by key, we saw this already in the r d d video,

where we use this to go through all our rewards and some, of our reward counts.

The result of this, reduce by key operation Is

a pair of key and values where, for each key,

we have a value which is the result of the reduction

of all of the Vs across our data set with these functions.

So if we a sum, it's going to be the sum of all our accounts across our data set.

Another interesting is Repartition.

So, repartition is very similar to coalesce.

However, while coalesce always tries to keep the data local, so

to reduce the number of partitions by merging them together locally.

Instead repartition, it does the same operation across the network so

a data is shuffled across the network to make

a even distribution of your partitions to all over your cluster of nodes.

So, when to use a coalesce, and when, partition.

So, generally, you should first try with coalesce,

if the local merging of partitions is good enough for your application to.

Balance more your data across your nodes.

If that's not necessary because your data are very unevenly distributed, for

example if you had a very strong filtering and this filtering

maybe applies a lot more on data on particular nodes,

then a partition is definitely a better way to go.

Now, I would like to go a little deeper in our shuffle works.

So why is the shuffle so important?

Because it's one of the most performance heavy

operations that you have in your data analysis pipeline.

And so how this works.

So let's take a look at, as usual, our couple of nodes.

So you see here that the key A is available on both nodes.

And this list will transferred over the network from the second node to the first.

And How does this actually happen?

So, the first step is that on the source side, the data are written to disk,

and they are written to local disk, already separated.

They're already ordered by the partition in which they're gonna be.

On the receiving side, on the final operation.

And then, the data are requested over the network

from the output nodes.

And this data is transferred from the source to

the final destination.

So, it's very important to know when you are getting a shuffle.

So first of all, you have to know which operations.

We saw the list of y transformations.

All of them are going to trigger a shuffle.

And it's very important to understand if it's really a necessary of, or

if you can avoid it.

One interesting sample is sometimes in your data processing

you are using groupbykey and so you are getting for the same key,

you are moving all your data to the same partition, the same node.

But sometimes in the latest steps of your

workflow you are going to call a reduction so

at this point you are calling this reduction so for simplicity let's

think for example this is a sum and we're going to sum all of our values.

But there is a better way of doing this which is using reduce by key so

let's see actually how this works in an example with our usual two nodes.

So if you do it first the group by key and then add a use what you're gonna do is

you are going to transfer your data from the in

this case your going to transfer 2 and 8 from the second node to the first node and

then In the reduction phase, you are going to sum the three of them.

Now, with reduceByKey, you can improve

this by doing, first, the sum.

Before transferring over the network and so now you are now,

you are transferring instead of two numbers, you are transferring one only.

And so, the good of reduce by key is that you can perform

a first reduction phase before actually shipping your data through network.

And then of course, you need a second reduction phase.

And to get to the same final results.

So it's very important to understand when you can use reduceByKey instead

of the goByKey.

and in the next lesson, we are going to look

at how the transformation are used

together by Spark to process your data analyzing

Explora nuestro catálogo

Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.

Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.

© 2019 Coursera Inc. Todos los derechos reservados.

Descargar en la App Store

Consíguelo en Google Play