0:01
Okay, in the previous video we talked about transformations,
in particular narrow transformations,
which are transformations that don't imply transferring data through the network.
Now let's talk about wide transformation instead.
So, let's first start with an example.
So I took the same example we used in the video about RDDs.
So, we have our input text file which is a dataset of where
each element is a line of text then we run flatMap and
we provide the function split_words that splits each line into the words.
And finally,
we call the map transformation with the function create pair.
And the output of this is a key value pair where the key is the word and
the value is always one.
Because then we want to sum all those ones to get the final counts for each word.
So let's call collect, so
that we can get the data back to the driver for inspection.
And you see that the output here is the least of words,
and each of them has the key one.
So the operation I would like to show you is groupByKey.
So what groupByKey does is it takes, as an input, an RDD of key value pairs.
And, as an output, transfers all the data
1:46
So let's see this example where on the top we have
the pair A1, and the bottom A1 to A5.
So they share the same key.
So the output we will have this key, with all of the three values.
In an iterable object.
In iterable object is an object can be the iterated through and
each time it will send back one of the value of our dataset.
2:20
So let's actually run this in practice on our VM,
and if we call groupBKey and
collect on our pairs_RDD, the results we have is unfortunately,
not very readable, because we have the keys that are our words, and
that's fine, but the values are these ResultIterable objects.
So we want to unpack those objects and the way of doing it is
we write a simple pi turn loop on the driver and
for which she's looping through each key value pair and
once we get the value we call the list functions that transforms this integrable
into a list and now the output is exactly what we were expecting.
3:43
A node needs to ship the data across the network to
the first node, and this operation is called a shuffle.
And it's extremely important because it's extremely resource intensive,
because you are because generally communication over the network
is one of the strongest bottlenecks so let's check side by side
in arrow against the Y confirmation so you see on the left that
A narrow transformation is local within each node,
so each node can run independently, and
the speed of this operation just depends on
the availability of local memory and on CPU speed.
Instead, in a groupbyKey operation on the right,
You have this transfer data through the network.
So interconnection speed between the nodes becomes an important factor.
4:49
So let's take a look at some other wide transformations, so
we know already groupByKey.
Another one is reduce by key, we saw this already in the r d d video,
where we use this to go through all our rewards and some, of our reward counts.
The result of this, reduce by key operation Is
a pair of key and values where, for each key,
we have a value which is the result of the reduction
of all of the Vs across our data set with these functions.
So if we a sum, it's going to be the sum of all our accounts across our data set.
Another interesting is Repartition.
So, repartition is very similar to coalesce.
However, while coalesce always tries to keep the data local, so
to reduce the number of partitions by merging them together locally.
Instead repartition, it does the same operation across the network so
a data is shuffled across the network to make
a even distribution of your partitions to all over your cluster of nodes.
So, when to use a coalesce, and when, partition.
So, generally, you should first try with coalesce,
if the local merging of partitions is good enough for your application to.
6:28
Balance more your data across your nodes.
If that's not necessary because your data are very unevenly distributed, for
example if you had a very strong filtering and this filtering
maybe applies a lot more on data on particular nodes,
then a partition is definitely a better way to go.
6:52
Now, I would like to go a little deeper in our shuffle works.
So why is the shuffle so important?
Because it's one of the most performance heavy
operations that you have in your data analysis pipeline.
And so how this works.
So let's take a look at, as usual, our couple of nodes.
So you see here that the key A is available on both nodes.
7:24
And this list will transferred over the network from the second node to the first.
And How does this actually happen?
So, the first step is that on the source side, the data are written to disk,
and they are written to local disk, already separated.
They're already ordered by the partition in which they're gonna be.
On the receiving side, on the final operation.
And then, the data are requested over the network
from the output nodes.
And this data is transferred from the source to
the final destination.
So, it's very important to know when you are getting a shuffle.
So first of all, you have to know which operations.
We saw the list of y transformations.
All of them are going to trigger a shuffle.
And it's very important to understand if it's really a necessary of, or
if you can avoid it.
One interesting sample is sometimes in your data processing
you are using groupbykey and so you are getting for the same key,
you are moving all your data to the same partition, the same node.
But sometimes in the latest steps of your
workflow you are going to call a reduction so
at this point you are calling this reduction so for simplicity let's
think for example this is a sum and we're going to sum all of our values.
But there is a better way of doing this which is using reduce by key so
let's see actually how this works in an example with our usual two nodes.
So if you do it first the group by key and then add a use what you're gonna do is
you are going to transfer your data from the in
this case your going to transfer 2 and 8 from the second node to the first node and
then In the reduction phase, you are going to sum the three of them.
Now, with reduceByKey, you can improve
this by doing, first, the sum.
Before transferring over the network and so now you are now,
you are transferring instead of two numbers, you are transferring one only.
And so, the good of reduce by key is that you can perform
a first reduction phase before actually shipping your data through network.
And then of course, you need a second reduction phase.
And to get to the same final results.
So it's very important to understand when you can use reduceByKey instead
of the goByKey.
Natasha BalacDirector, Predictive Analytics Center of Excellence (PACE)
Paul RodriguezResearch Programmer
Andrea ZoncaHPC Applications Specialist
