>>> By enrolling in this course you agree to the End User License Agreement as set out in the FAQ. Once enrolled you can access the license in the Resources area <<< This course, Applied Artificial Intelligence with DeepLearning, is part of the IBM Advanced Data Science Certificate which IBM is currently creating and gives you easy access to the invaluable insights into Deep Learning models used by experts in Natural Language Processing, Computer Vision, Time Series Analysis, and many other disciplines. We’ll learn about the fundamentals of Linear Algebra and Neural Networks. Then we introduce the most popular DeepLearning Frameworks like Keras, TensorFlow, PyTorch, DeepLearning4J and Apache SystemML. Keras and TensorFlow are making up the greatest portion of this course. We learn about Anomaly Detection, Time Series Forecasting, Image Recognition and Natural Language Processing by building up models using Keras on real-life examples from IoT (Internet of Things), Financial Marked Data, Literature or Image Databases. Finally, we learn how to scale those artificial brains using Kubernetes, Apache Spark and GPUs. IMPORTANT: THIS COURSE ALONE IS NOT SUFFICIENT TO OBTAIN THE "IBM Watson IoT Certified Data Scientist certificate". You need to take three other courses where two of them are currently built. The Specialization will be ready late spring, early summer 2018 Using these approaches, no matter what your skill levels in topics you would like to master, you can change your thinking and change your life. If you’re already an expert, this peep under the mental hood will give your ideas for turbocharging successful creation and deployment of DeepLearning models. If you’re struggling, you’ll see a structured treasure trove of practical techniques that walk you through what you need to do to get on track. If you’ve ever wanted to become better at anything, this course will help serve as your guide. Prerequisites: Some coding skills are necessary. Preferably python, but any other programming language will do fine. Also some basic understanding of math (linear algebra) is a plus, but we will cover that part in the first week as well. If you choose to take this course and earn the Coursera course certificate, you will also earn an IBM digital badge. To find out more about IBM digital badges follow the link ibm.biz/badging.
Autoencoder - understanding Word2Vec
Loading...
Del curso dictado por IBM
Applied AI with DeepLearning
336 calificaciones
De la lección
DeepLearning Applications
Conoce a los instructores
Romeo KienzlerChief Data Scientist, Course Lead
IBM Watson IoT
Niketan PansareSenior Software Engineer
IBM Research
Tom HanlonTraining Director
Skymind
Max PumperlaDeep Learning Engineer
Ilja RasinData Scientist
IBM Watson Health
Now it's time to do some NLP,
Natural Language Processing, and we will start with the famous word2vec example.
So word2vec is a way to compress
your multidimensional text data into smaller-sized vectors,
and with those vectors,
you can actually do calculations or further attach downstream neural network layers,
for example, for classification.
And in this case, we will just do both.
So we will start with some imports,
and we put here a seat that examples are replicatable.
So let's start with some sentences.
So this is a document which has five sentences,
and four are related to queens and kings and so on and men and women,
and one is unrelated.
We define vocabulary size 50.
That means this system here as it is will support 50 different words.
In a real-life scenario,
of course, you have millions of words.
Later, we do some calculations with the words.
Therefore, we already now get the representation of these words,
and I will show you what I mean by that.
So each word is just a sequence of characters and,
obviously, we cannot work with sequence of characters.
Therefore, we will convert each word into an integer number,
and this integer number is unique,
as long as we don't exceed the vocabulary size.
Note that the function is called one-hot,
but in my opinion, that's not correct.
It's not the one-hot encoding.
It's basically just the transformation from
a list of the words into a list of integer values.
As you can see here, I'm correct.
So, for example, king is number 23 and queen is number 26 or man is number 22, and so on.
So now we are actually one-hot encoding this stuff.
So this is now a single dimension with multiple values.
And one-hot encoding returns multiple dimensions,
and only one of those dimensions has a one.
The rest is zero.
That means we are getting sparse vector representations.
So that's actually what I'm talking about.
So you see here for each word,
we get a sparse vector.
And if the particular word is present at that position we have one,
otherwise we have zero.
So now we have done this for our example words,
but now let's do it for a complete document.
So as you can see here, we have some words again.
So those are the sentences.
Each element in this multidimension array is
a sentence and each integer is representing a word.
It's just an index which tells us at what position we should insert a particular word.
So 23, 42 and 22 are actually "king is man." So king is 23.
We see here, again, king,
a second word in the second sentence.
So here, we are actually padding that means,
if the sentence is not long enough,
we are just filling it with zeros.
This function is very,
very important and may be a bit complicated to understand,
but what we are doing is,
we're now creating tuples.
And we are creating tuples of a word and related words.
So in a sentence, for example,
"king is a man," if you look at the word king,
then we are creating tuples,
where "king" is the key and "is" is the value.
And "king" is the key and "man" is the value.
That means that's the embedding.
That means the neighbors of the word.
So we are creating a list of tuples where,
for each word, we also are assigning the neighbors of the word.
We're doing this, in this case, for two neighbors,
for two preceding neighbors and two succeeding neighbors.
Don't lose too much time with that.
That's the follow-up here.
Just look at recite.
So recite has dimension 38 by two.
That means you have 38 word pairs and the pairs,
as the name implies, have dimension two.
So, again, we have now two dimensions with integer keys.
And in order to make a neural network perform, we, again,
have to encode it as one-hot encoded vectors.
We first do it for the first dimension,
which basically resembles in the input extra neural network,
and then we do it for the second dimension,
which is our target.
So that's what we want to predict and that's why or the output of the neural network.
So here you see that, again,
we have 38 dimensions because,
in the previous example, we had 38 pairs,
but now we have 50 dimensions because now
the one-hot encoded vector representation is the sparse vector of size 50.
Remember, only one of those elements is one.
And that's the position which actually reflects the word.
So if you start with a dense input layer which has
50 internal neurons and also expects dimensionality of 50,
actually, that perfectly matches our one-hot encoded vectors of size 50.
And we use relu as activation.
So now it gets interesting.
Now we're increasing the bottleneck.
Remember bottlenecks. We have an auto encoder.
So this is an auto encoder of what we are building,
and the bottleneck has only two dimensions.
And, again, activation function is relu.
And in an auto encoder, of course,
we have to map again to the same dimensionality.
So because input was of dimension 50,
also the output has to be of dimension 50.
So we're mapping sparse vectors to sparse vectors.
That means we are mapping the words,
for example, a word in the center to words in the neighborhood.
And while we're doing this,
we are training the bottleneck layer,
and that's our low-dimensional representation.
And that's actually what word2vec does.
It creates a low-dimensional representation of our high-dimensional word embedding,
which then basically is a low-dimensional word embedding.
We compile, set optimizer,
and loss function, and then we call fit.
So that starts training on the one-hot encoded labels X and the one-hot encoded target Y.
So for this Mickey Mouse dataset,
this goes really fast,
and we are quite happy with our loss and accuracy.
So now it gets even more interesting.
Now we build a new neural network but we are using
two of the three already trained layers of the previous neural network.
So we start with a sequential model again,
and we add the already trained input layer,
and we add the already trained bottleneck layer.
So if we now compile this neural network,
the input still is 50 dimensional but the output now is only two dimensional.
So we take all five words of the beginning,
and let's see what the two-dimensional representations of those words are looking like.
So we see this is an array of two-dimensional vectors,
and in theory, we can now plot those words into 2D space.
But we can also calculate with those vectors.
So we now take the word king, we subtract man,
we add woman, and this should resemble nearly the same vector as the word "queen."
That means if we now subtract "queen" from it we should get a relatively low value.
Since this is only a try example with very little data and training takes a lot of time,
I'm using some online demo of this system.
So here, we actually take woman,
and here we say king minus man.
So we get somehow the meaning of king without a gender.
And if we add this meaning without a gender to a woman,
we should get "queen."
Let's see what happens. Yes, see. We are pretty close.
So that's actually really cool.
We can now calculate with words in a low-dimensional vector space.
Explora nuestro catálogo
Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.
Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.
© 2019 Coursera Inc. Todos los derechos reservados.
