Hello everyone. In this video, we're going to talk about unsupervised approach in deep learning. Here is a motivation why we want to have unsupervised approach in deep learning. In the past, researchers have really good success using supervised approach in deep learning because we had massive dataset with the labels such as ImageNet, powerful optimization techniques in deep learning tools, as well as a breakthrough in hardware like GPUs. We were able to have some good successes in supervised approach. However, later, people have found that supervised learning is not very efficient. That means in order to make a supervised learning model very successful, you need lots of data with the labels and this label quality must be very good too. The natural next step was to gather high-quality labels by hiring more people. However, it was not feasible always. When you think about medical data for example, getting labels are super expensive as well as the data volume itself is not that much. People figured out they needed to find out some better way to make the model learn better. A few years ago a famous machine learning researcher Yann LeCun made this analogy. This cake is called the LeCake. This cake analogy shows that the different machine learning tasks can have different data availability, for example, reinforcement learning is called the cherry on the cake because it is so expensive and difficult to get labeled data or useful signals for the reinforcement learning. Supervised learning is called the icing of the cake. It has a more data availability than reinforcement learning task, however, it's still not the majority of the cake. Unsupervised learning on the other hand is called the cake of cake, that means the majority of the data that's available is suitable for unsupervised task because it does not have labels. He was suggesting that the researchers should focus on unsupervised learning and learn how to harness this massive amount of data which don't have labels. People made a good progress in unsupervised approach in deep learning. There are so many techniques that fall into unsupervised approach, we'll talk about these three key points; one is called the density modeling which uses a generative task, which means that we generate the data, such that this generated data distribution resembles the real data. The second keyword is called the self supervision, and self supervision means that we can cheaply produce a label by using some another task instead of the intended original task, and then give a supervision signal to the model so that it can be trained on those but still those task is useful to learn the original feature that we wanted to learn. Adversarial training is a special instance of self-supervision, we use a task of classifying whether some output is real or fake. That has a name of adversarial training. We'll talk about those. One of the unsupervised approach is generative approach. We use machine learning model to generate the data, and then we require this generated data is similar to the training data. It can generate images, texts, and all kinds of other data types, audio, video and even some of the features of those data such as a style of image. Examples of this is density modeling, and autoencoders, and generative adversarial networks. A lot of all their research focused on how to generate data that looks very realistic, for example, these images they look okay but they are not perfect. Then as time goes, these generative models, especially GAN models became more refined and they were able to generate very realistic photos. People also look into how to catch some features in the images and then apply those and generate another image that captures their features, for example, this image on the bottom captures the zebra stripes from a training image and then it will apply this to a horse, that's called a style transfer and it's one example of this generative approach. Adversarial training was also important part of this generative adversarial network, GANs for short. They use both generative approach by generating the images and creating an adversarial task which classify whether the generated images are real or fake. It works like this, if you have dog's image and then we pass it onto the model then you will generate some image that may or may not look like dog. In GANs there is a classifier, which job is to tell whether this generated image is real or fake. Sometimes we send the real image and if the classifier learn the features very well then it should be able to tell whether the image input to the classifier is a real or fake. The reason why it is called adversarial training is that the generator that generates the data or fake images, and the classifier that discriminate whether the generated image is a real or fake, they compete each other. If the classifier or the discriminator is too strong that it will be able to discern all the generated images as fake then the generator may not have chance to learn, or if the discriminator is too bad then also the generator doesn't have a chance to learn because it's going to output some cropped images and it thinks it's always good. That's adversarial training and as you can see adversarial training usually produces a poor imaging in the early stage of the training, but as the epoch goes, it can create more refined images. Here is the website that shows the images from the current state of art GAN models. These images are generated images from the GAN models and not the real person but it looks very photorealistic and you might even think that this person exists, this person doesn't exist, this person doesn't exist, this person doesn't exist but they look very real. Next, let's briefly talk about self supervision. The key idea of self supervision is that we recreate the easy labels from the data itself. In many cases, getting large volume of good quality labels are very expensive, so instead of having human-generated labels we can auto-generate the labels from different tasks that are related to the original tasks that we wanted to do. This different task is called the surrogate task. In this example, they created a surrogate task of classifying the rotation angles instead of learning features from the object recognition task directly. They were able to train the neural network to learn better representation by having the neural network to learn about the image rotation. The advantage of this approach is that you can create the labels very easily and then you can train your model and once the model learns the useful features, you can take the part of the model by slicing the layers and the weight, and then you can retrain on the real task that has much less labels. The self supervision muscle that helps to train the model to learn about features even though we have a small number of labeled data. This is another example of self supervision approach. Surrogate task here is to classify whether the local feature of the one image input is the same as the global feature that is from another image input. To this encoder and this encoder, the upper encoder sees the global feature of the image, for example, if it was a dog's image, and then this bottom network sees the local features, that means it looks for some spots or some local features from the images, and we give the supervision signal that the answer is real if this image and this image are the same. If they are coming from different images like dog versus the jaguar then the answer should be fake. By enforcing the classifying surrogate task real or fake, whether the global feature and local features are coming from the same images. This gives a supervision signal, this forces the neural network to learn better. So far we had an overview of unsupervised approaches in deep learning, in the next videos we're going to talk about unsupervised deep-learning models.