Hello and welcome back. In this video we're going to be talking about image processing in a very particular application, medical imaging. And this is image processing for virus understanding. And in particular to understand the shape of the HIV virus. This actually brings a lot of challenges, as we are going to see. We're going to have to talk about many, many topics in image processing including image acquisition and let's start with that. There are many ways to acquire images. And before we do that, we have to understand. One of the basic things is the size of the structure that we're trying to acquire. And here we see a scale of very different structures. depending on what this type we want to acquire what type of technology, we're are going to be using to acquire that data? And of course, we are used to light, we are used to cameras. Basically, light reflecting from objects and then giving us a picture. But depending on the size of the objects and their scale, we might need different modities for image accusition. During this video we are interested in viruses and then we are going to be talking about nanometers or angstroms scale and in particular what we are going to use is not light we are going to use electrons to aquire our image because we are at a very, very small scale. Again, in the orders of nanometers and angstroms. Now, let me just tell you why we are interested in something very small in the HIV virus. And I hope this diagram, although a bit complicated. I think It illustrates a lot of the challenges that we're going to have and why is this so important what I'm going to explain next. So here we have an HIV virus. Remember we're talking again maybe 100 angstrom, you know just very, very small structures. And here basically, we have the host set. So basically, the virus is trying to latch into this host, and in order to do that, it uses things here which are called envelop or spikes. And then they go and try to latch here as showin in this diagram. And then basically transmission foes into the host's cell. So we're interested in this virus and in particular this might surprise you, and it will surprise you even more when you see the images. We are interested in these basically, these envelopes or spikes. They're called envelopes, and often, they're just read as ENV. And, once again, because that's what the HIV is using to latch itself into the membrane of the host cell and if we understand that structure, we might be able to stop and to block that connection and hopefully stop transmission. That's what we want to get. Now we are at those small scales, remember. And then what we need to use is not light. We won't be able with light to get to that scale. We use electrons and here is a diagram. Now once again your use to basically light. So light comes, reflects in objects like me and then regular cameras like the video camera that is recording right now, captures the reflection. In transmission electron microscopy, basically we have this specimen we basically have. A specimen with HIV, for example, but other specimens are possible as well. Microscopy is used all over for biological research. Electrons basically go, it's like you throw electrons into the specimen and you see, and you let them just go through. Basically you capture the image, you capture the image of the specimen by measuring how the electrons go through, and we see that in this diagram. There is a lens, and there is the image plane. That's where things basically get captured. This is a picture Of this device. The high end devices are not very cheap they can cost millions of dollars so this is not something that you carry with you like regular digital camera. Once again this uses electods are hiden in the speciman basically to go through and then we capture it. So for us, it's a different way of creating images, but the important way is that we can create images for these particles that are very, very, very, very small. So that's electron microscopy. And here is an image of what you will get from electron microscopy. Now remember, have always in mind, this is what we saw in the previous diagram. This is what we are looking for. One projection of that. So you took this electron microscopy and you basically threw electrons and you one projection of that. And that's basically what you have here so this, is this. Doesn't look very nice but we are going to have to deal with that and we are actually going to change the molality a little bit but it's not that it doesn't look very nice because the instrument isn't very good or the person taking the image is not Very good in doing that. This is actually extremely good quality for this technology. And we need this technology because we want to get these very tiny, very, very tiny elements. So we are going to have to deal with this. Before I do that, before I show you how to do that This is the beauty of image processing, that we are going to be able to deal with this. Let me just talk a bit more about tomography. Instead of just taking, putting the specimen and just taking one view, what, what you do in tomography is you rotate the specimen. So let me just show you this here. So you have the specimen and then yo rotate it and you rotate it and you take multiple images of this rotation and that's what's called tomography. Now, what do I mean by cryo? Okay. First of all we now know what's electron. Instead of electron microscopy we say electron tomography because we have multiple projections. Cryo is because, with the specimen, before we put here, we freeze it at prior temperatures. Now, this technology helps to reduce radiation. This is very important. When life is hitting me, you can take as many pictures as you want of me. I'm not going to get hurt by that. When you throw electrons at the specimen, you're hurting the specimen. You're changing it. So you're changing the shape of what you're trying to understand the shape of, and this is very dangerous. So this technique of cryo-electron tomography helps to reduce the rad iation of that and also helps you to obtain 3-dimension information because you have multiple, multiple tips. So that's basically what we do and let's see images in a second. But we are doing cryo-electron tomography. This is just one of those images. Again, remember we're interested in this kind of stuff. And actually we're interested in these spikes. You might be able to see some of them there. But that's what we want to get the 3-dimensional shape of that. They're very, very tiny and have received very, very noisy. So here's where you get the call to image processing. And just to say once again what kind of data we have. We got tiers. Basically we have the specimen. We rotate. We throw the electrons that did, we get one projection. We do a different rotation we get a different projection. A different rotation, a different projection. Then we put them together and we get the three dimensions. So I'm going to always show you slices of this, because, most of the time, I'm going to show you 3D when we get the reconstruction. But remember, this is supposed to represent this. And we obtain it so here was the specimen, very, very tiny, and here is the image in the computer. Now, the image processing starts with putting all these projections together to get the 3D. Normally, although there are alternatives, that's done with what is called fiducials. So, you basically embed in this specimen gold particles and then you have a gold particle, let's; say this pink and then you know where the gold particule is in every single one of the projections and then you align the images based on those cold particles. So you throw in a few and you put them together and that helps you to stack the images and to basically get the 3D in the computer of the real 3-dimensional virus. Now there's one thing I want to explain here. When you are doing the projection, you can see, this is very thin. And so, I'm illustrating it much wider. We're talking about very, very thin. So you cryo, micro-cryo-tomography, cryo-microscopy, you have a very thin slice at cryo-temperatures. So if I do the projection that I'm going to try to throw electrons from here. You're not going to get a lot because it's very, very thick. So look at my hand now. it's okay to have electrons going like this but not going like this. Okay. So if I have something very, very thin I can just do this. Remember, rotating the specimen is like rotating the raise. So I can have raise here but not raise here and maybe also not raise there. So we actually don't have the full 180 degrees. We normally, out of the 180 degrees. We might basically miss here 30 degrees and another 30 degrees there. And that's called a missing wedge. As we're going to see yet another problem. We actually don't have all the data we wanted to have because of this missing wedge. But now we are about to get into the image processing. We acquired the data. We align the projections using gold particles most of the time and now it's time to find those viruses and find those envelops. Now, this is what you see. You have a line and this is what you see. Remember, this is the schematic of what we are looking for. But these are slices of this 3D that we just have done. Now do you see here the envelopes or the spikes? No you dont see them. This is where we need image processing. We have extremly noisy data and we have the missing wedge. We have tons of problems.Are we going to be able to reconstruct these 3D. I am going to reassure you, yes that the answer is yes at a certain accuracy, of course. Again, these are just slices on the 3D volume that was reconstructed with cryo-tomography. This is what we have, this is the process that we have. We had basically a virus or a structure inside a specimen. We took multiple projections. Look how I represented here the missing wedge. We didn't take the whole around projection. We basically have missing wedge here so we took that and we kind of get a noisy version in this 3D volume. But there is 1 thing which is very important. We get multiple copies of these, as we have seen. Let me show you that again. We may have multiple copies. You see one. You see another one coming here. I'm going to show you more. We have multiple copies of these. And we also have multiple copies of this, and that's what we are going to exploit with image processing. We are going to exploit that we have not one very noisy example but multiple very noisy examples. So hopefully we can extract them, put them together, and get the three dimensions that we want. Remember one thing, when we talk about image denoising we already discussed that if we have multiple copies of the same object with random noise added, one of the best things you can do for de-noising is add all those multiple copies. If the noise has zero mean, and it's random, then it would just cancel each other. That basically was the concept behind local means. It's also the concept just behind local averaging. You basically want to average things that are repetitions of the same and that will help you to eliminate the noise. Remember if you average 10, the value of the noise goes down by the square of that. So a hundred, the more you average, the more you manage to de-noise. So that's great but what's the challenge here? Multiple challenges. These images are very noisy. The spikes, the envelopes, here, are rotated. We have to align them before we average them otherwise it will be a mess. Okay, if you average this with this, you let's say these two vectors you get something like this nothing to do with any of them. So, you have to first align them and I am going to demonstrate that in one of the next slides. So you have to align them and also are they identical copies? Maybe there are multiple different structures. Maybe slightly different, but still different. So we're going to have to rotate, align them, group only those that are the same. And only after we do all that, then we're allowed to average. So, here is the problem. Let me illustrate that to you. Let's say that we have three things as we see here. 1, 2, 3. Now let's simulate cryo electron tomography for this. We basically get multiple copies of each one. That's a good part. We get multiple copies of each one, very noisy. Okay? That's not so good. The multiple copies is good, the noise is not very good, but we have to deal with that. We get them rotated. That's also not very good, but we're going to have to deal with that. And then, we get the missing wedge. Okay? So and of course we don't have the originals, this is what we have. So somebody gave you this, very noisy data, with missing data, the missing wedge. They didn't give you in the columns, they are all unorganized. So you have to be able, from them, you have to be able to group the ones that came From basically the same building. Here we have to be able to group back here. You have to rotate back and all that from this noisy data with a lot of missing information. If we have 30 degrees missing. We have 60 degrees missing out of 180. That's a lot of missing information and that's represented here. So this holes is a realistic proportion. You have this group aligned, do everything. And hopefully after you're done, of course, you're doing all the rotations, all the groupings and everything together. Hopefully you are able to reconstruct back the virus or the images as we represent here. So, imagine that you take multiple cameras of the same thing and, but you occlude the cameras in a random fashion. you add a lot of noise. So now you have multiple pictures of the same scene. And now somebody comes and say hey, give me back the shape of that building. Very challenging but hopefully doable because you have multiple copies. So, what are some of the challenges that we have here? We have very low SNR. We have this problem with alignments and missing data. And, we need a lot of free time to remove the noise and basically we are going to have computational challenges. So lets see how we address each one of these challenges. In part together and in part separate and this is a very tough problem but I think your going to get happy to see the kind of resize that we can get. So lets get into that. The first thing that we have to do is define a distance. Remember we want to align this projections or this, this three dimensinal shapes but with missing Information with the missing wedge, and with noise. So, how do I design, going back to the images in 2D projections of the buildings. How do I design when these 2 are aligned, or they are not aligned? I need to basically define a distance. And we take any 2 images, we look at the 2 projections, or 2, 3 dimensions, depending. Depending where are we working, and how do I know if these two are aligned or not? That's the first thing we need to do, basically to define a distance. We are going to allow the images to rotate and also translate later on, and we want to be able to say, hey, keep rotating, keep rotating, stop. That's where you are align, so I need to define the distance between two images with to noise and with the missing . So let's illistrate that. That's the first step every time that we need to align images. How to decide when they are already aligned? ,, . So we have 1 image here, which I represent as a 1 dimension signal. Here, we have this 1 dimensional signal. And we represent it here. But we don't have the whole signal. We actually have the missing wedge. So we only have the signal in certain parts of it. Now we have another signal. And also for it, we have a missing wedge. Very important. We basically do'nt have the missing wedge in the same place. Remember, these particles were rotated. So the missing wedge is kind of fixed for a particular aquisition. But the particles are all rotated so with respect to them the missing wedge is all the time in a different position. Now if you actually know where the missing wedge is or you can guess. It's either you know from the instrument or you kind of guess it because there is a lack of signal to noise. There. And these are basically the two images that we need to basically decide a distance between them. We wish to have them like here, complete, but we only have portions of them. And the only place where we can compare them is when we have Portions of both. So here, we have information for the first one. But there's no information for the second one. Here, we had, for the second one, not for the first one. So the only place where we can compare is when we have information for both of them. So we can take a distance. Euclidean distance or any distance that you think is appropriate. But we need to only consider places where both of them are available and that's by this product. So this and this are basically square binary signals. 1 I have a signal 0, I don't have a signal. And I can only look at places where both of them are 1. So I have signals for both of them. again, you can just take that basically almost any type of metric that you wish, and illustrate it here with just a , Type of distances. And this is going to be kind of your distance. You take the difference between the images but you only consider where both of them are available and of course you have to normalize. By the amount of energy they have per signal, and that's basically written here. That's standard. Always a, a dis, distance with the pen of how much overlap you found, and we really do basically normalize for that. So that's the distance that we're going to use to illustrate the examples in the next few slides. Now this distance, actually, is not too hard to compute. You can actually compute in the Fourier domain. We haven't discussed this Course, too much about Fourier. For those that have a background in signal processing, you know what Fourier is. But we did talk about DCT, the Discrete Cosine Transform, so basically, you take the signals, go into a different domain, in the Fourier domain. And in that domain, you do this calculation. It's very convenient to do that for computational reasons. It's also very convenient to do that there, because it's very easy to represent a wedge than missing information there. So we have this distance which is a euclidean distance, but weighted for the regions where we have common information.
