Hello and welcome back. In this video we're going to be talking about image

processing in a very particular application, medical imaging. And this is

image processing for virus understanding. And in particular to understand the shape

of the HIV virus. This actually brings a lot of challenges, as we are going to see.

We're going to have to talk about many, many topics in image processing including

image acquisition and let's start with that. There are many ways to acquire

images. And before we do that, we have to understand. One of the basic things is the

size of the structure that we're trying to acquire. And here we see a scale of very

different structures. depending on what this type we want to acquire what type of

technology, we're are going to be using to acquire that data? And of course, we are

used to light, we are used to cameras. Basically, light reflecting from objects

and then giving us a picture. But depending on the size of the objects and

their scale, we might need different modities for image accusition. During this

video we are interested in viruses and then we are going to be talking about

nanometers or angstroms scale and in particular what we are going to use is not

light we are going to use electrons to aquire our image because we are at a very,

very small scale. Again, in the orders of nanometers and angstroms. Now, let me just

tell you why we are interested in something very small in the HIV virus. And

I hope this diagram, although a bit complicated. I think It illustrates a lot

of the challenges that we're going to have and why is this so important what I'm

going to explain next. So here we have an HIV virus. Remember we're talking again

maybe 100 angstrom, you know just very, very small structures. And here basically,

we have the host set. So basically, the virus is trying to latch into this host,

and in order to do that, it uses things here which are called envelop or spikes.

And then they go and try to latch here as showin in this diagram. And then basically

transmission foes into the host's cell. So we're interested in this virus and in

particular this might surprise you, and it will surprise you even more when you see

the images. We are interested in these basically, these envelopes or spikes.

They're called envelopes, and often, they're just read as ENV. And, once again,

because that's what the HIV is using to latch itself into the membrane of the host

cell and if we understand that structure, we might be able to stop and to block that

connection and hopefully stop transmission. That's what we want to get.

Now we are at those small scales, remember. And then what we need to use is

not light. We won't be able with light to get to that scale. We use electrons and

here is a diagram. Now once again your use to basically light. So light comes,

reflects in objects like me and then regular cameras like the video camera that

is recording right now, captures the reflection. In transmission electron

microscopy, basically we have this specimen we basically have. A specimen

with HIV, for example, but other specimens are possible as well. Microscopy is used

all over for biological research. Electrons basically go, it's like you

throw electrons into the specimen and you see, and you let them just go through.

Basically you capture the image, you capture the image of the specimen by

measuring how the electrons go through, and we see that in this diagram. There is

a lens, and there is the image plane. That's where things basically get

captured. This is a picture Of this device. The high end devices are not very

cheap they can cost millions of dollars so this is not something that you carry with

you like regular digital camera. Once again this uses electods are hiden in the

speciman basically to go through and then we capture it. So for us, it's a different

way of creating images, but the important way is that we can create images for these

particles that are very, very, very, very small. So that's electron microscopy. And

here is an image of what you will get from electron microscopy. Now remember, have

always in mind, this is what we saw in the previous diagram. This is what we are

looking for. One projection of that. So you took this electron microscopy and you

basically threw electrons and you one projection of that. And that's basically

what you have here so this, is this. Doesn't look very nice but we are going to

have to deal with that and we are actually going to change the molality a little bit

but it's not that it doesn't look very nice because the instrument isn't very

good or the person taking the image is not Very good in doing that. This is actually

extremely good quality for this technology. And we need this technology

because we want to get these very tiny, very, very tiny elements. So we are going

to have to deal with this. Before I do that, before I show you how to do that

This is the beauty of image processing, that we are going to be able to deal with

this. Let me just talk a bit more about tomography. Instead of just taking,

putting the specimen and just taking one view, what, what you do in tomography is

you rotate the specimen. So let me just show you this here. So you have the

specimen and then yo rotate it and you rotate it and you take multiple images of

this rotation and that's what's called tomography. Now, what do I mean by cryo?

Okay. First of all we now know what's electron. Instead of electron microscopy

we say electron tomography because we have multiple projections. Cryo is because,

with the specimen, before we put here, we freeze it at prior temperatures. Now, this

technology helps to reduce radiation. This is very important. When life is hitting

me, you can take as many pictures as you want of me. I'm not going to get hurt by

that. When you throw electrons at the specimen, you're hurting the specimen.

You're changing it. So you're changing the shape of what you're trying to understand

the shape of, and this is very dangerous. So this technique of cryo-electron

tomography helps to reduce the rad iation of that and also helps you to obtain

3-dimension information because you have multiple, multiple tips. So that's

basically what we do and let's see images in a second. But we are doing

cryo-electron tomography. This is just one of those images. Again, remember we're

interested in this kind of stuff. And actually we're interested in these spikes.

You might be able to see some of them there. But that's what we want to get the

3-dimensional shape of that. They're very, very tiny and have received very, very

noisy. So here's where you get the call to image processing. And just to say once

again what kind of data we have. We got tiers. Basically we have the specimen. We

rotate. We throw the electrons that did, we get

one projection. We do a different rotation we get a different projection. A different

rotation, a different projection. Then we put them together and we get the three

dimensions. So I'm going to always show you slices of this, because, most of the

time, I'm going to show you 3D when we get the reconstruction. But remember, this is

supposed to represent this. And we obtain it so here was the specimen, very, very

tiny, and here is the image in the computer. Now, the image processing starts

with putting all these projections together to get the 3D. Normally, although

there are alternatives, that's done with what is called fiducials. So, you

basically embed in this specimen gold particles and then you have a gold

particle, let's; say this pink and then you know where the gold particule is in

every single one of the projections and then you align the images based on those

cold particles. So you throw in a few and you put them together and that helps you

to stack the images and to basically get the 3D in the computer of the real

3-dimensional virus. Now there's one thing I want to explain here. When you are doing

the projection, you can see, this is very thin. And so, I'm illustrating it much

wider. We're talking about very, very thin. So you cryo, micro-cryo-tomography,

cryo-microscopy, you have a very thin slice at cryo-temperatures. So if I do the

projection that I'm going to try to throw electrons from here. You're not going to

get a lot because it's very, very thick. So look at my hand now. it's okay to have

electrons going like this but not going like this. Okay.

So if I have something very, very thin I can just do this. Remember, rotating the

specimen is like rotating the raise. So I can have raise here but not raise here and

maybe also not raise there. So we actually don't have the full 180 degrees. We

normally, out of the 180 degrees. We might basically miss here 30 degrees and another

30 degrees there. And that's called a missing wedge. As we're going to see yet

another problem. We actually don't have all the data we wanted to have because of

this missing wedge. But now we are about to get into the image processing. We

acquired the data. We align the projections using gold particles most of

the time and now it's time to find those viruses and find those envelops. Now, this

is what you see. You have a line and this is what you see. Remember, this is the

schematic of what we are looking for. But these are slices of this 3D that we just

have done. Now do you see here the envelopes or the spikes? No you dont see

them. This is where we need image processing. We have extremly noisy data

and we have the missing wedge. We have tons of problems.Are we going to be able

to reconstruct these 3D. I am going to reassure you, yes that the answer is yes

at a certain accuracy, of course. Again, these are just slices on the 3D volume

that was reconstructed with cryo-tomography. This is what we have,

this is the process that we have. We had basically a virus or a structure inside a

specimen. We took multiple projections. Look how I represented here the missing

wedge. We didn't take the whole around projection. We basically have missing

wedge here so we took that and we kind of get a noisy version in this 3D volume. But

there is 1 thing which is very important. We get multiple copies of these, as we

have seen. Let me show you that again. We may have multiple copies. You see one. You

see another one coming here. I'm going to show you more. We have multiple copies of

these. And we also have multiple copies of this, and that's what we are going to

exploit with image processing. We are going to exploit that we have not one very

noisy example but multiple very noisy examples. So hopefully we can extract

them, put them together, and get the three dimensions that we want. Remember one

thing, when we talk about image denoising we already discussed that if we have

multiple copies of the same object with random noise added, one of the best things

you can do for de-noising is add all those multiple copies. If the noise has zero

mean, and it's random, then it would just cancel each other. That basically was the

concept behind local means. It's also the concept just behind local averaging. You

basically want to average things that are repetitions of the same and that will help

you to eliminate the noise. Remember if you average 10, the value of the noise

goes down by the square of that. So a hundred, the more you average, the more

you manage to de-noise. So that's great but what's the challenge here? Multiple

challenges. These images are very noisy. The spikes, the envelopes, here, are

rotated. We have to align them before we average them otherwise it will be a mess.

Okay, if you average this with this, you let's say these two vectors you get

something like this nothing to do with any of them. So, you have to first align them

and I am going to demonstrate that in one of the next slides. So you have to align

them and also are they identical copies? Maybe there are multiple different

structures. Maybe slightly different, but still different. So we're going to have to

rotate, align them, group only those that are the same. And only after we do all

that, then we're allowed to average. So, here is the problem. Let me illustrate

that to you. Let's say that we have three things as we see here. 1, 2, 3. Now let's

simulate cryo electron tomography for this. We basically get multiple copies of

each one. That's a good part. We get multiple copies of each one, very noisy.

Okay? That's not so good. The multiple copies is good, the noise is not very

good, but we have to deal with that. We get them rotated. That's also not very

good, but we're going to have to deal with that. And then, we get the missing wedge.

Okay? So and of course we don't have the originals, this is what we have. So

somebody gave you this, very noisy data, with missing data, the missing wedge. They

didn't give you in the columns, they are all unorganized. So you have to be able,

from them, you have to be able to group the ones that came From basically the same

building. Here we have to be able to group back here. You have to rotate back and all

that from this noisy data with a lot of missing information. If we have 30 degrees

missing. We have 60 degrees missing out of 180. That's a lot of missing information

and that's represented here. So this holes is a realistic proportion. You have this

group aligned, do everything. And hopefully after you're done, of course,

you're doing all the rotations, all the groupings and everything together.

Hopefully you are able to reconstruct back the virus or the images as we represent

here. So, imagine that you take multiple cameras of the same thing and, but you

occlude the cameras in a random fashion. you add a lot of noise. So now you have

multiple pictures of the same scene. And now somebody comes and say hey, give me

back the shape of that building. Very challenging but hopefully doable because

you have multiple copies. So, what are some of the challenges that we have here?

We have very low SNR. We have this problem with alignments and missing data. And, we

need a lot of free time to remove the noise and basically we are going to have

computational challenges. So lets see how we address each one of these challenges.

In part together and in part separate and this is a very tough problem but I think

your going to get happy to see the kind of resize that we can get. So lets get into

that. The first thing that we have to do is define a distance. Remember we want to

align this projections or this, this three dimensinal shapes but with missing

Information with the missing wedge, and with noise. So, how do I design, going

back to the images in 2D projections of the buildings. How do I design when these

2 are aligned, or they are not aligned? I need to basically define a distance. And

we take any 2 images, we look at the 2 projections, or 2, 3 dimensions,

depending. Depending where are we working, and how do I know if these two are aligned

or not? That's the first thing we need to do, basically to define a distance. We are

going to allow the images to rotate and also translate later on, and we want to be

able to say, hey, keep rotating, keep rotating, stop. That's where you are

align, so I need to define the distance between two images with to noise and with

the missing . So let's illistrate that. That's the first step every time that we

need to align images. How to decide when they are already aligned? ,, . So we have

1 image here, which I represent as a 1 dimension signal. Here, we have this 1

dimensional signal. And we represent it here. But we don't have the whole signal.

We actually have the missing wedge. So we only have the signal in certain parts of

it. Now we have another signal. And also for it, we have a missing wedge. Very

important. We basically do'nt have the missing wedge

in the same place. Remember, these particles were rotated. So the missing

wedge is kind of fixed for a particular aquisition. But the particles are all

rotated so with respect to them the missing wedge is all the time in a

different position. Now if you actually know where the missing wedge is or you can

guess. It's either you know from the instrument or you kind of guess it because

there is a lack of signal to noise. There. And these are basically the two images

that we need to basically decide a distance between them. We wish to have

them like here, complete, but we only have portions of them. And the only place where

we can compare them is when we have Portions of both. So here, we have

information for the first one. But there's no information for the second one. Here,

we had, for the second one, not for the first one. So the only place where we can

compare is when we have information for both of them. So we can take a distance.

Euclidean distance or any distance that you think is appropriate. But we need to

only consider places where both of them are available and that's by this product.

So this and this are basically square binary signals. 1 I have a signal 0, I

don't have a signal. And I can only look at places where both of them are 1. So I

have signals for both of them. again, you can just take that basically almost any

type of metric that you wish, and illustrate it here with just a , Type of

distances. And this is going to be kind of your distance. You take the difference

between the images but you only consider where both of them are available and of

course you have to normalize. By the amount of energy they have per signal, and

that's basically written here. That's standard.

Always a, a dis, distance with the pen of how much overlap you found, and we really

do basically normalize for that. So that's the distance that we're going to use to

illustrate the examples in the next few slides. Now this distance, actually, is

not too hard to compute. You can actually compute in the Fourier domain. We haven't

discussed this Course, too much about Fourier. For those that have a background

in signal processing, you know what Fourier is. But we did talk about DCT, the

Discrete Cosine Transform, so basically, you take the signals, go into a different

domain, in the Fourier domain. And in that domain, you do this calculation. It's very

convenient to do that for computational reasons. It's also very convenient to do

that there, because it's very easy to represent a wedge than missing information

there. So we have this distance which is a euclidean distance, but weighted for the

regions where we have common information.