H.264

Loading...

Del curso dictado por Northwestern University

Fundamentals of Digital Image and Video Processing

867 calificaciones

Northwestern University

Fundamentals of Digital Image and Video Processing

867 calificaciones

In this class you will learn the basic principles and tools used to process images and videos, and how to apply them in solving practical problems of commercial and scientific interests. Digital images and videos are everywhere these days – in thousands of scientific (e.g., astronomical, bio-medical), consumer, industrial, and artistic applications. Moreover they come in a wide range of the electromagnetic spectrum - from visible light and infrared to gamma rays and beyond. The ability to process image and video signals is therefore an incredibly important skill to master for engineering/science students, software developers, and practicing scientists. Digital image and video processing continues to enable the multimedia technology revolution we are experiencing today. Some important examples of image and video processing include the removal of degradations images suffer during acquisition (e.g., removing blur from a picture of a fast moving car), and the compression and transmission of images and videos (if you watch videos online, or share photos via a social media website, you use this everyday!), for economical storage and efficient transmission. This course will cover the fundamentals of image and video processing. We will provide a mathematical framework to describe and analyze images and videos as two- and three-dimensional signals in the spatial, spatio-temporal, and frequency domains. In this class not only will you learn the theory behind fundamental processing tasks including image/video enhancement, recovery, and compression - but you will also learn how to perform these key processing tasks in practice using state-of-the-art techniques and tools. We will introduce and use a wide variety of such tools – from optimization toolboxes to statistical techniques. Emphasis on the special role sparsity plays in modern image and video processing will also be given. In all cases, example images and videos pertaining to specific application domains will be utilized.

De la lección

Video Compression

In this module we discus video compression with an emphasis on motion-compensated hybrid video encoding and video compression standards including H.261, H.263, H.264, H.265, MPEG-1, MPEG-2, and MPEG-4.

Motion-Compensated Hybrid Video Encoding 19:38

On Video Compression Standards 18:22

H.261, H.263, MPEG-1 and MPEG-2 28:32

Conoce a los instructores

Aggelos K. Katsaggelos
Joseph Cummings Professor
Department of Electrical Engineering and Computer Science

H.264 is the latest and bravest of the

video compression standards, although H.265 is along the way.

We described its major advances and differences.

Over preview standards in this segment.

By and large, there are no conceptual

differences with respect to the preview standards.

Again, this hybrid [INAUDIBLE] code that is used.

However, computer technology has advanced, and now it's possible to perform

considerably more complicated operations when a code is implemented even a on

a hand held device, which keep in mind is the code

is are computationally much lighter than

encoders because it involves motion estimation.

And this is why they're implemented first on a small portable devices, we will

see that there are advancement with respect

to motion estimation, special prediction, and entropy encoding.

Since by now, you have knowledge of over two

weeks of material uncompression, everything should make perfect sense.

So, let's check out the details of this particular topic.

Let's us now discuss H.264.

This is the latest and greatest offering by JVT.

There is actually H.265, but its in its phase of completion as

we speak and we'll say a few things potentially about it as well.

So as I mentioned earlier, the rule of thumb is to have twice the compression

efficiency as anything else and the last standard was established in about 1998.

So 264 has the same name as MPEG4 Part 10 because it's the collaboration between

ITU and MPEG and its also carries the name of H.264/AVC advanced video coding.

Initially at least it was carrying the name H.26L.

So if you look in the literature and see any of

these names, you should realize that, you're talking about the same standard.

So we're going to see next what are the distinguishing factors.

What are the similarities, but primarily part of the major differences that

offers so much improve in, improved performance in H.264.

Let us look now at the new features of H.264.

As you'll see these features do not deviate from that

basic block diagram I showed at the beginning of the course.

In other words it's a hybrid motion-compensated video encoder.

However, due to the advancement of, processors, now more

elaborate processing steps can be taken along the way.

And therefore for example additional considerations, additional situations

can be checked when we do motion estimation.

So, specifically, as we'll see, the

motion compensation and intra-prediction differ from previously.

We'll see that, motion compensation can be done, now, with respect to 30 frames in

the past, not just one frame, and

also there are additional intra-prediction modes per macroblock.

Also, the transforms have more flexibility, when DCT is used there

is an integer DCT, and some other differences that we'll talk about.

The deblocking fil, filters are back.

As I mentioned at an earlier point, they were 261 for example.

And then they were not used for a

while and now they, their benefit was again reevaluated.

So they're back into the new features of 8064.

Entropy coding is different.

It's actually, as you'll see,

a context-adaptive arithmetic coding, a topic

we covered two weeks ago when we talk about lossless encoding.

And also there's a difference in how the frames and slices can be defined.

You'll see that it's a mu, a much more flexible structure.

So let's go down the list and examine each one of them in some detail.

So one of the major differences is the user

variable block-size when motion estimation compensation is taking place.

This is not a new idea.

It has been used widely I would say in the literature.

However it was deemed that it was appropriate now again to the

advances in computational power to implement such a, a change while encoding.

So the motivation is clear and simple since

the moving or stationary objects are variable in size,.

If we use a fixed block size if it's

too small then we're losing too many bits to encode.

If it's too large then the prediction is not good.

So clearly we have to have a means to adapt to accommodate both situations.

And this is exactly what H.264 is doing.

So we start with a 16 x 16 macroblock as were

the case in the preview standards, but then we have choices.

So, it can be kept as a whole, or it can be divided horizontally,

or vertically into two sub blocks of size 16 x 8, or 8 x 16.

It could be divided into eight sub-blocks and then the last case, the

four sub-blocks may be divided once more into two or four smaller blocks.

So there are a number of choices, seven actually as

you see, and, but this choice are not arbitrary, they have

actually, a quarry structure review in vision of the splitting of

the 16 x 16 into the situation, then there's, [UNKNOWN] structure.

Of course the price you pay, you have to signal the decoder for

each of the particular blocks I use

when motion estimation compensation is taking place.

So here we see exactly the seven sizes,

different sizes of blocks that they mentioned earlier.

So.

The microblock, it's still at 16 x 16 block.

It's the same definition actually back from MPEG-1 and 2.

And then, the first choice is I just divide it horizontally.

It's shown here.

So this is now 8 x 16.

Or vertically, this is 16 x 8.

It will split this split into four, so each of

them now is 8 x 8, a block in essence.

Further subdivided, the way it is shown here, so this is now a 4 x 8 block.

This is an 8 x 4 block, and each of these little ones are 4 x 4.

And then you can have combinations as shown here.

So, this was divided in four.

And then we have the choice of the vertical and the quarter.

So, you see that, the, large number of combinations of the block

sizes and therefore we can accommodate any scene utilized in this way.

So it offers quite a lot of flexibility.

Again, it's not a arbitrary visual that would be very difficult to signal

to the decoder, but each one of these seven choices, so therefore we

need three beats to let the encode, the decoder know which of these

choices were picked, so which block was

used for motion estimation and motion compensation.

However, the benefit outweighs the cost and this is one of

the features in H.264 that provides quite a lot of benefit.

In this example we show the benefit of

the variable block size in doing motion estimation compensation.

So it's a specific sequence you use, the Bus sequence,

it's a CIF and it's at 30 frames per second.

So on the horizontal axis we see the bit rate; how many bits are sent per second.

And then vertically is the quality

of the constructed video, the [INAUDIBLE] moderation.

So what we see here is that if a fixed block

size is used, then we get this curve, the green curve.

If modes run in four from previously used we get the red curve.

One three the blue curve, and so on.

So, two main observations here.

In going from a fixed to a variable block size, you see quite a bit of benefit.

So, for example, if you take any point here, you can look at it either,

I fix the bit-rate, and then you see that, you gain about

a degree in performance, or if you think the quality, at this point for

example, we see that we say, this is about 200 kilobytes per second.

And the second observation is that the major benefit

comes from utilizing variable block size versus fixed and then

within the fixed it's, it's it's the situation of diminishing

returns like the improvement is not as great as possible.

However, in going from a fixed again this green curve.

To the combinationable seven, if I allow myself to use any

of the seven is shown before then we have taken this care.

And again, the, the, difference either, the difference in

bit rate is over 200 kilobytes per second or in

quality, you gain over 150 just by allowing the encoded

dis-flexibility of variable block sizes when motion estimation is performed.

So let us see with a little toy example here what

is the benefit of, the variable block size in doing motion estimation.

So you see here the two frames that times equals 1 and T equals 2.

And you see that the cloud moved to the right here away from the sun.

Now, this is a microblock, a 16 x 16 microblock that I tried to

match to the previous frame, so that

prediction based on the motion is taking place.

Using this fixed one, the best possible

match is achieved at the location shown here.

So we see that the best possible match does not really reflect what happened

because here the cloud and the sun

overlap, while here they, they're completely separated.

So this is really a very bad, terrible prediction, but

that's the best that we can do under this specific model of block matching.

So here we see the matching of the rest of the blocks.

In which case this is accurate so in

this case there is no need to consider, splitting

the blocks, but using just the fixed block size

can do a good job in, predicting the scene.

But now, if we are allowed to use variable block sizes then it would be reasonable

to split as shown here the block vertically so each of them is 16 x 8.

And then this one on the left is matched to this one.

It has much bigger overlap with the sun while this one to the right

is matched to this one since there is much bigger overlap with the cloud.

So it's again a toy example but I believe it strongly demonstrates the

benefit of this addition of flexibility of using a variable block size.

[BLANK_AUDIO].

One of the major innovations of H.264 in [UNKNOWN] from

previous standards is the use of an arbitrary reference frame.

So, in 263, as we saw, the reference

frame for prediction is always the previous frame.

And we also saw that in MPEG, in other version of H.26L, some frames are

predicted from both the previous and the

next frames and we call them bi-directional frames.

So this is what was done in the past.

Now in 264, any frame can be used as a reference.

So the encoder and decoder maintain synchronized

buffers of available frames that were previously

decoded and their reference frame is just specified by the index into the buffer.

So clearly this allows this tremendous flexibility, because you can match

one block if not in the previous frame if you're allowed to

look at 32 or so frames in the past then you

are bound to find a much better or the best possible match.

Clearly this Increases tremendously storage.

But storage as we know is very, very inexpensive.

So there is no major difficulty here.

But of course it increases the search size.

But this is because of the faster processors

we have available, does not present an issue either.

So we still have the bi-predictive mode.

And now, the, each macroblog can be predicted from one of the two

references and if predicted from both weighted

means of predictors is also a choice.

So this gets not changed, but instead of using two frames, one in

the front, one in the back, we can use arbitrary number of frames.

So here is a simple demonstration of

this multi reference frame motion compensated prediction.

So what has been done so far by all the standards?

I'm at the frame N and I consider just the previous

frame, so then for this specific macroblock here we look and find

the best match in the previous frame as we covered multiple times

and also go and talked about motion estimation earlier in the course.

So this is the best match for this particular case.

So this is where the frame is matched with this particular macroblock.

In the previous standards we only have available the previous

frame and therefore that's the best job we can do.

Now with 264 we're given the option to look at

frame and minus 2 and we find the best match there.

And let's say for example it was with this is a resulting motion vector.

And then we are able to compare if the match at frame N

minus 1 is better or worse than the match of frame N minus 2.

And we are allowed to pick the smaller one.

But we don't stop at N minus 2, we can [UNKNOWN]

additional frames it is a matter of factor allows to look

at 32 frames in the past,and it's should be intuitively clear

that you are bound to find by and large a better solution.

But in the worst case the solution you obtain by using

just frame N minus 1 is included in the other solution.

So you cannot do any worse.

You are only bound to do at least as good, but probably better.

So here's the look at the relative

performance comparison, we compare 263 if we

only use the previous frame, then we

intentionally modify 263 using up to five frames.

This is not allowed by the standard but we want to see even 263 how it will improve.

And then 264 using one frame and using five frames.

So the blue curve, the worst performance, is H.263

with one past frame and doing motion estimation compensation.

And we see that 263, if we're allowed to get the official

way to use up to five frames it moves now to this curve.

So, we see a benefit of about 1dB lets say at 1600 kilobytes per second we see

that, the performance improves by 1dB, with respect to

the video quality, in terms of PSNR of course.

And here we only computed with respect to the Y component.

Now, with, 264, just using one past frame, we have this

considerable jump in performance, and, actually, justifies this, rule of thumb.

That you get the same performance at half the bit-rate.

So it will look, for example, here, at 32 dB right

264 is around here and they should be around 600

or so kilobits per second while its over here for 263

and this is around 1200 kilobits per second.

So its roughly speaking again, twice their age for the same quality.

And then we see that the improvement in 264 by allowing multiple

frames for prediction is not as sizable as in jumping from 263 to 264.

And this would be expected because the, the, the, the higher

you increase the performance the smaller the room for improvement there is.

Intra Prediction is another major notation of

H.264, so we do temporal prediction through

motion estimation but no [INAUDIBLE] special prediction

when an intra-encoded macroblock is, is encountered.

So the motivation is that intraframes

in natural images exhibit strong correlations.

It's the same exact motivation we've been using when we talked about prediction.

This is actually implemented to some extent to 263 and [UNKNOWN]

but it's done in the transform, not in the spatial domain.

So the macroblocks and intra coded frames are based on previously coded ones.

And we need to have already coded pixels because

we want the decoded and the encoded to be synchronized.

To be able to operate on the same exact data, so there is no error [UNKNOWN].

And the past is above and to the left of the

current block, assuming a raster scan of the frame and those

of the macroblocks can be sub-divided as we'll see into 16

4 x 4 sub-blocks which are predicted in a cascading fa, fashion.

Thus shows it right away with an example.

Of course we have to let the decoder know which of

the neighborhoods, which of the modes is used in, doing the prediction.

So there's an overhead but again, the benefit outweighs the cost.

So here the macro block is shown.

So this is a 16 x 16 macro block.

And then as you can see its divided into 4 x 4 blocks.

So the nables to the left, and the Bov are used as predictions.

So this are also 4 x 4 blocks and

these representing coded intensity values so this is recursive computability.

Were using also the past to predict the current values.

So if vertical prediction is used, then

the values of these blocks are propagated vertically.

That's what we're trying to show here with these colored coding.

So this gray value

is propagated this way.

It's different value is propagated this way.

And so on.

So this for the prediction of this micro block under

consideration and we're going to take the difference between the

actual values and the predicted values to form the prediction

error which is going to be sent to the decoder.

So the assumption here is that there is continue

into your smoothness or strong correlation to vertical direction.

Alternatively we can use the horizontal mode

which means now these values are propagated

horizontally so these are all the same values, these are all the same values.

Again that's a prediction, I find the prediction error

and encode it and send it to the decoder.

This prediction along the main diagonal so these values now the same.

So in summary here we have all these modes that are shown here.

So we can predict actually at the 16 x 16 macroblock level.

The prediction is vertical, horizontal, the [UNKNOWN] and the three plane as

demonstrated here, all the prediction can be done at the 4x 4 level.

And it's the vertical, horizontal, the [UNKNOWN] down left diagonal.

Or downright, vertical-right, horizontal-down,

vertical left and horizontal up.

So you see that there are four 16 x 16 modes and then five

plus 49, 4 x 4 modes, and therefore, the idea goes

that each and every situation, I consider all these modes.

And of course we'll utilize the one that will

give you the best performance the smallest prediction error.

So there's a course of course again associated

with this that the decoder needs to be notified

as which one of these all, all, all

of these choices was used in doing the prediction.

But based on excessive, extensive experimentation.

The benefit outweighs the cost and that's why

it has been inter, included in the standard.

So from the list of new features that I

showed at the beginning of this part of the presentation.

We've talked about motion compensation, intra-prediction.

So let's look now at what is

different when image transformations are taking place.

So when it comes to the transformation, the

DCT is used again, as in the previous standards.

However, it's realized that when real number operations are used in fighting the

cosine transform, then inaccuracies can be used when intake the inverse DCT.

So in other words, if I take a block and take the forward and followed

by in the inverse DCT, I will not find exactly the block I started with.

So since now we are using variable

block sizes for motion estimation compensation, there

is less spatial correlation and therefore there is no need for the 8x8 transform.

So what 264 is doing is using a very simple integer 4x4 transform.

So it's an approximation to the 4x4

DCT, but it's implemented using only integerized [INAUDIBLE].

In other words,.

Matrix, matrices that contain only plus minus 1 and plus minus 2.

And then it can be computer clearly with only addition, subtraction and shifts.

So it's a very efficient implementation that does not take

anything away from the implementation of the DCT using real life method.

The loss in quality is negligible, at the level on the average of 0.02dB.

So the gain in, in speed of implementation outweighs the, the little marginal

of loss here in quality, due to this approximation of the DCT that is used.

The deblocking filter is a filter that you use in the feedback loop.

When the constructed pixels are used from

the past to predict the current pixel value.

So they had been used in the past, but, as

we know when we compress and especially at high compression ratio's.

We have these annoying blocking artifacts.

They are artificial, and they, distract from the quality of the image.

They're very visible to the human eye, especially at low bit-rates.

So, previous standards used a deblocking filter, but they used a very simple

one to, as I call it here, to smudge the edges between blocks.

No sophisticated filter was used, and therefore there was no benefit.

That's why they were used early on, and then they were abundant.

But here, what is done in 264, is that there is a choice of filters.

So there's some adaptivity and depending on the type

of edge one of five deblocking filters are used.

And this is just a very reasonable but also known thing that one can do.

You use adaptivity and you adapt to the content of the image

and based on the type of an edge at different filterings apply.

So, for example, if both blocks, both neighboring blocks

have the same motion vector, then less filtering is needed.

The experiment show that the quality improves, not just the PSNR.

So, for the same PSNR.

One can reduce the bit-rate by 7 to 9%, and of course, every little, bit helps.

I should add here that all these benefits and effects are not necessarily additive.

So, we mentioned here a benefit due to the deblocking, earlier,

we mentioned a benefit due to the multiframe motion estimation compensation.

But these two benefits again, they, they do not add, it's not a linear process

and therefore one has to include all of them to see what the benefit is.

So heres an example of the benefit of the use of the de-blocking filter.

This is a frame from foreman, so no deblocking filter on the left.

The PSNR is 35.07 dB.

And while the deblocking filter is used, [UNKNOWN] on the right,

then the PSNR increases by about 35 or so, point 35 that is dB.

Although in terms of PSNR, the, the benefit is not tremendous.

If you have a close look at the images, the visual quality does improve.

And this is one of the issues of how exactly

one can measure the, the visual quality of an image.

But by and large, the blocking is part of 264, and the benefits are there.

Moving down the list of the new features

in 264, let's see what's new in entropy coding.

We have covered already these three new features.

So with respect to the new features in entropy coding.

The, we have to look at what was done in the past

and in the past the traditional code use fixed, variable length codes.

The Huffman style codes they're non-adaptive and they cannot

encode symbols with probability greater than than a [INAUDIBLE] efficiently.

Since at least one bit is required.

And these are all points that we made when we were covering

[INAUDIBLE] image and data compression in week eight of the class.

263 defines an arithmetic coder.

But it is still non-adaptive and it uses multiple non-binary alphabets.

And this results in high, high computational complexity.

So,.

There was an improvement, improvement already in [UNKNOWN] 63.

But with [UNKNOWN] 64 this so called CABAC called the deductive binary code is used.

So this is a framework that was specifically designed for [UNKNOWN] 64.

And it binarizes all syntax symbols.

Which are translated into bit strings.

So, there are 399 predefined context models, and these

are used in groups to encode the various parameters.

So, for example, models 14 to 20 are

used to code the macroblock type for inter-frames.

If you recall we have all these possibilities in doing intraframe

prediction and therefore each of them needs to be signaled to

the decoder and the model to use is selected based on

previously encoded information as the context basic same idea that we covered.

But a number of times already.

So using the same test sequence as before.

This is the bus sequence CIF at 30 frames per second.

We show the relative gain for going from universal

variable length codes to this binary context adoptive binary.

So this is a [UNKNOWN] for VLC.

And this is the [UNKNOWN] for CABAC.

So, you do see some, some benefits certainly.

You might say it's not tremendous, but it's, it's sizeable.

So, for the same quality here, it will look at 32 dB PSNR.

We see that there is about 100 kilobits per second or so improvement.The

fileum, new feature we're going to address is that of frames and slices.

So along this lines in previous standards, 263 and

MPEG each frame is either P, interframe or I, intraframe.

Exception is that some macroblocks in P-frames

may be intra-coded, and are called I-blocks.

But now with 264, this is generalized.

And each frame now consists of one or more slices.

The slices are quite flexible.

They are formed by contiguous.

Groups of macroblocks, and they're processed in internal raster order.

The main concept of a slice, which was also included in previous standards, is

that this group of macroblocks can be independently encoded and decoded.

So they've used, been used by and large to stop error propagation.

Due to this independent encoding decoding, there is no, there are no dependencies

from one slice to anything outside the slice and now we have,

we can have intrapredictive and also bidirectoral, bidirectional

or B-slices, so looking at applications [UNKNOWN]

264 [UNKNOWN] demand a strong processor.

It should be clear based on all the

many computations and choices we have to perform.

It has been used in cellular products but usually the baseline.

So it's a simplified version of 264 it does not

have as the typically refer to all the bells and whistles.

And, but there are many variants to decode only.

Its also been used in set top boxes.

So broadcasting, in conjunction with transcoders and

for personal media, such as camcorder storage.

It's, you might say an overkill for surveillance applications.

But sometimes is used in high end systems so

there's no such great need for a survalence video

but typically stationary nothing is happening and only once

in a while an event of interest is taking place.

So in providing some commentary regarding 264 on one

hand is this magnificent latest and greatest of the standards.

It includes very competitive compression, application friendly user

uses, such as the flexible macroblock orbiting here.

And the flexible slicing.

And also it includes a large selection of optional tools

in the higher profiles, such as the film grain analysis,.

But it's also a cumbersome standard.

We, we see that there are so many computations that have to, to take place.

And it's a much higher complexity than 263+ or MPEG2.

And some of the complicated features is

the 4x4 transformation, the smaller inter prediction blocks.

And again there are many more choices, many more features to run such as

the multiple reference frame, the deblocking filter

that we talked about the quarter-pel motion estimation

motion conversation and then actually one can

use also [UNKNOWN] adaptive VLC's for half [UNKNOWN]

encoding but the content of that binary

[INAUDIBLE] code is the main feature of it.

So, in summary, with expect to 264, some of the very strong result is the quarter

pixel prediction, CABAC and the deblocking filters, and it's been the

major improvement since the 90s and here's this two-fold improvement in performance.

Explora nuestro catálogo

Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.

Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.

© 2019 Coursera Inc. Todos los derechos reservados.

Descargar en la App Store

Consíguelo en Google Play