0:00

The axioms of a vector space tell us how to combine vectors together.

So that a linear combination of vector

is the basic operation that we perform in vector spaces.

We take two or more vectors x and y.

We re-scale them by scalar multiplication.

And then we combine them together by a vector addition.

So when we combined vectors together to obtain new vectors in the space.

Initial questions is, whether we can find a minimal set of vectors.

So that we can express any vector in the space.

As a linear combination of this bases factor.

So this set of building blocks will be called the bases.

You can think back of the Lego block analogy that we drew before.

The question would be, can we find a set of

minimal building blocks with which we can build any other piece in the set?

A brief remark about notation where we want to indicate a family of vectors.

And we want to be able to index them.

Then we will use a superscript in parenthesis here.

That indicates the cardinal number of the vector in the family.

1:05

Let's start with the very simple example and introduced the canonical basis for

the Euclidean plane.

Do you remember any vector in the Euclidean plane?

Is the point on the plane.

And this can be express as a pair of coordinates x0 and x1 on the plane.

Now, it's easy to see that if you take two vectors shape like saw,

e0 will be the pair 1, 0 and e1 will be the pair 0, 1.

Then any vector in our R2 can be expressed as a linear combination of e0 and e1.

And the coefficients,

the scalars that you will have to use in this linear combination.

Are actually the values themselves in the original vector.

The fact that the scalars in the linear combination coincide

with the values of the elements of the vector.

Is why we call this basis the canonical basis.

As a graphical example, consider the vector x of coordinates 2 and 1,

so this guy here.

And we can see very easily that we can express this as the linear combination of

two times the horizontal basis vector, canonical basis vector,

plus one time the vertical canonical basis vector.

So if we sum this 2 times 0,

plus this, we have the usual parallelogram rule, and we get back our x.

However, there are other basis that we could choose for the Euclidean plane.

For instance, the basis given by the vectors 1, 0 and 1,

1 also represents a basis for the plane.

And any vector of coordinates x0 and x1,

can be expressed as a linear combination of v0 and v1.

How do we find the coefficients in the linear combination?

Well we have a system of two equations and two unknowns, pays Alpha 0 and Alpha 1.

And we get that Alpha 0 will be equal to x0 minor x1 and

alpha 1 will be equal to x1.

So again graphically the same vector of coordinate 2 and

1 can be expressed as a linear combination of v0 and v1.

And now the coefficients are different because they will be 1 and 1.

And you can see indeed if you sum this guy plus this guy,

again with a parallelogram rule you get back x.

There are in fact an infinite number of bases for R2.

But not all pairs of vectors represent the basis for the plane.

Here's a kind of example, if you take g(0),

the vector of coordinates 1 and 0, so equal to the first canonical basis vector.

And then you take g(1) as -1 0, then it's easy to show that it's not

possible to express any point on the plane as a linear combination of g(0) and g(1).

As a matter of fact, you cannot express any vector for

which the second component is not equal to 0.

Graphically, it's easy to see because you can see that g(0) and

g(1), are actually collinear.

They're in the same line on the plane.

So they can only capture one dimensional component of the plane.

So any linear combination of g(0) and g(1) will be a point on this line.

But you will not be able to reach the other points on the plane.

We express this by saying that the vectors are not linearly independent.

4:18

Okay, so the Euclidean plane, Euclidean space, and

in general, finite dimensional spaces, are well behaved animals.

And so it's kind of easy to believe that there will always be a basis for

this basis.

But what about interdimensional spaces?

We have seen, for instance, the space of infinite length symbols.

What is the basis for that?

Can we write linear combinations of an infinite number of basis factors?

So with some precautions, like for

instance in the case of square summable sequences, the answer is yes.

And the canonical basis for l2 of z for

instance is a collection of an infinite number of vectors.

In which there's only one non-zero component which is equal to one.

Even more interestingly, we can develop basis for function vectors basis.

Take for instance this base of square integral functions over an interval.

You can express any function that vector space

as a linear combination of basis vectors.

Where are these basis vectors?

Again, there exists an infinite number of basis for the space.

But one of the most famous ones is the Fourier basis.

In this case for say the interval- 1, 1.

We can order the basis vectors and

we have that basis vector 0 is a constant 1 over square root of 2.

Basis vector 1 is cosine of pi t.

Basis vector 2 is sine of pi t.

And so on and so forth.

Now, can we really represent any square integrable function over the -1,1

interval as a linear combination of these basis functions?

The answer is yes, and this leads to some surprising results.

For instance, we can approximate a discontinuous function.

Such as a square wave, as a linear combination of continuous functions,

a combination of sines.

6:06

Consider the following sum here for k that goes from

zero to capital N of sin of 2k+1 pi t over 2k+1.

If you considered the order of the basis functions that we showed in

the previous slide.

This is equal to the sum, for

k=0 to capital N of a basis function

W of 4 K + 2 divided by 2K + 1.

So, it is a linear combination of basis functions.

And the basis function of index 4K + 2 is normalize by the factor 1 over 2k +1.

So, in this figure here you see what happens when N is = to 0 so,

you have just to first determine the sum adjusted sin of period pi t.

As you increase the number of terms, so N = to 1.

You see that the shape starts to converge to that of a square wave.

Which to remind you will look exactly like this.

So 1 from 0 to 1 and -1 from 0 to -1.

So we increment the number of terms.

And the more we add,

the more we approximate to the function that we want to approximate.

Now of course, you can see that

8:19

Uniqueness of the representation

is equivalent to linear independence of the vector in the basis.

In other words, if we take linear combination of basis vectors.

And we obtain the null vector this implies that the coefficients in

the expansion are identical to zero and vice versa.

Of all the possible basis for this space.

The most important once are orthogonal basis.

In orthogonal basis, the basis vectors are mutually orthogonal.

And if were particular lucky this vectors will be also unit norm.

Which gives us an orthonormal basis.

We can express the key property of an Orthonormal basis in compact form,

like so.

The inner product between any two vectors in the basis is equal to the delta

of the difference between the indices.

So in other words,

if the vectors are not the same vector, the inner product will be equal to zero.

If it's a self inner product of a basis vector,

the inner product will be equal to one.

The good news is that we have an algorithm called the Gram-Schmidt orthonormalization

procedure.

That will allow us to turn any basis for

a space into an orthonormal basis, if we need it.

So if we have a basis, the questions is how do we find the expansion coefficients

of scalars that we have to use in this representation for

a vector that is a linear combination of basis vector?s.

In general the answer could be.

White involved, but

if we have a orthonormal bases then the procedure is extremely simple.

If we take the inner product between the bases vector number k,

and the original vector.

We directly obtain the expansion coefficient number k.

You can verify this very easily by replacing here x, with this expression.

And then consider the orthonormality of the bases.

Another common question when using basis is how to change

the representation of a vector, from one basis to another.

Take a vector x and consider its

representation as a linear combination of basis vectors from a basis w of k.

Now the question is to find the expansion coefficients of the same vector,

using a different basis v of k.

10:26

If v of k is orthonormal, the solution is extremely elegant.

The expansion coefficient number h and

the new basis can be obtained as we have seen on the previous slide.

As the inner product between the basis vector number h and

the new basis and the original vector.

Now, let's replace the expression for

x by the linear combination of basis vectors from the original basis.

So we get this expression here.

And now, we exploit the distributivity of the inner product to obtain this

expression here.

Where we can see that each new expansion coefficient is actually a linear

combination of the previous expansion coefficients.

What is remarkable however, is that the factors used in this linear combination do

not depend on the vectors that we're manipulating.

These are just inner products between vectors in the original basis and

a new basis.

As a matter of fact, we can write this sum here as matrix vector multiplication.

And this matrix here contains entries that are all the possible cross products.

Between vectors in the original and in the new basis.

In other words, given a starting basis and an orthogonal arrival basis.

You can compute this change of basis matrix once and for all.

And everytime you have to change the reference system for a vector,

you just need to compute a matrix vector multiplication.

We will see very soon that this is really what's behind

the discrete Fourier transform algorithm for finite like signals.

In the meantime,

lets just show a simple example using an elementary rotation of the Euclidean plan.

Lets start with the canonical basis e0,e1.

We know the space is to be orthonormal.

Any vector in the plane can be expressed as linear combination of the two

canonical vectors.

Now, we rotate the plane.

So we rotate the reference system by considering a new basis v0 and v1.

With v0 equals cosine of theta sine of theta.

And v1 minus sine of theta cosine of theta.

Theta is the angle that we rotate the plane by.

We want to find the coordinates of a vector

in this new rotated reference system.

It is easy to verify with basic trigonometry the new basis is also

orthonormal.

So we can use the previous result and try to compute this change of basis matrix.

It will be a two by two matrix.

Where the entries are going to be given by the inner product.

Between every vector in the rotated reference system and

a vector in the canonical system.

Now, because these vectors are very simple, they are just one, zero and

zero, one.

These inner products are going to be very easy to compute.

And the change of basis matrix has this form here.

The coefficients in the rotated reference system

will be given by the product between the change of basis matrix.

And the coefficients in the original system.

Well, we were starting from the canonical reference system.

So these are also the coordinates of the point in the plane.