Signal vector spaceSignal vector space

Def, Vector space, V over a field (scalar),F

• Vector addition, VxV->V, and scalar multiplication FxV-p>V, plus

• 8 axioms– 1) Commutativity of addition1) Commutativity of addition– 2) Associativity of addition– 3) Identity: There exists an element of V, denoted by 0, such that

v+0=v, for any element v of V.v 0 v, for any element v of V.– 4) Inverse: For each element v of V, there exists an element,

which we denote as –v, such that v + (-v) = 0– 5) Scalar associativity) y– 6) 1v=v– 7) a(v+u)=av+au– 8) (a+b)v=av+bv8) (a b)v av bv

Uniqueness of the identityUniqueness of the identityProposition: Let , be elements of such that . Then, 0.P f

v w V v w v w+ = =Proof:

( ) ( )0 0

v w vv v w v v

+ =− + + = − +

0 00 Q.E.D.

This proposition implies that the identity is unique.

ww

+ ==

Proposition: Let , be elements of such that 0. Then, .Proof:

v w V v w w v+ = = −

0( ) ( ) 00

v wv v w vw v

+ =− + + = − ++ = −

Q.E.D.This proposition implies that for each , its inverse is unique.w v

v= −

Basic Algebra: GroupBasic Algebra: Group

• A group is defined as a set of elements GA group is defined as a set of elements Gand a binary operation, denoted by · for which the following properties are satisfiedg p p– For any element a, b, in the set, a·b is in the

set.– The associative law is satisfied; that is for

a,b,c in the set (a·b)·c= a·(b·c)There is an identity element e in the set such– There is an identity element, e, in the set such that a·e= e·a=a for all a in the set.

– For each element a in the set there is anFor each element a in the set, there is an inverse element a-1 in the set satisfying a· a-1

= a-1 ·a=e.

Group: exampleGroup: example

• A set of non-singular n×n matrices of real numbers, with matrix multiplication

• Note; the operation does not have to be commutative to be a Group.

• Example of non-group: a set of non-Example of non group: a set of nonnegative integers, with +

Unique identity? Unique inverse of h l ?each element?

• a·x=a Then a-1·a·x=a-1·a=e so x=ea x a. Then, a a x a a e, so x e.• x·a=a

• a·x=e. Then, a-1·a·x=a-1·e=a-1, so x=a-1.

Abelian groupAbelian group

• If the operation is commutative the groupIf the operation is commutative, the group is an Abelian group.

The set of m×n real matrices with +– The set of m×n real matrices, with + .– The set of integers, with + .

Application?Application?

• In channel coding (for error correction orIn channel coding (for error correction or error detection).

Algebra: fieldAlgebra: field

A fi ld i t f t l t• A field is a set of two or more elements closed under two operations, + (addition) and * (multiplication) with the(addition) and (multiplication) with the following properties– F is an Abelian group under additionF is an Abelian group under addition– The set F−{0} is an Abelian group under

multiplication, where 0 denotes the identity p , yunder addition.

– The distributive law is satisfied: (α+β)∗γ = α∗γ+β∗γ

Immediately following propertiesImmediately following properties

Proposition: α∗β=0 implies α=0 or β=0Proposition: α∗β=0 implies α=0 or β=0Proposition: For any non-zero α, α∗0= 0

P f 0 0 1 (0 1) 1 Proof: α∗0 + α = α∗0 + α ∗1= α∗(0 +1)= α∗1=α; therefore α∗0 =0

P iti 0 0 0Proposition: 0∗0 =0Proof: For a non-zero α, its additive inverse is

0 0 ( ( ) ) 0 0 ( ) 0 non-zero. 0∗0=(α+(− α) )∗0 = α∗0+(− α)∗0 =0+0=0

Examples:Examples:

• the set of real numbersthe set of real numbers• The set of complex numbers

L t fi it fi ld (G l i fi ld ) ill b• Later, finite fields (Galois fields) will be studied for channel coding– E.g., {0,1} with + (exclusive OR), * (AND)

Vector space

• A vector space V over a given field F is a set of p gelements (called vectors) closed under and operation + called vector addition. There is also an operation * called scalar multiplication, which p p ,operates on an element of F (called scalar) and an element of V to produce an element of V. The following properties are satisfied:following properties are satisfied:– V is an Abelian group under +. Let 0 denote the additive

identity.For every v w in V and every α β in F we have– For every v,w in V and every α,β in F, we have

• (α∗β)∗v= α∗(β∗v)• (α+β)∗v= α∗v+β∗v

α∗( v+w)=α∗v+ α ∗wα∗( v+w) α∗v+ α ∗w• 1*v=v

Examples of vector spaceExamples of vector space

• Rn over RR over R• Cn over C

L R L C• L2 over R, L2 over C

SubspaceSubspace.

Let be a vector space and . If is also a vector space with the same operations as ,h S i ll d b f

V S VS V

V

⊂

then S is called a subspace of .

S i b if

V

S is a subspace if , v w S av bw S∈ ⇒ + ∈

Linear independence of vectorsLinear independence of vectors

1 2

Def)A set of vectors , ,..., are linearly independent iffnv v v V∈

BasisBasis

C id t ( fi ld)V F

0 0

Consider vector space over (a field).We say that a set (finite or infinite) is a basis, if * for every finite subset , the vectors in are linearly independent, and

V FB V

B B B⊂

⊂ * for every x

1 1

1 1

, it is possible to choose , ..., and , ...,

such thatn n

Va a F v v B

x a v a v

∈∈ ∈

= + +1 1 such that ... .

The sums in the above definition are all finite because without ddi i l h i

n nx a v a v+ +

f d iadditional structure the axioms of a vector space do not permit us to meaningfully speak about an infinite sum of vectors.

Vector spaceVector space

A set of vectors is said to span ifv v v V V∈1 2

1 2

A set of vectors , ,... is said to span if every vector is a linear combination of , , ..., .

n

n

v v v V Vu V v v v

∈∈

Example: nR

Finite dimensional vector spaceFinite dimensional vector space

• A vector space V is finite dimensional ifA vector space V is finite dimensional if there is a finite set of vectors u1, u2, …, un that span Vthat span V.

Finite dimensional vector spaceFinite dimensional vector space

Let V be a finite dimensional vector space. Then

If are linearly independent but do not span thenv v v V V• 1 2

1 2

If , ,..., are linearly independent but do not span , then has a basis with vectors ( ) that include , , .

m

m

v v v V Vn n m v v v

•>

1 2If , , ..., span anmv v v V•

1 2

d but are linearly dependent, then a subset of , ,..., is a basis for with vectors ( ) .mv v v V n n m<

Every basis of contains the same number of vectors. V•

Dimension of a finiate dimensional vector space.

Example: Rn and its Basis VectorsExample: R and its Basis Vectors

•••

Inner product space: for length and langle

Vector space over . Inner product is a mapping such that

V CV V C× →

*1) , ,2) , , ,

u v v uu v w u w v wα β α β

< >=< >< + >= < > + < >

* * (Conequently, , , , )3) , 0, with equality if and only if 0.

w u v w u w vu u v

α β α β< + >= < > + < >< > ≥ =

Example: RnExample: R

•••

•••

•••

•••

Follows the notion of orthogonality, ( i )norm (metric)

Def)

Orthonormal set and projection htheorem

Def)A non-empty subset of an inner product space is said to beorthonormal iff

Sorthonormal iff1) , , 1 and2) If , and , then , 0.

x S x xx y S x y x y

∀ ∈ < >=∈ ≠ < >=

Projection onto a finite dimensional bsubspace

Def) If is a subspace of an inner product space , and , the projection of on is defined to be a vector such that is orthogonal to all vectors in .S S

S V u V u Su V u S u u S

∈∈ ∈ −

Projection Theorem (Gallage

1

r Thm 5.1)Let be an n-demensional subspace of an inner product space and assume that , ,is an orthonormal basis of S Then any may be composed as where

nS Vu V u u u

φ φ∈ = +is an orthonormal basis of S. Then, any may be composed as where

and , 0 SS

SS

u V u u u

u S u s s S⊥

⊥

∈ = +

∈ = ∀ ∈

1

. Furthermore, is uniquely determined by

, .

S

nj jS j

u

u u φ φ=

= ∑ 1 j jj=∑

Projection onto a finite dimensional bsubspace

22 222 2

2 2

Form Pythagorean theorem .SSu u u⊥= +

2Norm bounds: 0

with equality on the right iff and equality on thh left iff is orthogonal to all vectors in

Su u

u Su S

≤ ≤

∈

1

equality on thh left iff is orthogonal to all vectors in .

Bessel's inequality: 0 ,njj

u S

u φ=

≤ ∑2 2 u≤

1 jj=∑ with equality on the right iff and equality on thh left iff is orthogonal to all vectors in .

u Su S

∈

Least squared error property: ,Su u u s s S− ≤ − ∀ ∈

Gram –Schmidt orthonormalizationGram Schmidt orthonormalization

{ }1

1

Consider linearly independent , ..., , and inner product space.We can construct an orthonormal set , ..., so that

n

n

s s VVφ φ

∈

∈

{ }1 1 { , ..., } , ...,n nspan s s span φ φ=

Gram-Schmidt Orthog. Procedure

ExamplesExamples

• Euclidian space Rn (over R)Euclidian space, R , (over R)• Cn, (over C, over R)

L2 f l f ti ( R)• L2 of real functions, (over R)• L2 of complex functions, (over C, over R)• Real random variables, (over R)• more abstract examples… more abstract examples…

Application: Detection in Gaussian noiseA di A 2Appendix A.2

Application: linear least square estimation of d t Xrandom parameter X

1 1 1Y h X W= +

( ) ( )2 2 2

and are uncorrelated. 0i i

Y h X WX W E X E W

= +

= =

Linear least square estimation of .X

Application: linear least square i i f destimation of random parameters

( ) ( )d l d 0

Y HX W

X W E X E W i j

= +

∀( ) ( )( )

and are uncorrelated. 0, ,

ˆDesign linear least square estimator s.t.i j i jX W E X E W i j

x y Ay

= = ∀

=

( ){ }2

1ˆ is minimized.n

i iiE X x Y

= − ∑

Application: linear least square i i f destimation of random parameters

( ) ( )and are uncorrelated 0

Y HX W

X W E X E W i j

= +

= = ∀( ) ( )( )

( ){ }2

and are uncorrelated. 0, ,

ˆDesign linear least square estimator s.t.

ˆ i i i i d

i j i j

n

X W E X E W i j

x y Ay

E X Y

= = ∀

=

∑ ( ){ }1ˆ is minimized.

Concept of sufficient information: projection onto the direction of H

i iiE X x Y

= − ∑

Concept of sufficient information: projection onto the direction of H.(Start the discussion with the real randomvariables first.)