Vittal Rao Lecture 1

Linear Algebra by Prof. R. Vittal Rao

Lecture 1: August 05, 2010

1.1 Linear Systems

Historically, the subject of linear algebra has its origins in thestudy of linear systems of equations. It is therefore natural that webegin our discussions with a look at linear systems.

Example 1.1 Consider the system of equations

x + 3y = 6x y = 2

}(1.1)

It is easy to see that x = 3, y = 1 is a solution of this system, andthat this is the only solution. We can view this geometrically asfollows:

The first equation represents a line L1 whose X and Y interceptsare 6 and 2 respectively. The second equation represents the lineL2 whose X and Y intercepts are 2 and 2 respectively. They meetat the point whose co-ordinates are x = 3 and y = 1.

0 1 2 3 4 5 6 7

-2-10123

(3, 1)

Thus, in the example, the solution to the system is given by the

co-ordinates of the point of intersection of the lines represented bythe two equations. Viewed from this geometric point of view, wesee that we can encounter the following situations:

1. The two lines intersect at a point, in which case we get exactlyone solution given by the coordinates of the the point of inter-section,

2. The two lines coincide with each other, in which case we get aninfinite number of solutions, given by the coordinates of everypoint on this line, and

3. The two lines are distinct and parallel to each other, in which casethere is no solution since the parallel lines do not intersect atany point.

The example we considered earlier corresponds to the case 1.

Example 1.2 Consider the system

x + 3y = 62x + 6y = 12

}(1.2)

This corresponds to case 2.

1

1.2 The Basic Questions

Example 1.3 Consider the system

x + 3y = 62x + 6y = 10

}(1.3)

This corresponds to case 3.

In general, we would like to look at m equations in n unknowns.We can write such a linear system as

a11x1+ a12x2 + + a1nxn = b1,a21x1+ a22x2 + + a2nxn = b2,...

......

...am1x1+ am2x2 + + amnxn = bm.

In matrix notation, the same can be written as

Ax = b (1.4)

where A = (ai j)1im,1 jn is a known matrix, and for any givenb = (bi)1im we have to find x = (x j)1 jn.

We have to make this a little more precise, as we have to mentionwhere the entries ai j of the matrix A, the entries bi of b are fromand where we are seeking the entries xi of x. For the moment, letus say all are real or complex. We shall denote by F either R or C.Thus, the problem can be precisely stated as

Problem 1.1 Given A Fmn, for b Fm given, to find x Fn suchthat

Ax = b

is the problem of linear systems.


We have seen systems (1.2) and (1.3) which can be written inmatrix notation as

Ax =(

612

)and Ax =

(6

10

)where A =

(1 32 6

). Even though the matrix of the coefficients

is the same, we found out that the system (1.2) has solution(s),whereas the system (1.3) has no solution. Thus, we see that for agiven m n matrix A, the system Ax = b may have solution forsome b Rm and may not have solution for some other b Rm.Thus, the first question we have to ask is the following:

Question 1.1 Given A Rmn, what criterion b Fm should satisfyin order that the system Ax = b can have a solution?

We shall denote such a criterion by [C] and we call it consistencycondition(s). When b satisfies [C] we say the system is consistent.This leads to a series of questions as given in Figure 1.1.

1.2.1 Least Square Solution

By b not satisfying [C], we mean that Ax = b has no solution. Thismeans that whatever x we choose in Rn, Ax will not be equal to b.This means

b Ax , 0 for every x Rn.We define

Eb(x) = b Ax

2


Does b satisfy [C]?

YES

There exists solutionfor problem 1.1

How many?

Exactly One

Under whatconditions

Find theunique solution

Infinitely many


Characterizeall solutions

Find a criterion andpick a unique rep-resentative solution

Find the unique rep-resentative solution

NO (No solution exists)

What can be done?

Can find leastsquare solution

(see section 1.2.1)

How many?

Exactly one


Find the uniqueleast square solution

Infinitely many


Characterize all leastsquare solutions

Find a criterionto pick a unique

representative leastsquare solution

Find the uniquerepresentative least

square solution

Figure 1.1: Series of questions from the consistency condition(s) [C]

3

1.3 Easy Systems

as the errorwe get by taking x as the solution forAx = b. Now, Eb(x)is an m 1 matrix. We quantify this error by the sum of squares ofits entries by defining

eb(x) =mi=1

(bi (Ax)i)2 .

What we can show is that we can find xl Rn such that this erroris minimized, that is,

eb(xl) eb(x) x Rn.Any such xl is called a least square solution.

1.2.2 The Pseudo Inverse

We observe that in the string of questions we have raised, in eachcase, ultimately, we want to find a unique vector as our suitablesolution, as the case may be. This can be viewed as follows:

Consider a system for which we input an x Rn and get anoutput Ax Rm.

x Rn System Matrix,A Rmn Ax Rm

I/P O/P

When we say we want to solve Ax = b, what we are seekingis a desired output b and we want to find which input can thisoutput b. The system Ax = b is consistent means that there is aninput x which will produce exactly the output b. The system isinconsistent means that there is no input which can produce thedesired output b, but we can find input xl which will produce anoutput Axl will error eb(xl) smaller than or equal to the error forany other input.

This means, we have to start from the known b and constructxl. So, we have to construct a new system, for which inputs arefrom Rm and outputs are in Rn. Let the matrix of this system be A(should be n m).

b Rm A Ax RnI/P O/P

We must construct A in such a way that Ab is the solution weare looking for in each of the cases. Such a matrix A is called thepseudo inverse of A.

Remark 1.1 We shall see that

1. When A is n n and A1 exists then A = A1, and2. WhenA is nn andA1 does not exist orA ismn, the pseudo

inverse A still can be constructed.

Thus, the notion of pseudo inverse is more general than that ofan inverse of a matrix.

1.3 Easy Systems

Let us now look at those systems which are relatively easy to beanalyzed. A general system Ax = b is such that the unknown areall coupled in all the equations. Hence the system becomes easierwhen there is less coupling. The system in which the unknownsare not coupled at all is the easiest to treat. So, let us considera system of n equations in n unknowns in which the ith equationinvolves only the ith unknown, xi. Such a system is of the form

aiixi = bi, 1 i n.

4

1.4 Exercises

The matrix of such a system is

D = (aii)1in =

a11 0 . . . 00 a22 . . . 0...

.... . .

...0 0 . . . ann

is a diagonal matrix. Thus, a system whose coefficient matrix isa diagonal matrix is the easiest system. We shall soon see how toanalyze such a system.

1.3.1 Reduction to a diagonal system

The question that arises naturally is whether we can reduce ageneral system of n equations in n unknowns to a diagonal system.Consider,

Ax = b where A is n n matrix.Let us introduce a change of variables,

y = x,c = b

where is an invertible n n matrix. Then, we havex = 1y,b = 1c.

Let P = 1. Then, x = Py, b = Pc. The system Ax = b hencebecomes

P1APy = c.

If we can choose P such that P1AP = D, a diagonal matrix, thenthe system for y is a diagonal system

Dy = c

and hence an easy system. Thus, we can analyze y and hencex = Py. So, the basic question that arises is the following:

Question 1.2 Let F = R or C. Given any nn matrix A over F, canwe find an n n matrix over F which is invertible and such thatP1AP is a diagonal matrix?

If such a P exists, we say that A is diagonalizable over F.

1.4 Exercises

1. Discuss, geometrically, the various alternatives that may arisefor the solution of the system of the type

a1x + a2y + a3z = b11x + 2y + 3z = b2

}where a1, a2, a3, 1, 2, 3, b1, b2 R.

2. For the diagonal system (over R), Dx = b where

D =

2 0 0 00 1 0 00 0 4 00 0 0 6

, b =b1b2b3b4

,discuss the solution of the system.

3. Do the same as above for

D =

2 0 0 00 1 0 00 0 0 00 0 0 0

.

5

August 05, 2010Linear SystemsThe Basic QuestionsLeast Square SolutionThe Pseudo Inverse

Easy SystemsReduction to a diagonal system

Exercises

Documents

Vittal Rao Lecture 1