Notation

: the set of real numbers

: the set of vectors with real components

: the subset of of vectors whose components are all

: the set of integers

: the set of nonnegative integers

: the vector of with components . All vectors are assumed to be column vec-tors unless otherwise specified.

, or : the inner product of and , .

: Euclidean norm of the vector , .

: every component of the vector is larger than or equal to the corresponding component of .

: every component of the vector is larger than the corresponding component of .

OR-1 2015 1

(continued)

, or : transpose of matrix

rank(): rank of matrix

: the empty set (without any element)

: the set consisting of three elements and

: the set of elements such that …

: is an element of the set

: is not an element of the set

: is contained in (and possibly )

: is strictly contained in

: the number of elements in the set , the cardinality of

: the union of the sets and

: the intersection of the sets and

, or : the set of the elements of which do not belong to

OR-1 2015 2

(continued)

such that: there exists an element such that

such that: there does not exist an element such that

: for any element of …

(P) (Q): the property (P) implies the property (Q). If (P) holds, then (Q) holds. (P) is sufficient condition for (Q). (Q) is necessary condition for (P).

(P) (Q): the property (P) holds if and only if the property (Q) holds

, or : graph which consists of the set of nodes and the set of arcs (directed)

, or : graph which consists of the set of nodes and the set of edges (undi-rected)

: maximum value of the numbers and

: the element among which attains the value

OR-1 2015 3

OR-1 2015 4

Backgrounds

Def: line segment joining two points is the collection of points .

(same as

(Generally, , called convex combination)

1x

2x

1x

2x

21212 )1()( xxxxx

)( 21 xx

OR-1 2015 5

Def: is called convex set if and only if whenever , and .

Convex sets Nonconvex set

OR-1 2015 6

Def: The convex hull of a set is the set of all points that are convex combina-tions of points in S, i.e.

conv(S) =

Picture: , for all i, .

(assuming )

x

y

z

OR-1 2015 7

Proposition: Let be a convex set and for , define .

Then is a convex set.

Pf) If , is convex. Suppose .

For any , .

Then .

But , hence .

Hence the property of convexity of a set is preserved under scalar multi-plication.

Consider other operations that preserve convexity.

OR-1 2015 8

Convex function Def: Function is called a convex function if for all and , satisfies , .

Also called strictly convex function if satisfies

, .

1x 2x21 )1( xx

))1(( 21 xxf

)()1()( 21 xfxf

111 ))(,( nRxfx ))(,( 22 xfx

𝑥 (𝑅𝑛)

𝑓 (𝑥 )(𝑅)

Meaning: The line segment joining and is above or on the locus of points of function values.

OR-1 2015 9

OR-1 2015 10

Def: Let . Define epigraph of as epi. Equivalent definition of convex function: is a convex function if and only if epi

is a convex set.

Def: is a concave function if is a convex function.

Def: is an extreme point of a convex set if x cannot be expressed as for dis-tinct

(equivalently, x does not lie on any line segment that joins two other points in the set C.)

: extreme points

OR-1 2015 11

Review-Linear Algebra 2052 4321 xxxx

30545 4321 xxxx

20263 4321 xxxx

2613

5451

1512

A

4

3

2

1

x

x

x

x

x

20

30

20

b

notation vectormatrix,in bAx

inner product of two column vectors : .

If , then are said to be orthogonal. In 3-D, the angle between the two vec-tors is 90 degrees.

( Vectors are column vectors unless specified otherwise. But, our text does not differentiate it.)

OR-1 2015 12

Submatrices multiplication

2221

1211

34333231

24232221

14131211

AA

Aa

aaaa

aaaa

aaaa

A

2

1

4

3

2

1

B

b

b

b

b

b

B

222121

212111

BAbA

BAbaAB

OR-1 2015 13

submatrix multiplications which will be frequently used.

'

'

'

2

1

21

21

22221

11211

m

n

mnmm

n

n

a

a

a

AAA

aaa

aaa

aaa

A

bxAxAxAxA

x

x

x

x

AAAbAx jnj jnnn

12211

4

3

2

1

21

nn AyAyAyAAAyAy ''''' 2121

mi iimm

m

m 'ay'ay'ay'ay

'a

'a

'a

yyyA'y 122112

1

21

:

OR-1 2015 14

Def: is said to be linearly dependent if , not all equal to 0, such that .

( i.e., there exists a vector in which can be expressed as a linear combination of the other vectors. )

Def: linearly independent if not linearly dependent.

In other words, implies for all .

(i.e., none of the vectors in can be expressed as a linear combination of the re-maining vectors.)

Def: Rank of a set of vectors : maximum number of linearly independent vectors in the set.

Def: Basis for a set of vectors : collection of linearly independent vectors from the set such that every vector in the set can be expressed as a linear combination of them. (maximal linearly independent subset, minimal generators of the set)

OR-1 2015 15

Thm: r linearly independent vectors form a basis if and only if the set has rank r.

Def: row rank of a matrix : rank of its set of row vectors

column rank of a matrix : rank of its set of column vectors

Thm: for a matrix A, row rank = column rank

Def : nonsingular matrix : rank = number of rows = number of columns. (determinant of a nonsingular matrix?) Otherwise, called singular

Thm: Let be an matrix. Then has a unique solution if and only if is nonsingular.

Thm: If A is nonsingular, then unique inverse exists.

OR-1 2015 16

Simultaneous Linear Equations

Thm: Ax = b has at least one solution if and only if rank(A) = rank( [A, b] )

Pf) ) rank( [A, b] ) rank(A). Suppose rank( [A, b] ) > rank(A).

Then b is linearly independent of the column vectors of A, i,e., b can’t be expressed as a linear combination of columns of A. Hence does not have a solution.

) There exists a basis in columns of A which generates b. So has a solu-tion.

Thm: Suppose matrix , rank(A) = rank ([A, b]) = r. Then Ax = b has a unique solution if (and only if) r = n.Pf) Let be any two solutions of . Then , or . . Since column vectors of are linearly independent, we have for all j. Hence . (Note that m may be greater than n.)

OR-1 2015 17

Thm: Suppose matrix , rank(A) = rank ([A, b]) = . Then has infinitely many solutions if . (In this case, if , some equations are redundant.)

Pf) Let . Then is nonempty.

Suppose that the first rows of are linearly independent. (Otherwise, we rearrange the rows of without loss of generality.)

Consider . Then .

(Pf: Clearly since any element of automatically satisfies the constraints defining . We will show that .

Since rank(, the row space of has dimension and the rows forms a basis of the row space. Therefore, every row of can be expressed in the form , for some scalars .

Let be an element of and note that

,

(continued)Consider now an element of . We will show that it belongs to .For any , , which establish that and .)Let be expressed as , where is the submatrix of which has linearly inde-pendent rows of as its rows ( is of full row rank). Then and has the same set of solutions. Let after permuting the columns of , where is a matrix with rank and full-column rank. Also let , which corresponds to the partition of as .

Then .

We may assign any values to , then and is uniquely determined since is nonsingular. Hence there exist infinitely many solutions. Note that this also provides a proof for the necessity part of the previous theorem.

OR-1 2015 18

OR-1 2015 19

Operations that do not change the solution set of the linear equations

(Elementary row operations)Change the position of the equationsMultiply a nonzero scalar k to both sides of an equationMultiply a scalar k to an equation and add it to another equation

Outline of the proof for the third operation:

Let ,

Show that implies (which means )

implies (which means )

Hence . Solution sets are same.

The operations can be performed only on the coefficient matrix , for .

OR-1 2015 20

Solving systems of linear equations (Gauss-Jordan Elimination, 변수의 치환 ) (will be used in the simplex method to solve LP problems)

2252

103

104

32

21

321

xx

xx

xxx

2252

2042

104

32

32

321

xx

xx

xxx

2252

102

104

32

32

321

xx

xx

xxx

2

102

206

3

32

31

x

xx

xx

2

6

8

3

2

1

x

x

x

OR-1 2015 21

Infinitely many solutions case

2252

103

104

432

421

4321

xxx

xxx

xxxx

2252

20242

104

432

432

4321

xxx

xxx

xxxx

2252

102

104

432

432

4321

xxx

xxx

xxxx

2

102

2026

43

432

431

xx

xxx

xxx

2

63

88

43

42

41

xx

xx

xx

43

42

41

2

36

88

xx

xx

xx

txtxtx

ttx

2 ,36 ,88

get and arbitrary for Assign

321

4

OR-1 2015 22

43

42

41

2

36

88

xx

xx

xx

method)simplex in the used be (will

0 2, 6, 8, Here

solution. basic a called is variablesdependent for the solving and

0 toariablesindepent v thesettingby obtainedsolution thely,Particular

variables.dependent are ,, and t variableindependen is

4321

3214

xxxx

xxxx

OR-1 2015 23

2

102

2026

43

432

431

xx

xxx

xxx

2

63

88

43

42

41

xx

xx

xx

2

123

248

43

32

31

xx

xx

xx

434241 2 ,36 ,88 xxxxxx 343231 2 ,312 ,824 xxxxxx

different istion representabut solutions, ofset same thehave systemsBoth

0,2,6,8.). 4321 xxxxge

OR-1 2015 24

Elementary row operations are equivalent to premultiplying a nonsingular square matrix to both sides of the equations

2252

103

104

32

21

321

xx

xx

xxx

22

10

10

520

031

411

3

2

1

x

x

x

2252

2042

104

32

32

321

xx

xx

xxx

22

10

10

1

11

1

520

031

411

1

11

1

3

2

1

x

x

x

22

20

10

520

420

411

3

2

1

x

x

x

𝐴𝑥=𝑏

OR-1 2015 25

2252

102

104

32

32

321

xx

xx

xxx

2252

2042

104

32

32

321

xx

xx

xxx

22

10

10

1

11

1

520

031

411

1

11

1

3

2

1

x

x

x

22

10

10

1

11

1

1

2/1

1

520

031

411

1

11

1

1

2/1

1

3

2

1

x

x

x

OR-1 2015 26

22

10

10

1

11

1

1

2/1

1

520

031

411

1

11

1

1

2/1

1

3

2

1

x

x

x

22

10

10

1

2/12/1

1

520

031

411

1

2/12/1

1

3

2

1

x

x

x

22

20

10

1

2/1

1

520

420

411

1

2/1

1

3

2

1

x

x

x

OR-1 2015 27

So if we multiply all elementary row operation matrices, we get the ma-trix having the information about the elementary row operations we per-formed

2252

103

104

32

21

321

xx

xx

xxx

2

6

8

3

2

1

x

x

x

22

10

10

111

25.25.2

65.65.7

520

031

411

111

25.25.2

65.65.7

3

2

1

x

x

x

2

6

8

100

010

001

3

2

1

x

x

x

111

25.25.2

65.65.71A

OR-1 2015 28

Finding inverse of a nonsingular matrix .

Perform elementary row operations (premultiply elementary row opera-tion matrices) to make to for some . Then is .

Let the product of the elementary row operation matrices which converts to be denoted by C.

Then

Hence .

OR-1 2015 29

The form we will see in the simplex method.

Consider , where is matrix with rank . Suppose after permuting col-umns of , where is and nonsingular, and is matrix.

Hence is now expressed as .

Now premultiplying on both sides of the equation is equivalent to per-forming elementary row operations on the equations which converts the coefficient matrix to identity matrix.

Let . Then . The solution is called a basic solution which is considered in the simplex method. By different choice of matrix (nonsingular), we may obtain different solution.