Matrix Multiplication

Chapter III – General Linear Systems

By Gokturk Poyrazoglu

The State University of New York at Buffalo – BEST Group – Winter Lecture Series

General Outline

1. Triangular Systems

2. The LU Factorization

3. Pivoting

Triangular Systems Outline

1. Row oriented Forward Substitution

2. Row oriented Back Substitution

3. Column oriented Forward Substitution

4. Column oriented Back Substitution

5. Multiple Right-Hand Sides

6. Nonsquare Triangular Systems Solving

7. Algebra of Triangular Matrices

Forward Substitution

Consider a lower triangular system

The unknowns are:

General Procedure:

Row Oriented Forward Substitution

L is a square lower triangular matrix

b is a vector

Overwrite b with the solution of Lx=b

Row Oriented Back Substitution

U is a square upper triangular matrix

b is a vector

Solution x:

Overwrite b with the solution of Ux=b

Column Oriented Versions

Example:

Solve for x1 (x1=3); remove from the equations by

Column Oriented Forward Substitution

L is a square lower triangular matrix

b is a vector

Overwrite b with the solution of Lx=b

Column Oriented Back Substitution

U is a square upper triangular matrix

b is a vector

Overwrite b with the solution of Ux=b

Multiple Right-Hand Sides

Consider block matrices L, X, and B

Solve L11X1=B1 for X1.

Remove X1 from block equations as follows :

Nonsquare Triangular System Solving

Consider a block matrix L where m > n

Assume L11 is nonsingular and lower triangular.

Solve L11x=b1 for x

Then x should solve the system 𝐿21 𝐿−111𝑏1 = 𝑏2

Otherwise, there is no solution to the overall system.

Triangular Matrix Properties

Unit Triangular Matrix:

Triangular matrix with 1’s on the diagonal.

Other Properties

Inverse of an upper triangular is an upper triangular matrix.

The product of two upper triangular is an upper triangular.

Inverse of a unit upper triangular is a unit upper triangular.

The product of two unit upper triangular is an upper triangular.

The LU Factorization

Outline

Background

Gauss transformations

Application

Upper Triangularizing

Existence of LU

Other versions of LU

Rectangular Matrix

Block LU

Background

Example :

Matrix Notation:

Multiply the 1st equation by 2,

Subtract it from the 2nd equation.

Gauss Transformations

Consider a vector V (a stack of 2 block vectors v1 and v2)

Suppose , and define Gauss Vector

Define Gauss Transformation matrix

Application

Consider a matrix C, and apply an outer product update

Repeat the process;

Algorithm

Upper Triangularizing

Consider a square matrix-A, then

Example:

Note :Diagonal components of A (pivots) should be zero

Existence of LU

If no zero pivots are encountered; then

and

so that

LU factorization does NOT exist if is singular.

That means kth pivot is zero.

Construction of matrix-L

Consider the example

kth column of L is defined by the multipliers from kth step

Outer Product Point of View

Consider a matrix A as;

Gauss Elimination results:

where

Hence;

LU Factorization of a Rectangular Matrix

Such matrices L and U exist if is nonsingular.

Examples:

Algorithm for the 1st example:

Operation: nr2-r3/3 flops

Roundoff Error in Gaussian Elimination

Triangular Solving with Inexact Triangles

If a small pivot is encountered, then we can expect large numbers to be

present in L and U.

Example :

Solution is in contrast to exact solution

Pivoting

Outline

Interchange Permutations

Partial Pivoting

Complete pivoting

Rook Pivoting

Interchange Permutations

Consider a permutation matrix:

If we multiply matrix A from the left,

rows 1 and 4 interchanged

If we multiply matrix A from the right;

columns 1 and 4 interchanged

Partial Pivoting

Motivation: To guarantee that no multiplier is greater

than 1 in absolute value.

Example : Consider matrix-A, get the largest entry in

the first column to a11

Partial Pivoting

Particular row interchange strategy is called partial

pivoting.

In general :

where U is an upper triangular, and no multiplier is greater

than 1 in absolute value as a consequence of partial

pivoting.

Complete Pivoting

Partial Pivoting was scanning the current subcolumn for

maximal element;

Complete Pivoting scans the current submatrix to find the

largest entry to pivot.

Hence;

Complete Pivoting Properties

Gaussian Elimination with Complete Pivoting is STABLE.

No significant reason to choose Complete pivoting over

Partial pivoting.

Only if matrix A is rank deficient.

In principal, when the pivot of current submatrix is ZERO

at the beginning of step r+1; that indicates that the

rank(A) =r;

In practice, encountering an exactly zero pivot is lesslikely.

Rook Pivoting

Computes the factorization :

Choosing Pivot: Search for an element of current submatrix

that is maximal in BOTH its ROW and COLUMN.

Complete Pivoting

Rook Pivoting Candidate

Comparison

Flops:

Partial pivoting : O(n2)

Complete Pivoting O(n3)

Rook Pivoting O(n2)

Rook has same level of reliability as complete pivoting

and represents same O(n2) overhead as partial pivoting.

Complete Pivoting may be used for rank identification in

principal.

All pivoting methods are stable.

Extra Proof Slides

Proof of LU Factorization Theorem (slide 18)

Proof of Slide 23