LINEAR ALGEBRA - CHAPTER 1: VECTORS

LINEAR ALGEBRA - CHAPTER 1: VECTORS

A game to introduce Linear Algebra

In measurement, there are many quantities whose description entirely rely onmagnitude, i.e., length, area, volume, mass and temperature. Alternatively, thereare quantities whose magnitude do not entirely determine their nature, velocity, ac-celeration and force all depend on the magnitude and their direction to be describedfully. Geometrically these are represented as arrows or a directed line segment of aspecific length. Originally introduced in the nineteenth century, vectors have a wideapplication outside of mathematics, in particular in physics,engineering, economics,computer science, statistics and the life and social sciences.

In this chapter we will introduce vectors and consider their geometric and al-gebraic properties. As an alternative viewpoint we will introduce a non-geometricapplication with particular utility to computer science. However as a first intro-duction we examine the rules for a simple game played on graph paper. This gamewill be a racing game, and hence requires a track with starting and finishing lines,the track may be as complicated as one likes as long as it may accommodate eachplayer at the starting line. As an example, we will play this game with two playersA and B.

To start, both players begin by drawing a point on the starting line to indicatewhere they are at time zero. They then take turns moving to a new point subjectto the rules [1]:

(1) Each point and the line segment connecting it to the previous point mustlie entirely within the track.

(2) No two players may occupy the same point on the same turn.(3) Each new move is related to the previous move as follows: If a player moves

a units horizontally and b units vertically on one move, then on the nextmove the player must move between a− 1 and a+ 1 units horizontally andbetween b− 1 and b+ 1 units vertically.

1

2 LINEAR ALGEBRA - CHAPTER 1: VECTORS

Figure 1. A sample game of Race track.

To better understand this game, there are some questions we can ask about it:

• Question Use the notation [a, b] to denote a move that is a units horizontallyand b units vertically. If [3, 4] has been made, what is the region containingthe points that can be reached on the next move?• Question What is the net effect of two successive moves?• Question Assuming player A starts at the origin (0, 0) can we express their

moves using the [a, b] notation?• If the axes were translated so that player A begins at (2, 3) what is the

coordinate of the final point on player A’s trajectory?

This game actually contains the core ideas of this section, as it contains an alge-braic and geometric interpretation of vectors that may be generalized to higherdimensions than a two-dimensional plane.

Vectors as geometric and algebraic objects

Vectors in two-dimensions: the plane. To introduce the concepts of vectors,we start by considering a familiar example, the Cartesian plane. A vector is definedas a line between two points in the plane along with a direction, i.e, a directed line

segment. The vector from A to B is denoted−→AB - the point A is called the initial

point or tail, while B is the terminal point or head. Frequently the labels forthe points are omited and instead we denote a vector in bold as v or −→v in thehand-written figures.

The collection of all points in the plane is equivalent to the set of all vectorswhose tails are at the origin O = (0, 0); for each point A, there is a corresponding

vector a =−→OA and vice versa. It is always helpful to work with coordinates, one

LINEAR ALGEBRA - CHAPTER 1: VECTORS 3

may give a precise representation of vectors. For example, in the following figure

A = (1, 3) while B = (4, 5) yielding the vector−→AB = [3, 2], alternatively the point

C = (−2, 2) gives the vector−→OC = [−2, 2]. Although the zero vector cannot be

drawn, it is perfectly acceptable to include it as a vector as the zero vector,

0 =−→OO = [0, 0].

Figure 2. Vectors or line segments - both are acceptable.

The individual coordinates are called the components of the vector; the positionof the components is vital as the vectors with [a, b] and [b, a] a, b ∈ R will genericallybe different except in the trivial case with a = b. We say two vectors are equalif and only if their corresponding components are equal. In calculations it will

be convenient to use column vectors instead of row vectors, i.e.

[ab

]instead

of [a, b]. These two representations are related through the transpose operation,

[a, b]T =

[ab

], due to this fact we will use both representations. The set of all

vectors with two components will be denoted as R2 where R is the set of all realnumbers (the rational and irrational numbers).

Returning to the race track game, we can interpret vectors whose tails are notat the origin in the context of the game. A vector [a, b] may be interpreted as aplayers first move from the origin where they travel a units horizontally and b unitsvertically. The same displacement may be applied with a different starting point;the following figure (3) shows two equivalent displacements:


Figure 3. A vector in standard position and the same vector atdifferent points for its tail.

We say two vectors are equal if they have the same length and direction. In figure

(3)−→AB =

−→CD. In a geometric sense, two vectors are equal if one can translate one

vector so that it overlaps the other. We will say a vector such as−→OB with a tail

at the origin is said to be in standard position. Furthermore we now know thatevery vector in the plane can be drawn as a vector in standard position and anystandard position vector may be translated so that its tail is at any point.

Example 0.1. Q: If A = (−1, 1) and B = (2, 3), find−→AB and redraw this vector in

standard position and with its tail at the point C = (2, 2)

A: Calculating the difference in the components−→AB = [2 − (−1), 3 − 1] = [3, 2].

Translating−→AB to

−→CD: D = (3 + 2, 2 + 2) = (5, 4).

Figure 4. Addition of vectors


One vector, two vector: new vector. If one is playing the race track game,at each turn, one must follow one vector by another. If one were to combine twomoves in terms of vector notation, can we add them to get another vector? Theanswer is yes, in fact vector addition is one of the most basic vector operations.As an example consider u = [1, 3] and v = [3, 2] as two moves in the game, we candetermine the total displacement as a third vector u + v. In figure (5) we see thatthe sum will be u + v = [4, 5] which may be seen geometrically From this examplewe may derive a simple formula for the vector sum in terms of the components:

u + v = [u+v1, u2 + v2].

Alternatively by translating u and v parallel to themselves we produce a par-allelogram. Moving the vectors to standard position we have the parallelogramdetermined by u and v.

Theorem 0.2. The Parallelogram Rule Given vectors u and v in R2, their sumu + v is the vector in standard position along the diagonal of the parallelogramdetermined by the vectors.

Figure 5. The Parallelogram Rule - when it looks like a square.

Example 0.3. Q: If u = [2, 3] and v = [4,−2] computer their sum and draw it.A: Adding the components of the vectors, u + v = [2 + 4, 3 + (−2)] = [6, 1]. Usingthe parallelogram rule we have:

Figure 6. The sum of u and v.


The next vector operation is scalar multiplication; given a vector v and areal number c, the scalar multiple cv. This is computed by multiplying eachcomponent of the vector by c:

cv = c[v1, v2] = [cv1, cv2]

Example 0.4. Q: If v = [1, 3] what is 2v, 12v and − 1

2v. Draw these.A: These quantities are easily calculated using coordinates,

2v = [2, 6],1

2v = [

1

2,

3

2], −1

2v = [−1

2,−3

2].

Figure 7. Scalar multiples of v.

Notice that cv has the same direction as the original vector if c > 0 and theopposite direction if c < 0; furthermore cv is |c| times as long as v. In the contextof vectors, constants will be called scalars. Taking into account that we may alwaystranslate vectors, we say they are parallel if and only if they are scalar multiplesof each other.

Combining these two operations we may now define vector subtractions as

u− v = u + (−1)v

Figure (8) in the following example shows u − v will be the other diagonal of theparallelogram determined by u and v.

Example 0.5. Q: If u = [3, 1] and v = [2, 3] what is u− v?A: Choosing coordinates the sum will be u + (−v) = [3− 2, 1− 3] = [1,−2].


Figure 8. Geometric derivative of u− v.

Vectors in Rn. By adding another component to our vectors we may extend thework done to three dimensions, by considering the ordered triples of real numbersdenoted as R3 Points and vectors may now be defined by choosing coordinatesand identifying their position on the x, y and z axes. One may verify that vectoraddition and scalar multiplication behave as one would expect by actually drawingthe usual two-dimensional immersion of R3.

If we want to work in spaces like Rn with n > 3, we can no longer resort todrawing the vectors and points in space. Instead, we must resort to more symboliccalculations, we define Rn as the set of all ordered n-tuples of real numbers writtenas either row or column vectors,

[v1, v2, ..., vn] or

v1v2...vn

.If i ∈ [1, n] we say vi is the i-th component. Then the addition of two vectors u andv consists of adding each component, ui + vi, while scalar multiplication is thencui.

As we can no longer draw n-dimensional vectors, we must have some sure wayto calculate with vectors. To do so we explore their algebraic properties. As anexample, we notice that addition is commutative that is u + v = v + u for anytwo vectors. In two and three dimensions this may be seen and verified directly, inhigher dimensions one must resort to the formulas to prove this fact. In this mannerwe may prove the following theorem listing the algebraic properties of vectors.

Theorem 0.6. Let u,v and w be vectors in Rn, and c and d be scalars.

(1) u+v= v+u(Commutativity)(2) (u+ v)+w= u+(v+w) (Associativity)(3) u+ 0 = u(4) u+ (-u) = 0(5) c(u+v)=cu+ cv(Distributivity)(6) (c+d)u= cu+ du(Distributivity)(7) c(du) = cd u(8) 1u= u


+ 0 1

0 0 1

1 1 0

* 0 1

0 0 0

1 0 1

Here is an example to illustrate the utility of Theorem (0.6) for algebraic compu-tations of vectors

Example 0.7. Q: Let uand vand wdenote vectors in Rn. Simplify the expression,3u+ (5v−2u) + 2(v−u) and solve for w from the expression 5w−u = 2(u+ 2w).A: In the first case by applying the above rules we find this may be written as7v − u. In the second case this first becomes 3w = 3u, then dividing by 3 yieldsw = u.

Linear Combinations of vectors. If a vector may be expressed as a sum ofscalar multiples of other known vectors, we say this is a linear combination of thosevectors. More formally we have the definition

Definition 0.8. A vector v is a linear combination of vectors v1,v2, ...,vn ifthere are scalars c1, c2, ..., cn such that v = c1v1 + c2v2 + ...+ cnvn. The scalars ciare called the coefficients of the vectors in the linear combination.

Example 0.9. The vector

13−2

is a linear combination of

011

, 2

3−1

and

134

as:

13−2

= 3

011

+

23−1

−1

34

.Example 0.10. If u =

[21

]and v =

[14

]we can express any vector in R2 in terms

of u and v, instead of the usual vector basis e1 =

[10

]and e2 =

[01

]. For example

if we suppose w = −2u + v we see that

w =

[−32

]= −3e1 + 2e2

Modular Arithmetic and Binary Vectors. In computer science one often en-counters a vector which has no geometric interpretation. As a computer representsdata in terms of 1s and 0s, binary vectors are vectors each of whose componentsis a 0 or a 1. In this setting the usual rules of arithmetic must be changed as anycalculation must yield a 0 or a 1. The group tables for addition and multiplicationare simply: Notice here that 1 + 1 = 0, this may appear odd, however if we say 0corresponds to even and 1 to odd these tables summarize the fact the usual parity


+ 0 1 2

0 0 1 2

1 1 2 0

2 2 0 1

* 0 1 2

0 0 0 0

1 1 1 2

2 0 2 1

rules. i.e. the sum of two odd or even numbers is even, or the sum of an even andodd number is odd.

With these rules and the set {0, 1} is denoted by Z2 and we call this the set ofintegers modulo 2. More generally we call the order n-tuples with componentsin Z2 binary vectors of length n and denote this as Zn2 .

Example 0.11. The vectors in Z22 consist of [0, 0], [0, 1], [1, 0] and [1, 1]. For Zn2 we

have n2 vectors.

This idea can be extended to produce ordered n-tuples whose components arefrom a finite set {0, 1, 2, ...k}, k > 1. To do so we recall the concept of a remainder,for any integer n we may write n = mk+ l where m is some integer and 0 ≤ l < k;by ignoring the mk term, the binary operations of addition and multiplicationproduce elements in the original set - the operations are closed. We will denotethis as Zk and call it the integers modulo k. For technical reasons which willnot be stated here, we will restrict k to prime numbers to discuss vectors whosecomponents belong to Zk, called k-ary vector of length n.

Example 0.12. Consider the integers modulo 3, Z3 = {0, 1, 2} with addition andmultiplication tables:

Example 0.13. Q: 16661 is the 1928th prime, what is its value in Z3?A: Since 16661 = 5553*3+2, we conclude that this number is equivalent to 2 in Z3.

Example 0.14. Q: In Z53, let u = [1, 1, 2, 1, 2] and v = [0, 1, 2, 2, 0], computer their

sum.A: Adding each component and using the addition and multiplication tables, u+v =[1, 2, 1, 0, 2].

Length and Angle via the Dot Product

Since vectors have a magnitude and a direction, the familiar ideas of length, dis-tance and angle may be expressed in terms of vectors. In fact by generalizing theseideas from two and three dimensions we may define these quantities independentof dimension.


The dot product of two vectors. The vector equivalent of length, distance andangle depend on the concept of a dot product

Definition 0.15. If uT = [u1, u2, ..., un] and vT = [v1, v2, ..., vn] then the dotproduct u · v of u and v is defined as

u · v = u1v1 + u2v2 + ...+ unvn.

So the dot product of u and v is the sum of the products of the components ofthese two vectors. Unlike addition and scalar multiplication this operation takestwo vectors and produces a scalar. Furthermore this operation is defined only forvectors with the same number of components.

Example 0.16. Q: Compute u · v where u =

1−12

, v =

−354

. A: u · v =

1(−3) + (−1)5 + 2(4) = 0.

The dot product is commutative, as the dot product of v and u in this casebecause v · u = 0. However, this will happen if u · v 6= 0 because the componentsbelong to R which is commutative. Again, knowing the properties of the dot productwill facilitate calculations with vectors.

Theorem 0.17. Let u, v and w be vectors in Rn and let c be a scalar. The dotproduct satisfies:

(1) u · v = v · u(2) u · (v + w) = u · v + u ·w(3) (cu) · v = c(u · v)(4) u · u ≥ 0 and u · u = 0 if and only if u = 0

Example 0.18. The dot product of the sum of two vectors may be simplified:

(u + v) · (u + v) = (u + v) · u + (u + v) · v= u · u + v · u + u · v + v · v= u · u + u · v + u · v + v · v= u · u + 2u · v + v · v

Length. To illustrate how an idea of length may be derived from the dot product,we return to R2 and the Pythagoras theorem. Here, the length of a vector uT = [a, b]is the distance from the origin to the head of the vector at (a, b). Using Pythagoras,

we see that the distance is√a2 + b2. Noting that u · u = a2 + b2 we have a simple

definition for distance

Definition 0.19. The length or norm of a vector uT = [u1, u2, ..., un] in Rn isthe non-negative scalar ||u|| given by

||u|| =√

u · u =√u21 + u22 + ...+ u2n

Example 0.20. Consider the length of the vector uT = [0, 0, 4, 3],

||u|| =√

02 + 02 + 42 + 32 =√

25 = 5

Using the properties of the dot product we have two helpful facts for the length ofan arbitrary vector:


Theorem 0.21. Let u be a vector in Rn and c a scalar.

(1) ||u|| = 0 if and only if u = 0(2) ||cu|| = c||u||

We call any vector of length 1 a unit vector, in R2 and R3 the set of all unitvectors can be identified with the unit circle and unit sphere respectively (i.e. withradius 1 centered at the origin). Given any non-zero vector u, we may produce aunit vector v by taking the norm of the vector and dividing each component, i.e.v = 1

||u||u. It is easily verified that this is indeed a unit vector

||u|| = 1

||u||||u|| = 1

Notice that v is in the same direction as the original vector as ||u|| > 0 We call thisprocedure normalizing a vector.

Figure 9. Unit vectors in the plane.

Example 0.22. Q: Normalize the vector, uT = [0, 3,−4]A: The norm of u is ||u|| = 5 and so

v =u

||u||=

1

5[0, 3,−4].


Figure 10. Normalizing a vector

As the rule describing length and its change under scalar multiplication, onewonders if length and vector addition produce a similar rule. For example, whenis the identity ||u + v|| = ||u|| + ||v|| true? For almost any choice of u and v thisidentity will not hold. Instead if we replace the equality for an inequality:

||u + v|| ≤ ||u||+ ||v||this identity holds true, and is called the Triangle inequality.

In two and three dimensions the triangle inequality may be checked visuallyusing geometry. To prove this rigorously for any dimension, we must introduceanother inequality

Theorem 0.23. The Cauchy-Schwarz Inequality: For all vectors u and v inRn,

|u · v| ≤ ||u||||v||

Invoking this inequality we have

Theorem 0.24. The Triangle Inequality: For all vectors u and v in Rn,

||u + v|| ≤ ||u||+ ||v||

Proof. Both sides of the inequality are non-negative, implying that the square ofone side is less than or equal to the square of the other side. Thus we may compute

||u + v||2 = (u + v) · (u + v)

= u · u + 2u · v + v · v≤ ||u||2 + 2|u · v|+ ||v||2

≤ ||u||2 + 2||u||||v||+ ||v||2 = (||u||+ ||v||)2

Taking the square root completes the proof. �

Distance. Just as how vectors are directed line segments between points, distancebetween two vectors is the direct analogue of the distance between two points onthe real number line R or two points in R2. In R the distance between points Aand B is simply |B − A| (distances must be positive, so we must use the absolutevalue).


Figure 11. Normalizing a vector u.

In R2 the distance betweenA = (a1, a2) andB = (b1, b2) is simply d =√

(b1 − a21 + (b2 − a2)2

Figure 12. Distance on the plane.

In terms of vectors uT = [a1, a2] and vT = [b1, b2] then the distance will just bethe length of the vector u− v.

Definition 0.25. The distance d(u,v) between two vectors in Rn is defined by

d(u,v) = ||u− v||.

Example 0.26. Q: Find the distance between uT = [√

3, 1, 3] and vT = [0, 1, 2]

A: Computing (u− v)T = [√

3, 0, 1] we find that d(u,v) =√

3 + 0 + 1 = 2.

Angles. Just as the dot product is used to calculate length and distances in termsof vectors, it may be used to calculate the angle between a pair of vectors. In twoand three dimensions, the angle between two non-zero vectors will refer to the angleθ ∈ [0, 2π] between the two vectors.


Figure 13. The angle between u and v.

Figure 14. Always look for the angle θ ∈ [0, π]

Consider the triangle with sides u,v and u − v given in figure (13), we denotethe angle between u and v as θ. Applying the law of cosines to this triangle wefind

||u− v||2 = ||u||2 + ||v||2 − 2||u||||v||cosθ

then by expanding the left hand side and noting ||v||2 = v · v we obtain

||u||2 − 2(u · v) + ||v||2 = ||u||2 + ||v||2 − 2||u||||v||cosθ.

Simplifying we find that u · v = ||u||||v||cosθ, we have found a simple formula forθ in terms of dot products of vectors.

Definition 0.27. For any two non-zero vectors u and v in Rn the angle betweenthem is defined as

cosθ =u · v||u|||||v||

Example 0.28. Q: Compute the angle between the vectors uT = [1,−1, 0] andvT = [1, 0, 0].

A: Calculating u · v = 1, ||u|| =√

2 and ||v|| = 1. Therefore cosθ = 1√2

implying

that θ = π4 .


θ 0 π6

π4

π3

π2

cosθ 1√32

1√2

12 0

Table 1. Cosines of Special Angles

Orthogonal vectors. So far we have seen that vectors which are parallel to eachother must be scalar multiples of each other. With the dot product we say thatthese vectors have θ = 0 or π. What about perpendicular vectors? This is animportant tool in geometry, and so it will be helpful to generalize this conceptto vectors in Rn. In two and three dimensions two non-zero vectors u and v areperpendicular if the angle between them is a right angle, θ = π

2 . Applying theformula for θ this implies

u · v||u||||v||

= 0

Thus the dot product of u and v must vanish.

Definition 0.29. Two vectors u and v in Rn are orthogonal if u · v = 0

Notice that 0 · u = 0, so that every vector in Rn is orthogonal to the zero vector.

Example 0.30. Consider the basis vectors in R3, eT1 = [1, 0, 0] and eT2 = [0, 1, 0],clearly the dot product of these two is zero. What of u = e1− e2 and v = e1 + e2?

To illustrate the utility of orthogonal vectors, we easily prove Pythagoras theoremin arbitrary dimension,

Theorem 0.31. For all vectors u and v in Rn, ||u + v||2 = ||u||2 + ||v||2 if andonly if u and v are orthogonal.

Proof. Noting that ||u+v||2 = ||u||2+2(u·v)+||v||2, if these vectors are orthogonalthe term, u · v = 0 giving the desired equality. �

As an application of this idea, we use this to find the distance from a line to a pointin Rn.

Projection of a Vector onto Another. In two dimensions, figure (15) summa-rizes the problem of finding the distance from a point B to a line L. This canalways be reduced to finding the length of the perpendicular line segment from P

to B, or alternatively the length of the vector−→AB. Picking another point A on L

we may draw a right-angled triangle ∆APB with two new vectors−→AP and

−→AB. We

say−→AP is the projection of

−→AB onto the line L. We will now interpret this in terms

of n-dimensional vectors.


Figure 15

Given two non-zero vectors u and v, let p be the vector obtained by droppinga perpendicular from the head of v onto u and denote θ as the angle between u

and v as in figure (16). We may express p as p = ||p||||u||u, furthermore using simple

trigonometry, ||p|| = ||v||cosθ where cosθ = u·v||u||||v|| . Combining these facts we find

p =(u · v

u · u

)u,

from this derivation we now have a helpful tool.

Definition 0.32. If u and v are vectors in Rn and u 6= 0 then the projection ofv onto u is the vector proju(v) defined by

proju(v) =(u · v

u · u

)u.

Returning to the question of the distance from the point B to a line `, we see thatthe shortest distance between B and ` is the magnitude of the vector or directedline segment between B and P in figure (16); if we could calculate its magnitude we

would have the distance. Since we know v and can calculate p =−→AP = projd(v),

The vector−→PB will be the difference:

−→PB = v−p, and taking the magnitude gives

us the distance.

Figure 16

Definition 0.33. The distance d(B, `) between a point B and a line ` in R2 isdefined as the magnitude of the vector:

d(B, `) = projd(v)


where v =−→AB is the directed line segment between B and another point on ` and

d is the direction vector of `.

References

[1] D. Poole, Linear Algebra: A modern introduction - 3rd Edition, Brooks/Cole (2012).

Documents

LINEAR ALGEBRA - CHAPTER 1: VECTORS