245893173 Nonlinear Optimization in Electrical Engineering With Applications in MATLAB 2013

8/20/2019 245893173 Nonlinear Optimization in Electrical Engineering With Applications in MATLAB 2013

1/325

Renewable Energy Series 17

Nonlinear Optimization inElectrical Engineering with

Applications in MATLAB®

Mohamed Bakr


2/325


Applications in MATLAB‡


3/325


4/325


Applications in MATLAB‡

Mohamed Bakr

The Institution of Engineering and Technology


5/325

Published by The Institution of Engineering and Technology, London, United Kingdom

The Institution of Engineering and Technology is registered as a Charity in England &Wales (no. 211014) and Scotland (no. SC038698).

†

The Institution of Engineering and Technology 2013First published 2013

This publication is copyright under the Berne Convention and the Universal CopyrightConvention. All rights reserved. Apart from any fair dealing for the purposes of researchor private study, or criticism or review, as permitted under the Copyright, Designs andPatents Act 1988, this publication may be reproduced, stored or transmitted, in any form or by any means, only with the prior permission in writing of the publishers, or inthe case of reprographic reproduction in accordance with the terms of licences issuedby the Copyright Licensing Agency. Enquiries concerning reproduction outside thoseterms should be sent to the publisher at the undermentioned address:

The Institution of Engineering and Technology Michael Faraday HouseSix Hills Way, StevenageHerts, SG1 2AY, United Kingdom

www.theiet.org

While the author and publisher believe that the information and guidance given in this work are correct, all parties must rely upon their own skill and judgement when makinguse of them. Neither the author nor publisher assumes any liability to anyone for any loss or damage caused by any error or omission in the work, whether such an error oromission is the result of negligence or any other cause. Any and all such liability isdisclaimed.

The moral rights of the author to be identified as author of this work have beenasserted by him in accordance with the Copyright, Designs and Patents Act 1988.

British Library Cataloguing in Publication Data

A catalogue record for this product is available from the British Library

ISBN 978-1-84919-543-0 (hardback)ISBN 978-1-84919-544-7 (PDF)

Typeset in India by MPS LimitedPrinted in the UK by CPI Group (UK) Ltd, Croydon


6/325

To my wife Mahetab, my children Jannah, Omar, and Youssef, and

to my parents to whom I am indebted for as long as I live


7/325


8/325

Contents

Preface xi

Acknowledgments xv

1 Mathematical background 11.1 Introduction 11.2 Vectors 11.3 Matrices 31.4 Thesolution of linear systems of equations 61.5 Derivatives 11

1.5.1 Derivative approximation 111.5.2 The gradient 121.5.3 The Jacobian 141.5.4 Second-order derivatives 151.5.5 Derivatives of vectors and matrices 16

1.6 Subspaces 181.7 Convergence rates 201.8 Functions and sets 201.9 Solutions of systems of nonlinear equations 221.10 Optimization problem definition 25References 25Problems 25

2 Anintroductionto linear programming 29

2.1 Introduction 292.2 Examples of linear programs 292.2.1 A farming example 292.2.2 A production example 302.2.3 Power generation example 312.2.4 Wireless communication example 322.2.5 A battery charging example 32

2.3 Standard form of an LP 332.4 Optimality conditions 372.5 The matrix form 39

2.6 Canonical augmented form 402.7 Moving from one basic feasible solution to another 422.8 Cost reduction 452.9 The classical Simplex method 462.10 Starting the Simplex method 49

2.10.1 Endless pivoting 512.10.2 The big M approach 512.10.3 The two-phase Simplex 52


9/325

2.11 Advanced topics 55A2.1 Minimax optimization 55

A2.1.1 Minimax problem definition 55A2.1.2 Minimax solution using linear programming 57

A2.1.3 A microwave filter example 59A2.1.4 The design of coupled microcavities optical filter 61

References 65Problems 65

3 Classical optimization 693.1 Introduction 693.2 Single-variable Taylor expansion 693.3 Multidimensional Taylor expansion 71

3.4 Meaning of the gradient 733.5 Optimality conditions 763.6 Unconstrained optimization 763.7 Optimization with equality constraints 78

3.7.1 Method of direct substitution 793.7.2 Method of constrained variation 80

3.8 Lagrange multipliers 843.9 Optimization with inequality constraints 863.10 Optimization with mixed constraints 92

A3.1 Quadratic programming 92A3.2 Sequential quadratic programming 95References 99Problems 99

4 One-dimensional optimization-Linesearch 1014.1 Introduction 1014.2 Bracketing approaches 102

4.2.1 Fixed line search 103

4.2.2 Accelerated line search 1044.3 Derivative-free line search 105

4.3.1 Dichotomous line search 1054.3.2 The interval-halving method 1064.3.3 The Fibonacci search 1084.3.4 The Golden Section method 111

4.4 Interpolation approaches 1124.4.1 Quadratic models 1134.4.2 Cubic interpolation 116

4.5 Derivative-based approaches 1194.5.1 The classical Newton method 1194.5.2 A quasi-Newton method 1214.5.3 The Secant method 122

4.6 Inexact line search 123A4.1 Tuning of electric circuits 124

viii Nonlinear optimization in electri cal engineering with appli cations in MATLAB


10/325

A4.1.1 Tuning of a current source 125A4.1.2 Coupling of nanowires 127A4.1.3 Matching of microwave antennas 128

References 129

Problems 130

5 Derivative-freeunconstrained techniques 1315.1 Why unconstrained optimization? 1315.2 Classification of unconstrained optimization techniques 1315.3 The random jump technique 1325.4 The random walk method 1335.5 Grid search method 1345.6 The univariate method 135

5.7 The pattern search method 1375.8 The Simplex method 1405.9 Response surface approximation 143A5.1 Electrical application: impedance transformers 146A5.2 Electrical application: the design of photonic devices 149References 151Problems 152

6 First-order unconstrained optimization techniques 1536.1 Introduction 1536.2 The steepest descent method 1536.3 The conjugate directions method 156

6.3.1 Definition of conjugacy 1576.3.2 Powell’s method of conjugate directions 158

6.4 Conjugate gradient methods 162A6.1 Solution of large systems of linear equations 164A6.2 The design of digital FIR filters 169References 173Problems 173

7 Second-order unconstrained optimization techniques 1757.1 Introduction 1757.2 Newton’s method 1757.3 The Levenberg–Marquardt method 1787.4 Quasi-Newton methods 179

7.4.1 Broyden’s rank-1 update 1807.4.2 The Davidon–Fletcher–Powell (DFP) formula 1827.4.3 The Broyden–Fletcher–Goldfarb–Shanno method 1847.4.4 The Gauss–Newton method 185

A7.1 Wireless channel characterization 188A7.2 The parameter extraction problem 189A7.3 A rtificial neural networks training 193References 201Problems 201

Contents ix


11/325

8 Constrainedoptimization techniques 2038.1 Introduction 2038.2 Problem definition 2038.3 Possible optimization scenarios 204

8.4 A random search method 2068.5 Finding a feasible starting point 2088.6 The Complex method 2108.7 Sequential linear programming 2128.8 Method of feasible directions 2158.9 Rosen’s projection method 2188.10 Barrier and penalty methods 221A8.1 Electrical engineering application: analog filter design 224A8.2 Spectroscopy 227

References 230Problems 231

9 Introduction to global optimization techniques 2339.1 Introduction 2339.2 Statistical optimization 2339.3 Nature-inspired global techniques 236

9.3.1 Simulated annealing 2379.3.2 Genetic algorithms 2409.3.3 Particle swarm optimization 246

A9.1 Least p th optimization of filters 247A9.2 Pattern recognition 252References 257Problems 258

10 Adjoint sensitivity analysis 26110.1 Introduction 26110.2 Tellegen’s theorem 26210.3 Adjoint network method 264

10.4 Adjoint sensitivity analysis of a linear system of equations 27810.5 Time-domain adjoint sensitivity analysis 282A10.1 Sensitivity analysis of high-frequency structures 295References 298Problems 299

Index 303

x Nonlinear optimization in electri cal engineering with applications in MATLAB


12/325

Preface

In 2008, I was asked to givea sequenceof lectures about optimization to graduateelectrical engineering students in University of Waterloo, Canada. These lectureswereattended by over 40 students, postdoctoral fellows, andindustry CAD people.I tried in this course to teachoptimization, whichis a highly mathematical subject,with more focus on applications. The course was well received by all attendees. The feedback I received from them regarding other optimization courses, in com-parison to my course, was surprising. Most of them attended other courses inoptimization that contributed very little to their understanding of the subject. Mostof thesecourses weretoo abstract with very little connectionto practical examples. There were so many confusing mathematical proofs that distracted them from theactual meanings and implementations. Engineering students, both graduate andundergraduate, would like to get a better focus on the meaning hidden behind thelengthy proofs. They are interested in real-life engineering applications rather thanabstract material that add very little to their global understanding of the subject.

I was encouraged by this experiencetowritean optimization book dedicated toelectrical engineers; abook that combines simplicity inintroducingthesubjectwithmathematical rigor; a book that focuses more on examples and applications ratherthan onlengthy mathematical proofs. I useMATLAB as atool to show simplifiedimplementations of the discussed algorithms. This should help in removing theabstractness of the theorems and improveunderstanding of the different subjects.

This book is intended as an initial book in optimization for undergraduatestudents, graduate students, and industry professionals. Third-year engineeringstudents should have the necessary background to understand different concepts.Special focus is put ongivingas many examples and applications as possible. Onlyabsolutely necessary mathematical proofs are given. Theseproofs, usually short innature, serve the purpose of increasing the depth of understanding of the subject.For lengthy proofs, the interested reader is referred to other more advanced booksand published research papers in the subject.

For theapplicationsgiven, I try to showaMATLAB implementationasmuchaspossible. In more advanced applications that require non-MATLAB commercialcodes, I just report on the problem formulation and the results. All the MATLABcodes given in this book are only for educational purposes. The students should beable to fully understand these codes, change their parameters, and apply them to

differentproblemsof interest.Thesecodescanalsobedevelopedfurther forresearch-and industry-related applications. The full MATLAB codes can be downloadedthroughthelink www.optimizationisfun.com.

Even though most of the addressed examples and applications are related toelectrical engineering, students, researchers, and technical people from otherengineeringareas will findthematerial also useful. Electrical engineeringhas beenintegrated with other areas resulting in innovative programs in, for example,


13/325

mechatronics andbiomedical engineering. I giveas muchexplanation to thetheorybehind every applicationso that theunfamiliar reader will grasp thebasic concepts.

This book is organized as follows. In Chapter 1, we review the basic mathe-matical background needed in this book. Most third-year electrical engineering

students will have good familiarity with the content of this chapter. We discuss inthis chapter basic rules of differentiation of vectors and matrices. I added morecontent related to linear algebra that is useful in understanding many of the dis-cussed optimization subjects.

Chapter 2 gives an overview of the linear programming approach. As wellknown in the area of engineering, nonlinear problems can be solved by convertingthemto a sequence of linear problems. Linear programming addresses the issue of optimizing a linear objective function subject to linear constraints. I explain in thischapter thebasic Simplex methodandthetheory behindit. I give several examples

and applications relevant to the area of electrical engineering.Chapter 3 reviews some classical optimization techniques. These techniques

found wide applications before the advent of the digital computer era. The math-ematical foundations of thesetechniques formthe bases for many of the numericaloptimization techniques.

In Chapter 4, we address the one-dimensional line search problem. This pro-blem is very important in optimization theory as it is routinely utilized by otherlinear and nonlinear optimization techniques. I introduce different approaches forsolving this problem including derivative-free techniques, first-order line searchtechniques, and Newton-based techniques.

Derivative-free nonlinear optimization techniques for unconstrained problemsare discussed in Chapter 5. These techniques do not require any derivative infor-mation for solvinggeneral multidimensional problems. They areuseful inproblemswhere sensitivity information may not be available or costly to obtain. Onlyobjective function values are used to guide the optimization iterations.

Chapter 6 discusses gradient-based techniques for unconstrained optimization. These are techniques that assume the availability of first-order sensitivity infor-mation. We discuss the basic steepest descent method and a number of conjugateoptimization techniques. These techniques have robust convergence proofs under

certain assumptions on the objective function.Quasi-Newton techniques for the unconstrained optimization of general non-

linear problemare discussed in Chapter 7. These techniques aim at approximatingsecond-order sensitivity information and using them in guiding the optimizationiterations. These techniques are known for their good convergence rate. I illustratethesetechniques through examples and applications.

Chapter 8 reviews some constrained optimization techniques. Some of thesetechniques build on different concepts of unconstrained optimization. Other tech-niques sequentially approximate the nonlinear problem by a corresponding linearproblem.

All the techniques discussed in Chapters 1 through 8 are local optimizationtechniques. They obtain a minimum of the optimization problem that satisfiescertain optimality conditions only locally. Chapter 9 introduces a number of tech-niques for obtaining the global minimum of an optimization problem. This is theminimumthat has thelowest valueof theobjectivefunctionover all other minima.A number of nature-inspired techniques are discussed and illustrated throughexamples and applications.

xii Nonlinear optimization in electrical engineering with appli cations in MATLAB


14/325

Chapter 10addressesadjoint-basedtechniques for estimatingthegradient of anobjective function. I show in this chapter how, by using only one extrasimulation,thesensitivitiesof anobjectivefunctionwith respect toall parameters areestimatedregardless of their number. I illustrate this approach for the adjoint network

method, the frequency-domain case, and the time-domain case. Several examplesand applications are presented.

Finally, I should say that the area of optimization is ahugefield with so manytechniques and applications. No one book can cover all material by its own. I justattempted hereto help thereaders scratch thesurfaceof this area. I encouragethemtobuildonthematerial in this book andcontinuetoseek moreadvanced learning inthis area.

Dr. Mohamed Bakr

Professor, Department of Electrical and Computer Engineering,McMaster University, Hamilton, Ontario, Canada

July 2013

Preface xiii


15/325


16/325

Acknowledgments

I would like to acknowledge the mentorship of Drs. Hany L. Abdel Malek andAbdel Karim Hassan from Cairo University, Egypt. They were the first ones tointroduce me as a Master student to the fascinating area of optimization. Beingelectrical engineers themselves, they have always attempted to focus on applica-tions without abandoning the mathematical rigor.

I would like also to thank Drs. John Bandler from McMaster University andRadek Biernacki (currently withAgilent technologies) for their guidanceduringmygraduate studies in Canada. I have learnt a lot frommy Ph.D. supervisor Dr. JohnBandler. Heis adedicated person withhigh commitment toexcellence. His work incomputer aided design (CAD) of microwave structures has helped shape the fieldand motivated all his students. It was also my pleasurethat I helped teachEE3KB3course with Dr. Biernacki. Dr. Biernacki gave me the opportunity to conduct thetutorials for this applied optimization course. This was an amazing experience tolearn about applications of optimization theory to the design of microwavecircuits

and antennas. I include the material of this course as a reference in some of thechapters because they helped inspire several of the presented examples.

I would like to thank all the good researchers who worked with me over theyearsandhelpedshapemy experienceintheareasof optimizationandcomputationalelectromagnetics. These excellent researchers include Dr. Payam Abolghasem,Dr. Peter Basl, Dr. Ezzeldin Soliman, Dr. MahmoudElSabbagh, Dr. AhmedRadwan,Dr. MohamedSwillam, Dr. OsmanAhmed, Peipei Zhao, Kai Wang, LalehKalantari, Y u Zhang, Mohamed Negm, Harman Malhi, and Mohamed Elsherif.

I would also like to thank my wife and children for their patience during thedevelopment of this book. Last but foremost, I would liketo thank God for givingme the strength and energy to continue this work to its end.


17/325


18/325

Chapter 1

Mathematical background

1.1 Introduction

Mathematics is the language weuse to express different concepts and theorems in

optimization theory. Likeall languages, onehas tomaster thebasic building blocksof a language suchas the letters, the words, and the symbols in order to gain somefluency in a language.

The main target of this chapter is to briefly review some of the mathematicalconceptsneededfor subsequentchapters. Wewill review somepropertiesof vectorsand matrices. I will introducesome of the vocabulary used in optimization theory.Wewill also discuss theproperties of thesolutionof asystemof linear equations.

1.2 Vectors

A vector expresses a number of variables or parameters combined together.It offers a more compact way for manipulating these parameters. A generaln -dimensional real vector x 2


19/325

The parameters in (1.2) can have completely different orders of magnitude. Theresistors can be in the K W range, the capacitors are in the mF range, while theinductors may be in the mH range. This difference in magnitudes can cause ill-conditioning in optimization algorithms that usually assume that all parameters

havethe same order of magnitude. To fix this issue, weuse scaling. For example,within the optimization algorithm the used values of the parameters can bex ¼ ½1:0 3:0 1:5 2:0 2:0 1:5T . These are scaled values of the parameterswithadifferent scalefactor for eachparameter. Every timeit is required toevaluatethe response of the circuit, the parameters are scaled back to their physical values. Thisapproach allows usto usethesameoptimization algorithmfor parameters withdifferent orders.

During an optimization algorithm, the values of the parameters are iterativelyadjusted until an optimal design is reached. We denote by x (k ) the value of the

optimizableparameters at thek thiteration. Theoptimal designis denoted by x *

. Anoptimization algorithm aims at obtaining a good approximation of x * within anumber of iterations. We denote the final design achieved by an optimizationalgorithmas x which is expected to be very close to x *.

The iterative adjustment done to the variables at the k th iteration involvesdetermining aperturbationDx (k ) of the parameters. Theparameters at the (k þ 1)stiteration are given by:

xðk þ1Þ ¼ xðk Þ þ Dxðk Þ ð1:3Þ

where vector addition implies adjusting every component of the vector x (k ) by thecorresponding component in the vector Dx (k ) to give the new parameters x (k þ1).

Another important vector operation is the inner product of two vectors. If x , y 2


20/325

xyT ¼x 1

x 2

.

.

.

x n

26664

37775

y 1 y 2 . . . y m ½ ¼x 1y 1 x 1y 2 . . . x 1y m

x 2y 1 x 2y 2 . . . x 2y m

.

.

...

...

...

.

x n y 1 x n y 2 x n y m

26664

37775

ð1:7Þ

This matrix has n rows and m columns. Notice that all the rows of the matrix in(1.7) are multiples of the vector y . Such a matrix is called a rank-one matrixbecauseit has only onelinearly independentrow. In later chapters, I will show thatsuchmatrices play arole in optimizingnonlinear problems. Thefollowingexampleillustrates some concepts in vector operations.

Example 1.1: Consider the two vectors x ¼ ½1:0 2:0 1:0T and y

¼ ½3:0 2:0 2:0

T . Evaluate x

þy , x

y , x T y , xy T , and the angle between

these two vectors.

Solution : Theresults for this example areas follows:

Addition: x þ y ¼12

1

24

35þ 32

2

24

35 ¼ 24

1

24

35

Subtraction: x

y

¼

12

124 353

2

224 35 ¼40

324 35Inner product: xT y ¼ 1 2 1½

322

24

35 ¼ 1, and

Multiplication: xyT ¼12

1

2

4

3

5 3 2 2½ ¼

3 2 26 4 4

3 2 2

2

4

3

5Noticethat inthelast result, thematrix isarank-onematrix asall rowsaremultiplesof the vector y . Because the inner product of the two vectorsis negative, these twovectorsmakean obtuseangle. To determine this angleweuse(1.6) to get:

cosðqÞ ¼ xT y

kxkk yk¼ 1

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffið1Þ2þð2Þ2þð1Þ2q ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffið3Þ

2þð2Þ2þð2Þ2q ¼ 1

ffiffiffi6

p

ffiffiffiffiffi17

p ¼ 0:099

+q¼ cos1ð0:099Þ ¼ 1:67radians¼ 95:68

ð1:8Þ

1.3 Matrices

Matrices are another means of manipulating a group of components together.A matrix can belooked at as anumber of column vectors arranged in columns or as

M athematical background 3


21/325

the transpose of column vectors arranged in rows. These two different points of view are useful in different manipulations. A matrix A 2


22/325

b ¼ Ax ¼a 11 a 12 . . . a 1n a 21 a 22 . . . a 2n

.

.

...

...

...

.

a m 1 a m 2 . . . a mn

2

6664

3

7775

x 1

x 2

.

.

.

x n

2

6664

3

7775¼

a 11x 1 þ a 12x 2 þ þ a 1n x n a 21x 1 þ a 22x 2 þ þ a 2n x n

.

.

.

a m 1x 1 þ a m 2x 2 þ þ a mn x n

2

6664

3

7775ð1:14Þ

The product of a matrix A 2


23/325

Example 1.2: Consider the matrix:

A ¼2 1 0

1 2 10 1 224 35and the vectors:

x ¼1

12

24

35 and b ¼ 23

2

24

35

Evaluate the quadratic forms x T Ax and b T Ab .

Solution : In this example, we check the definiteness of the matrix by consideringtwo vector samples. For the vector x , wehave:

y ¼ Ax ¼2 1 0

1 2 10 1 2

24

35 11

2

24

35 ¼ 35

5

24

35 ) xT y ¼ xT Ax

¼ 1 1 2½ 3

55

24 35 ¼ 18 ð1:18ÞFor the vector b we have:

c ¼ Ab ¼2 1 0

1 2 10 1 2

2

4

3

523

2

2

4

3

5 ¼

16

7

2

4

3

5 ) bT c ¼ bT Ab

¼ 2 3 2½ 16

7

24 35 ¼ 34 ð1:19ÞFor both cases, it is evident that both vectors arerotated by less than 90. For

this matrix, it can be shown that its eigenvalues are all positive and given by0.5858, 2.0000, 3.4142. It follows that indeed this matrix is a positive definitematrix.

1.4 Thesolution of linear systemsof equations

One of the main problems that frequently appears in electrical engineering is thesolution of a linear system of equations. When writing the nodal equations of acircuit, we solve a linear system of equations for the unknown currents. Whenwritingtheloopequationsof anelectric circuit wesolvealinear systemof equations

6 Nonlinear optimization in electr ical engineering with applications in M ATLAB


24/325

for thedifferent unknownvoltages. Similar applicationscanbefoundinmany otherelectrical applications. A linear systemof n equations in n unknowns is given by:

a 11x 1 þ a 12x 2 þ þ a 1n x n ¼ b 1a 21x 1 þ a 22x 2 þ þ a 2n x n ¼ b 2

.

.

.

a n 1x 1 þ a n 2x 2 þ þ a nn x n ¼ b n

ð1:20Þ

These equations impose n constraints on the vector of unknowns x . In matrixform, (1.20) is given by:

Ax ¼ b ð1:21ÞIf the n equations in (1.20) are independent fromone another, then the rows of the

matrix A are linearly independent. In this case, the matrix A is invertible andthe solutionof the linear systemof equations is obtained by multiplying both sidesof (1.21) by the inverseA1 to get:

x ¼ A1b ð1:22ÞAnother approach for solving the system (1.20) that does not require direct

evaluation of the inverse matrix is to apply elementary row operations on (1.20). Thebasic concept of theseoperations is that asystemof equations does not changeby dividing an equation by a constant or by adding a multiple of one equation to

another equation. Usingtheseoperationswemakeonevariableresponsiblefor onlyoneequation. Wescale this equation to makethecoefficient of this variable in thisequation unity. We add to all other equations multiple of this equationto makethecoefficients of that variable in all other equations zero. Repeating this for allvariables one by one weend up with the diagonalized systemof equations:

x 1 ¼ x 1x 2 ¼ x 2

..

.

x n ¼ x 3

ð1:23Þ

The elimination technique is illustrated through the following example.

Example 1.3: Utilize basic row operations to solve the system of equations:

2x 1 þ 3x 2 þ x 3 þ x 4 ¼ 4x 1 þ x 2 þ 2x 3 x 4 ¼ 1

3x 1 2x 2 þ 2x 3 þ 3x 4 ¼ 182x 1

x 2

2x 3

þ2x 4

¼ 5

ð1:24Þ

Solution : We first build the augmented matrix by adding the vector b as the(n þ 1)st column of the matrix A to get Tableau 1.

The variable x 1 is assigned to the first equation. We thus make the coefficientof this variable in this equation unity by dividing the first equation by 2. We thenforce the coefficients of x 1 in all other equations to zero by multiplying the scaledfirst equationby 1and adding it tothesecond equation, multiplyingit by 3and



25/325

addingit to thethird equation, and multiplying it by 2 and adding it to thefourthequation. This gives Tableau 2.

Wethen assign thevariablex 2 tothesecond equation. Wemakethecoefficient

of this variable in this equation unity by multiplying the second equation by2.Wethen forcethecoefficientsof x 2 inall other equations tozero by multiplying thescaled second equationby3/2and addingit to thefirst equation, multiplyingit by13/2 and addingit to thethird equation, and multiplyingit by 4 and addingit to thefourth equation. This gives Tableau 3.

Similarly, weassign the variable x 3 to the third equationand the variable x 4 toget the Tableaus 4 and 5:

Tableau 1

x 1 x 2 x 3 x 4 RHS

2 3 1 1 41 1 2 1 13 2 2 3 182 1 2 2 5

Tableau 2

x 1 x 2 x 3 x 4 RHS

1 3/2 1/2 1/2 20 1/2 3/2 3/2 10 13/2 1/2 3/2 120 4 3 1 1

Tableau 3

x 1 x 2 x 3 x 4 RHS1 0 5 4 10 1 3 3 20 0 19 21 250 0 15 13 9

Tableau 4

x 1 x 2 x 3 x 4 RHS

1 0 0 29/19 106/190 1 0 6/19 37/190 0 1 21/19 25/190 0 0 68/19 204/19

8 Nonlinear optimization in electr ical engineering with applications in M ATLAB


26/325

It follows that the solution is x ¼ ½1 1 2 3T . This means that thecoefficients in the last column give the expansion of the vector b in the initialtableau in terms of the columns of the matrix in the final tableau, i.e.:

41

185

26643775 ¼ 1 213

2

26643775 1 312

1

26643775þ 2 122

2

26643775þ 3 113

2

26643775 ð1:25Þ

Actually, if the original matrix is augmented by more columns other than b , thecomponents of eachcolumn at thefinal tableau aretheexpansionof that column interms of thecolumns correspondingtotheidentity matrix in theinitial tableau. Thispropertyisimportant intheSimplex algorithmwhichwill bediscussedinChapter 2.

The same concept applies as well to rectangular full-rank matrices. Forexample, consider the system of m equations in n unknowns with m < n :

a 11x 1 þ a 12x 2 þ þ a 1n x n ¼ b 1a 21x 1 þ a 22x 2 þ þ a 2n x n ¼ b 2

.

.

.

a m 1x 1 þ a m 2x 2 þ þ a mn x n ¼ b m

ð1:26Þ

Pivoting can be applied to this system similar to the square matrix case. If weassign the first m variables to the first m equations, one gets:

x 1þ a 1;m þ1x m þ1 a 1;m þ2x m þ2 . . . a 1;n x n ¼ x 1x 2þ ..

..

.

...

.

..

...

..

.

...

.

x m þ a m ;m þ1x m þ1 a m ;m þ2x m þ2 a m ;n x n ¼ x m ð1:27Þ

Thissystemhas infinitenumber of solutions. Wecanobtainsomeof thesesolutions

by assigning values to (n m ) variables and then solving for the remainingm variables. For example, in (1.27) if we set the last (n m ) variables to zero,weobtain the solutionx ¼ x 1 x 2 . . . x m 0T n m

T , where0n m is a vector of

zeros of size (n m ).In (1.23) and (1.27), we assume that the i th variable is assigned to the i th

equation resulting in an identity in the first m rows. However, this is not a limita-tion. The p th variable can be assigned to the q th equation. Pivoting in thiscase results in a unity coefficient at the location a qp and zero in all other

Tableau 5

x 1 x 2 x 3 x 4 RHS

1 0 0 0 10 1 0 0 10 0 1 0 20 0 0 1 3



27/325

components a jp , j = q . This results in a permuted identity matrix, but all the con-cepts discussed earlier still hold.

The following example illustrates the solution of rectangular systems of equations.

Example 1.4: Obtain a solution of the underdetermined system:

3:0x 1 x 2 þ 3x 3 ¼ 250:4x 1 þ 0:6x 2 þ x 4 ¼ 8:53:0x 1 þ 6:0x 2 þ x 5 ¼ 70

through basic row operations.

Solution : This problem has three equations and fiveunknowns. It thus has infinitenumber of solutions. Tableau 1 of this problem is first constructed as shown:

First, weassign x 2 (p ¼ 2) to the first equation (q ¼ 1) to get Tableau 2:

By setting x 1

¼x 3

¼0, wehave the solutionx

¼ ½0

25 0 23:5 220

T .

As stated earlier, thecoefficientsof thefirst column inthelast tableau represent theexpansion coefficients of this column in the initial tableau in terms of the columnscorresponding to the identity in the first tableau, i.e.:

3:00:4

3

24

35 ¼ 3 10:6

6

24

35þ 2:2 01

0

24

35þ 21 00

1

24

35

Similarly for the third and the RHS columns, wehave, respectively:

3:000

24

35 ¼ 3 10:6

6

24

35þ 1:8 01

0

24

35þ 18 00

1

24

35

258:570

24

35 ¼ 25 10:6

6

24

35þ 23:5 01

0

24

35þ 220 00

1

24

35

Tableau 1

x 1 x 2 x 3 x 4 x 5 RHS

3.0 1.0 3.0 0 0 250.4 0.6 0 1.0 0 8.53.0 6.0 0 0 1.0 70

Tableau 2

x 1 x 2 x 3 x 4 x 5 RHS

3.0 1.0 3.0 0 0 252.2 0 1.8 1.0 0 23.5

21.0 0 18 0 1.0 220

10 Nonlinear optimization in electr ical engineer ing with applications in M ATLAB


28/325

These properties of the basic row operations will be utilized later in the Simplexmethod for solving linear programs.

1.5 Derivatives

The concepts associated with derivatives of scalars, vectors, and matrices areimportant in optimization theory and they find applications in many proofs. In thisbook, we consider, in general, differentiable and continuous functions of n variables x ¼ [x 1 x 2 . . . x n ]T . These functions are denoted as f (x ) or f (x 1,x 2, . . . , x n ). The function f may be the gain of an amplifier, the return loss of amicrowavecircuit, or thereal part of theinput impedanceof an antenna. Changingone of the parameters affects the value of f. The derivative with respect to the i thparameter tells us how sensitive the function is with respect to that parameter.

1.5.1 Der ivative approximation

Mathematically speaking, the first-order derivatives are obtained by perturbing theparameters one at a time and determining the correspondingrelative change of thefunction. The derivative with respect to the i th parameter is given by:

@ f

@ x i ¼ limit

Dx i !0f ðx 1; x 2; . . .; x i þ Dx i ; . . .; x n Þ f ðx 1; x 2; . . .; x i ; . . .; x n Þ

Dx i ð1:28Þ

Theformula(1.28) means that theperturbation shouldbeas small as possibleto getan accurate result. The formula (1.28) can be approximated by the forward differ-ence formula:

@ f

@ x i f ðx 1; x 2; . . .; x i þ Dx i ; . . .; x n Þ f ðx 1; x 2; . . .; x i ; . . .; x n Þ

Dx i ð1:29Þ

whereafinitesmall perturbationDx i is used. This formulais denoted as theforwarddifference finite difference (FFD) formula. Other formulas that can be utilized forapproximating the first-order derivatives include the backward finite difference

(BFD) and central finite difference (CFD) given by, respectively:

@ f

@ x i f ðx 1; x 2; . . .; x i ; . . .; x n Þ f ðx 1; x 2; . . .; x i Dx i ; . . .; x n Þ

Dx i ð1:30Þ

@ f

@ x i f ðx 1; x 2; . . .; x i þ Dx i ; . . .; x n Þ f ðx 1; x 2; . . .; x i Dx i ; . . .; x n Þ

2Dx i ð1:31Þ

The FFD approach requires one extra function evaluation per parameter in theforward direction. The BFD formula requires one extra function evaluation perparameter in the backward direction. The CFD approximation is the average of

both the FFD and BFD approximations. It offers a better accuracy for derivativeestimation at theexpenseof using two extrafunctionevaluations per parameter. Toestimatederivatives relativeto all parameters, theFFD and theBFD requiren extrafunction evaluations while the CFD requires 2n extrafunction evaluations.

In many practical engineering problems, the function is evaluated through anumerical simulation with a finite accuracy. It follows that the finite differenceformulas (1.29)(1.31) should be used with a carefully selected perturbation.Using a too small perturbation may result in an inaccurate result because of the

M athemati cal background 11


29/325

numerical noise of the simulator. Using a large perturbation will also result in aninaccurate result. Also, the computational cost of one simulation may be tooexpensive. In that case the less accurateFFD or BFD is preferable over the costlybut more accurate CFD. We will discuss in Chapter 10 the adjoint sensitivity

approaches that offer moreefficient approaches for sensitivity estimation.

1.5.2 The gradient

The gradient of a function rf (x ) is the column vector of its derivatives relative toall parameters. It is given by:

rf ¼ @ f @ x 1

@ f

@ x 2 @ f

@ x n

T ð1:32Þ

The gradient varies in general from one point in the parameter space to another. This vector plays an important role in optimization theory as will become clear inthe following chapters.

A differentiable nonlinear functioncan always beapproximated within asmallneighborhood of a point x o by the first-order Taylor expansion:

f ðx þ DxÞ ¼ LðDxÞ ¼ f ðxoÞ þ rf T Dx

¼ f ðxoÞ þ @ f @ x 1

Dx 1 þ @ f @ x 2

Dx 2 þ þ @ f @ x n

Dx n ð1:33Þ

where Dx ¼ x xo ¼ ½Dx 1 Dx 2 . . . Dx n T . All the first derivatives in (1.33)are evaluated at the point x o. The linearization L(Dx ) approximates the originalnonlinear functiononly over asmall neighborhood of thepoint x o. Wewill discussin Chapter 3 in detail the different properties of the gradient.

The following example illustrates the concepts of the gradient and lineariza-tions of nonlinear functions.

Example 1.5: Consider the function:

f ðxÞ ¼ x 1 2ð Þ2 þ x 2 1ð Þ2 x 1x 2 ð1:34ÞEvaluatethegradient of thefunctionat thepointxo ¼ ½1:0 1:0T both analyticallyand numerically. Construct alinearization of the function f (x ) at this point and testits accuracy.

Solution : Thegradient of this function at any point x is given by:

rf

¼

@ f

@ x 1

@ f @ x 2

26664

37775 ¼ 2ð

x 1

2

Þ x 2

2ðx 2 1Þ x 1 ð1:35ÞAt the point xo ¼ ½1:0 1:0T , the gradient is obtained by setting x 1¼ 1.0 andx 2¼ 1.0 in (1.35) to get:

rf ¼ 2ð1 2Þ 12ð1 1Þ 1

¼ 31

ð1:36Þ



30/325

Thesameanswer can beobtained through FFD, BFD, or CFD approximations. Weutilize a perturbation of Dx ¼ 0.05 for both parameters. The function values at theoriginal point and the perturbed points are given by:

f ð1:0;1:0Þ ¼ 0f ðxo þ Dx e1Þ ¼ f ð1:05; 1Þ ¼ 0:1475f ðxo Dx e1Þ ¼ f ð0:95;1Þ ¼ 0:1525

f ðxo þ Dx e2Þ ¼ f ð1:0;1:05Þ ¼ 0:0475f ðxo Dx e2Þ ¼ f ð1:0; 0:95Þ ¼ 0:0525

ð1:37Þ

where e 1 and e 2 are the first and second column of the identity matrix I 2. Theforward, backward, and central finitedifferenceapproximations of thegradient arethus given by:

FFD ¼f ðxo þ Dx e1Þ f ðxoÞ

Dx

f ðxo þ Dx e2Þ f ðxoÞDx

2664

3775 ¼

0:1475 00:05

0:0475 00:05

2664

3775 ¼ 2:950:95

BFD ¼f ðxoÞ f ðxo Dx e1Þ

Dx

f

ðxo

Þ f

ðxo

Dx e2

ÞDx

2

664

3

775¼

0 0:15250:05

0

0:0525

0:05

2

664

3

775¼ 3:05

1:05 ð1:38Þ

CFD ¼f ðxo þ Dx e1Þ f ðxo Dx e1Þ

2Dx

f ðxo þ Dx e2Þ f ðxo Dx e2Þ2Dx

2664

3775 ¼

0:1475 0:15250:1

0:0475 0:05250:1

2664

3775

¼ 3:0

1:0

It can be seen the gradient estimated through the CFD approximation is moreaccurate than the other approaches. Actually, in this example, it is identical tothe analytical gradient. This comes at the expense of requiring four extrafunction evaluations.

At the point x o, a linearization of the function is given by (1.33) as follows:

LðxÞ ¼ LðDxÞ ¼ f ðxoÞ þ rf T Dx ¼ 0 3Dx 1 Dx 2

¼ 3ð

x 1

1Þ

1ð

x 2

1Þ ð

1:39Þ

Notice that Dx ¼ x x o is the deviation from the expansion point. Notice alsothat L(x ¼ x o)¼ 0 implying that linearized function has the same value as the ori-ginal function at the expansion point. Also we have Lðx ¼ ½1:1 1:1T Þ ¼ 0:4. The original function value at this point is f ðx ¼ ½1:1 1:1T Þ ¼ 0:39. It followsthat the linearization approximates the original function well at this point. At thepoint x ¼ ½2:0 2:0T , both the linearization and the original function have thevalues Lðx ¼ ½2:0 2:0T Þ ¼ 4 and f ðx ¼ ½2:0 2:0T Þ ¼ 3. It is obvious that



31/325

as the perturbation Dx becomes larger, the accuracy of the linearization starts todrop. Linearizations are usually used within a ‘‘trust region’’ around the expansionpoint to guarantee reasonable accuracy.

1.5.3 The Jacobian

In many engineering applications, we consider at the same time a number of functions. These functions may, for example, represent the response of a circuit ata number of frequencies. In this case, we have a vector of functions where eachfunction depends on the parameters. A vector of m functions is given by:

f ðxÞ ¼ f 1 f 2 . . . f m ½ T ð1:40Þ

Every one of these functions has its gradient with respect to the parameters x .

The Jacobian matrix is the matrix combining all these gradients together and isgiven by:

J ðxÞ ¼

rf T 1rf T 2

.

.

.

rf T m

26666664

37777775

¼

@ f 1@ x 1

@ f 1@ x 2

@ f 1@ x n

@ f 2@ x 1

@ f 2@ x 2

@ f 2@ x n

.

.

...

...

...

.

@ f m @ x 1

@ f m @ x 2

@ f m @ x n

26666666666664

37777777777775

ð1:41Þ

As evident from (1.41), the Jacobian matrix is a rectangular matrix of m rowsand n columns. The i th row of the Jacobian is the transpose of the gradient of thefunction f i . The linearization (1.33) can be generalized for the vector f to get theexpansion:

f

ðx

þDx

Þ f

ðx

Þ þJ

ðx

ÞDx

ð1:42

Þ The following example illustrates the calculation of the Jacobian and its uti-

lization in approximating functions.

Example 1.6: Consider the vector of functions f ðxÞ ¼ ½3x 21 þ x 32 x 31 2x 3T .Evaluate the Jacobian at the point xo ¼ ½2:0 1:0 1:0T . Utilize this Jacobian toestimatethe value of this vector function at the point x1 ¼ ½2:1 1:2 0:9T .Solution : For this problem, we have two functions (m ¼ 2) and three variables(n ¼ 3). It follows that the Jacobian matrix has two rows and three columns. Eachrow is the transpose of the gradient of the corresponding function. The value of vector of functionsat the expansion point is f ðxoÞ ¼ ½13:0 6:0T . TheJacobian ata general point x is thus given by:

J ðxÞ ¼rf T 1rf T 2

" # ¼

6x 1 3x 22 0

3x 21 0 2

" # ð1:43Þ

At the point x o, the Jacobian is given by:



32/325

J ðxoÞ ¼rf T 1rf T 2

" # ¼ 12 3 0

12 0 2

ð1:44Þ

Utilizing theformula (1.42), thevalueof thefunction vector at thenew pointisgiven by:

f ðx1Þ f ðxoÞ þ J ðxoÞðx1 xoÞ+

f ðx1Þ 136

þ 12 3 012 0 2

2:11:20:9

24

35 2:01:0

1:0

24

35

0@

1A ¼ 14:8

7:4

ð1:45Þ

The exact value of the function at the new point is f ðx1Þ ¼ ½14:958 7:46T . Theaccuracy of the linearization formula (1.42) continues to deteriorate as we movefurther away from the expansion point as expected.

1.5.4 Second-order der i vati ves

The derivatives discussed so far are first-order derivatives. They require only per-turbingoneparameter at atime. Thesecond-order derivatives are, by definition, therateof change of the first-order derivatives. For a function f (x ) with n -parameters,the mixed second-order derivative with respect to the i th and j th parameters isgiven by:

@ 2f

@ x i @ x j ¼ limit

Dx j !0

@ f ðx 1; x 2; . . .; x j þ Dx j ; . . .; x n Þ@ x i

@ f ðx 1; x 2; . . .; x i ; . . .; x n Þ@ x i

Dx j ð1:46Þ

In other words, the derivative @ f /@ x i is calculated for two different values of theparameter x j and their relative difference is calculated. The same value can beobtained if the derivative @ f /@ x j was differentiated relative to x i as the order of differentiation does not matter. These second-order derivatives play an importantrole in some optimization algorithms. They are also used in deriving optimality

conditions as will become clear in following chapters.All the second-order derivatives of a function f (x ) are combined in the com-

pact Hessian matrix:

H

¼

@ 2f

@ x 21

@ 2f

@ x 1@ x 2 @

2f

@ x 1@ x n

@ 2f

@ x 2@ x 1

@ 2f

@ x 22 @

2f

@ x 2@ x n

.

.

. ... .

.

.

@ 2f

@ x n @ x 1

@ 2f

@ x n @ x 2 @

2f

@ x 2n

2666666666666664

3777777777777775

ð1:47

Þ

TheHessian matrix is asymmetric matrix because theorder of differentiation doesnot make a difference. The following example illustrates the calculation of theHessian matrix.



33/325

Example 1.7: Evaluate the gradient and the Hessian of the functionf ðxÞ ¼ 3x 21 þ 2x 1x 2 þ x 1x 3 þ 2:5x 22 þ 2x 2x 3 þ 2x 23 8x 1 3x 2 3x 3 at the pointx o¼ [1.0 1.0 1.0]T .

Solution : Utilizing the gradient definition (1.32) weget the gradient at an arbitrarypoint:

rf ðxÞ ¼6x 1 þ 2x 2 þ x 3 8

2x 1 þ 5x 2 þ 2x 3 3x 1 þ 2x 2 þ 4x 3 3

24

35 ) rf ðxoÞ ¼ 34

0

24

35 ð1:48Þ

The Hessian can be obtained using (1.47) which is equivalent to differentiatingeach component of the transpose of gradient with respect to all parameters. It fol-lows that wehave:

H ¼ @ rf T

@ x ¼

6 2 12 5 21 2 4

24

35 ð1:49Þ

Inthiscase, theHessiandoesnotchangewithparametersasthefunctionisquadratic.

1.5.5 Deri vati ves of vector s and matri ces

In most undergraduate courses, basic rules for differentiating products or summa-tions of functions arediscussed. Thesamerules can beextended toscalars, vectors,

and matrices that are functions of optimization variables. I show here the way thederivatives are calculated for a number of important cases.

The derivatives of a scalar function f (x ) with respect to the vector of para-metersx isacolumn vector givenby thegradient (1.32). If f ¼ a T x , alinear functionof x , then its gradient is obtained by applying (1.32) and is given by:

@ f

@ x ¼ @ ða

T xÞ@ x

¼ @ ðxT aÞ

@ x ¼ a ð1:50Þ

wherea is aconstant column vector. Similarly, thederivative of arow vectorg T of m components with respect to the variables x is defined as:

@ gT

@ x ¼ @

@ x g 1 g 2 . . . g m ½ ¼ @ g 1

@ x

@ g 2@ x

@ g m @ x

+

@ gT

@ x ¼ rg 1 rg 2 . . . rg m ½

ð1:51Þ

The derivative is thus a matrix with n rows and m columns. Notice that a columnvector cannot bedifferentiated with respect to another column vector. A byproductof this result is the derivative of a row vector that is a linear function of the vari-ables of the form:

@ ðxT AÞ@ x

¼ A ð1:52Þ

where A is a constant matrix. Therule (1.52) can be derived by differentiating eachcomponentof therowvector separately relativetoall parametersandputtingtheresult



34/325

in amatrix form. A useful formulathat can bederived using thepreviously discussedrules is the derivativeof the quadratic termx T Ax withA being a squarematrix. Thisderivativeisobtainedby applyingdifferentiationandmultiplicationrules:

@ ðxT AxÞ@ x

¼ @ ðxT AÞ@ x

x þ @ ðxT Þ@ x

ðxT AÞT ¼ Ax þ AT x ð1:53Þ

For a symmetric matrix A, (1.41) gives 2Ax .Differentiating a matrix with respect to a variable implies differentiating all

elements of the matrix. The derivative of a matrix A with respect to the i th para-meter x i is given by:

@ A

@ x i ¼

@ a 11@ x i

@ a 12@ x i

@ a 1n @ x i

@ a 21@ x i

@ a 22@ x i

@ a 2n @ x i

.

.

...

...

...

.

@ a m 1@ x i

@ a m 2@ x i

@ a mn @ x i

26666666666664

37777777777775ð1:54Þ

The following example illustrates differentiation rules of vectors and matrices.

Example 1.8: Repeat Example 1.7 using vector differentiation rules.

Solution : Theobjective function in Example 1.7 can be written as:

f ðxÞ ¼ 0:5 x 1 x 2 x 3½ 6 2 1

2 5 2

1 2 4

264

375 x 1x 2

x 3

264

375þ 8 3 2½ x 1x 2

x 3

264

375

¼ 0:5xT Axþ bT x ð1:55Þ

Applying the vector differentiation rules, the gradient at an arbitrary point isgiven by:

rf ¼ @ f @ x

¼ Ax þ b ¼6 2 12 5 21 2 4

24

35 x 1x 2

x 3

24

35þ 83

2

24

35

¼6x 1 þ 2x 2 þ x 3 8

2x 1 þ 5x 2 þ 2x 3 3x 1

þ2x 2

þ4x 3

2

24

35

ð1:56Þ

which is identical to the result obtained earlier. The Hessian (1.47) can be eval-uated as:

H ðxÞ ¼ @ rf T

@ x ¼ @

@ xðxT AT þ bT Þ ¼ AT ¼

6 2 12 5 21 2 4

24

35 ð1:57Þ

which is identical to (1.49).



35/325

1.6 Subspaces

Linear algebra concepts appear in some theorems or proofs of convergence of optimization techniques. Wecover heresomeof theessentials of theseconceptsforgood understanding of therest of thematerial.

A fundamental concept in linear algebrais theconcept of asubspace. A subsetS of then -dimensional spaceis asubspaceif 8x , y 2 S ,ax þ by 2 S , wherea and bare any scalars. In other words, any combination of two vector members of thesubset must also belong to this subset. Noticethat implies that the origin must alsobelong to this subset. There are many examples of such subsets. For example,consider the hyperplane a T x ¼ 0 in a general n -dimensional space where a is aconstant vector. Thepointsx 1andx 2areonthis hyperplanesatisfyinga

T x 1¼ 0 and

a T x 2

¼0. If weconsider any pointx 3 that isalinear combinationof thesetwopoints

with x 3¼ax 1þ bx 2, then it also satisfies:aT x3 ¼ aT ðax1 þ bx2Þ ¼ aaT x1 þ baT x2 ¼ 0 ð1:58Þ

It follows that x 3 also belongs to this hyperplane. This hyperplane is thus a sub-space. Notice that the hyperplanea T x ¼ b , b =0 is not a subspace because it doesnot include the origin.

An important subspacethat is utilized frequently in optimizationproblems withconstraints, linear ornonlinear, isthenull subspaceof amatrixA, denotedby Null(A). Todefinethissubspace, westartby consideringasetof m givenvectorsa i 2


36/325

RangeðAT Þ NullðAÞ ¼ R n ð1:62Þ

It can be shown that all vectors in the Null(A) are orthogonal to all vectors in the

Range(AT

). In other words, (1.62) shows a decomposition of avector in <n

in twoorthogonal components.If the matrix A has a full rank, it can be shown that the projection of a vector

into the range of AT is given by:

xp ¼ Px ¼ AT ðAAT Þ1Ax ð1:63Þ

where P is a projection matrix. The component of x in the null subspace of A isgiven by:

xn ¼ x xp ¼ x Px ¼ ðI AT ðAAT Þ1AÞx ð1:64Þ

The concepts of null subspace and range are illustrated by the followingexamples.

Example 1.9: Consider the matrix:

A

¼

1 1 31

1 1

1 3 124 35Show that the vector v ¼ ½2 1 1T lies in the null spaceof this matrix. Showthat this vector is normal to all vectors in Range(AT ). Get the projection of thevector w ¼ ½3 2 1T in the null subspaceof A .Solution : Multiplying the vector v by the matrix A we get:

Av ¼1 1 3

1 1 11 3 1

24 35 211

24 35 ¼ 000

24 35It follows that v is normal to all the rows of the matrix A and thus belongs toNull(A). All vectorsin therangeof AT can bewritten as alinear combinationof therows of A as follows:

d

¼ a1

11

324 35þ a21

1

124 35þ a31

3

124 35+

vT d ¼ 2 1 1½ a1113

24

35þ a2 11

1

24

35þ a3 13

1

24

35

0@

1A ¼ 0

regardless of thevalues of thecoefficientsa1, a2, anda3. As thematrixA has onlytwo independent rows, we consider only the first two rows to construct the



37/325

projection matrix. Using(1.64), thematrix projection inthenull spaceof thematrixA is given by:

P n ¼

1 0 0

0 1 00 0 1

264 3751 1

1 13 1

264 375 1 1 31 1 1" #

1 1

1 13 1

264 3750B@ 1CA1

1 1 3

1 1 1" #

P n ¼1 0 0

0 1 0

0 0 1

264

375

1=3 1=3 1=31=3 5=6 1=6

1=3 1=6 5=6

264

375 ¼

2=3 1=3 1=31=3 1=6 1=6

1=3 1=6 1=6

264

375

The projection of the vector w in the null spaceof the vector A is thus given by:

P n w ¼2=3 1=3 1=31=3 1=6 1=61=3 1=6 1=624 35

3

2124 35 ¼3

3=23=224 35It can beeasily verified that this vector is indeed in the null space of the matrix A.

1.7 Convergencerates

Optimization algorithms carry out a sequence of iterations. Starting with an initialguessof theoptimal parametersx (0), thealgorithmcreatesasequenceof pointsx (k ) thatshould convergeto an optimal set of variablesx * satisfying certain optimality condi-

tions. The rateat which the iterations approach the optimal point is called the con-vergencerateof the algorithm. Thefaster thealgorithmapproaches the solution, thebetter thealgorithmis.

An algorithm is said to havelinear rate of convergence if it satisfies:

kxðk þ1Þ xk r kxðk Þ xk ð1:65Þwherer is apositivescalar with r


38/325

f ð lx1 þ ð1 lÞx2Þ lf ðx1Þ þ ð1 lÞf ðx2Þ ð1:67Þ

with 0 l 1.0. In other words, the valueof a function at the line connecting twopoints is less than the convex combination of their function values. Figure 1.1shows an example of a convex function and a non-convex function for the one-dimensional case.

Convexity can also bedefined for sets. If x 1andx 2arepointsin aconvex set S ,then any point onthe line connecting them, x ¼ lx 1þ (1 l)x 2, 0 l 1.0, is alsoin that set. Interestingproperties can beproven for problems with convex objectivefunctions and convex constraints. Figure 1.2 shows an illustration of a convex andnon-convex sets for the two-dimensional case.

An extreme point of a set is a point that cannot be written as a convex combi-nationof any twoother points. Anexampleof thataretheverticesof atriangular area.

x

x1 x2

f ( x2)

x

(a) (b)

f ( x)

f ( x1)l f ( x1) + (1 – l ) f ( x2) l f ( x1) + (1 – l ) f ( x2)

f ( x)

f ( x1)

f ( x2)

x1 x2

Fi gure 1.1 An il lustr ation of convex and non-convex functions. (a) A convex

function is shown where the function value at any point x ¼ lx 1 þ(1 l )x 2 between x 1 and x 2 (0 l 1) is below the line connecting their function values lf(x 1 ) þ (1 l )f(x 2 ) and (b) a non-convex function wher e some of the points have function values higher than the

line lf(x 1 ) þ (1 l )f(x 2 ). Noti ce how the convex functi on has onl y one minimum while the non-convex function has two minima

S S

(a) (b)

Fi gure 1.2 An i l lustration of convex and non-convex sets. (a) A convex set S that

is defined by three constraints y i ( x ) 0, i ¼ 1, 2, 3. The line connecting any two points belonging to S lies entirely in this set and

(b) a non-convex where the line connecting two points have some

points outside the set



39/325

All points inside the triangle can be written as a convex combination of two otherpoints except the vertices. This definition of extreme points will be utilized inChapter 2 when discussinglinear programs.

Another important property of functions is the concept of a constant value

surface. For an n -dimensional function f (x ), the surface f (x )¼ C , with C being aconstant, defines a surface in the n -dimensional space. All points on that surfacehave the same objective function value (C ). For two-dimensional problems, thissurfacebecomes acontour. Thesecontours areusually useful inobtaininggraphicalsolutions of someproblems. As will beseen later, theconstant surfacecontours arerelated to the gradient at every point in the parameter space.

The following example helps illustrate the concepts of convexity and constantvaluesurfaces.

Example 1.10: Consider the two-dimensional function f

ðx 1; x 2

Þ ¼ x 21

þx 22. Obtain

the constant value contours of this function. Determine the minimum of thisfunction by visually inspecting the contours.

Solution : This function represents the square of the L2 normof a point. All normscanbeshownto beconvex. Thecontour plot of this functionis shownin Figure1.3.We see that as we move away from the origin, the values associated with thecontours increase. It can beseen fromthe contours that the minimum, as expected,is at the origin x ¼ [0 0]T .

1.9 Solutionsof systemsof nonlinear equations

In Section 1.4, we addressed the solution of linear systems of equations. Manyengineeringproblemsmay, however, result inasystemof nonlinear equationsof theformf (x )¼ 0, wheref 2


40/325

If m > n , the system is overdetermined and an exact solution may not exist. The problem should then be treated as an optimization problem in which we aimatminimizingk f k atthesolution. Wewill addresstheoverdeterminedcaseinChapter 7.

We address here the Newton–Raphson technique for solving the case m

¼n .

The algorithm assumes that at the k th iteration, we have the approximation to thesolution x (k ). Writing the Taylor expansion (1.42) at this point, we have:

f ðxðk Þ þ Dx Þ f ðxðk ÞÞ þ J ðxðk ÞÞDx ð1:68Þ

ThealgorithmselectsaperturbationDx (k )suchthatthefunctionatthenewx (k )þDx (k )satisfies f (x (k )þDx (k ))¼ 0. Imposingthis conditionin(1.68) gives theequations:

f ðxðk ÞÞ þ J ðxðk ÞÞDxðk Þ ¼ 0) Dxðk Þ ¼ J ðxðk ÞÞ1 f ðxðk ÞÞ ð1:69Þ

It is assumed here that the Jacobian matrix is J 2


41/325

The new point is thus given by:

x 1ð Þ ¼ xð0Þ þ Dx 0ð Þ ¼ 24 þ

0:76471

1:05882 ¼

1:235292:94118

At the point x (1), wehave:

f ðxð1ÞÞ ¼ 0:584781:12111

; J ðxð1ÞÞ ¼ 2:47059 1:00000001:000000 1:88235

Noticethat thealgorithmis converging ask f ðxð1ÞÞk k f ðxð0ÞÞk. Thestep taken inthe second iteration is given by:

Dxð1Þ ¼ 2:47059 1:00000001:000000 1:88235

1 0:584781:12111

¼ 0:33313 0:176970:17697 0:43723

0:584781:12111

¼ 0:393210:38669

The new point is thus given by:

x

2

ð Þ ¼ xð1

Þ þ Dx 1

ð Þ ¼ 1:23529

2:94118 þ 0:393210:38669 ¼ 0:842082:55448 Notice how the step taken is getting smaller which is also a characteristic of con-verging iterations. At the point x (2), wehave:

f ðxð2ÞÞ ¼ 0:154620:14953

; J ðxð2ÞÞ ¼ 1:684161 1:000001:00000 1:10897

The step taken in the third iteration is given by:

Dxð2Þ ¼ 1:684161 1:000001:00000 1:10897

1 0:154620:14953

¼ 0:38671 0:348710:34871 0:58729

0:154620:14953

¼ 0:111940:03390

The last point is given by:

x 3ð Þ ¼ xð2Þ þ Dx 2ð Þ ¼ 0:842082:55448

þ 0:111940:03390 ¼ 0:730142:52058 The value of the nonlinear equations at this point is:

f ðxð3ÞÞ ¼ 0:012530:00115



42/325

It follows that the normof the error has dropped to only 0.01258 at the last itera-tion. The exact solution of this problem is x *¼ [0.72449 2.52489]T .

1.10 Optimization problemdefinition

For the remainder of this book, we are concerned with techniques for solvingoptimization problems of the form:

x ¼ minx

f ðxÞ

subject to:y j ðxÞ 0; j ¼ 1;2; . . . ; m h k ðxÞ ¼ 0; k ¼ 1; 2; . . . ; p ð1:70Þ

We are searching for an optimal point x *

that has the minimum value of theobjectivefunctionover all points satisfyingthegivenconstraints. Theconstraintsin(1.70) consist of m inequality constraints and p equality constraints. The nature of the constraints and the objective function determine how the problem should besolved. We will focus mainly on problems with nonlinear differentiable objectivefunctions. Constrained and unconstrained problems will be addressed.

Most of the algorithms discussed in this book are local optimization techni-ques. They obtain a minimum x * satisfying f (x *) f (x ), x 2 N (x *), the neighbor-hood of the point x *. In other words, the optimal point has the best value of theobjectivefunctionover all closepoints. Other minimamay also exist and may have

a better value of the objective function. A global optimization technique aims atfinding the point x * that has the best objective function value over all pointssatisfying the constraints not only within a small neighborhood. Such a pointis called the global minimum of the problem. Convex functions discussed inSection 1.8 enjoy the characteristic that a local minimum of a convex function isalso a global minimum.

As will be shown in later chapters, local techniques require much less numberof function evaluations than global techniques.

References

1. Paul McDougle, Vector Algebra , Wadsworth Publishing Company, Belmont,CA, 1971

2. John W. Bandler and R.M. Biernacki, ECE3KB3 Courseware on Optimizati on Theory , McMaster University, Canada, 1997

3. Singiresu S. Rao, Engineeri ng Optimization Theory and Practice , ThirdEdition, John Wiley & Sons Inc, New Y ork, 1996

Problems

1.1 Obtain two different solutions of the underdetermined system of equations:

x 1 2x 2 þ x 3 x 4 ¼ 42x 1 þ x 2 x 3 þ 2x 4 ¼ 1x 1 þ x 2 þ 2x 3 þ x 4 ¼ 3

through basic row operations.



43/325

1.2 Evaluate analytically the gradient of the function f ðx 1; x 2Þ ¼x 1 þ x 22 þ e x 1sinðx 2Þ at the point x ¼ ½1:0 0:5T . Confirm your answer byusing FFD, BFD, and CFD approximations with Dx ¼ 0.05 for bothparameters.

1.3 Evaluate analytically the gradient and the Hessian of the functionf ðx 1; x 2; x 3Þ ¼ 3x 21 þ 6x 22 þ 2x 23 2x 1 þ x 2 þ x 3 5 at the pointx ¼ ½2:0 2:0T . Confirm your result using FFD approximation withDx ¼ 0.02.

1.4 Repeat Problem 1.3 by putting the function in a matrix form similar to(1.55). Apply vector differentiation rules.

1.5 Consider the circuit shown in Figure1.4.By applying Kirchhoff’s voltage and current laws, express the powerdelivered to the load resistor R L as a function of the other circuit parameter.

Obtain analytically the gradient of power relative to the parametersx ¼ [R 1 R s R L]T . Evaluate the gradient for the shown values. Confirmyour answer by usingFFD withaperturbationof Dx ¼ 2.0W for all resistors.

1.6 Consider the matrix:

A ¼1 3 5

3 10 85 8 7

24 35

Evaluate the quadratic forms x T Ax and y T Ay for x ¼ [1 0 0]T andy ¼ [0 0 1]T . What is your conclusion of the definiteness of this matrix?

1.7 Using elementary row operations, obtain two solutions of the under-determined system:

2x 1þ

x 2

4x 3

¼ 8

3x 1 x 2 þ 2x 3 ¼ 1

1.8 Project the vector x ¼ [1.0 1 1.0]T in the null subspace of the matrixA ¼ 1 3 4

1 1 1

. Find its component in the Range(AT ). Show that both

vectors are orthogonal to each other.

Rs = 40Ω

R1 = 75 Ω R L = 50ΩV s = 1.0 V +

–

Fi gure 1.4 The circuit of Problem 1.5



44/325

1.9 Express analytically the Jacobian of the vector function

f ¼ ½3x 21 þ 2x 42 3x 31 2x 2T . Evaluate the Jacobian at the pointx ¼ ½2 2:0T . Writethelinearization of this vector function at this point.Compare its value at the points ½2:1 1:9

T

and ½3:0 3:0T

with thatobtained directly fromthe vector function.1.10 Consider the nonlinear circuit of Figure 1.5. The circuit elements have the

values V s ¼ 5.0 V, R s ¼ 1.0 K W, R 1¼ 300W, R 2¼ 500W, R 3¼ 200W, and

R 4¼ 1.0 K W. The two diodes follow the model i D ¼ I s exp(v D /V T ), withI s ¼ 1.0e13 A and V T ¼ 0.025 V.(a) Write the two nonlinear nodal equations at nodes 1 and 2 in terms of

the two unknown diode voltages.(b) Derive the J acobian of the two nonlinear equations with respect to

x ¼ ½v D 1 v D 2T .(c) Write a MATLAB program that implements the NewtonRaphson

method to obtain the solution of this circuit starting with initial solu-tionxð0Þ ¼ ½0:4 V 0:4 VT . Does the algorithm converge? Repeat forthe starting pointxð0Þ ¼ ½0:5 V 0:5 VT . What is x *?

1 2 Rs R2

R4

R3 R1

V s +

–

Figure 1.5 The circuit of Problem 1.10



45/325


46/325

Chapter 2

An introduction to linear programming

2.1 Introduction

Many engineering optimization problem can be cast as a linear Program. By a

Linear Programming (LP) I mean an optimization problem where the objectivefunction and the constraints are linear functions of the optimization variables. Inaddition, several nonlinear optimization problems can be solved by iterativelysolving linearized versions of the original problem. Several examples of suchproblems will beillustrated in this chapter and in thefollowingchapters. Theword‘‘programming’’ in this context does not mean writing computer codes usingprogramming languages but rather it means ‘‘scheduling.’’

It was not until the end of the Second World War that significant progress inthe solution of linear programs was achieved. Dantzig developed his famous Sim-

plex approach. Following his pioneering work, several variations of the Simplexmethod were introduced. The more efficient interior-point methods (IPMs) forsolving linear programs were developed later.

In this chapter, we focus on the solution of linear programs. Different state-ments of the LP problemareintroduced. The Simplex method is explained in boththetabular andmatrix forms. Several approaches for starting theSimplex approachare discussed.

2.2 Examplesof linear programs

Linear programs arise in many fields when there is a needed compromise betweenthe linear cost or resources in a problem and the expected profit. I give here fewsimple examples.

2.2.1 A farming example

A farmer owns apiece of land of area 5.0 km2. Hewants to grow in his land eitherwheat or barley or a combination of both. He has only 100.0 kg of fertilizer and20.0 kg of pesticides. Eachsquarekilometer of barely requires 30.0 kg of fertilizerand 4.0 kg of pesticide, while each square kilometer of wheat requires 25.0 kg

of fertilizer and 6.0 kg of pesticide. He is expected to sell his crop for $1.0 perkilogramof barely and $1.25 per kilogramof wheat. Eachsquarekilometer of landshould deliver 1 ton of barely or 0.75 ton of wheat. Our target is to formulate thisproblem as a linear program.

The farmer aims at maximizing his profit within the limited resources he has.Wedenotetheareaassignedtobarely by x b andtheareaassignedtowheat asx w . Theexpected income is thus f ¼ 1.00 1000 x b þ 1.25 750 x w . As the farmercannotusemorethan100 kgof fertilizer, it followsthat (30 x b ) þ (25 x w ) 100.


47/325

For the pesticide, the physical constraint (4.0 x b ) þ (6.0 x w ) 20 applies. Also,the sumof the two areas should not exceed the area of the pieceof land (5 km2). Itfollows that the corresponding linear programis given by:

maximize 1:00 1000 x b þ 1:25 750 x w subject to ð30 x b Þ þ ð25 x w Þ 100

ð4 x b Þ þ ð6 x w Þ 20x b þ x w 5

ð2:1Þ

In matrix form, (2.1) can be written as:

minx

cT x

subject to Ax b; x 0ð2:2Þ

where x ¼ ½x b x w T , b ¼ ½100 20 5T , and c ¼ 1:0 ½1000 ð1:25750ÞT .

The matrix A is given by:

A ¼30 254 61 1

24

35 ð2:3Þ

Notice that maximizing a quantity is equivalent to minimizing its negative. Also,the non-negativity constraints in (2.2) are needed to ensure a feasible answer with

non-negative areas.

2.2.2 A production example

A company produces two types of products. Projections indicate an expecteddemand of at least 75 units for the first product and 50 units for the second productper day. Becauseof physical limitations, no morethan 150 units of thefirst productand100unitsof thesecondproduct canbeproduceddaily. Thecompany shouldshipat least 200 units per day of both types tofulfill its commitments. Theprofit per unitof the first product is $3.00 while the profit per unit of the second product is $5.00.

How many units of each type should be produced per day to maximize profit?Similar to the previous example, the target is to maximize the profit or alter-

natively minimizethenegativeof theprofit subjecttotheconstraints. Wedenotebyx 1 thenumber of produced units of thefirst productper day andby x 2 thenumber of produced units of the second type. Our target is to maximize the objectivefunction f (x 1,x 2) ¼ 3 x 1þ 5 x 2 subject to the constraints 75 x 1 150 and50 x 2 100. Also, the shipments’ constraint implies that 200 (x 1þ x 2). Col-lecting all these equations in one linear program, weget:

subject to

minx cT

xx 1 150

x 1 75

x 2 100

x 2 50

x 2 x 2 200

x 0

ð2:4Þ

30 Nonlinear optimization in electri cal engineer ing with applications in M ATLAB


48/325

wherex ¼ ½x 1 x 2T and c ¼ ½3:0 5:0T . Noticethat all constraints, except for

the non-negativity constraints, arecast in the less than or equal to form. This is theformadopted throughout this text. Inmatrix format, (2.4) canbewritten intheform(2.2) with:

A ¼

1 01 0

0 10 1

1 1

266664

377775; b ¼

15075100

50200

266664

377775 ð2:5Þ

2.2.3 Power gener ation example

Two thermal power plants A and B use two different types of coal C 1

and C 2. The

minimumpower tobegenerated by eachplant is 40.0 and 80.0 MWh, respectively. The quantities of each grade of coal to generate 1.0 MWh and the cost per ton forboth plants are given by the following table:

A (tons) B (tons) A (Cost/ton) B (Cost/ton)

C 1 2.5 1.5 20 18C 2 1.0 2.0 25 28

Our target is todeterminetheamount of eachgradeof coal to beused in generatingpower so as to minimize the total production cost.

We denote by x 1 and x 2 the quantities in tons of the first type of coal usedin both stations. We also denote by x 3 and x 4 the quantities of the second typeof coal used in both stations. It follows that the cost of operation is given byf ¼ 20x 1þ 18x 2þ 25x 3þ 28x 4. The constraints imply minimum power productionin both stations. These constraints are thus formulated as:

x 1=2:5þ x 3=1:0 40

x 2=1:5þ x 4=2:0 80x 0

ð2:6Þ

The problem can be then written in the following form:

minimize 20x 1 þ 18x 2 þ 25x 3 þ 28x 4subject to x 1=2:5 x 3=1:0 40

x 2=1:5 x 4=2:0 80x 0

ð2:7Þ

Notice that all constraints are put in the less than or equal to formby multiplyingboth sides by a negative sign. Also, all quantities areforced to be positive throughthe non-negativity constraint to ensure a physical solution. This system has theform (2.2) with the parameters x ¼ x 1 x 2 x 3 x 4½

T , b ¼ ½40 80T ;c ¼ 20 18 25 28½ T , and:

A ¼ 0:4 0 1 0

0 0:6667 0 0:5

An introduction to linear programming 31


49/325

2.2.4 Wir eless communication example

Consider the wireless communication system shown in Figure 2.1. There are n users connected to a base station. The i th user transmits fromhis cell phone with a

power of p i . Thepower transmitted by the i thuser facesattenuation h i by thetimeitreaches thebasestation. When thestation is receiving fromthe i thuser, thesumof powers received from all other users is considered as interference. For reliablecommunication, the signal to interference ratio (SNR) must exceed a certainthreshold gi . Our target is to minimize the powers sent by all users while main-taining a reliable communication.

Thetotal power transmitted by all users is f ¼ p 1þ p 2þ þ p n . TheSNR forthe i th user is SNRi ¼ h i p i =

P j ; j 6¼ i h j p j , 8i . The condition on the SNR ratio implies

that SNRi gi or that h i p i gi

P j ; j 6¼ i h j p j 0; i ¼ 1; 2; . . .; n . TheLP problemto

solve this problem can thus be cast in the form:

minimize cT xAx bx 0

wherex ¼ ½p 1 p 2 . .. p n T ; c ¼ ½1 1 . . . 1T ; A ¼

h 1 g1h 2 . . . g1h n g2h 1 h 2 . . . g2h n

.

.

....

.

.

....

gn h 1 gn h 2 . . . h n

26664

37775;

b ¼ 0T .

2.2.5 A batter y chargi ng example

The electric circuit shown in Figure 2.2 uses a source of 30 V to charge threebatteries of values 5.0, 10.0, and 20.0 V. The currents in the branches have thefollowing physical constraints I 1 4.0 A, I 2 3.0 A, I 3 3.0 A, I 4 2.0 A, andI 5 2.0A. All currents I i , 8i must bepositive to prevent batteries fromdischarging.Weaimat formulatingan LP that determines thevalues of thecurrentsmaximizingthe total power transferred to the charged batteries.

The power transferred to each battery is equal to the product of its current (in

the charging direction) by the voltage of the battery. The total power is thus givenby f ¼ 5I 2þ 10I 4þ 20I 5. In addition, circuit current lawsimposethe constraint that

Fi gure 2.1 A wir eless communication example

32 Nonlinear optimization in electri cal engineer ing with applications in M ATLAB


50/325

the current flowing into every node must be equal to the currents flowingout fromthat node. It follows that the charging problem can be cast in the following LP:

maximize 5I 2 þ 10I 4 þ 20I 5

subject to I 1 ¼ I 2 þ I 3

I 3 ¼ I 4 þ I 5

I 1 4:0; I 2 3:0; I 3 3:0

I 4 2:0; I 5 2:0; and

I 0

ð2:8Þ

whereI is avector whosecomponentsarethecurrents. This problemcan becast inthe compact form:

maximize cT x

subject to Ax ¼ b

x xu

x 0

ð2:9Þ

wherex ¼ ½I 1 I 2 I 3 I 4 I 5T ; c ¼ ½0 5 0 10 20T ;

A ¼ 1 1 1 0 0

0 0 1 1 1

; b ¼ ½0 0T ; and xu ¼ ½4 3 3 2 2

T .

2.3 Standard formof an LP

LP problems are usually cast in the standard form:

minx

cT x

Ax ¼ bx 0

ð2:10Þ

Here A 2


51/325

These equations can be cast in the standard form by adding an extra ‘‘slack’’variable to every equation. These variables are positive and they ‘‘balance’’ theequations into equalities. The system of equations in (2.11) then becomes:

Ax þ I xs ¼ bx; xs 0

ð2:12Þ

Notice that in (2.12), one positive slack variable is added to each equation with a

unity coefficient. We can thus define a new extended system matrix A ¼ A I ½

and anew extended vector of unknowns x ¼ xT xT s T

toconvert (2.12) into the

standard form:

minx

cT x

Ax ¼ b

x 0ð2:13Þ

where cT ¼ ½cT 0T . The original problem (2.11) has been converted into thestandard form(2.10) at theexpenseof increasing thedimensionality of theproblemby extra m variables with zero cost coefficients.

Similarly, problems with ‘‘greater than or equal to’’ constraints of the form:

Ax bx 0

ð2:14Þ

can be converted into the standard form by subtracting a positive slack variablefrom every equation to get the equality equations:

Ax I xs ¼ bx; xs 0

ð2:15Þ

Anaugmented systemmatrix can then bedefined asin(2.13). If thevariables of theproblem are not constrained to be positive, these variables can be written as thedifference between two positive variables. For example, consider an LP with con-

straints of the form:

Ax ¼ b ð2:16Þ

Notethat thevariables in (2.16) do nothavenon-negativity constraints. Constraintsof the form(2.16) can be converted into the standard formby defining x ¼ x p x n ,where x p and x n are vectors of non-negative variables. In other words, the uncon-strained original problem can be cast in the standard formby adding n more vari-ables. Thenew system to be solved has the form:

A A½ xp

xn

" # ¼ b

xp 0; xn 0

ð2:17Þ

It follows from (2.11)–(2.17) that any LP can be converted to the standard form(2.10). Wewill thus focus onthesolutionof thestandard problem(2.10) in therestof this chapter.

The following examples illustrate the graphical solutions of LP fortwo-dimensional problem

Documents

245893173 Nonlinear Optimization in Electrical Engineering With Applications in MATLAB 2013