1. 1. S M A R T G R I D S Clouds, Communications, Open Source, and Automation
2. 2. Devices, Circuits, and Systems Series Editor Krzysztof Iniewski CMOS Emerging Technologies Research Inc., Vancouver, British Columbia, Canada PUBLISHED TITLES: Atomic Nanoscale Technology in the Nuclear Industry Taeho Woo Biological and Medical Sensor Technologies Krzysztof Iniewski Building Sensor Networks: From Design to Applications Ioanis Nikolaidis and Krzysztof Iniewski Circuits at the Nanoscale: Communications, Imaging, and Sensing Krzysztof Iniewski Electrical Solitons: Theory, Design, and Applications David Ricketts and Donhee Ham Electronics for Radiation Detection Krzysztof Iniewski Embedded and Networking Systems: Design, Software, and Implementation Gul N. Khan and Krzysztof Iniewski Energy Harvesting with Functional Materials and Microsystems Madhu Bhaskaran, Sharath Sriram, and Krzysztof Iniewski Graphene, Carbon Nanotubes, and Nanostuctures: Techniques and Applications James E. Morris and Krzysztof Iniewski High-Speed Photonics Interconnects Lukas Chrostowski and Krzysztof Iniewski Integrated Microsystems: Electronics, Photonics, and Biotechnology Krzysztof Iniewski Integrated Power Devices and TCAD Simulation Yue Fu, Zhanming Li, Wai Tung Ng, and Johnny K.O. Sin Internet Networks: Wired, Wireless, and Optical Technologies Krzysztof Iniewski Low Power Emerging Wireless Technologies Reza Mahmoudi and Krzysztof Iniewski Medical Imaging: Technology and Applications Troy Farncombe and Krzysztof Iniewski
3. 3. MEMS: Fundamental Technology and Applications Vikas Choudhary and Krzysztof Iniewski Microfluidics and Nanotechnology: Biosensing to the Single Molecule Limit Eric Lagally and Krzysztof Iniewski MIMO Power Line Communications: Narrow and Broadband Standards, EMC, and Advanced Processing Lars Torsten Berger, Andreas Schwager, Pascal Pagani, and Daniel Schneider Nano-Semiconductors: Devices and Technology Krzysztof Iniewski Nanoelectronic Device Applications Handbook James E. Morris and Krzysztof Iniewski Nanoplasmonics: Advanced Device Applications James W. M. Chon and Krzysztof Iniewski Nanoscale Semiconductor Memories: Technology and Applications Santosh K. Kurinec and Krzysztof Iniewski Novel Advances in Microsystems Technologies and Their Applications Laurent A. Francis and Krzysztof Iniewski Optical, Acoustic, Magnetic, and Mechanical Sensor Technologies Krzysztof Iniewski Radiation Effects in Semiconductors Krzysztof Iniewski Semiconductor Radiation Detection Systems Krzysztof Iniewski Smart Grids: Clouds, Communications, Open Source, and Automation David Bakken and Krzysztof Iniewski Smart Sensors for Industrial Applications Krzysztof Iniewski Technologies for Smart Sensors and Sensor Fusion Kevin Yallup and Krzysztof Iniewski Telecommunication Networks Eugenio Iannone Testing for Small-Delay Defects in Nanoscale CMOS Integrated Circuits Sandeep K. Goel and Krishnendu Chakrabarty Wireless Technologies: Circuits, Systems, and Devices Krzysztof Iniewski PUBLISHED TITLES:
4. 4. FORTHCOMING TITLES: 3D Circuit and System Design: Multicore Architecture, Thermal Management, and Reliability Rohit Sharma and Krzysztof Iniewski Circuits and Systems for Security and Privacy Farhana Sheikh and Leonel Sousa CMOS: Front-End Electronics for Radiation Sensors Angelo Rivetti Gallium Nitride (GaN): Physics, Devices, and Technology Farid Medjdoub and Krzysztof Iniewski High Frequency Communication and Sensing: Traveling-Wave Techniques Ahmet Tekin and Ahmed Emira High-Speed Devices and Circuits with THz Applications Jung Han Choi and Krzysztof Iniewski Labs-on-Chip: Physics, Design and Technology Eugenio Iannone Laser-Based Optical Detection of Explosives Paul M. Pellegrino, Ellen L. Holthoff, and Mikella E. Farrell Metallic Spintronic Devices Xiaobin Wang Mobile Point-of-Care Monitors and Diagnostic Device Design Walter Karlen and Krzysztof Iniewski Nanoelectronics: Devices, Circuits, and Systems Nikos Konofaos Nanomaterials: A Guide to Fabrication and Applications Gordon Harling and Krzysztof Iniewski Nanopatterning and Nanoscale Devices for Biological Applications Krzysztof Iniewski and Seila Selimovic Optical Fiber Sensors and Applications Ginu Rajan and Krzysztof Iniewski Organic Solar Cells: Materials, Devices, Interfaces, and Modeling Qiquan Qiao and Krzysztof Iniewski Power Management Integrated Circuits and Technologies Mona M. Hella and Patrick Mercier Radio Frequency Integrated Circuit Design Sebastian Magierowski Semiconductor Device Technology: Silicon and Materials Tomasz Brozek and Krzysztof Iniewski
5. 5. FORTHCOMING TITLES: Soft Errors: From Particles to Circuits Jean-Luc Autran and Daniela Munteanu VLSI: Circuits for Emerging Applications Tomasz Wojcicki and Krzysztof Iniewski Wireless Transceiver Circuits: System Perspectives and Design Aspects Woogeun Rhee and Krzysztof Iniewski
6. 6. CRC Press is an imprint of the Taylor & Francis Group, an informa business Boca Raton London NewYork E D I T E D B Y David Bakken Washington State University School of Electrical Engineering and Computer Science M A N A G I N G E D I T O R Krzysztof Iniewski CMOS Emerging Technologies Research Inc. Vancouver, British Columbia, Canada S M A R T G R I D S Clouds, Communications, Open Source, and Automation
7. 7. MATLAB is a trademark of The MathWorks, Inc. and is used with permission. The MathWorks does not warrant the accuracy of the text or exercises in this book. This books use or discussion of MATLAB soft- ware or related products does not constitute endorsement or sponsorship by The MathWorks of a particular pedagogical approach or particular use of the MATLAB software. CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 2014 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Version Date: 20140305 International Standard Book Number-13: 978-1-4822-0612-8 (eBook - PDF) This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint. Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmit- ted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright. com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com
8. 8. ix Contents Preface.................................................................................................................... xiii Editors.......................................................................................................................xv Contributors............................................................................................................xvii Chapter 1 Mission-Critical Cloud Computing for Critical Infrastructures...........1 Thoshitha Gamage, David Anderson, David Bakken, Kenneth Birman, Anjan Bose, Carl Hauser, Ketan Maheshwari, and Robbert van Renesse Chapter 2 Power Application Possibilities with Mission-Critical Cloud Computing........................................................................................... 17 David Bakken, Pranavamoorthy Balasubramanian, Thoshitha Gamage, Santiago Grijalva, Kory W. Hedman, Yilu Liu, Vaithianathan Venkatasubramanian, and Hao Zho Chapter 3 Emerging Wide-Area Power Applications with Mission-Critical Data Delivery Requirements...............................................................33 Greg Zweigle Chapter 4 GridStat: High Availability, Low Latency, and Adaptive Sensor Data Delivery for Smart Generation and Transmission......................55 David E. Bakken, Harald Gjermundrd, and Ioanna Dionysiou Chapter 5 A Distributed Framework for Smart Grid Modeling, Monitoring, and Control.................................................................... 115 Alfredo Vaccaro and Eugenio Zimeo Chapter 6 Role of PLC Technology in Smart Grid Communication Networks........................................................................................ 133 Angeliki M. Sarafi, Artemis C. Voulkidis, Spiros Livieratos, and Panayotis G. Cottis Chapter 7 Power Grid Network Analysis for Smart Grid Applications............. 151 Zhifang Wang, Anna Scaglione, and Robert J. Thomas
9. 9. x Contents Chapter 8 Open Source Software, an Enabling Technology for Smart Grid Evolution................................................................................... 179 Russell Robertson, Fred Elmendorf, and Shawn Williams Chapter 9 Contribution of Microgrids to the Development of the Smart Grid............................................................................... 191 Tine L. Vandoorn and Lieven Vandevelde Chapter 10 Microgrids......................................................................................... 213 Mietek Glinkowski, Adam Guglielmo, Alexandre Oudalov, Gary Rackliffe, Bill Rose, Ernst Scholtz, Lokesh Verma, and Fang Yang Chapter 11 Integrating Consumer Advance Demand Data in Smart Grid Energy Supply Chain........................................................................ 251 Tongdan Jin, Chongqing Kang, and Heping Chen Chapter 12 Photovoltaic Energy Generation and Control for an Autonomous Shunt Active Power Filter.................................................................. 275 Ayman Blorfan, Damien Flieller, Patrice Wira, Guy Sturtzer, and Jean Merckl Chapter 13 Self-Tuning and Self-Diagnosing Simulation.................................... 311 Jin Ma Chapter 14 A Consensus-Based Fully Distributed Load Management Algorithm for Smart Grid................................................................. 333 Yinliang Xu, Wei Zhang, and Wenxin Liu Chapter 15 Expert Systems Application for the Reconfiguration of Electric Distribution Systems......................................................................... 359 Horacio Tovar-Hernndez and Guillermo Gutierrez-Alcaraz Chapter 16 Load Data Cleansing and Bus Load Coincidence Factors................ 375 Wenyuan Li, Ke Wang, and Wijarn Wangdee Chapter 17 Smart Metering and Infrastructure...................................................399 Wenpeng Luan and Wenyuan Li
10. 10. xiContents Chapter 18 Vision of Future Control Centers in Smart Grids............................. 421 Fangxing Li, Pei Zhang, Sarina Adhikari, Yanli Wei, and Qinran Hu Index....................................................................................................................... 435
11. 11. xiii Preface While electric interconnections have had different kinds and levels of intelligence in them for many decades, in the last 6years the notion of the smart grid has come seemingly out of nowhere to be on the minds of not just power engineers but policy makers, regulators, rate commissions, and the general public. Inherent in the notion of the smart grid is the ability to communicate much more sensor data and have far more computations at many more locations using these data. The purpose of this book is to give power engineers, information technology workers in the electric sector, and others a snapshot of the state of the art and practice today as well as a peek into the future regarding the smart grid. There is a special focus on new kinds of communications and computations enabled or necessitated by the smart grid. This book is divided into four parts. Part I deals with cloud computing, whose use is being seriously considered for planning and operational use in a number of utilities and independent system operators/regional transmission organizations as of March 2014. Cloud computing has the potential to deploy massive amounts of computational resources to help grid operations, especially under contingency situations. Chapter1 describes the mission-critical features that cloud computing infrastructures must sup- port in order to be appropriate for operational use in power grids. It also describes the Advanced Research Projects Agency-Energy GridCloud project to develop such technologies. Chapter 2 describes a handful of killer apps for cloud computing in power grid operations. It has been written by leading power researchers. Part II deals with wide-area communications for power grids. Chapter 3 describes a wide range of power application programs that have extreme communications require- ments over wide distances. Such applications are becoming more widely deployed as grids come under more pressure with every passing year. Chapter 4 describes GridStat, a middleware communications framework designed from the ground up to meet these challenging requirements. The chapter includes a detailed analysis of how different technologies used in todays grids such as multiprotocol label switching, Internet pro- tocol multicast, IEC 61850, and others are inadequate for the applications described in Chapter 3 and the requirements derived from them in Chapter 4. Chapter 5 presents an advanced framework based on the service-oriented architecture approach for integrated modeling, monitoring, and control. Chapter 6 analyzes the role of power line commu- nication, which is also called broadband over power lines, in the smart grid. Power line communication/broadband over power lines technologies can provide additional redun- dant paths for data delivery in a grid, and ones that have failure characteristics other than traditional network communications infrastructures. Finally, Chapter 7 describes a novel approach for estimating the statistical properties of power grids. This is an impor- tant first step toward having more broadly reusable power algorithms with greater con- fidence, as computer scientists and mathematicians have done for centuries. Part III deals with open source, something common in other industries that is start- ing to draw great interest from utilities and has great potential to help stimulate inno- vation in power grids (which suffer from a far higher degree of vendor lock-in than
12. 12. xiv Preface most other industries). Chapter 8 explains what open source software is and its history. It then overviews a number of freely available open source power application programs. Part IV deals with the broad category of automation. Chapter 9 explains how microgrids fit into the smart grid landscape and how they can contribute to its operations. Chapter 10 describes in detail the design and operation of microgrids. Chapter11 introduces a virtual energy provisioning concept by which utilities can collect and aggregate advanced demand information in order to better manage smart grid supply chains. Chapter 12 describes a new technique for better managing pho- tovoltaic energy while limiting harmonic pollution. Chapter 13 provides an approach for two-way interactions between simulations and an operational wide-area measure- ment system that is both self-tuning and self-diagnosing. Chapter 14 describes an approach for load management in smart grids that is stable, distributed and employs multiagent techniques. Chapter 15 details the use of an expert system application to enable electric distribution systems to be reconfigured in new and advantageous way. Chapter 16 describes an approach for both cleansing the load curve data and calculat- ing bus load coincidence factors in order to better exploit smart meter data. Chapter 17 overviews an advanced metering infrastructure system and its components, discusses its benefits, and summarizes a variety of applications by which smart metering and infrastructure supports both planning and operations. Finally, Chapter 18 offers a vision of how smart grid control centers may look in the future. David E. Bakken Pullman, Washington Krzysztof (Kris) Iniewski Vancouver, British Columbia MATLAB is a registered trademark of The MathWorks, Inc. For product informa- tion, please contact: The MathWorks, Inc. 3 Apple Hill Drive Natick, MA 01760-2098 USA Tel: 508 647 7000 Fax: 508-647-7001 E-mail: info@mathworks.com Web: www.mathworks.com MATLAB and Simulink are trademarks of the MathWorks, Inc. and are used with permission. The MathWorks does not warrant the accuracy of the text or exercises in this book. This books use or discussion of MATLAB and Simulink software or related products does not constitute endorsement or sponsorship by the MathWorks of a particu- lar pedagogical approach or particular use of the MATLAB and Simulink software.
13. 13. xv Editors David Bakken is a professor of computer science in the School of Electrical Engineering and Computer Science at Washington State University and chief scien- tist at GridStat, Inc. His research interests include wide-area distributed computing systems, middleware implementation, and dependable computing. Since 1999, he has been working closely with researchers in his departments very strong electric power group on helping rethink the way data delivery is done in power grids over the wide area, and is considered the worlds leading expert on this. His GridStat data-delivery software has influenced the shape of the emerging NASPInet. He is a frequent visitor and lecturer at utilities, electrical engineering departments, and power meetings worldwide. Prior to Washington State University, Dr. Bakken was a research scientist at BBN (Cambridge, MA), which built the first Internet in 1969. There he was coinventor of the Quality Objects middleware framework, in which the Defense Advanced Research Projects Agency invested more than 50 BBN person-years, which was integrated with approximately 10 other research projects in various demonstrations, and which flew in Boeing experimental aircraft. Dr. Bakken has worked for Boeing and consulted for Amazon.com, Harris Corp., Realtime Innovations, Intel, TriGeo Network Security, and others. He holds a MS (1990) and a PhD (1994) in computer science from the University of Arizona, and Bachelor of Science degrees in com- puter science and mathematics from Washington State University (1985). He is the author of over 100 publications and coinventor of three patents. Krzysztof (Kris) Iniewski manages R&D at Redlen Technologies Inc., a start- up company in Vancouver, Canada. Redlens revolutionary production process for advanced semiconductor materials enables a new generation of more accurate, all- digital, radiation-based imaging solutions. Kris is also president of CMOS Emerging Technologies Research Inc. (www.cmosetr.com), an organization of high-tech events covering communications, microsystems, optoelectronics, and sensors. In his career, Dr. Iniewski has held numerous faculty and management positions at the University of Toronto, the University of Alberta, Simon Fraser University, and PMC-Sierra Inc. He has published over 100 research papers in international journals and con- ferences. He holds 18 international patents granted in the United States, Canada, France, Germany, and Japan. He is a frequent invited speaker and has consulted for multiple organizations internationally. He has written and edited several books for CRC Press, Cambridge University Press, IEEE Press, Wiley, McGraw-Hill, Artech House, and Springer. His personal goal is to contribute to healthy living and sus- tainability through innovative engineering solutions. In his leisure time, Kris can be found hiking, sailing, skiing, or biking in beautiful British Columbia. He can be reached at kris.iniewski@gmail.com.
14. 14. xvii Sarina Adhikari Department of Electrical Engineering and Computer Science University of Tennessee Knoxville, Tennessee David Anderson School of Electrical Engineering and Computer Science Washington State University Pullman, Washington David Bakken School of Electrical Engineering and Computer Science Washington State University Pullman, Washington Pranavamoorthy Balasubramanian School of Electrical, Computer, and Energy Engineering Arizona State University Tempe, Arizona Kenneth Birman Department of Computer Science Cornell University Ithaca, New York Ayman Blorfan Modelling, Intelligence, Process and Systems Laboratory Universit de Haute Alsace Mulhouse, France and National Institute of Applied Science Strasbourg, France Anjan Bose School of Electrical Engineering and Computer Science Washington State University Pullman, Washington Heping Chen Ingram School of Engineering Texas State University San Marcos, Texas Panayotis G. Cottis School of Electrical and Computer Engineering National Technical University of Athens Athens, Greece Ioanna Dionysiou Department of Computer Science University of Nicosia Nicosia, Cyprus Fred Elmendorf Grid Protection Alliance Chattanooga, Tennessee Damien Flieller National Institute of Applied Science Research Group of Electrical and Electronics in Nancy Strasbourg, France Thoshitha Gamage School of Electrical Engineering and Computer Science Washington State University Pullman, Washington Contributors
15. 15. xviii Contributors Harald Gjermundrd Department of Computer Science University of Nicosia Nicosia, Cyprus Mietek Glinkowski ABB Inc. Raleigh, North Carolina Santiago Grijalva School of Electrical and Computer Engineering Georgia Institute of Technology Atlanta, Georgia Adam Guglielmo ABB Inc. Raleigh, North Carolina Guillermo Gutierrez-Alcaraz Department of Electrical Engineering Instituto Tecnolgico de Morelia Morelia, Mexico Carl Hauser School of Electrical Engineering and Computer Science Washington State University Pullman, Washington Kory W. Hedman School of Electrical, Computer, and Energy Engineering Arizona State University Tempe, Arizona Qinran Hu Department of Electrical Engineering and Computer Science University of Tennessee Knoxville, Tennessee Tongdan Jin Ingram School of Engineering Texas State University San Marcos, Texas Chongqing Kang Department of Electrical Engineering Tsinghua University Beijing, China Fangxing Li Department of Electrical Engineering and Computer Science University of Tennessee Knoxville, Tennessee Wenyuan Li School of Electrical Engineering Chongqing University Chongqing, China and BC Hydro Vancouver, Canada Yilu Liu Department of Electrical Engineering and Computer Science University of Tennessee Knoxville, Tennessee Wenxin Liu Klipsch School of Electrical and Computer Engineering New Mexico State University Las Cruces, New Mexico Spiros Livieratos School of Pedagogical and Technological Education Athens, Greece Wenpeng Luan State Grid Smart Grid Research Institute Beijing, China
16. 16. xixContributors Jin Ma School of Electrical and Information Engineering The University of Sydney New South Wales, Australia Ketan Maheshwari Argonne National Laboratory Lemont, Illinois Jean Merckl Modelling, Intelligence, Process and Systems Laboratory Universit de Haute Alsace Mulhouse, France Alexandre Oudalov ABB Inc. Raleigh, North Carolina Gary Rackliffe ABB Inc. Raleigh, North Carolina Robbert van Renesse Department of Computer Science Cornell University Ithaca, New York Russell Robertson Grid Protection Alliance Chattanooga, Tennessee Bill Rose ABB Inc. Raleigh, North Carolina Angeliki M. Sarafi School of Electrical and Computer Engineering National Technical University of Athens Athens, Greece Anna Scaglione Department of Electrical and Computer Engineering University of California Davis, California Ernst Scholtz ABB Inc. Raleigh, North Carolina Guy Sturtzer National Institute of Applied Science Research Group of Electrical and Electronics in Nancy Strasbourg, France Robert J. Thomas Department of Electrical and Computer Engineering Cornell University Ithaca, New York Horacio Tovar-Hernndez Department of Electrical Engineering Instituto Tecnolgico de Morelia Morelia, Mexico Alfredo Vaccaro Department of Engineering University of Sannio Benevento, Italy Lieven Vandevelde Department of Electrical Energy Ghent University Ghent, Belgium Tine L. Vandoorn Department of Electrical Energy Ghent University Ghent, Belgium
17. 17. xx Contributors Vaithianathan Venkatasubramanian School of Electrical Engineering and Computer Science Washington State University Pullman, Washington Lokesh Verma ABB Inc. Raleigh, North Carolina Artemis C. Voulkidis School of Electrical and Computer Engineering National Technical University of Athens Athens, Greece Ke Wang School of Computing Science Simon Fraser University Vancouver, Canada Zhifang Wang Department of Electrical and Computer Engineering Virginia Commonwealth University Richmond, Virginia Wijarn Wangdee The Sirindhorn International ThaiGerman Graduate School of Engineering King Mongkuts University of Technology North Bangkok Bangkok, Thailand Yanli Wei Department of Electrical Engineering and Computer Science University of Tennessee Knoxville, Tennessee Shawn Williams Grid Protection Alliance Chattanooga, Tennessee Patrice Wira Modelling, Intelligence, Process and Systems Laboratory Universit de Haute Alsace Mulhouse, France Yinliang Xu Klipsch School of Electrical and Computer Engineering New Mexico State University Las Cruces, New Mexico Fang Yang ABB Inc. Raleigh, North Carolina Pei Zhang Grid Operations and Planning Electric Power Research Institute Palo Alto, California Wei Zhang Klipsch School of Electrical and Computer Engineering New Mexico State University Las Cruces, New Mexico Hao Zhu Department of Electrical and Computer Engineering University of Illinois Champaign, Illinois Eugenio Zimeo Department of Engineering University of Sannio Benevento, Italy Greg Zweigle Schweitzer Engineering Laboratories,Inc. Pullman, Washington
18. 18. 1 1 Mission-Critical Cloud Computing for Critical Infrastructures Thoshitha Gamage, David Anderson, David Bakken, Kenneth Birman, Anjan Bose, Carl Hauser, Ketan Maheshwari, and Robbert van Renesse 1.1 INTRODUCTION The term cloud is becoming prevalent in nearly every facet of day-to-day life, bring- ing up an imperative research question: how can the cloud improve future critical infrastructures? Certainly, cloud computing has already made a huge impact on the computing landscape and has permanently incorporated itself into almost all sec- tors of industry. The same, however, cannot be said of critical infrastructures. Most notably, the power industry has been very cautious regarding cloud-based computing capabilities. This is not a total surprise: the power industry is notoriously conservative about changing its operating model, and its rate commissions are generally focused on short-term goals. With thousands of moving parts, owned and operated by just as many stakeholders, even modest changes are difficult. Furthermore, continuing to CONTENTS 1.1 Introduction ......................................................................................................1 1.1.1 Cloud Computing..................................................................................2 1.1.2 Advanced Power Grid...........................................................................4 1.2 Cloud Computings Role in the Advanced Power Grid....................................5 1.2.1 Berkeley Grand Challenges and the Power Grid..................................7 1.3 Model for Cloud-Based Power Grid Applications............................................8 1.4 GridCloud: A Capability Demonstration Case Study ......................................9 1.4.1 GridStat.................................................................................................9 1.4.2 Isis2......................................................................................................10 1.4.3 TCP-R................................................................................................. 11 1.4.4 GridSim ..............................................................................................12 1.4.5 GridCloud Architecture......................................................................12 1.5 Conclusions.....................................................................................................13 References................................................................................................................15
19. 19. 2 Smart Grids operate while incorporating large paradigm shifts is neither a straightforward nor a risk-free process. In addition to industry conservatism, progress is slowed by the lack of comprehensive cloud-based solutions meeting current and future power grid application requirements. Nevertheless, there are numerous opportunities on many frontsfrom bulk power generation, through wide-area transmission, to residential distribution, including at the microgrid levelwhere cloud technologies can bolster power grid operations and improve the grids efficiency, security, and reliability. The impact of cloud computing is best exemplified by the recent boom in e-commerce and online shopping. The cloud has empowered modern customers with outstanding bargaining power in making their purchasing choices by provid- ing up-to-date pricing information on products from a wide array of sources whose computing infrastructure is cost-effective and scalable on demand. For example, not long ago air travelers relied on local travel agents to get the best prices on their reser- vations. Cloud computing has revolutionized this market, allowing vendors to easily provide customers with web-based reservation services. In fact, a recent study shows that online travel e-commerce skyrocketed from a mere $30 billion in 2002 to a staggering $103 billion, breaking the $100 billion mark for the first time in the United States in 2012 [1]. A similar phenomenon applies to retail shopping. Nowadays, online retail shops offer a variety of products, ranging from consumer electronics, clothing, books, jewelry, and video games to event tick- ets, digital media, and lots more at competitive prices. Mainstream online shops such as Amazon, eBay, Etsy, and so on provide customers with an unprecedented global marketplace to both buy and sell items. Almost all major US retail giants, such as Walmart, Macys, BestBuy, Target, and so on, have adopted a hybrid sales model, providing online shops to complement the traditional in-store shopping experience. A more recent trend is flash sale sites (Fab, Woot, Deals2Buy, Totsy, MyHabit, etc.), which offer limited-time deals and offers. All in all, retail e-commerce in the United States increased by as much as 15% in 2012, totaling $289 billion. To put this into perspective, the total was $72 billion 10 years earlier. Such rapid growth relied heav- ily on cloud-based technology to provide the massive computing resources behind online shopping. 1.1.1 CLOUD COMPUTING What truly characterizes cloud computing is its business model. The cloud provides on-demand access to virtually limitless hardware and software resources meeting the users requirements. Furthermore, users only pay for resources they use, based on the time of use and capacity. The National Institute of Standards and Technology (NIST) defines five essential cloud characteristics: on-demand self-service, broad network access, resource pooling, rapid elasticity, and measured service [2]. The computational model of the cloud features two key characteristics abstraction and virtualization. The cloud provides its end users with well-defined application programming interfaces (APIs) that support requests to a wide range of hardware and software resources. Cloud computing supports various configura- tions (central processing unit [CPU], memory, platform, input/output [I/O], network- ing, storage, servers) and capacities (scale) while abstracting resource management
20. 20. 3Mission-Critical Cloud Computing for Critical Infrastructures (setup, startup, maintenance, etc.), underlying infrastructure technology, physical space, and human labor requirements. The end users see only APIs when they access services on the cloud. For example, users of Dropbox, the popular cloud-based online storage, only need to know that their stored items are accessible through the API; they do not need any knowledge of the underlying infrastructure supporting the service. Furthermore, end users are relieved of owning large computing resources that are often underused. Instead, resources are housed in large data centers as a shared resource pool serving multiple users, thus optimizing their use and amortiz- ing the cost of maintenance. At the same time, end users are unaware of where their resources physically reside, effectively virtualizing the computing resources. Cloud computing provides three service models: software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS). Each of these service models provides unique APIs. Services can be purchased separately, but are typically purchased as a solution stack. The SaaS model offers end-point business applications which are customizable and configurable based on specific needs. One good example is the Google Apps framework, which offers a large suite of end-user applications (email, online storage, streaming channels, domain names, messaging, web hosting, etc.) that individuals, businesses, universities, and other organizations can purchase individually or in combination. Software offered in this manner has a shorter development life cycle, resulting in frequent updates and up-to-date versions. The life-cycle maintenance is explicitly handled by the service provider, who offers the software on a pay-per-use basis. Since the software is hosted in the cloud, there is no explicit installation or maintenance process for the end users in their native environment. Some of the prominent SaaS providers include Salesforce, Google, Microsoft, Intuit, Oracle, and so on (Figure 1.1). The PaaS model offers a development environment, middleware capabilities, and a deployment stack for application developers to build tailor-made applications or host prepurchased SaaS. Amazon Web Services (AWS), Google App Engine, and Microsoft Azure are a few examples of PaaS. In contrast to SaaS, PaaS does not abstract development life-cycle support, given that most end users in this model are application developers. Nevertheless, the abstraction aspect of cloud computing is Software as a service (application) Platform as a service (operating system) Infrastructure as a service (hardware) FIGURE 1.1 Cloud service models as a stack.
21. 21. 4 Smart Grids still present in PaaS, where developers rely on underlying abstracted features such as infrastructure, operating system, backup and version control features, development and testing tools, runtime environment, workflow management, code security, and collaborative facilities. The IaaS model offers the fundamental hardware, networking, and storage capa- bilities needed to host PaaS or custom user platforms. Services offered in IaaS include hardware-level provisioning, public and private network connectivity, (redundant) load balancing, replication, data center space, and firewalls. IaaS relieves end users of operational and capital expenses. While the other two models also provide these features, here they are much more prominent, since IaaS is the closest model to actual hardware. Moreover, since the actual hardware is virtualized in climate-controlled data centers, IaaS can shield end users from eventual hardware failures, greatly increasing availability and eliminating repair and maintenance costs. A popular IaaS provider, Amazon Elastic Compute Cloud (EC2), offers 9 hardware instance fami- lies in 18 types [3]. Some of the other IaaS providers include GoGrid, Elastic Hosts, AppNexus, and Mosso [4]. 1.1.2 ADVANCED POWER GRID Online shopping is just one of many instances where cloud computing is making its mark on society. The power grid, in fact, is currently at an interesting cross- roads in this technological space. One fundamental capability that engineers are striving to improve is the grids situational awarenessits real-time knowledge of grid statethrough highly time-synchronized phasor measurement units (PMUs), accurate digital fault recorders (DFRs), advanced metering infrastructure (AMI), smart meters, and significantly better communication. The industry is also facing a massive influx of ubiquitous household devices that exchange information related to energy consumption. In light of these new technologies, the traditional power grid is being transformed into what is popularly known as the smart grid or the advanced power grid. The evolution of the power grid brings its own share of challenges. The newly introduced data have the potential to dramatically increase accuracy, but only if pro- cessed quickly and correctly. True situational awareness and real-time control deci- sions go hand in hand. The feasibility of achieving these two objectives, however, heavily depends on three key features: 1. The ability to capture the power grid state accurately and synchronously 2. The ability to deliver grid state data reliably and in a timely manner over a (potentially) wide area 3. The ability to rapidly process large quantities of state data and redirect the resulting information to appropriate power application(s), and, to a lesser extent, the ability to rapidly acquire computing resources for on-demand data processing Emerging power applications are the direct beneficiaries of rapid data cap- ture, delivery, processing, and retrieval. One such example is the transition from
22. 22. 5Mission-Critical Cloud Computing for Critical Infrastructures conventional state estimation to direct state calculation. Beginning in the early 1960s, the power grid has been employing supervisory control and data acquisi- tion (SCADA) technology for many of its real-time requisites, such as balancing load against supply, demand response, and contingency detection and analysis. SCADA uses a slow, cyclic polling architecture in which decisions are based on unsynchronized measurements that may be several seconds old. Consequently, the estimated state lags the actual state most. Thus, state estimation gives very lim- ited insight and visibility into the grids actual operational status. In contrast, tightly time-synchronized PMU data streams deliver data under strict quality of service (QoS) guaranteeslow latency and high availabilityallowing control centers to perform direct state calculations and measurements. The capabilities that come with the availability of status data make creating a real-time picture of the grids opera- tional state much more realistic [5]. There are also many myths surrounding the operations of a power grid in con- junction with big data and its efficient use. The following is a nonexhaustive list of some of these myths. 1. Timeliness: Real-time data is a relative term. Often the application require- ments dictate the timeliness needs. Modern software and hardware tech- nologies provide many workarounds on the timeliness of data availability on wide area networks with average bandwidths. One of them is selective packet dropping. This technique guarantees a minimum QoS while deliv- ering information to recipients in a timely manner. Smart power grids will greatly benefit from these techniques. 2. Security and Safety: Security and safety are concerns often cited by deci- sion makers when considering new technologies. While absolute security is impossible, most concerns arising from data security issues have been technically addressed. One large factor that affects security is human errors and oversights. Often, insufficient emphasis is given to this side of security. More and more emphasis is given to the communication channels. Securing an already secure channel only results in performance losses and overheads. 3. Cost: The cost of maintaining information infrastructures has become a major portion of budgets for large industries, and is a substantial challenge in running a sustainable, data-centered architecture. Thanks to data centers and cloud computing infrastructures, these challenges are being success- fully addressed. Clouds facilitate outsourcing of large-scale computational infrastructures while achieving provably reliable QoS. 1.2 CLOUD COMPUTINGS ROLE IN THE ADVANCED POWER GRID Cloud computing can play a vital role in improving the advanced power grids situational awareness and the ability to derive better control decisions. As men- tioned earlier, emerging power applications will leverage large amounts of data in making control decisions affecting the stability and reliability of the grid.
23. 23. 6 Smart Grids Analyzingandprocessing such large amounts of data require data parallelism and massive computational capabilities well beyond general-purpose computing. Beyond data analysis, the future grid can greatly benefit from much more extensive simula- tion and analysis to remediate stressful situations. These are spontaneous special purpose applications (e.g., system integrity protection schemes [SIPS], also known as remedial action schemes [RASs] or special protection schemes [SPSs]) [6], each with different needsreal time, scaling, and computationalthat are triggered by grid disturbances such as frequency oscillations, voltage fluctuations, line overloads, and blackouts. Moreover, the number of power grid applications and their computational needs can only be expected to increase as the grid evolves. Managing this variety of applications and needs presents a challenge. Keeping these applications running idle on dedicated hardware until the specific condition is triggered is both inefficient and expensive. An elegant solution is presented here which utilizes cloud computing and its rapid elasticity. Power grid applications can utilize the cloud to rapidly deploy an application-specific infrastructure using IaaS and PaaS to achieve new levels of availability and scalability. Availability and scalability are properties that are much harder to meet in a piecemeal fashion, but are inherent features of the cloud and easily adoptable. Cloud-based solutions also benefit entities at different levels of the control hierarchy, giving them the ability to perform an independent, replicated analysis on the same sensor data. The ability to elastically manage resources in the presence of a grid disturbance is extremely attractive in comparison with in-house solutions, which could be overprovisioned or underprovisioned at the time of need. Another area where cloud computing performs well is in supporting the varying needs of the growing ecosystem of power applications. Both PaaS and SaaS will be useful for developing and supporting power applications. PaaS for the power grid will need to encompass industry best practices, engineer- ing standards, compliance requirements, and data privacy and security requirements as properties of the platform itself. The CAP theorem [7] argues that simultaneously achieving three key propertiesconsistency, availability, and partition tolerance is impossible in distributed systems. As a result, and especially since their apps are not mission critical, present-day commercial clouds often sacrifice consistency in favor of availability. Cloud environments that are used for power applications must be able to guarantee high-assurance properties, including consistency, fault tolerance, and real-time responsiveness, in order to support the anticipated needs of power applications. While PaaS enables power researchers and developers to expand the power appli- cation ecosystem, SaaS can abstract essential properties and requirements to provide end-user application solutions. Grid incident-specific applications can be offered as SaaS, readily deployable by power utilities. The success of power grid SaaS depends heavily on the completeness and the richness of power grid PaaS. The overarching challenge lies in ensuring that power applications delivered across SaaS/PaaS mod- els inherently carry the necessary high-assurance properties. The subtle intricacies of high-assurance properties, which are often outside the power engineering realm, will necessitate a different approach to cloud computing as well as a stronger mesh between power engineering and computer science.
24. 24. 7Mission-Critical Cloud Computing for Critical Infrastructures 1.2.1 BERKELEY GRAND CHALLENGES AND THE POWER GRID The Berkeley view of the cloud [8,9] outlines 10 grand challenges and opportunities for cloud computing. The following list reviews some of these challenges and their implications for cloud-based power grid applications: 1. Ensuring High Service Availability: Consistency is arguably one of the most critical requirements for cloud-based power applications [10], but availability is a close second. Many of the early adopters of cloud technol- ogy support availability as a liveness property, while smart-grid applica- tions depend on availability as a safety property. Availability also relates to the issue of whether cloud-based power grid applications should follow stateful or stateless models. The ability to design stateful applications often depends on the availability of state information. Achieving high availability requires avoiding single point of failure scenarios and potential bottlenecks. The general consensus is that the cloud promotes and provides high availability. However, using cloud ser- vices from a single service provider allows a single point of failure [9]. Interoperability between different cloud vendors for the sake of availability for power grid applications merely because of the many proprietary and market advantages is a far-fetched ambition. Perhaps one solution would be for the power grid community to manage and operate its own cloud, either overlaying existing commercial clouds or as a private infrastructure with built-in replication at all levels. Such an initiative, however, would be dictated by the many economic drivers. 2. Eliminating Data Transfer Bottlenecks: Large amounts of high-frequency data must cross the cloud boundary to reach power applications running within the cloud. Application responsiveness is directly tied to the time lines with which data reach their destination. The outermost layer of the cloud can have a dual role as a sensor data aggregator for sources outside the cloud and as a multiplexer toward the applications within the cloud. Thus, a sufficiently large number of replicated cloud end points for sensor data must be provided in order to prevent a potential data transfer bottleneck. 3. Assuring Data Confidentiality and Availability: For a community histori- cally notorious for a conservative modus operandi, sharing sensor data is a frightening proposition. Power grid entities operate under industry regula- tions and standards that can prevent data sharing in many circumstances. Additionally, companies are reluctant to share market-sensitive data that could give away economic advantage. Power application data travers- ing the cloud must be secured, meeting all compliance requirements, so that they cannot be used by unintended parties. Thus, the cloud will need to provide adequate data sanitization and filtering capabilities to protect applicationdata. 4. Performance Predictability under Scaling: The enormous amount of data that some power applications require, combined with the impacts of vir- tualized I/O channels and interrupts, leads to unpredictable performance
25. 25. 8 Smart Grids during elastic scaling. Different IaaS vendors exhibit different I/O perfor- mance, resource acquisition, and release time characteristics. The computa- tional performance on current cloud computing infrastructures also shows signs of strain under scaling [11]. High-end batch processing applications will require improved resource sharing and scheduling capabilities for vir- tual machines to ensure strict QoS demands. 1.3 MODEL FOR CLOUD-BASED POWER GRID APPLICATIONS Many of the cloud adoption challenges outlined in Section 1.2 are essentially about supporting highly scalable, highly assured behaviors and stringent communication guarantees. These are properties that are rarely found in todays commercial cloud infrastructures, which are optimized to support mobile applications and web-based e-commerce applications. The notions of real-time responsiveness, guaranteed con- sistency, data security, and fault tolerance are significantly more forgiving in these applications than in infrastructure control, supplying little incentive for current com- mercial clouds to embrace the type of changes necessary to support critical infra- structure systems. Figure 1.2 visually represents an abstract architectural model for cloud-based power applications. The architecture includes three basic components: 1. A real-time data collection and transport infrastructure 2. A soft state, elastic outer cloud tier that supports data collection, data pre- processing, temporary archiving, data sanitization and filtering, and multi- plexing to services residing in interior tiers 3. Interior cloud tiers hosting services and applications, and supporting data processing, analysis, batch processing, persistent storage, and visualization functions The data collection and transportation infrastructure sits between the physical sensors and the outermost tier of the cloud, and is the communication backbone of Data sanitization Soft archiving Visualization Hard archiving Interior cloud tier (applications) Outermost cloud tier (data collectors) Analysis Computation Data multiplexing Data aggregation Batch processing Data ltering Data collection and transportationSensors FIGURE 1.2 An abstract architectural model for a cloud-based power grid application.
26. 26. 9Mission-Critical Cloud Computing for Critical Infrastructures the overall architecture. This component is responsible for delivering data that are produced outside the cloud to the first-tier cloud collectors with strong QoS guaran- tees such as guaranteed delivery, ultrahigh availability, ultralow latency, and guar- anteed latency. The soft state, outermost cloud tier provides the interface to data flowing to the applications hosted in the interior tiers. The primary objective of this tier is to provide high availability, to exhibit rapid elasticity, and to forward correct data to the appropriate applications. To aid in this process, this tier will also host auxiliary applications that provide data sanitization, filtering of bad data, buffering (or soft achieving), data preprocessing, and forwarding capabilities. Availability and fault tolerance are heightened by replicated shardsnodes that collect data from a group of sensorsand by mapping sensor data sources appropriately to the shards. The interior cloud tiers host the actual applications that consume data from the shards and perform analysis, computation, and batch processing tasks. Additionally, the results of these deeper computations may be delivered at high rates to visualiza- tion applications residing inside and outside the cloud. 1.4 GRIDCLOUD: A CAPABILITY DEMONSTRATION CASE STUDY An Advanced Research Projects Agency-Energy (ARPA-E)-funded, high-profile research collaboration between Cornell University and Washington State University is spearheading efforts to develop, prototype, and demonstrate a powerful and com- prehensive software platform realizing the cloud computing needs of the future power grid. Appropriately named GridCloud [12], this research project aims to bring together best-of-breed, already existing high-assurance distributed system technolo- gies as a basis to innovate new cloud architectural models for the monitoring, man- agement, and control of power systems. The technologies integrated in this effort include GridStat [13,14], Isis2 [15,16], TCP-R [17], and GridSim [18]. A brief descrip- tion of each of these technologies is presented here. 1.4.1 GRIDSTAT GridStat implements a data delivery overlay network framework designed from the bottom up to meet the challenging requirements of the electric power grid (see Chapter 4). Power grids today are increasingly populated with high-rate, time-synchro- nized sensors that include PMUs and DFRs, whose functionalities are actually blurring. High-rate, time-synchronized data are expected to form the basis of many monitor- ing and control applications with a wide variety of delivery requirements and config- urations across such dimensions as geographic scope, latency, volume, and required availability[19]. These needs cannot be met by Internet protocol (IP) multicast, which forces all subscribers of a given sensor variable to get all updates at the highest rate that any subscriber requires. They also cannot be met by multiprotocol label switching (MPLS), which is not designed to provide per-message guarantees (only overall statisti- cal guarantees) and also only has three bits (eight categories) with which to categorize the millions of different sensor flows that will likely be deployed in 510 years. GridStat delivers rate-based updates of sensor variables with a wide range of QoS+ guarantees (latency, rate, availability) that include support for ultralow latency
27. 27. 10 Smart Grids and ultrahigh availability, which are implemented by sending updates over redundant disjoint paths, each of which meets the end-to-end latency requirements for the given subscription. Additionally, GridStat enables different subscribers to a given sensor variable to require different QoS+ guarantees, which can greatly reduce bandwidth requirements and improve scalability. GridStats data delivery plane is a flat graph of forwarding engines (FEs), each of which stores the state for every subscription whose updates it forwards. FEs for- ward sensor updates on each outgoing link at the highest rate that any downstream subscriber requires. They drop updates that are not needed downstream, based on the expressed rate requirements of subscribers. GridStats management plane is implemented as a hierarchy of QoS brokers that can be mapped onto the natural hierarchy of the power grid. Each node in the hierarchy is designed to contain poli- cies for resource permissions, security permissions, aggregation, and adaptations to anomalies. With these policies, the management plane calculates the paths required for the data delivery (with the given number of disjoint paths) and updates the for- warding tables in the FEs. Applications interact with GridStat using publisher and subscriber software libraries through which the applications requirements for QoS are conveyed to the management plane. GridStat incorporates mechanisms for secur- ing communication between the management plane entities and those of the data plane. Security mechanisms for end-to-end message security between publishers and subscribers are modular and configurable, allowing different data streams and appli- cations to fulfill different security and real-time requirements [20]. GridStat in the power grid provides the opportunity to respond to different power system operating conditions with different communication configurations. GridStat provides a mecha- nism by which communication patterns can be rapidly changed among multiple pre- configured modes in response to anticipated power system contingencies. 1.4.2 ISIS2 Isis2 is a high-assurance replication and coordination technology that makes it easy to capture information at one location and share it in a consistent, fault-tolerant, secure manner with applications running at other locationsperhaps great num- bers of them. This system revisits a powerful and widely accepted technology for replicating objects or computations, but with a new focus on running at cloud scale, where the system might be deployed onto thousands of nodes and supporting new styles of machine-learning algorithms. Isis2 enables massive parallelism, strong con- sistency, and automated fault tolerance, and requires little sophistication on the part of its users. With Isis2, all computational nodes and applications sharing the same data see it [the data?] evolve in the same manner and at nearly the same time, with delays often measured in hundreds of microseconds. The system also supports repli- cated computation and coordination: with Isis2 one could marshal 10,000 machines to jointly perform a computation, search a massive database, or simulate the conse- quences of control actions, all in a manner that is fast, secure against attack or intru- sion, and correct even if some crashes occur. The form of assurance offered by Isis2 is associated with a formal model that merges two important prior modelsvirtual synchrony [21] and Paxos [22]. Isis2
28. 28. 11Mission-Critical Cloud Computing for Critical Infrastructures embeds these ideas into modern object-oriented programming languages. Isis2 is used to create two new components for GridCloud: a version of the technology spe- cialized for use in wide-area power systems networks, and support for high-assurance smart-grid applications that are hosted in cloud computing data centers. The GridCloud researchers believe that Isis2 can be used to support services that run on standard cloud infrastructures and yet (unlike todays cloud solutions) are able to guarantee continuous availability, automatically adapting under attack so that intruders cannot disrupt the grid even if a few nodes are compromised. They are also analyzing and demonstrating the best options for building cloud services that respond to requests in a time-critical manner. 1.4.3 TCP-R GridCloud will tie together a very large number of components, including sensors, actuators, forwarding elements and aggregators, cloud-based services, and so on, using Internet standards. For best performance, it is important that related components communicate using persistent, stateful connections. Stateful connections reduce retransmissions and wasteful connections, and provide better flow control. The standard for stateful connections in the Internet is transmission control protocol (TCP). TCP provides network connections that provide reliable first in, first out (FIFO) communication as well as fair flow provisioning using adaptive congestion windows. Consider a cloud service that sends commands to a large number of actuators. The cloud service consists of a cluster of a few hundred servers. To keep actuators simple, and also to allow flexibility in evolving the cloud service, the cloud service should appear to the actuators as a single end point with a single TCP/IP address. While an actuator will receive commands from a particular server machine in the cluster, it appears to the actuators (and their software) as if the cloud service is a single, highly reliable, and fast machine. It is desirable to maintain this illusion even when connections migrate between server machines for load balancing, for hardware or software upgrades, or when rebooting cloud servers. TCP connections, unfortu- nately, are between socket end points, and, using current operating systems abstrac- tions, socket end points cannot migrate or survive process failures. Also, the cloud service would have to maintain a TCP socket for every actuator. This does not scale well, as each TCP socket involves storing a lot of state information. Replacing TCP with a radically different protocol would not be feasible today. Operating systems and even networking hardware implement TCP connections very efficiently. TCP is the dominant communication protocol on the Internet, and Internet routers have evolved to support TCP efficiently, easily scaling to many mil- lions of simultaneous TCP connections. TCP-R proposes to support standard TCP connections, but to extend them with a technology that addresses the shortcomings mentioned above. The essential idea is to extend the cloud service with a filter that intercepts and preprocesses TCP packets. The filter is scalable and maintains little state per TCP connection (on the order of 32 bytes). It has only soft state (i.e., it does not have to store its state persistently across crashes, greatly simplifying fault toler- ance). The filter allows servers to migrate TCP connections, and TCP connections
29. 29. 12 Smart Grids to survive server failure and recovery. Originally developed to maintain TCP con- nections between border gateway protocol (BGP) (Internet routing) servers across failures and subsequent recovery [23], TCP-R is extended into a scalable technology for a cluster serving client end points and also to park connections that are not currently live. 1.4.4 GRIDSIM GridSim is a real-time, end-to-end power grid simulation package that is unique in its integration of a real-time simulator, data delivery infrastructure, and multiple applications all running in real time. The goal of this project is to simulate power grid operation, control, and communications on a grid-wide scale (e.g., the Western Interconnection), as well as to provide utilities with a way to explore new equipment deployments and predict reactions to contingencies. The ability to simulate opera- tion under different equipment deployment configurations includes large-scale con- figurations of PMUs. With the objective of simulating real-world equipment usage, and usage in conjunction with readily available industry equipment, the GridSim simulation package uses the industry standard C37.118 data format for all streaming measurement data. The first element in the GridSim platform is a transient power stability simula- tor, specially modified to output streaming data in real time. The output data are encoded into C37.118 and sent to a huge number of substation processes. At each of these processes, the data are aggregated, as would be done in a real power utility sub- station. The data are also sent to any of the substation-level power applications that are running. Both the raw substation data as well as any power application outputs are then published to GridStat. GridStat allows the substation data to be distributed as they would be in the real world. Published data can be sent via redundant paths, 1many communication (publish-subscribe, whose degenerate version is network-level multicast), and so on. The flexibility provided by the GridStat data delivery middleware allows subscrip- tion applications to be easily integrated into the system with minimal reconfigura- tion. Published data are available to any subscribers of GridStat, including the two applications included in the GridSim simulation, the hierarchical state estimator (HSE), and the oscillation and damping monitor (Figure 1.3). 1.4.5 GRIDCLOUD ARCHITECTURE GridCloud was designed with the expectation that the developers of the advanced power grid will require easy access to large computing resources. Tasks may require large-scale computation, or may involve such large amounts of data that simply host- ing and accessing the data will pose a substantial scalability challenge. This leads us to believe that cloud computing will play an important role in the future grid, supplementing the roles played by existing data center architectures. The compel- ling economics of cloud computing, the ease of creating apps that might control household power consumption (not a subject that has been mentioned yet), and the remarkable scalability of the cloud all support this conclusion.
30. 30. 13Mission-Critical Cloud Computing for Critical Infrastructures Figure 1.4 shows the architecture of GridCloud. The representative application used in this case is a HSE [18,24,25]. Data sources represent PMUs, which stream data to data collectors across a wide area using GridStat. The HSE comprises several substation-level state estimators that aggregate, filter, and process PMU data before forwarding them to a control center-level state estimator. The input and the first-level computation are inherently sharded at substation granularity. Furthermore, compu- tations are inherently parallel between substations. Thus, the HSE has a natural map- ping in GridCloud with substation state estimators residing in the outermost tier of the cloud while the control center state estimator is moved to the interior tier. The substation state estimators are replicated to increase fault tolerance and availability. The consistency of the replicas is managed through Isis2. TCP-R is used to provide fail-over capabilities for connections across the cloud. 1.5 CONCLUSIONS This chapter presents a roadmap of how cloud computing can be used to support the computational needs of the advanced power grid. Todays commercial cloud comput- ing infrastructure lacks the essential properties required by power grid applications. These deficiencies are explained and a cloud-based power grid application architec- ture is presented which overcomes these difficulties using well-known distributed system constructs. Furthermore, the GridSim project, which instantiates this model, is presented as a case study example. Control- level applications OpenPDC Oscillation monitor State estimator Substation gateway Static data generator C37.118 generator Measurement generator Powertech TSAT simulator Substation SE Subs SE Sub SE Su SE Su SE Substation OM Substation OM Substation OM Substation O Substation O Substation 1 Substation N Simulated power system GridStat FE FE FE FE FE FESubstation- level simulation FIGURE 1.3 The GridSim architecture. (From Anderson, D., Zhao, C., Hauser, C., Venkatasubramanian, V., Bakken, D., and Bose, A., IEEE Power and Energy Magazine, 10, 4957, 2012.)
31. 31. 14 Smart Grids COL N M COL 2 M SKPM P2 Dataorigin P1 S2 TCP-R S1 COL 1 M COL N 2 COL 2 2 COL 1 2 COL N 1 COL 2 1 COL 1 1 FE FE FE FE FE FE FE FE GridStat (UDP) S-SE1 M S-SE 1 2 S-SE 2 2 S-SE N 2 S-SE 1 1 S-SE 2 1 EC2cloud GridCloudOutput Localhost S-SE N 1 S-SE 2 M S-SE N M S-SE N S-SE 2 S-SE 1 ControlcenterSE Localclient and visualizer Computation Results Isis 2 FIGURE1.4GridCloudarchitecture.
32. 32. 15Mission-Critical Cloud Computing for Critical Infrastructures REFERENCES 1. G. Fulgoni, A. Lipsman and I. Essling, State of the U.S. Online Retail Economy in Q4 2012, February 2013. Available at: http://goo.gl/BCnbn. 2. P. Mell and T. Grance, The NIST Definition of Cloud Computing: Recommendations of the National Institute of Standards and Technology. The National Institute of Standards and Technology, Gaithersburg, MD, 2011. 3. A. W. Services, Amazon elastic compute cloud: User guide API version 2013-02-01. http://awsdocs.s3.amazonaws.com/EC2/latest/ec2-ug.pdf (accessed 3 June 2013). 4. R. Prodan and S. Ostermann, A survey and taxonomy of infrastructure as a service and web hosting cloud providers, in Proceedings of the 10th IEEE/ACM International Conference on Grid Computing, October 1315, 2009, Banff, AB. 5. D. Novosel, V. Madani, B. Bhargava, K. Vu and J. Cole, Dawn of the grid synchroniza- tion, IEEE Power and Energy Magazine, 6(1), 4960, 2008. 6. S. Horowitz, D. Novosel, V. Madani and M. Adamiak, System-wide protection, IEEE Power and Energy Magazine, 6(September), 3442, 2008. 7. S. Gilbert and N. Lynch, Brewers conjecture and the feasibility of consistent, available, partition-tolerant web services, ACM SIGACT News, 33(2), 5159, 2002. 8. M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski, G. Lee, et al., A view of cloud computing, Communications of the ACM, 53(4), 5058, 2010. 9. M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski, G. Lee, et al., Above the clouds: A Berkeley view of cloud computing, Department of Electrical Engineering and Computer Sciences, University of California Berkeley, Report UCB/ EECS 28, 2009. 10. K. Birman, L. Ganesh and R. V. Renesse, Running smart grid control software on cloud computing architectures, in Proceedings of the Computational Needs for the Next Generation Electric Grid Workshop, Cornell University, April, 2011, New York. 11. A. Iosup, S. Ostermann, M. Yigitbasi, R. Prodan, T. Fahringer and D. H. J. Epema, Performance analysis of cloud computing services for many-tasks scientific computing, IEEE Transactions on Parallel and Distributed Systems, 22(6), 931945, 2011. 12. K. Birman, A. Bose, D. Bakken and C. Hauser, GridControl: A software platform to support the smart grid, A Cornell University and Washington State University Research Collaboration, 2011. Available at: http://www.cs.cornell.edu/Projects/gridcontrol/index. html#gridcloud (accessed 3 June 2013). 13. H. Gjermundrod, D. Bakken, C. Hauser and A. Bose, GridStat: A flexible QoS-managed data dissemination framework for the power grid, IEEE Transactions on Power Delivery, 24(1), 136143, 2009. 14. C. Hauser, D. Bakken and A. Bose, A failure to communicate: Next generation commu- nication requirements, technologies, and architecture for the electric power grid, IEEE Power and Energy Magazine, 3(2), 4755, 2005. 15. K. P. Birman, D. A. Freedman, Q. Huang and P. Dowell, Overcoming CAP with consis- tent soft-state replication, IEEE Computer, 45(2), 5058, 2012. 16. K. Birman, Isis2 cloud computing library, 2013. Available at: http://isis2.codeplex.com/. 17. A. Agapi, K. Birman, R. M. Broberg, C. Cotton, T. Kielmann, M. Millnert, R. Payne, R. Surton and R. van Renesse, Routers for the cloud: Can the Internet achieve 5-nines availability?, IEEE Internet Computing, 15, 7277, 2011. 18. D. Anderson, C. Zhao, C. Hauser, V. Venkatasubramanian, D. Bakken and A. Bose, A virtual smart grid: Real-time simulation for smart grid control and communications design, IEEE Power and Energy Magazine, 10(1), 4957, 2012. 19. D. E. Bakken, A. Bose, C. H. Hauser, D. E. Whitehead and G. C. Zweigle, Smart gen- eration and transmission with coherent, real-time data, Proceedings of the IEEE, 99(6), 928951, 2011.
33. 33. 16 Smart Grids 20. E. Solum, C. Hauser, R. Chakravarthy and D. Bakken, Modular over-the-wire configurable security for long-lived critical infrastructure monitoring systems, in Proceedings of the Third ACM International Conference on Distributed Event-Based Systems (DEBS), Nashville, TN, 2009. 21. K. Birman, Reliable Distributed Systems: Technologies, Web Services, and Applications, Springer, New York, 2005. 22. L. Lamport, Paxos made simple, ACM SIGACT News, 32(4), 1825, 2001. 23. R. Surton, K. Birman, R. Broberg, T. Marian and R. van Renesse, Man-in-the-Middle TCP recovery, Cornell University. Available at: http://goo.gl/9S4Jz (accessed 3 June 2013). 24. W. Jiang, V. Vittal and G. Heydt, A distributed state estimator utilizing synchronized phasor measurements, IEEE Transactions on Power Systems, 22(2), 563571, 2007. 25. M. Zhao and A. Abur, Multi area state estimation using synchronized phasor measure- ments, IEEE Transactions on Power Systems, 20(2), 611617, 2005.
34. 34. 17 2 Power Application Possibilities with Mission-Critical Cloud Computing David Bakken, Pranavamoorthy Balasubramanian, Thoshitha Gamage, Santiago Grijalva, Kory W. Hedman, Yilu Liu, Vaithianathan Venkatasubramanian, and Hao Zho CONTENTS 2.1Overview.......................................................................................................... 18 2.2 Robust Adaptive Topology Control................................................................. 18 References.................................................................................................................19 2.3 Adaptive Real-Time Transient Stability Controls............................................20 Reference..................................................................................................................21 2.4 Prosumer-Based Power Grid............................................................................21 2.4.1Introduction......................................................................................... 21 2.4.2 Prosumer-Based Control Architecture................................................22 2.4.3 Computational Challenges...................................................................23 2.4.4 Cloud Computing in the Future Electric Grid.....................................23 2.4.5 Economic Dispatch of Stochastic Energy Resources..........................23 2.4.6 Cloud-Based Apps...............................................................................24 2.4.7 Scenario Analysis and Transmission Planning...................................24 2.4.8 Model Integration................................................................................24 2.4.9 Exploiting and Abstracting Self-Similarity.........................................25 References.................................................................................................................25 2.5 Wide-Area Frequency Monitoring..................................................................26 2.5.1Introduction.........................................................................................26 2.5.2 FNET Architecture..............................................................................26 2.5.3 FNET Applications..............................................................................27 2.5.4 Cloud Computing and FNET...............................................................27 2.5.5 Rapidly Elastic Data Concentrators.....................................................27 2.5.6 Computational Requirement Flexibility..............................................28 2.5.7 Cloud-Based Applications...................................................................28 References.................................................................................................................29
35. 35. 18 Smart Grids 2.1OVERVIEW As we have seen in Chapter 1, not only is cloud computing coming to the grid, but mission-critical implementations such as GridCloud can provide mission-critical properties. This chapter explores new applications enabled by such technology. While this chapter is only scratching the surface of what is likely to be routine in a decade, we hope that it provides a tantalizing glimpse of what is possible. What, then, is mission-critical cloud computing? To recap, in a nutshell, it Keeps the same fast throughput as generic commercial cloud platforms Does not deliberately trade off this throughput to allow inconsistencies, for example, when a replica does a state update on a copy of the state but this update is forgotten Is much more predictable (and faster) in terms of ramp-up time, central processing unit (CPU) performance per node, and number of nodes Therefore, the question for power application developers is how they can use: Hundreds of processors in steady state. Thousands or tens of thousands of processors when a contingency is reached or is being approached. Note: often there are many minutes of advanced warning of this, sometimes an hour or more. Data from all participants in a grid that is enabled quickly when a crisis is approached (though, for market reasons, not necessarily during steadystate). With this in mind, we now present groundbreaking applications that can exploit such mission-critical cloud platforms. 2.2 ROBUST ADAPTIVE TOPOLOGY CONTROL Balasubramanian and Hedman The electric power transmission system is one of the most complex systems available today. Traditionally, bulk power transmission systems (lines and transformers) are treated as static assets, even though these resources are controllable. However, it is known that transmission topology control has been used in the past and is still being used for corrective-based applications; for example, PJM uses corrective topology control as a special protection scheme (SPS) [1]. These switching actions are primarily taken on an ad hoc basis, determined by the system operators based on past historical data rather than in an automated way based on decision support tools. Past research has demon- strated the ability of topology control to help improve voltage profiles, increase transfer capacity, improve system reliability, and provide cost benefits to the system [28]. Even 2.6 Oscillation Mitigation Strategies.....................................................................29 References.................................................................................................................30 2.7Automatic Network Partitioning for Steady-State Analysis............................30 References................................................................................................................. 31
36. 36. 19Power Application Possibilities with Mission-Critical Cloud Computing though transmission topology control can provide these benefits, harnessing such flex- ibility from the transmission network in existing operational procedures is limited due to the computational challenges of optimizing the transmission topology. More recently, sensitivity-based methods have been proposed as a mechanism to reduce the computational complexity [912]. The robust adaptive topology control method develops a sensitivity-based heuristic, which reduces the computational time of the topology control problem. An expression is derived indicating the impact of changing the state of a transmission line on the objective. This expression is used to generate a line-ranking system with the potential candidate lines for switching based on a direct current (dc) optimal power flow, which builds on the work of [12]. This approach selects a single feasible switching action per iteration, which provides an improvement to the system. The advantage of this method is that it solves linear programs iteratively to come up with a beneficial line-switching solution, which is computationally simple as compared with other methods employing mixed integer programming. All the possible switching solutions are lined up in the ranked list, with the switching action most likely to be beneficial placed at the top of the list. As the list is formed based on a sensitivity study, the switching action is not guaranteed to improve the system. Hence, the switching actions need to be checked for alternating current (ac) feasibility and whether they truly provide an improvement in the objective before they are implemented. This is done by selecting the first action from the ranked list and simulating the switching to find the improvement in the system. If the switch- ing is not beneficial, the next action in the ranked list is checked for improvement. This process is continued until a beneficial switching action is found. While such a procedure is a heuristic, prior work has shown substantial economic savings [9] as well as strong performance in comparison with global optimization techniques [12]. The processing time taken to come up with a beneficial switching action could be significantly reduced if this process were parallelized so that all the proposed switching solutions could be checked at once. This opens up enormous opportunities for the application of cloud computing to transmission-switching applications, which would drastically reduce the computational time and improve the solution quality, as the best solution from the ranking list could be identified very quickly. With prior research demonstrating cost savings of close to 4% for a $500 billion industry [3], there is a great opportunity for advanced decision support tools to fill this tech- nological need, in terms of both algorithm sophistication and advanced computing capabilities, such as cloud computing. REFERENCES 1. PJM, Manual 3: Transmission Operations, Revision: 40, 2012. Available at: http://www. pjm.com/~/media/documents/manuals/m03.ashx. 2.W. Shao and V. Vittal, Corrective switching algorithm for relieving overloads and volt- age violations, IEEE Transactions on Power Systems, 20(4), 18771885, 2005. 3. K. W. Hedman, M. C. Ferris, R. P. ONeill, E. B. Fisher, and S. S. Oren, Co-optimization of generation unit commitment and transmission switching with N-1 reliability, IEEE Transactions on Power Systems, 25(2), 10521063, 2010. 4. K. W. Hedman, R. P. ONeill, E. B. Fisher, and S. S. Oren, Optimal transmission switching with contingency analysis, IEEE Transactions on Power Systems, 24(3), 15771586, 2009.
37. 37. 20 Smart Grids 5. A. Korad and K. W. Hedman, Robust corrective topology control for system reliability, IEEE Transactions on Power Systems, 28(4), 40424051, 2013. 6. K. W. Hedman, R. P. ONeill, E. B. Fisher, and S. S. Oren, Smart flexible just-in-time trans- mission and flowgate bidding, IEEE Transactions on Power Systems, 26(1), 93102, 2011. 7.E. B. Fisher, R. P. ONeill, and M. C. Ferris, Optimal transmission switching, IEEE Transactions on Power Systems, 23(3), 13461355, 2008. 8.K. W. Hedman, S. S. Oren, and R. P. ONeill, A review of transmission switching and network topology optimization, in Proceedings of IEEE Power and Energy Society General Meeting, July 2011, Detroit, MI. 9.P. A. Ruiz, J. M. Foster, A. Rudkevich, and M. C. Caramanis, On fast transmission topology control heuristics, in Proceedings of IEEE Power and Energy Society General Meeting, July 2011, Detroit, MI. 10. J. M. Foster, P. A. Ruiz, A. Rudkevich, and M. C. Caramanis, Economic and corrective applications of tractable transmission topology control, in Proceedings of 49th Annual Allerton Conference on Communication, Control, and Computing, pp. 13021309, September 2011, Monticello, IL. 11.P. A. Ruiz, J. M. Foster, A. Rudkevich, and M. C. Caramanis, Tractable transmission topology control using sensitivity analysis, IEEE Transactions on Power Systems, 27(3), 15501559, 2012. 12.J. D. Fuller, R. Ramasra, and A. Cha, Fast heuristics for transmission line switching, IEEE Transactions on Power Systems, 27(3), 13771386, 2012. 2.3 ADAPTIVE REAL-TIME TRANSIENT STABILITY CONTROLS Venkatasubramanian The power system is expected to undergo major changes in the next decade, resulting from rapid growth in system loads (such as electric cars) and from increased depen- dence on renewable intermittent generation. To face up to these challenges, power utilities are making major upgrades to wide-area monitoring and control technolo- gies, with impetus from major federal investments in the past few years. Power system operation is designed to withstand small- and large-scale distur- bances. However, when the system is subjected to several large disturbances in a short span of time, it may become vulnerable to blackouts. Some recent events, such as the 2012 San Diego blackout and the 2003 Northeastern blackout, point to the need for adaptive real-time transient stability control designs that are specifically designed on an adaptive premise of making control decisions during the evolution of the event. In the present-day power system, wide-area transient stability controls such as reme- dial action schemes (RAS) or SPS are hard-coded control algorithms that are triggered by a central controller in response to the occurrence of specific contingencies based on preset switching logic. When the system is subject to any unknown set of contingen- cies that is not part of the RAS controller logic, the system operation typically switches to a safe mode whereby interarea power transfers are limited to low conservative settings. The tie-line transfers remain at safe low values until the reliability coordinator completes a new set of transient stability simulation studies, which results in significant economic losses due to operation at nonoptimal power transfer levels. Cloud computing emerges as an ideal platform for handling transient stabil- ity mitigation issues, both for the present-day power system and for future control designs. In present-day operation, whenever the system operation is found to be in
38. 38. 21Power Application Possibilities with Mission-Critical Cloud Computing one of the unknown operating conditions, the reliability coordinator can dial in a vast amount of cloud-based processing power to carry out the massive number of new transient stability simulations needed for determining the safe transfer limits. In the future, we need to rethink the design of transient stability controls such as RAS or SPS schemes. The massive computational capability offered by cloud com- puting opens up truly novel futuristic control schemes for mitigating transient stabil- ity events, as proposed in [1]. In the present-day power system, simulation studies are performed off-line for a guesstimated list of potential contingencies, and RAS schemes are implemented for a subset of problematic N2 or higher-order contingen- cies whenever needed. Such RAS schemes, then, only work for a limited number of potential scenarios. Moreover, the respective control actions in these RAS schemes are also designed to be conservative, being based on off-line studies. Zweigle and Venkatasubramanian [1] propose to select and implement transient sta- bility controls based on simulations of the system in real time during the evolution of the events themselves. Wide-area monitoring from an abundance of phasor measurement units (PMU) in the future will pave the way for real-time monitoring of the state and sys- tem topology of the full-size power system. Combining this real-time state information with real-time simulations will allow us to evaluate which control actions are optimally suited to the system at the present time, and the decisions are fully adaptive to what- ever the system conditions are. Since the controller continues to monitor the system in a closed-loop fashion, the proposed control schemes are also robust with respect to simula- tion errors and communication or actuation failures. The formulation is not restrictive to any subset of contingencies, and can handle low-probability events consisting of multiple outages, such as those that have served as precursors to large blackouts in the past. In this proposed formulation, denoted as adaptive real-time transient stability con- trols, massive processing power is needed to carry out what if simulations of many potential control candidates in parallel before deciding on whether any control action is needed and which specific action(s) will be implemented. The system monitoring and simulations of what if scenarios will continue throughout the event until the system has been stabilized. Once the controller recognizes that the system has returned to its normal state, the controller returns to dormant system monitoring mode, and cloud resources can be released. Details of the control algorithms can be seen in [1]. REFERENCE 1.G. Zweigle and V. Venkatasubramanian, Wide-area optimal control of electric power systems with application to transient stability for higher order contingencies, IEEE Transactions on Power Systems, 28(3), 23132320, 2013. 2.4 PROSUMER-BASED POWER GRID Gamage and Grijalva 2.4.1 Introduction The electric power grid, in a bid to improve its sustainability, is aggressively explor- ing ways to integrate distributed renewable energy generation and storage devices
39. 39. 22 Smart Grids at many levels. The most obvious integration is at the level of generation, where renewable generation sources such as large wind turbine and solar panel farms will supplement and eventually (it is hoped) supplant traditional nonrenewable power generation sources. Another natural integration is at the distribution level, where relatively smaller-scale renewable energy generation by utilities and other power dis- tribution entities offers cheaper and greener energy options to customers. While not on the same bulk scale as the generation or distribution level, an emerging trend in recent years is for end consumers who are typically below the distribution level (e.g.,households, microgrids, and energy buildings) to generate their own power using renewable sources and become self-sustainable and energy-independent of the grid. A fascinating aspect of this changing energy landscape is the drastic changes in the roles of the players involved. For example, end consumers, in addition to their typical energy consumption role, are economically motivated to sell excess energy and provide energy storage services to the grid. Modern utilities also go beyond their traditional energy distribution role in buying energy from end consumers when available. Similar role augmentations can be observed at all levels of the modern electric power grid [1]. As a consequence, what traditionally has been a one-way energy transferfrom bulk generation, through transmission and distribution, to end-user consumptionis transfor