Next Generation Automation Final

  • Published on

  • View

  • Download


This paper describes a service-oriented architecture for data acquisition and control leveraging some of the latest concepts in software engineering. By addressing the business requirements to support role-based applicationsand services, the platform removes the islands of automation issue and allows a vertical decomposition of the application stack to lower the barrier to entry for new applications


1. Next Generation Automation Effective Platform Design and PracticalImplementation Daniel Evans, Sam Hendley, Adam Crain, John S. CamilleriNCSU Centennial Park Campus Venture IV Building 1730 Varsity Drive, Suite 500Raleigh, NC,,, jcamilleri@greenenergycorp.comKeywords: Automation, SOA, ESB, Messaging 1. KEY CHALLENGES The Electric Utility Industry is under an ever increasingAbstract pressure to support the demands of the Smart Grid. TheThere are a number of challenges in meeting the definedchallenge is complex due to the variety of systems andNIST architecture while driving functions that exist today and those that are beingApplication modularity, integration with existing enterprise considered the smarter models, leveraging existing enterprise architectureand supporting increasing volumes of diverse field dataSome of these challenges are identified as the following:(including meter data, phasor measurements, traditionaltelemetry for operations, power quality and new remote Bridging the operational technologies (OT) andsensing devices) are among some of the key challenges information technologies (IT) to support increasingtoday.interoperability needs Avoid architecture brittlenessThis paper describes a service-oriented architecture for dataRemove the island of automationacquisition and control leveraging some of the latestExponential growth in remote sensing dataconcepts in software engineering. By addressing the Manage the effects of a diminishing workforcebusiness requirements to support role-based applications Harmonization of enterprise data modelsand services, the platform removes the islands ofautomation issue and allows a vertical decomposition of Leveraging/Enhancing the legacy IT investmentthe application stack to lower the barrier to entry for newapplications. The platform supports horizontal scalability for The Guidelines for Smart Grid Cyber Security - NIST IRperformance and is distributed, by design, for reliability 7628 [1] help identify the overall number of systems,and high availability. In addition, this paper addresses the interoperability and guidelines for cyber securitydetailed design issues around support, performance,requirements. The main thrust of this paper is to describe anreliability, security and scalability. open platform approach that addresses the challenges identified above and leverages the NIST IR as a standardFinally, we will showcase a practical implementation of thereference model.basic concepts of data acquisition and control whileleveraging an industry standard enterprise data model to 2. DESIGN PRICINPLESsupport interoperability between heterogeneous systems.In order to address the key challenges, the platform supportsThis platform design shows how todays well definedthe following design principles. With these principles instandards coupled with emerging standards can be leveraged place, barriers to innovation are a modular approach to minimize the customers risk andreduce overall development support for vendors.Grid-Interop Forum 2011 2. Evans, Hendley, Crain, Camilleri2.1. Open Platform abandon their use. The principle use of open source withThe concept of openness is typically focused on a specific this platform is to allow community involvement across theprotocol, standard or programming interface rather than acore platform components and support liberal licensingholistic view of a system. The word open has become apolicies for independent software vendors to use themarketing buzzword with no real meaning. Therefore itplatform.requires a clear understanding of the platform lifecycle toevaluate whether open in the context of the specific 2.3. Cross Platformsystem or platform really is a feature or red herring. In order to support a diverse set of applications, platforms and human resource skill sets the platform should leverageThe main objective of an open platform is to foster adoption architectures and technologies that allow multipleby those entities that can support and employ the platform. languages, operating systems, and client architectures. ThisAdoption is encouraged by common standards that a approach enables another opportunity for innovation bycommunity agrees to use. There are numerous standards increasing the tool sets and approaches of domain experts.that are currently being reviewed by the Smart GridInteroperability Panel. Many of these standards, like IEC61850, DNP3, C22.12,are specific to the electric utility.Other standard bodies like IETF, OASIS, and W3C support2.4. Service Oriented Architecturea vast array of technologies in the software community. TheA service oriented architecture (SOA) provides for modularprinciple of this platform is to leverage informationgrowth capabilities and is especially designed fortechnology standards across the software and industrycontinuous integration support [4]. Figure 1 shows thedomains. publish/subscribe model supported in this platform. The SOA supports modularity and endpoint abstraction. The message broker supports pub/sub and event driven systems. Adherence to this type of architecture promotes composition and reusability. Architecture brittleness is reduced and nicely supports a dynamic business and technical environment such as the evolution to a smarter grid. This principle supports innovation, serviceability, continuous integration. 3. DOMAIN TRIAD DESCRIPTION In order to support continuous integration of business objectives related to managing the electric power system, providing value added solutions to the end customers, and managing the rate of change on business requirements we first haveto identify the domaintriad.Figure 1: Publish/Subscribe Model2.2. Open SourceThe use of open source in commercial software is growing.In many cases, the end customer does not even realize thatsome of the code it is running is based on open source.Many companies leverage stable open source projects tosupport their product. Running stable projects [2] dependson a complex set of community interactions and solidsoftware engineering lifecycle principles. Many of theseprojects rival some of most complex propriety code bases inthe world due to the large number of developers and level ofcomplexity. One key use of open source in this platform isto leverage hundreds of thousands of lines of code fromstable community projects addressing common issues.Licensing options add to the complexity of open source.Figure 2: Domain TriadThe open source initiative [3] maintains a list of approvedlicenses. Some licenses are incompatible with each otherand require users to package their solution differently or For simplicity, we can identify three main domains of systems, business processes and technologies for a Utility.Grid-Interop Forum 2011 3. Evans, Hendley, Crain, Camilleri Back Office systems are dedicated to running the business messages/sec on an 8-core box using Gigabit Ethernet [8].of the utility such as accounting, billing, and account There are a number of other industries adopting AMQP asmanagement. Front Office systems are customer facing the transport protocol for the same reasons required by thesuch as support and new services. Grid Operations refers toUtility industry.the management of the power system. The conventional Brokers and clients communicate with each other by thedomain model from SGIP contains seven domains, but for AMQP specification. Because the specification is open,simplicity the three mentioned above are sufficient for multiple vendor implementations can be deployed to workdiscussions in this paper. Figure 2 shows a simple depiction together which gives the end customer true interoperabilityof thedomaintriad. and the ability to choose best-in-class client and broker solutions avoiding vendor lock-in.To build out a platform architecture that is open, leverages The function of the broker is to receive messages from thebest practices and solutions from other industries, and clients and create routes, queues and generally manage allsupports a robust ecosystem of solution providers requires incoming/outgoing messages of the system. Clients arebasic architecture building blocks to interface across all services and/or applications that use them.three domains. In this paper we will focus primarily on thebuilding blocks for the Grid Operations Domain. 4.2. Configuration Brokers can be configured in clusters for High Availability4. ARCHITECTURE BUILDING BLOCKS Messaging. Clients, also referred to as consumers andIn many cases the IT Enterprise solves integration needs for producers, are really an extension of a simple client serverbusiness systems using Enterprise Integration Patterns [4]. model. When a client publishes messages to the broker, theThe common term for this type of integration architecture is broker, for instance, might configure an exchange withcalled an Enterprise Service Bus (ESB) [5]. An ESB topics for subscription by other clients. For this platform aleverages service oriented architecture (SOA) [6] for simple example is the collection of measurements from animplementing the interaction and communication of services RTU in a substation where the client subscribes to alland applications. The SOA design also supports distributed measurement in that particular substation.computing and promotes an asynchronous message orienteddesign.The information technology fabric used to bridge thedomain triad is called a message oriented middleware. Theheart of the fabric is the Advanced Messaging QueuingProtocol (AMQP) which is an open standard applicationlayer protocol. The deployment models of the brokers and clients are driven4.1. AMQPby the requirements of the architecture. Clustering,AMQP is the heart of the message-oriented middleware federation, message persistence (durability), and otherplatform that bridges the domain triad. AMQP providesimplementation can be tuned to support the deploymentqueuing, routing, message orientation, reliability and The specification [7] calls out Ubiquity, Safety,Fidelity, Applicability, Interoperability, and Manageability The configuration supports how the messages moveas key requirement areas. The open standard originated inthroughout the system, their availability, and reliability. Thethe financial industry. Their objective was to address a payload of the messaging system for this type of systemcritical business need of providing high speed reliablerequires a terse payload envelope to maintain highmessaging with an open standard to avoid vendor-lock in. performance. The payload schema in this platformThis work started in the mid 2000s and AMQP 1.0 is just leverages protocol buffers developed by being ratified underOASIS. 4.3. Protocol BuffersThe requirements of AMQP from the financial industry are Google protocol buffers are licensed under an Apache 2.0applicable to the utility industry as well. The two majorlicense and have C++, .Net, Java and Pythonrequirement areas for supporting ubiquitous messagingimplementations among others. Protocol buffers use awithin the Grid Operations domain are performance andschema that is compiled to multiple language bindings,security.AMQP addresses these requirements in an providing a simple, yet effective, interface. The lightweightabstract fashion. In a cited performance test, a commercialschema format loosely resembles structs in C.implementation of AMQP achieved over 6 millionGrid-Interop Forum 2011 4. Evans, Hendley, Crain, CamilleriThe following example shows how measurement quality is GET, PUT, POST, DELETE) which decouplesencoded as a protocol buffer. In this case, the IEC 61850client server development.definition is used. The reef project uses protocol buffers to Opaque transport - Clients are unaware of whetherdefine service resources and the REST framework. they are directly connected to the end service or a proxy. Proxies can be used for things like load// mirror the iec61850 quality (CIM uses these too)balancing or to enforce security.message Quality { Stateless - Every request contains all of the enum Validity {// No abnormal condition of the acquisition function orinformation necessary to satisfy the request. Any// the information source is detectedserver-side state must be addressable in some way.GOOD = 0;// Abnormal condition ""5. CORE SERVICESINVALID = 1;Within this platform there are a number of services. Wewill focus a set of core services that support the automation// supervision function detects abnormal behavior,functions of the platform. Figure 3 shows the basic// however value could still be valid.// Up to client how to interpret. services of Reef.QUESTIONABLE = 2;}enum Source {// value is provided by an input function from// the process I/O or calculated by applicationPROCESS = 0;// value is provided by input on an operator// or by an automatic sourceSUBSTITUTED = 1;}optional Validity validity = 1 [default = GOOD];optional Source source = 2 [default = PROCESS]; Figure 3: Basic Services of Reefoptional DetailQual detail_qual = 3;// classifies a value as a test value, not to be used for 5.1. Measurementsoperational purposesMeasurements are the basic units of data published by dataoptional bool test = 4 [default = false]; sources which are processed and monitored by the system.Often, measurements are acquired using communications// further update of the value has been blocked by an// operator. if set, DetailQual::oldData should be setprotocols, and are used to represent the state of remote fieldoptional bool operator_blocked = 5 [default = false]; devices. Measurements may also be generated or manually} entered by agents. The basic fields provided by themeasurement service are Value, Quality, Time, and Unit.Measurements enter the system either from communicationfront-ends or directly using the Reef Project APIs. Themeasurement stream is then processed -- configured4.4. REST transformations are applied (scaling, value mapping) andRepresentational State Transfer (REST) is an architecturalside-effects are triggered (event/alarm generation) . Finally,style for distributed systems. REST is commonly thought ofmeasurements are stored in the database and published toas a "nouns and verbs" alternative to Remote Procedure Call the bus. The system also maintains measurement history,(RPC), but it is actually a proper subset of RPC that hasthe chronological stream of previous measurement values .some interesting properties. Factoring a distributed systemin a RESTful style has the following advantages (which can5.2. Commandsalso be thought of as constraints) as applicable to theCommands are the indications agents use to interact withdiscussed platform:and modify the state of the system. Commonly, commandsare tunneled by the communications processors to remote Uniform interface - Service consumers/providers field devices in order to exercise control over theircommunicate through a uniform interface (i.e. functions. Whereas measurements constitute the flow ofGrid-Interop Forum 2011 5. Evans, Hendley, Crain, Camilleridata from the field, commands form the information moving may contain a further description, and contain any relevantoutwards. data attributes.Commands may or may not contain an associated value. In Ultimately, the definition of events, as well as thethe field communications world, commands with values areconditions under which they are triggered, is highlyusually referred to as setpoints, and may refer to a target configurable. Events, as a whole, are a tool system designersvalue the system or end device is intended to reach. Values use to characterize system behavior and to provide clientsare therefore used when a simple imperative cannot convey and administrators necessary information.the proper message, such as "set temperature to 65 degreesCelsius. Many or most commands issued by clients will qualify asevents. Other events may not be tied to operational data, butWhen multiple agents/clients will be using the system instead will record system activity such as application errorsconcurrently, simultaneously making modifications to theand agent authentication actions.same subsystems can lead to undesired results andindeterminate behavior. Furthermore, it is frequently thecase that some field devices need to be declared "off limits"for safety or maintenance reasons. The following objects areused to regulate access to commands:Selects are acquired by clients to grant exclusive access to a command or set of commands.Blocks are used to prevent any client from accessing a command or set of commands.The architecture allows for a unique opportunity to supportmultiple clients in the grid operations domain. A real worldexample would be clients like load shedding, remedialaction schemes, and switch order management applications.In this case the architecture allows for each client to be fromdifferent vendors since they are operating on the sameequipment and communication models of the platform.They can subscribe to the measurement stream needed andperform automatic /manual operations without concern ofindeterminate behaviors.Another capability of this architecture is a term calledscatter-gather. This is an Enterprise Integration Pattern thatFigure 4: Alarm State Diagramallows a client to broadcast many requests and receive anaggregate message when all the recipients have responded. 5.4. AlarmsA practical application is broadcasting a select command to Alarms are a refinement of events which identify systemmultiple switching devices and then waiting for the responseoccurrences that require operator intervention. Alarmfrom all devices. The client does not need to manage this objects are tied closely to event objects. All alarms arebut instead allows the platform to perform the work.associated with events, but not all events cause alarms.In contrast to events, alarms have persistent state. The three5.3. Events principal alarm states are unacknowledged, acknowledged,Events are objects that record a meaningful occurrence or and removed. The transitions between these states constitutechange of state in the system. They are used both to monitorthe alarm lifecycle, and manipulation of the states involvesthe system in real-time and to provide an audit log of system workflow. Transitions in alarm state may themselves, as they are part of the record of client operations.Events are configured with the parameters type and severity,and contain context information such as the originatingsubsystem and associated agent. The "message" of the eventGrid-Interop Forum 2011 6. Evans, Hendley, Crain, Camillerioperation personnel can access real time data for analysis6. PROJECT SERVICE LISTand planning.Table I: Reef Project Service List8. MODELINGFrom an automation context, modeling refers to the act ofNamerepresenting the power system data acquisition Modeling of grid operations systems has always been acomplex design issue due to user experience (UX), federation, and integration of other sources of recordAllScadaService.javasystems such as a Geographical Information System / AssetManagement System. The modeling system uses a directed graph to support relationships like own, uses, source and many others. This approach is not specific tothe electric utility. With this approach the platform support modeling for water, gas, building, manufacturing,electric automation system individually or combined. ForCommunicationChannelService.javainstance, a Municipal Utility may have operational supportConfigFileService.javafor both the electric and water systems for their customerbase. Modeling for both systems can reside in the sameEndpointManagementService.javasystem while still supporting different operational businessEntityService.javaunits and processes. Another more interesting case is theintegrated utility services of a microgrid. The allows simpler integration of applications that multiple and interdependent 9. DEPLOYMENT CONTAINERAnother feature of the platform is the deployment containerMeasurementOverrideService.javabased on a specification of the Open Services Initiative (OSGi). The components of the Reef project aredeployed as bundles within this container. Figure 5 the Reef Karaf shell.Table I list the services in the Reef Project at These services provide the automationplumbing and core services.7. SERVICE EVENTS & SUBSCRIPTIONSIn order to create a highly scalable system, the platform hasto be event driven. Clients can subscribe to various eventsfrom services based on what information they require. Subscribing reduces the wasted conversation time and CPUoverhead of a reply/request method. The method used istypically referredto as pub/sub.Figure 5: Reef Karaf shellAnother aspect of this approach is that multiple clients cansubscribe to the same service topics (routing keys). ForThe Karaf provides easy interface to load services, accessexample, a client could be any number of thick or thinresources and overall management of the platform. Thisapplications all subscribing to the same information. technology supports the following key features:This is an extremely powerful tool in the grid operationCross platform support (Windows and Linux)domain to share automation data between many clientsHot code swap supportwithin and across domains. With appropriate authenticationCommand Line Interface (CLI) to serviceand authorization access to the automation data, non interfacesGrid-Interop Forum 2011 7. Evans, Hendley, Crain, Camilleri Protocol Apache 2.0Google Protocol BuffersThe ability to load and unload bundles while the system is Buffersrunning is a very powerful feature. A practical applicationof this feature is the ability to patch a protocol bundle likeDNP3 without shutting down the system. Since the system13. IMPLEMENTATIONcan be distributed to multiple nodes, the operational impact The following are various implementations of the reefof such an act can be limited. project.Another feature that supports developers is the ability to 13.1. FREEDM Lab - MicroGrid Implementationexpose service interface through the karaf shell. Client The NSCU FREEDM Lab ( )application can leverage the same interfaces; therefore theuses the Reef platform to preform data acquisition to variousdeveloper can explore the interfaces through the CLI.components in the microgrid. The figure below represents a conceptual model of the microgrid. DNP3 is used to10. PERFORMANCEcommunicate to the field devices. This is an NSF fundedPerformance of the platform can be explained in terms IO,site that supports research in solid state devices to controlCPU, and network bound. Persistence of time series data is the flow of power. The platform allows new applicationsthe typical performance bottleneck. This type of IOdevelopment and leverages a real time HMI to manage theconstraint has been addressed by leveraging newnetwork. The configuration of the platform is a single nodetechnologies like Cassandra, which is a highly scalableablerunning on Ubuntu 10.4 LTS. The model is less than 100NOSQL solution.measurements, but supports a number of control options and can be configured to support the new solid state devices.CPU and Network performance bottlenecks are addressedthrough horizontal scaling of commodity hardware. Theservice oriented architecture allows the design of this typeof linear expansion.11. SECURITYFor message oriented middleware, a fundamental securityrequirement is to ensure safe messaging. The AMQPspecification calls out Simple Authentication and SecurityLayer (SASL).This method provides authentication support for the clients. Figure 6: FREEDM MicrogridThere are optional authentication methods and capabilitiesto negotiate protection on other protocol interactions. 13.2. EV Charging StationThe underpinnings Role Based access is support for The figure below shows the automation platform Reef withmultiple users with course permission sets to servicea special type of protocol connected to the frontendresources and/or the entity model. Granular permission willprocessor (FEP) of Reef. The protocol bundle provides thebe supported in future releases. mapping and the specific charging station protocol. A state machine is used as a poller. The Spring Web Service12. OPEN SOURCE MATRIX provides the interface to the charging station web serviceAll of the software components discussed in this paper are headend. Within the karaf shell the protocol and spring arelicensed under various open source licenses. The following bundled to provide a deployable package.Thetable is a partial list of the main components.Configuration file is a general XML modeling file for the system. It provides the modeling parameters for the FEP Table II: Platform Open License Listand protocol bundle. This implementation provides demandComponentLicense Description information and is processed by the platform so applicationsReef CoreAGPL on the platform can monitor and control the chargingReef API Apache 2.0 station. Technologies like SOAP, WS-Security and HTTPSDNP3 Apache 2.0 are used to leverage a standard reusable approach and aCassandraApache 2.0 authenticated connection.PostgreqlGPL www.postgresql.orgqpid Apache 2.0qpid.apache.orgSpring www.springsource.orgGrid-Interop Forum 2011 8. Evans, Hendley, Crain, CamilleriFigure 7: Charging Station FEP13.3. Grid Operation / Back Office IntegrationAnother integration problem is providing back officeintegration with an outage management system (OMS) andthe Reef platform. In this configuration the web servicestack can use approaches like a JMS connector tocommunicate with the platform and subscribe to breakerstatus changes to support the OMS fault location algorithms.The data model used is the point-to-point standard isMultiSpeak. The platform provides a number ofadvantages over a straight application to applicationconnection.First, the integration of the MultiSpeak client does notimpact the operation of the Reef platform. Therefore, theclient can be modified and / or patched without any directimpact on the platforms ability to communicate with fielddevices. Second, the integration cost of standing up anotherversion of the same interface is minimized to just days ofintegration and testing. Finally, configuration issues arefocused on the client and not the platform.14. CONCLUSIONThe discussed message oriented middleware automationplatform leverages a next generation messaging protocoland diverse IT technologies to support ubiquitousmessaging across multiple domains. The platform is basedon open source components to drive innovation and providea path for industry adoption. The purpose built services andmessaging infrastructure also provide a reliable andpreformat platform to enable utilities to build out a non-brittle IT infrastructure for the realization of a smarter grid.Grid-Interop Forum 2011 9. Evans, Hendley, Crain, Camilleria number of open source projects necessary to use testing15. REFERENCESand coverage tools on the RIM Blackberry device. SinceMid 2006 Mr. Hendley has been a software developer at[1] NISTIR-7628 Vol 1.- GEC. Mr.Hendley has worked on many projects including Advanced Applications built on top of a large utility7628_vol1.pdf EMS, a custom control system for a sun tracking solarcollector, a web-based HMI system for a renewable energy [2] Chris DiBona, Danese Cooper, Mark Stone, Openplant and a number of communication protocol stacks.Sources 2.0: The Continuing Evolution, OReilly, ISBN: 0-596-00802-3, 2006.Mr. Crain leads the GreenBus platform development andopen source efforts. His experience in the power industry[3] OSI - SCADA systems, substation automation, control[4] Gregor Hohpe, Bobby Woolf, Enterprise Integration centers, field devices, protocol stacks, and concentratingPatterns, Designing, Building and Deploying Messaging photovoltatic systems. He is the principal author of GreenSolutions, Addison-Wesley, 2004.Energy Corps first open source release, DNP3, a highperformance implementation of Distributed Network[5] David A. Chappell, Theory in Practice EnterpriseProtocol. Mr. Crain was the principal software engineerService Bus, OReilly Media, 2004.behind Skynet, a control system for arrays of robotic[6] Thomas Erl, Service-Oriented Architecture Concepts, telescopes. Skynet ( is now anTechnology, and Design, Printice Hall, ISBN-10 0-13-integrated part of the high school and college curriculum in185858-0, 2005. North Carolina. Mr. Crain holds a BS in Physics andComputer Science from the University of North Carolina at[7] AMQP Specification V1.0, Revision:1350, 07 Oct 2011.Chapel Hill. Camilleri is the EVP of Product Development and[8] Chief Product Owner of GreenBus at Green Energy Corp.He has worked as a distribution and transmission engineerfor PECO Energy. During his time with ALSTOM-GRID16. BIOGRAPHY Mr. Camilleri integrated SCADA/EMS/DMS systems tonumerous utilities. Later he served in the Operations andR&D divisions providing leadership to the engineering andMr. Evans is a Software Engineer working on Green product development organizations as well as workingEnergy Corps Green Bus Platform and open sourceclosely with utility customers. For Microsoft, Mr. Camilleriinitiatives. Mr. Evans holds a Bachelor of Science in managed developments of security features and operatingComputer Science from the University of North Carolina- system security architecture for several windows mobileChapel Hill. Since 2006, Mr. Evans has worked on a varietyproduct lines. Mr. Camilleri holds both a BS and MS inof efforts as part of Green Energys work in power systemsElectrical Engineering from Tennessee Technologicalautomation. His domain experience includes software University and is a Senior Member of for substation integration, the development ofsmart grid load reduction automation for a major utility,and the development and open-source release of theindustry-standard DNP3 protocol. He has been responsiblefor quickly prototyping proof of concepts for thin-clientSCADA interfaces, rich internet application-based operatorHMIs, and distributed field device communications usinglow-power digital radios. Mr. Evans is currently developingcore applications and services as part of the Green BusPlatform team.Mr. Hendley is a development lead within the ProductionDevelopment group of Green Energy Corp. He holds aBachelor of Science in Physics from the University of NorthCarolina-Chapel Hill. Mr. Hendley worked on a number ofgames and products and built an extensive framework forporting games to a range of phone hardware increasing theteam productivity. Mr. Hendley also made contributions toGrid-Interop Forum 2011


View more >