A Study on Group Communication in Distributed Wide-AreaMeasurement System Networks in Large Power Systems

Yufeng Xin and Aranya Chakrabortty

Abstract— Future wide-area measurement and control ap-plications in large electric power systems will require a newdecentralized architecture that scales up with the rapidlygrowing deployment of Phasor Measurement Units (PMUs).The emerging cloud computing paradigm that allows dynamiccreation of virtual machines to form virtual data centers wouldhelp better support this new architecture through more efficientand flexible use of the networking and computing resources.However, this paradigm shift poses new technical challenges tothe underneath communication and computing infrastructureleading to new problem formulations and solution approaches.Given that the primary communication pattern in the de-centralized system will consist of various types of real-timegroup communication methods, in this paper we present apreliminary study on two problems, namely communicationgroup formation and routing, that are fundamental to theenvisioned new communication architecture.

Index Terms— Wide-area measurements, phasor data con-centrator, multicast communication, distributed algorithms

I. INTRODUCTION

The wide-area measurement system (WAMS) technologyusing high sampling-rate (6-60 samples/sec) phasor measure-ment units (PMU) has developed tremendously over the pastdecade as the leading measurement technology for large-scale electric power transmission systems. PMUs, which aretypically installed at the high voltage buses of transmissionsubstations, can measure high-resolution measurements ofvoltage, phase angle, frequency and current phasors fromdifferent parts of the power grid, and export these mea-surements to centralized phasor data concentrators (PDC),which may be located remotely at the control centers. ThePDC runs centralized algorithms using these measurementsfor either online situational awareness, voltage stability,phasor state estimation, and visualization, or for offlinedisturbance analysis such modal decomposition. However,with the exponential growth in the number of PMUs thiscentralized communication and computing architecture isenvisioned to become much more distributed over the nextthree years [1], [2]. Emerging cloud computing paradigmsthat allow dynamic creation of virtual machines to formvirtual data centers would be instrumental in supporting thisnew architecture through more efficient and flexible use ofthe networking and computing resources. In this distributedarchitecture a group of PDCs and application servers, located

Y. Xin is with the Renaissance Computing Institute, University ofNorth Carolina Chapel Hill, NC. A. Chakrabortty is with Electrical andComputer Engineering Dept., North Carolina State University, Raleigh, NC.yxin@renci.org, aranya.chakrabortty@ncsu.eduSupport from the NSF ECCS grant number 1054394 is gratefully acknowl-edged.

at different parts of an interconnection, will run a distributedalgorithm cohesively using real-time PMU data from a subsetof PMUs located both in the vicinity of as well as remotely tothe PDCs. Furthermore, with the fast evolving cloud basedIT infrastructure provisioning model becoming a reality inrecent years, it is possible to build on-demand software PDCsand application servers in virtual machines with sufficientcommunication capability in an on-demand fashion to adaptto the high scalability and granularity of the underneathpower grid system [3].

The communication need of the existing and emergingWAMS system architectures can be served by different net-working technologies. To achieve the high inter-operabilityand efficiency, the current census is that the smart gridnetwork will be based on the Internet Protocol (IP), thoughthe underneath communication medium could be optical,copper, wireless, or power line [4]. The dissemination ofphasor data from PMUs can be fulfilled by different transporttechnologies on top of the unified IP layer. A typical patternin PMU networks is group communication, in which a PMUneeds to stream the phasor data to a group of receivers.The traditional point-to-point unicast is not an economicsolution for group communication as multiple redundantcopies exist on the common links shared by the paths toindividual receivers. It also becomes a great burden for thePMUs to maintain multiple connections to all the receivers.The one-to-many or many-to-many multicasting, in whichthe receivers are connected to the source in a tree topology,in contrast, is a much more feasible choice. The IP networkitself provides native unicast and multicast functions in asingle domain or cross multiple domains that can be used asthe foundation for PMU communication [6].

However, adaptation of IP multicasting in current Internethas been extremely slow due to various technical, adminis-trative and commercial reasons. Technically, IP multicastingrequires all the routers in the network to run the multicas-ting protocols, which may cause great traffic managementdifficulties and security concerns. As an alternative solution,overlay multicasting has been gaining significant supportby constructing an overlay virtual topology first, on top ofwhich multicasting trees are created. The application layermulticasting (ALM) conducts group communication onlybetween the end hosts in the system on top of the unicastingIP network [5]. While ALM remedies the key shortcomingof the IP Multicasting through easier and possibly immediatedeployment over the wide-area network, its major drawbackis the limited bandwidth, which always becomes the bottle-neck when the end host needs to transport multiple copies of

543978-1-4799-0248-4/13/$31.00 ©2013 IEEE GlobalSIP 2013

the data in the middle of the multicasting tree. In the serviceoverlay multicasting approach, some multicasting-capableapplication core routers will be deployed to form an overlaynetwork to connect the end hosts in the group, therebyallowing multicasting only among the overlay routers. Ageneral problem with the overlay networking approaches isthat multiple overlay links may actually share some com-mon underneath network links, which may cause resourceutilization and congestion problems.

There have been intensive research on single sourcemulticasting over the past decade. Theoretically, the basicmulticasting tree construction problem is equivalent to thethe well-known NP-Hard Steiner tree problem whose goal isto build the minimal cost tree to connect a group of nodes.There are many variants of the problem under differentconstraints such as the delay, delay variants, and nodal degreeetc. Several multicasting routing protocols have been definedand developed to manage the dynamic group membershipand the update the multicasting trees. Multicasting have beenstudied in the context of many applications, e.g., the multi-medium content distribution system and peer-to-peer system.Only recently multi-source multicasting has attracted someattention. In the multi-source wireless sensor networks, thegoal is to minimize the energy consumption. In [7], forexample, the authors have studied the multi-source IPTVsystem where the goal is to minimize the overall resourceusage.

However, the existing studies are not sufficient for theemerging distributed PMU-PDC network in wide-area powersystems applications due to some of its unique communi-cation properties and requirements such as: (1) The mul-ticasting groups could be very dynamic, including groupsthemselves and the group members, especially for wide-areacontrol where PMUs and their corresponding connectivityto the PDCs may have to be chosen adaptively; (2) a largenumber of PMU sources need to simultaneously send phasordata to a relatively smaller group of designated receivers.It is essentially similar to the multi-sink sensor networks,but does not have the energy consumption concerns and thebroadcast nature; (3) it is very delay sensitive as the receiversin the same group needs to receive the time- stamped phasordata from all the interested PMUs with tight delay anddelay variance constraints. It is similar to medium streamingapplications but the latter normally does not require the delayvariance constraints.

With the aforesaid communication challenges in mind, inthis paper, we attempt to model the distributed PMU-PDCcommunication framework into two related problems: (1)PMU-PDC communication group formation problem, and (2)a dynamic multi-source multi-rate multicasting problem. Wealso present some preliminary algorithmic analysis.

The remainder of the paper is organized as follows. InSection II, the Synchrophasor data dissemination in thedistributed PMU-PDC system is first modeled as a uniquecommunication group formation problem. Section III ad-dresses the general group communication problem, especiallythe group member management problem. The focus is on

multiple source multicasting under real-time requirement.Section IV concludes the paper.

II. COMMUNICATION GROUP FORMATION

Every WAMS application, such as wide-area oscillationmonitoring, voltage monitoring, state estimation, wide-areaprotection, or even wide-area oscillation damping or voltagecontrol, involve the computation of some metrics in real-time. For monitoring purposes these metrics can be, forexample, transient energy functions or loadability factors,while for control applications they can simply be the es-timates of oscillation damping and voltage stability margins.When these applications are distributed over multiple geo-graphically distributed servers, the first problem that needsto be answered is how the underlying communication groupis formed, i.e., how many PDCs are needed and where theyneed to be deployed.

We define the PMU-PDC network as a networkP (U,C,G), where U is the set of PMUs and C is the set ofPDCs deployed in the system. G(V,E) is the communicationnetwork that connects U and C. A PMU ui ∈ U or a PDCci ∈ C connects to the network via a router v(ui) ∈ Vor v(ci) ∈ V by their physical networking proximate.The number of PMUs is typically much larger than thenumber of PDCs. We assume the set of PMUs and theirdistributions to be relatively static as part of the powergrid infrastructure. The set of PDCs and their distributionscould be more dynamic as in responding to state changes ofthe power grid, eg. turning-on and off of new loads suchas PHEVs, intermittent dispatch due to renewable powersources, and deployment of new advanced applications. Aswe have introduced in previous section, the PDCs could becreated in the form of virtual machines in data centers in anon-demand fashion.

Depending on which PMUs may be chosen as sourcesand which PDCs as receivers, there can be two types ofcommunication groups in the system. One is the controlgroup, where one or a group of PDCs need to send controlcommands to all the PMUs that they receive the phasor datafrom. The commands may direct the PMUs to start or pausestreaming data, change the sampling rate or the resolutionof the measurement metrics, etc. The second type is thedata dissemination group, where a PMU needs to stream itsmeasurement data to a group of receivers. For a particularapplication deployed in one or a number of PDCs, the groupbecomes one with a large number of PMU sources that needto stream the phasor data to the receivers in a synchronousfashion in real-time.

A. PDC coverage and PMU assignment problem for thecontrol groups

The solution of this problem includes the PMU assignmentplan, i.e., the assignment of PMUs to each of the PDCsdeploying the application. A particular application can beimplemented in a distributed fashion in multiple PDCs withthe fundamental requirement of receiving phasor data fromPMUs covering all the interesting buses. Due to the physical

544

u1 u2 u3 u4 u5 u6

1

2 3

c1 c2 c3

Fig. 1. PMU Network Example

location proximate, co-existence of multiple applications,redundancy consideration, and system configuration, thegroups of PMUs streaming data to PDCs always overlap.The other way to look at the problem is to think that eachPDC has a stringent requirement on the data disseminationdelays from the PMUs. By taking the networking delays intoconsideration in the graph P (U,C,G), each PMU can finda group of PDCs to safely send its measurement data underthe delay requirement.

To better illustrate the problem, we use Fig. 1 to showan example of a PMU network where three PDCs receivephasor data from the 6 PMUs through the communicationnetwork in the middle. PMUs are divided into three groups,each of which sends to a PDC indexed by the correspondinggroup number. We see that PMU u3 streams data to PDCsc1, c2 and c3, and u4 streams data to the PDCs c2 and c3,simultaneously.

We first consider a ‘static’ PMU-PDC assignment problemwhere the set of PDCs is pre-selected, and each PDC in theset may receive data from a pre-selected subset of PMUs.

Problem 2.1: PDC covering problem: Given a dis-tributed WAMS application that needs phasor data from aspecific set of PMUs, find the minimal number of PDCs thatreceive the phasor data from all the required PMUs.

This problem is a variant of the well-known set coverproblem, and therefore is NP-hard. In the example shownin Fig. 1, the minimal PDC set consists of c1 and c2 inorder to receive data from all the six PMUs. In reality,there will be a number of locations, but not all may beavailable for PDC deployment, e.g., , by the local accessnetwork bandwidth, administrative and security constraints,or business cost consideration. Furthermore, receiving andprocessing phasor data will cost the processing power in thePDC servers, and incur processing delays. If we describethe deployment of PDCs and the processing load in eachPDC with cost functions, we can define a more fundamentalPMU assignment problem with the objective that the PMUdata can be balanced among the PDCs while achieving theoverall cost minimization.

Problem 2.2: PMU assignment problem: Assuming thata PDC can be created in any of the |V | router locations inthe network G with a cost function fi and capacity constraint

si, and a PMU j incurs a communication delay di,j to theith router location, i ∈ [1, |V |], j ∈ U , Find a set of PDC Clocated in V (C) ∈ V and a PMU assignment π : U 7→ C,so that the overall cost

∑i inC fi +

∑j inU α ∗ dj,π(j) is

minimized and∑

π(j)=i dj ≤ si, dj = 1. α is a constant thatcan be used to balance between the fixed location cost andthe overall delay.

We can assign arbitrary large values to the location cost forlocations not allowing PDC deployment. The delay betweena pair of PDC and PMU can be approximated by the shortestpath delay in the network. This problem is NP-complete as itcan be derived from the well-known unsplitable capacitatednetwork facility problem [8].

B. Data dissemination group problem

In order to make meaningful distributed computing forWAMS applications, the power grid system in interest has tobe partitioned in certain ways depending on the bus topology,and generation and load distributions. Phasor data from onlyone or a few PMUs from each area may not be usefulfor a global control action. In other words, in certain partof the grid, there exists a minimum set of PMUs whosemeasurements have to be collected together in one PDCto fulfill a part of the distributed computing. The size ofthese minimal PMU partitions will be different and maysubject to change due to the topology or state changes ofthe power grid. Based on above observation, and assumingeach possible PDC can take data from up to S PMUs, wecan formulate a PMU group packing problem as follows.

Problem 2.3: PMU group packing problem: Supposewe divide the PMUs in U into M non-overlapping groups,Ui, i ∈ [1,M ], and

∑i = U , and S > |Ui|. Find the

minimum number of PDCs that can pack all the PMU groups.It is easy to see that this problem is NP-complete as it can

be deducted from the Bin Packing Problem where the PDCsare the bins. This problem can have multiple flavors whenconsidering different realistic and performance constraints ontop of the basic problem setting. For example, we can havethe delay constraints and differentiated PDC capacities.

In general, the distributed PDC-PMU system presentsseveral interesting but difficult problems that different appli-cations may need to solve. By formulating these problems inthe way that they are equivalent to well-studied combinatorialoptimization problems, operators may be able to compareand pick the most efficient solutions and approximate algo-rithms.

III. REAL-TIME MULTI-SOURCE MULTICASTING

After the decisions on the communication group formationbeing made by solving the PDC location and PMU assign-ment problems presented in Section II, the next question thatneeds to be answered is the follow-up group communicationrouting problem, i.e., how to efficiently transport data fromthe source(s) to destinations. As part of the solution, othermiddle nodes in the network need to be included in thegroup to facilitate the data transport because the sources anddestinations are normally not directly connected.

545

In Section I, we briefly overviewed the existing multicas-ting solutions. Depending on the application and networkproperties, these solutions differ from each other in groupconfiguration, routing protocols, and other characteristicsthat typically lead to trade-offs in these design decisions.Given the wide-area nature of the PMU-PDC network underour study, we think the application-based service overlymulticasting is a very likely choice to avoid the problemsof IP multicasting. Nevertheless, in this section we willfocus on the general multicasting routing problem when theunderneath network topology has been given to be either a IPnetwork or the overlay virtual topology. The data routing isnormally conducted in the form of multicasting tree, in whichthe problem is how the tree is constructed and maintained.We further assume that there exists a control plane forall the participants (PDCs and PMUs) and other nodes inthe network to exchange commands for group membershipmanagement (join/leave).

Following what we discussed in previous section, we areparticularly interested in two types of communication groups:the control group with the PDCs as the sources and the datadissemination groups with the PMUs as the sources.

A. PDC source multicasting

A typical scenario for a particular distributed applicationis that control commands need to be sent to all the PDCs inthe group from time to time. Each PDC in the group willbe the source of a multicasting tree to send the commandmessage to those other PDCs, and perhaps also to the PMUsassigned to it. As a result, there will be |C| separate trees ifthere are |C| PDCs. Regardless of how these trees are built,it is a waste of resource to maintain multiple trees as theyare sending the same command message, and they will haveseveral overlapping links.

Problem 3.1: Multiple-source multicasting forest prob-lem: Develop a real-time algorithm such that a groupof PMU sources simultaneously send the same measure-ments/messages to a group of PDC receivers.

To solve this problem, we can add a pseudo super sourcenode connecting to all the PDCs via equal weight links in thenetwork. Then this problem becomes a regular Steiner treeproblem, and therefore all the variants of Steiner tree modelssuch as with delay, delay variant, or degree constraints canbe adapted.

Problem 3.2: Multiple multicasting tree packing prob-lem: When there are multiple different communicationgroups, construct the network trees in a load balanced wayso that bottlenecks in the underneath network links can beavoided as much as possible.This is a hard problem that may be solved using a heuristicapproach. Lower bounds on the optimal solution for packingmultiple multicasts that minimizes the network congestioncan also be found.

B. PMU source multicasting

Compared to PDC source multicasting, PMU data dissem-ination multicasting is much more complex as there may

be thousands of PMU sources. In addition to the basic treecreation and packing problem, the stringent real-time require-ment on the data arrival in PDCs need to be respected. Thedelay variation requirement is for synchronization amongthe PDCs running the same distributed application to avoidextra queueing and computing delays. A big difference ofthe PMU source multicasting is that the data from eachPMU is different, and needs to be combined in the PDCs.Therefore, the simple solution of adding a pseudo supersource connecting to all the sources in order to build a singleSteiner tree would not work.

Problem 3.3: Multiple source multicasting problem un-der delay and delay variation constraints: For a particularcommunication group, develop a strategy by which the datafrom all the PMUs need to arrive in the PDCs within a certaindelay and/or delay variation threshold.

Problem 3.4: Multiple source multicasting tree-packing problem under delay and delay variationconstraints: Given a set of communication groups, buildthe multicasting trees in a load balanced way so thatbottleneck links can be avoided.

IV. CONCLUSION

In this paper, we presented several challenging communi-cation problems that need to be solved for next-generationdistributed WAMS. We have focused on two fundamentalproblem sets: communication group formation, and multi-source group communication routing problem. We formu-lated a number of optimization problems that are unique tothe decentralized architecture under different system settingsand constraints. Many of these problems are NP-complete,and deserves further investigation to find the most adequateand efficient approximation solutions. Related problems ongroup membership management, fault tolerance, and securemulti-source multicasting are also important directions forfuture studies on this subject.

REFERENCES

[1] A. Chakrabortty, G. Michailidis, and Y. Xin, “A Decentralized IDAlgorithm for Detecting Slow-fast Oscillations in Power Systems fromOverwhelming Volumes of Phasor Data,” in IEEE CDC, 2012.

[2] A. Chakrabortty, “Handling the Data Explosion in Tomorrow’S PowerSystems,” IEEE Smart Grid Newsletter, Sep. 2011.

[3] Y. Xin, I. Baldine, J. Chase, T. Beyene, B. Parkhurst, and A.Chakrabortty, “Virtual smart grid architecture and control framework,”in IEEE Smartgridcomm, Oct. 2011.

[4] Challenges for the Smart Grid, Technical Report, NIST Workshop ontechnology, measurement, and standards, Special Publication 1108R2,March 2013.

[5] M. Hosseini, D. T. Ahmed, S. Shirmohammadi, and N. D. Georganas,“A Survey of Application-Layer Multicast Protocols,” IEEE Communi-cation Surveys and Tutorials, vol. 9, no. 3, 2007.

[6] X. Tang, J. Pu, Y. Gao, Z. Xiong, and Y. Weng, “Energy-EfficientMulticast Routing Scheme for Wireless Sensor Networks,” —it IEEETransactions on Emerging Telecommunication Technologies, 2013.

[7] S. R. D. Bailey, Y.-R. Chen and S. Karabuk, “Multi-Source IPTV net-works: Zap time and bandwidth optimization,” International Conferenceon Computing, Networking and Communications, 2013.

[8] M. Bateni and M. Hajiaghayi, “Assignment problem in content dis-tribution networks: unsplittable hard-capacitated facility location,” pro-ceedings of ACM-SIAM Symposium on Discrete Algorithms, New York,2009.

546