An Autonomic Service Delivery Platformfor Service-Oriented Network

Environments

by

Robert David Callaway

Department of Electrical and Computer Engineering

College Of Engineering

North Carolina State University

PhD Preliminary Examination

November 2, 2007

Advisory Committee:

Dr. Michael Devetsikiotis, Chair

Dr. Yannis Viniotis, Co-Chair

Dr. Adolfo Rodriguez, Member

Dr. Andrew Rindos, Member

Dr. Mihail Sichitiu, Member

Dr. Sharon Setzer, Graduate School Representative

ABSTRACT

CALLAWAY, ROBERT DAVID. An Autonomic Service Delivery Platform for Service-

Oriented Network Environments. (Under the direction of Professors Michael Devetsikiotis

and Yannis Viniotis).

Service-oriented architectures (SOA) offer a more effective and flexible approach

to integrating technology with business processes than traditional IT architectures. SOA is

the foundation for both next-generation telecommunications and middleware architectures,

which are rapidly converging together on top of commodity transport services. Services such

as triple/quadruple play, multimedia messaging and presence are enabled by the emerging

service-oriented IP Multimedia Subsystem, and allow telecommunication service providers

to maintain, if not improve, their position in the marketplace. SOA is aggressively leveraged

in next-generation middleware systems as the system model of choice to interconnect service

consumers and providers within and between enterprises.

We leverage previous research in active, overlay, and peer-to-peer networking tech-

nologies, along with recent advances in XML and Web Services, to create the paradigm of

”service-oriented networking” (SON). SON is an emerging architecture that enables net-

work devices to operate at the application layer to provide functions such as service-based

routing, content transformation, and protocol integration to consumers and providers. By

adding application-awareness into the network fabric, SON can act as a next-generation fed-

erated enterprise service bus which provides vast gains in overall performance and efficiency

and enables the integration of heterogeneous environments.

The contributions of this research are threefold: first, we formalize SON as an

architecture and discuss the challenges in building SON devices. Second, we discuss issues

in interconnecting SON devices together to create large-scale service-oriented middleware

and telecommunications systems. In particular, we discuss the concept of federations of en-

terprise service buses, and present two protocols that enable a distributed service registry to

support federation. Finally, we propose an autonomic service delivery platform for service-

oriented network environments. The platform enables a self-optimizing infrastructure that

balances the goals of maximizing the business value derived from processing service requests

and the optimal utilization of IT resources.

Acknowledgments

The author would like to acknowledge financial support from the WebSphere Technology

Institute and RTP Center for Advanced Studies of IBM Software Group, as well as an IBM

PhD Fellowship for 2007-2008.

iii

Table of Contents

List of Figures v

1 Introduction & Motivation 11.1 Telecommunications Services . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Integration Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Service-Oriented Architectures . . . . . . . . . . . . . . . . . . . . . . . . . 21.4 Service-Orientation in Next-Generation Networks . . . . . . . . . . . . . . . 31.5 Contributions of Our Research . . . . . . . . . . . . . . . . . . . . . . . . . 31.6 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Service-Oriented Networking 62.1 Application-Aware Networks . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.1.1 Active Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.1.2 Overlay Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 Service-Oriented Networking . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2.1 Benefits of Service-Oriented Networking . . . . . . . . . . . . . . . . 82.2.2 Service-Oriented Networking Functions . . . . . . . . . . . . . . . . 9

2.3 Research Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.3.1 Implementation Considerations . . . . . . . . . . . . . . . . . . . . . 132.3.2 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.3.3 Security and Management . . . . . . . . . . . . . . . . . . . . . . . . 16

3 Large-Scale Service-Oriented Systems 173.1 Introduction & Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183.2 Current Approaches to ESB Federation . . . . . . . . . . . . . . . . . . . . 19

3.2.1 Manual Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 193.2.2 Broker ESB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.2.3 Centralized Registry . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.3 Federation Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.3.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.4 Building An Autonomous Federation . . . . . . . . . . . . . . . . . . . . . . 253.4.1 Service Request Forwarding . . . . . . . . . . . . . . . . . . . . . . . 29

iv

3.5 Interconnecting Autonomous Federations . . . . . . . . . . . . . . . . . . . 303.5.1 Service Request Forwarding . . . . . . . . . . . . . . . . . . . . . . . 36

3.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4 An Autonomic Service Delivery Platform 384.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384.2 Architecture of Service Delivery Platform . . . . . . . . . . . . . . . . . . . 40

4.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404.2.2 Key Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.2.3 Methodologies Integrated in the Platform . . . . . . . . . . . . . . . 424.2.4 Related Work in Service Systems . . . . . . . . . . . . . . . . . . . . 46

4.3 Analytic Framework of Service Delivery Platform . . . . . . . . . . . . . . . 474.3.1 Solution of Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

4.4 Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514.4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

4.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

5 Conclusions 585.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585.2 Remaining Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

5.2.1 Relaxing Assumptions on Concavity of Utility & Cost Functions . . 595.2.2 Multipath XML-Based Service Routing Protocols . . . . . . . . . . . 595.2.3 Minimizing Optimization Computations using Wavelet-Based Traffic

Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605.2.4 Measurement of Effective Capacity of Resources . . . . . . . . . . . 615.2.5 Investigate Alternative Solution Methods . . . . . . . . . . . . . . . 62

Bibliography 63

v

List of Figures

2.1 Example of Functional Offloading: Decryption and Authentication via WS-Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.2 Example of Service Integration: Document Transformation via XML SchemaValidation and XSLT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.3 Example of Intelligent Routing: Routing Requests to Logical Partitions usingXPath Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.4 Comparison of Software and Appliance Approaches . . . . . . . . . . . . . . 132.5 Example of Adaptive Admission Control: SEDA Response Time Controller 15

3.1 Example Topology of Multiple ESB Deployments - Hub & Spokes . . . . . 213.2 Example Topology of Multiple ESB Deployments - Peer Business Divisions 213.3 Example Topology of Interconnected Autonomous Federations . . . . . . . 223.4 Message Exchange Between Two ESBs Within a Federation . . . . . . . . . 263.5 Example of Contents of Hello XML Message . . . . . . . . . . . . . . . . . . 273.6 Example of Contents of Database Description XML Message . . . . . . . . 273.7 Example of Contents of Acknowledgement Database Description XML Message 283.8 Example of Contents of Service State Update XML Message . . . . . . . . . 293.9 Flowchart for Routing/Forwarding Service Requests within a Federation of

Enterprise Service Buses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.10 Message Exchange Between Two Autonomous Federations . . . . . . . . . . 323.11 Example of Contents of Open XML Message . . . . . . . . . . . . . . . . . 323.12 Example of Contents of KeepAlive XML Message . . . . . . . . . . . . . . . 323.13 Example of Contents of Update XML Message . . . . . . . . . . . . . . . . 333.14 Example of Contents of Notification XML Message . . . . . . . . . . . . . . 333.15 Message Exchange Between Three Autonomous Federations . . . . . . . . . 343.16 Example of Contents of Open XML Message . . . . . . . . . . . . . . . . . 343.17 Example of Contents of KeepAlive XML Message . . . . . . . . . . . . . . . 353.18 Example of Contents of Update XML Message . . . . . . . . . . . . . . . . 353.19 Example of Contents of Update XML Message . . . . . . . . . . . . . . . . 353.20 Flowchart for Routing/Forwarding Service Requests Between Autnomous

Federations of Enterprise Service Buses . . . . . . . . . . . . . . . . . . . . 36

vi

4.1 Example of Service-Oriented Network Topology with Multiple Service Providers 414.2 Plot of Utility Functions for Browse and Order Services . . . . . . . . . . . 534.3 Logical Diagram of Topology of the Service-Oriented Network . . . . . . . . 544.4 Plot of Offered Load for Browse and Order Services . . . . . . . . . . . . . 544.5 Plot of Supported Load for Browse and Order Services . . . . . . . . . . . . 554.6 Plot of Per-Flow Supported for Browse and Order Services . . . . . . . . . 564.7 Total Utility Gained from Browse and Order Services . . . . . . . . . . . . 56

5.1 Various Types of Utility Functions . . . . . . . . . . . . . . . . . . . . . . . 605.2 Using Traffic Prediction Algorithms to Minimze Optimization Calculations 61

1

Chapter 1

Introduction & Motivation

1.1 Telecommunications Services

Telecommunications services are becoming to a commoditized to a large extent;

network service providers are seeing low margins with high investments, and are required to

provide a near-perfect quality-of-service in order to satisfy their customers. Basic economic

theory states that profit and degree of commoditization are inversely proportional to one

another.

The basic landline telephone system in the US is a good example of such a service.

Local calling (basic voice transport service) is a commodity which is experiencing compe-

tition from VoIP providers. Traditional telephone providers have been long dependent on

value added services (such as long distance, caller ID, and voicemail, for example) to gen-

erate additional operational profits, as they allow the provider to differentiate themselves

in the marketplace.

1.2 Integration Services

Due to the pervasive nature of IT in corporations, the problems of heterogeneity

and change are the greatest issues facing IT managers today [1]. Even with wider adoption of

2

open standards, it remains a daunting task to make IT systems communicate across vendor,

protocol, and software differences. The rate of change in available hardware and software

products makes it even more difficult to continue to support a dynamic infrastructure which

is adaptable to business requirements and industry trends.

Furthermore, the pervasive nature of the Internet into global culture is futher

advancing the requirements for businesses to be able to adapt to a quality-sensitive, content-

driven customer base. The ability to interconnect systems in an attempt to maintain a

cohesive product offering is crucial.

1.3 Service-Oriented Architectures

Heterogeneity and change are the problems that are directly addressed by service-

oriented architectures (SOA). Services, the core unit of an SOA, are defined in [1] as “a

course-grained, discoverable software entity that exists as a single instance and interacts

with applications and other services through a loosely coupled, message-based communi-

cation model”. Services are based on the idea that IT infrastructures should be directly

aligned with relevant business processes, rather than more traditional horizontal or vertical

alignment. Services are comprised of the combination of different software components that

together execute a reusable business function.

One key property of services is that they are loosely coupled with one another

within the SOA. Loosely coupled is defined in [2] as having “no tight transactional properties

among the components.” This property is essential to SOA because it removes dependences

on implementation specifics by relying on interaction between services through standardized

interfaces. Services can be implemented in different languages and deployed on different

platforms. The use of standardized interfaces is the key to the enablement of SOA as a

flexible architecture.

Service-oriented architectures were designed to directly address the issues in open

systems [3]. If adopted and implemented correctly, SOA can provide a framework that can

leverage elements of an existing IT infrastructure which will reduce costs and provide a

more flexible and robust environment for the integration of IT and business processes.

3

1.4 Service-Orientation in Next-Generation Networks

The concept of service-orientation is being adopted in several next-generation net-

work architectures. In the IP Multimedia System designed by the 3rd Generation Part-

nership Project, service orientation is key to the interconnection and layering components

of the architecture. In parallel, the International Telecommunications Union has generated

the Next Generation Network standards whose main goal is the convergence of networks

and services. Furthermore, there has been a recent trend in the application/integration

middleware space towards XML-aware networking. This architecture assumes that XML is

now the lingua franca of network communication, and leverages XML-aware devices placed

in the network fabric to perform content-based routing, among many other functions.

Next-generation telecommunications and middleware systems are converging to-

gether under the theme of service-orientation. With this convergence, the properties of

network devices and the larger service-oriented network architecture is an open research

area, that draws from a diverse background of prior work. It is the goal of our research to

summarize the breadth of the research area, and make substantial in-depth contributions

to particular problems that are currently open in the literature.

1.5 Contributions of Our Research

The contributions of our research are as follows:

• Three conference papers

– R. D. Callaway, A. Rodriguez, M. Devetsikiotis, and G. Cuomo. “Challenges in

Service-Oriented Networking”, Proceedings of IEEE GLOBECOM, November

2006.

– R. D. Callaway, M. Devetsikiotis, Y. Viniotis, and A. Rodriguez. “An Auto-

nomic Service Delivery Platform for Service-Oriented Network Environments”,

Submitted to IEEE ICC, May 2008.

– R. D. Callaway, Y. Viniotis, A. Rodriguez, K. Brown, and R. Robinson. “En-

abling Federations of Enterprise Service Buses Using a Distributed Service Reg-

istry”, in preparation for submission.

4

• One journal paper

– R. D. Callaway, M. Devetsikiotis, Y. Viniotis, and A. Rodriguez. “An Auto-

nomic Service Delivery Platform for Service-Oriented Network Environments”,

in preparation for submission to IEEE Transactions on Network and Ser-

vice Management.

• Two patent applications

– Methods and protocols for enabling the dynamic and scalable federation of en-

terprise service buses

– Methods and protocols for enabling the dynamic and hierarchical interconnection

of autonomous federations of enterprise service buses

• One book chapter

– M. Devetsikiotis and R. D. Callaway. “The Role of the Enterprise Service Bus in

IP Multimedia System Architectures”, invited chapter of “The IP Multimedia

Systems Handbook”, edited by S. Ahson and M. Ilyas, CRC Press, 2008

• One invited presentation

– R. D. Callaway and A. Rodriguez. “Service-Oriented Systems in Next Generation

Middleware and Telecommunications Architectures”, Hot Topic Presentation at

1st IEEE Workshop on Enabling the Future Service-Oriented Internet,

November 26th, 2007.

1.6 Outline

The outline of the remainder of this report is as follows:

• In Chapter 2, we formally propose Service-Oriented Networking as an emerging ar-

chitecture. We discuss the challenges both in building SON devices as well as in

interconnecting the devices together to form a true network.

5

• In Chapter 3, we continue the discussion regarding large-scale service-oriented net-

works. We explicitly discuss a use case for SON, federations of enterprise service

buses. We describe how federations are enabled by a distributed service registry, and

provide details and examples of two protocols, based upon Internet routing protocols,

that enable a robust, scalable and dynamic infrastructure.

• In Chapter 4, we present our autonomic service delivery platform. The goal of this

platform is to optimally route requests from service consumers to providers. We

provide details of the underlying utility-based analytic framework, as well as results

from an initial experiment which shows the ability of the framework to optimally route

and throttle load under resource constraints.

• In Chapter 5, we summarize our work to this point and propose items for remaining

work on the autonomic service delivery platform.

6

Chapter 2

Service-Oriented Networking

2.1 Application-Aware Networks

Active and overlay networks have both attempted to provide application-aware

functionality in the network without an open standard for application-to-application com-

munication.

2.1.1 Active Networks

Active networks sought to improve the deployment of emerging networking tech-

nologies and protocols by adding application layer functionality in specific active nodes.

While data would still be passed in packets as in a traditional packet-switched network,

active networks would support “smart” packets, which would carry bytecode, along with

data, to be executed in active nodes. The main assumption underlying active network re-

search is that the network layer is inflexible and can not adapt to the dynamic requirements

of emerging network services. Since standardization of new protocols is often a lengthy

process, active network technology attempted to leverage advances in compilers, operating

systems, and programming languages that would enable running user-supplied code in ac-

tive nodes [4, 5]. Several groups [6, 7, 8] have proposed examples of potential architectures

for the organization of information and program code into the packet headers, showing

7

results in which active networks only suffer a slight degradation in performance versus a

software router.

However, active networks were not widely deployed due to issues with security,

resource allocation, and the substantial cost of deployment. Since packets could contain

arbitrary code to be executed on an active node, precautions must be taken to ensure that

an rogue user could not execute code that would corrupt the operations of other users. It

is essential to manage the computing resources of the node to ensure that programs are fair

to each other. Furthermore, the deployment of active network technology in the network

would require a substantial investment for network operators in order to support this new

architecture.

2.1.2 Overlay Networks

The development and subsequent deployment of active networks showed that en-

abling application-awareness in the network by executing user-supplied code in the network

layer is infeasible. Overlay networks sought to provide application-aware functionality by

pushing the complexity of such algorithms towards the end users of the network. The ma-

jor assumption in overlay networks is that application-aware functionality should not reside

in the network layer due to the issues presented in active networks; rather, application-

awareness should be enabled in the application layer where issues of security and resource

allocation could be more easily addressed. Overlay networks consist of peer nodes that self-

organize into a distributed data structure based on application criteria. Strategically placed

application-level agents serve as intermediaries for forwarding data from a source to a set

of destinations, in effect forming an overlay on top of the underlying IP substrate. Overlay

networks can be used to deploy new protocols, such as multicast [9], or enable application-

aware routing, where messages are forwarded based on application data or state.

2.2 Service-Oriented Networking

The rapid adoption of XML and acceptance of Web Services and SOA have enabled

network components to offload portions of application data processing or decision-making

8

outside of the traditional data-center. Differences in application data-encoding that once

hindered the network’s ability to comprehend true application intent are now described by

XML. Routing has become XML-oriented with the use of functions such as XPath routing

[10] to direct traffic based on XML content. Web Services Security (WS-Security) defines

security criteria within XML Web Services envelopes across service invocations. Further,

additional offload capabilities are now possible, such as XML transformation (XSLT) [11]

to change XML content as it traverses the network and service mediation to enable the

interoperability of Web Services in heterogeneous environments. These have key benefits

to SOA architectures as they enable services to be integrated in a loosely-coupled manner

where implementation details of components are hidden from the requester of the service.

Service-Oriented Networking (SON) is an emerging architecture that enables net-

work devices to operate at the application layer with ESB-like features such as offloading,

protocol integration, and content-based routing. By adding application-awareness into the

network fabric, SON can provide vast gains in overall performance and efficiency and en-

able the integration of heterogeneous environments. We refer to this collection of network-

resident application-level operations as SON functions. Among others, SON functionality

provides three key benefits: service virtualization, locality exploitation and improved man-

ageability.

2.2.1 Benefits of Service-Oriented Networking

Service Virtualization

Service virtualization transparently maps a set of services to the protected back-

end resources which actually provide the service. A SON device can serve as a proxy for

actual services by masking internal resources via XML transformation and routing tech-

niques. The SON device could also be leveraged to manage security and denial-of-service

(DoS) policies for incoming requests.

Locality Exploitation

By deploying certain functions in the network fabric, SON devices can be pro-

visioned and customized to handle unique workloads. For example, these systems can be

9

provisioned with cryptographic hardware assist for SSL or other security functions. Simi-

larly, domain-specific hardware to optimize XML processing can be installed to offset the

cost of processing Web Services or XML transformation functions. Provisioning and cus-

tomizing these SOA servers lead to greater efficiencies and is much more cost-effective than

provisioning the entire enterprise with these capabilities. Lastly, a potential performance

benefit is gained from exploiting locality within co-located SON functions. For example,

consider a function executing an XSLT schema transformation while another is performing

XPath routing. The two functions can communicate to avoid parsing the request twice.

Locality also has benefits at lower levels of the system, such as in cache utilization.

Improved Manageability

Offloading function into the network enables centralized, and therefore simplified,

management of the function and corresponding configuration. For example, style sheets,

security, caching and routing policies can all be centrally managed at SON devices versus

decentralized across a cluster of enterprise servers.

2.2.2 Service-Oriented Networking Functions

Three examples of SON functions include functional offloading, service integration,

and intelligent routing [12].

Functional Offloading

Offloading security-related operations has been a common practice for Internet-

based application environments. This practice applies equally well to document-centric

service-oriented environments. Like the HTTP server, a SON device can be specially provi-

sioned to handle cryptographic functions. This enables the device to optimize the validation

of digital certificates in the context of WS-Security. We illustrate this in Figure 2.1, where

the SON appliance intercepts WS-Security SOAP envelopes, performs the appropriate cryp-

tographic functions, and forwards the requests to the service provider.

A SON device can also perform a firewall-like security function to validate service

requests (for example, against a corresponding WSDL document) before forwarding them

10

ServiceProvider

Encrypted &Signed

SOAP/XML

Decrypted & AuthenticatedSOAP/XML

WS-Security:Decryption & Authentication

ServiceOriginator

SON Appliance

Figure 2.1: Example of Functional Offloading: Decryption and Authentication via WS-Security

to the enterprise server for processing. These checks would ensure that only well-formed

service requests are forwarded. This prevents DoS attacks and ensures enterprise servers

are encountering only valid service requests.

The most efficient form of offloading is in full-function offload where the service re-

quest can be satisfied completely within the SON device. Dynamic service response caching,

a technique that accomplishes this, is most effective for read-mostly interactions where re-

quests do not update back-end states or databases. For example, service requests that

retrieve stock quotes, where ticker values are updated every five minutes, are well suited

for this type of offload. If done correctly, a large proportion of the read traffic can be

completely serviced by the appropriate caching component, thereby reducing the load on

enterprise database servers. A cache policy contains rules that define how results from

specified services requests are cached.

Service Integration

Figure 2.2 illustrates the service integration aspect of the SON device in which a

widget retail store (Widgets, Inc.) is ordering a collection of parts by invoking a service

request back at the home office. The home office has deployed the SON device in the

network fabric that chooses the best parts supplier and forwards the service request to that

supplier. However, in this case, the XML schema of Widgets, Inc. is different depending

on the chosen supplier. The SON device is capable of transforming the original order to

11

WidgetsRUSServiceProvider

Purchase Order in Widgets, Inc.XML Schema

Purchase Order in WidgetsRUSXML Schema

XSLTransformationWidgets, Inc.

ServiceOriginator

SON Appliance

Figure 2.2: Example of Service Integration: Document Transformation via XML SchemaValidation and XSLT

schemas of participating providers, in this case WidgetsRUS. Other widget manufacturers

would likely require different schemas, requiring the SON function to apply the appropriate

XSLT transformation dependant on the supplier.

Since the majority of corporate data today exists in mainframe databases, service

integration also provides the ability to interface with existing legacy systems, giving a

system architect much more flexibility to migrate towards a service-oriented environment.

This increases the number of service consumers that can take advantage of these programs

and data and extends the reach of SON further into the enterprise.

Intelligent Routing

Content-based routing (CBR), like priority-based routing, is driven from policy

documents. The policies typically apply a rule against some part of a service request

(header or content) and derive a token as a result. The token is then used to look up a

corresponding enterprise server address in a routing table. For example, a CBR policy might

be created by combining the port-type and operation-name of a service and mapping it to

a specific enterprise server. In a SON device, CBR can be realized by using XPath-based

expressions to determine the destination of the request as shown in Figure 2.3.

CBR also allows an affinity between a class of services and the enterprise server

that services the request, called service partitioning. Figure 2.3 also shows this service

partitioning pattern. Service partitioning can be used as the foundation to address bottle-

necks that occur in high volume Online Transaction Processing (OLTP) applications that

12

ServiceProviders

Unclassified Requests

XPathRouting

SON Appliance

Figure 2.3: Example of Intelligent Routing: Routing Requests to Logical Partitions usingXPath Routing

intensively read and write data to databases and require the utmost in data consistency

and availability. Examples of such systems include trading, banking, reservation, and online

auctioning systems. While a strategically located SON device enables service partitioning,

the value is actually garnered on the enterprise application servers where the services are

run. For example, service-based applications can now assume that their variation of the

service is not running anywhere else in the enterprise server cluster. The applications can

then aggressively cache interactions without the processing overhead of maintaining data

consistency within that cluster. Service partitioning also enables other optimization tech-

niques such as data batching where insertions, updates, and deletions can be done in bulk

to the database.

2.3 Research Challenges

We believe that service-oriented networking is an exciting new research area that

can have a dramatic impact on the design, performance, integration, and management of

service-oriented environments. Therefore, we believe that significant research is needed in

the following areas in order to create an adaptive and robust SON device which can provide

the benefits of service-oriented networking as we have described in this paper:

13

2.3.1 Implementation Considerations

A tradeoff exists between the performance of implementing SON functions in a net-

work appliance versus software, the extensibility of arbitrary programs versus the hardened

security of an appliance based upon standardized security mechanisms, and the flexibility

of a software solution versus the increased consumability of an appliance. This tradeoff is

depicted in Figure 2.4. Since care must be taken to ensure that the SON function improves

SecurityExtensibility

AppliancePerformance

Software Performance

Consumability

Flexibility

SON Appliance

Figure 2.4: Comparison of Software and Appliance Approaches

the overall performance of the architecture rather than degrading it, we believe that net-

work appliances that host SON functionality can leverage hardened software and specialized

hardware solutions and overcome the limitations experienced in previous attempts to in-

troduce application-awareness into the network fabric. SON functions could be collocated

within a switch or router, as in the Cisco line of AON products [13]. The SON functionality

can also be deployed in a standalone hardened appliance as in several products sold by

DataPower, recently acquired by IBM [14].

A further discussion of issues concerning implementing SON functions in a network

appliance can be found in Section 1.5.

14

2.3.2 Performance

Robustness

The SON device should scale to support a large number of requests to be processed

concurrently. It should be robust to overload conditions, continuing to prioritize and pro-

cess high priority requests and shed low priority requests while operating in an overloaded

regime. Admission control ensures that a server always operates in a stable regime; even

in overloaded conditions, the server can scale and continue to provide differentiated service

to its users. In order for an SON device to provide services that are fair to the requesting

users, a policy should be defined that enumerates the differential treatment that requests

are to receive. This policy should define strategies to prioritize traffic under both normal

and overloaded conditions. Since requests must be classified before they can be prioritized,

it is essential in overloaded conditions that the system can continue to process high pri-

ority requests while possibly shedding lower priority requests. Therefore, fast methods for

classifying incoming requests are needed. The classifications could be based on network

layer information or upon information residing within the XML content. Algorithms for

executing XPath expressions on streaming XML such as QuickXScan [15] could be useful

in such situations.

Resource Allocation

The SON device should be adaptive, changing its underlying execution model to

support different types of software components in order to maximize the efficiency of the

system. We believe that concurrency mechanisms will be a significant component of resource

allocation within a scalable and adaptive SON device. Concurrency mechanisms have a

dramatic effect on the overall performance and efficiency of a device. Internet services are

unique because they require massive concurrency but also block while waiting on unavailable

resources. It is this unique combination of requirements that suggests a hybrid architecture

could be used to exploit the benefits of different concurrency mechanisms. Models such

as the Staged Event-Driven Architecture (SEDA) [16] could prove useful in building an

adaptive resource allocation system for an SON device.

15

SEDA is an architecture that separates functions within applications into stages,

which each have their own thread pool and are connected together as a network of queues

in order to provide the desired application functionality. Admission control is used at each

stage, and adaptive controllers that can modify the thread pool size or the amount of

requests that are processed by each thread (batching) are included. Figure 2.5 (reprinted

from [17]) shows how admission control is performed on requests to a SEDA stage using a

response time controller.

Figure 2.5: Example of Adaptive Admission Control: SEDA Response Time Controller

Performance Optimization

One main benefit of SON is that it can leverage specialized hardware, such as

hardware-accelerated cryptographic or XML processing functionality, to enhance the overall

performance of the device. However, the SON device will contain software components

that will process requests in conjunction with the available hardware devices. Since these

components could block upon the remove invocation of services, it will be important to

ensure an efficient and robust cooperation scheme between these hardware and software

components exists as this scheme will be crucial to the overall stability and performance of

the SON device.

16

2.3.3 Security and Management

Security

As in active networks, SON provides software functionality that will be executed

in the network fabric. However, with the introduction of open standards such as XML

and WS-Security, we believe that SON devices will not suffer from the same security issues

as active networks. The use of XML in network operations raises new research questions

regarding how open standards such as XML and Web Services could be leveraged together

in an SON appliance in order to create a device which is hardened against XML and Web

Services-based DoS attacks. One approach is to leverage well-formedness checking and

XML schema validation against all incoming documents in order to ensure that only valid

requests proceed within the device for further processing.

Manageability

The initial work presented here concentrates on enablement technologies that logis-

tically deliver and deploy SON functions manually; however, we look toward the autonomic

configuration and coordination amongst these functions. Specifically, we anticipate that

enterprise applications of the future will begin to leverage distributed SON design patterns

where large numbers of SON devices coordinate with peers using network-wide application-

specific policies. Manual configurations are not able to scale with these environments,

nor can they adapt the configuration to dynamic network and application conditions. For

example, a large-scale SON deployment could be leveraged to enable application-specific

multicast. SON devices should coordinate with their peers to determine the appropriate

points in the network to perform configuration changes based on prevailing network and

application conditions.

17

Chapter 3

Large-Scale Service-Oriented

Systems

The enterprise service bus (ESB) acts as the integration and communications plat-

form for connecting service consumers and providers. It is often desirable to have multiple

ESB deployments federate with one another to provide a distributed integration platform

that promotes the reuse of services within and across enterprises. However, the existing

solutions to federate ESBs are limited by their inflexibility to change and inability to scale.

In this chapter, we propose the enablement of a federation of enterprise service buses via

a distributed service registry that distributes policy-appropriate service metadata to feder-

ation members. We provide a high-level description of two protocols which maintain the

state of the distributed registry within and between autonomous federations. We argue

that these application of a distributed service registry and the enabling protocols is a novel

application of existing technology that creates a robust, scalable, and flexible federation of

ESBs that is needed in the next generation of large-scale SOA deployments.

18

3.1 Introduction & Motivation

As a critical infrastructural component of service-oriented architectures (SOA),

the enterprise service bus (ESB) acts as the integration and communications platform for

connecting service consumers and providers [18]. As such, the ESB is responsible for, along

with many other functions, the enforcement of policies, routing of service requests, and

performing content and/or transport protocol transformation.

As the number of services in an organization increases, the need for a service

discovery and governance platform arises. The service registry enables consumers to find

available services and providers to advertise available service instances. The registry can

optionally serve as a repository for governance metadata, policy documents, and XML

schemas.

Instantiating the enterprise service bus in a message-oriented middleware product

along with deploying a service registry provides an intuitive solution towards implementing a

small to medium size SOA. However, recent market trends show that SOA is being rapidly

adopted; therefore, strategies for creating more large-scale deployments are needed. A

typical approach to transitioning from a moderate-scale to a large-scale SOA deployment

is to “scale-up”; that is, leave the topology of the architecture fundamentally unaltered

while adding additional resources to the individual architectural components. “Scaling-

out” yields a distributed approach to the large-scale problem that involves altering the

topology of interconnected architectural components. Furthermore, we argue that the rapid

adoption of SOA is causing an increase in the number of business-to-business transactions

between autonomous SOA deployments. Primarily for reasons of governance, these types of

interactions exemplify the need for a large-scale distributed ESB; we refer to such a system

as a federation of enterprise service buses.

In a federation of ESBs, the primary problem is to appropriately disseminate infor-

mation throughout nodes that comprise the ESB to enable policy-driven service discovery

and routing. We propose that a distributed service registry is a scalable and robust ap-

proach to enabling federations of enterprise service buses. The distributed service registry is

hierarchical in nature, and is maintained by two protocols that synchronize relevant service

metadata amongst ESB deployments as appropriate under defined business policies. There

are three main advantages to our proposal over existing approaches:

19

• We enable the federation of ESB deployments within an enterprise in a flexible and

scalable manner.

• The distributed service registry and the supporting protocol allow our solution to

adapt autonomically to dynamic network and service conditions.

• This architecture provides the capability to support on-demand techniques such as

fast failover or priority-based load shedding in an autonomic fashion.

The remainder of the chapter is structured as follows: in the following section, we

review existing approaches to the ESB federation problem. In Section 3.3, we explicitly pro-

pose our architecture that enables the dynamic and scalable federation of ESBs. In Section

3.4, we present an overview of the first of two protocols which maintains the consistency

and availability of service metadata within an autonomous federation. In Section 3.5, we

present the second protocol which is responsible for the maintenance of interconnections of

autonomous federations. In Section 3.6, we provide a brief review of related literature in

the service discovery, naming, and registry areas.

3.2 Current Approaches to ESB Federation

Currently, there are three approaches to addressing the problem of policy-driven

service metadata dissemination: manual configuring interconnections, deploying a broker

ESB, and utilizing a centralized registry across or between enterprises.

3.2.1 Manual Configuration

One way of federating ESBs is by manually configuring functionality within an ESB

that serves as a ”proxy” to other ESBs in the federation. For each service that is managed

by a remote ESB, a mediation must be defined that selects appropriate requests to be

forwarded to the remote ESB, performs necessary content/protocol transformations, and

then forwards the request onto the remote ESB. Matching mediations must exist on remote

ESBs in order to support bidirectional communication in this case. Since this configuration

must be done manually by a systems administrator at each ESB, the configuration of such

20

a solution is tedious and prone to error (for S services and N ESBs, there are possibly

SN proxies to be configured). There is also no mechanism to change the properties of this

mediation based on changes in network or service availability. Manual configuration allows

basic federation of multiple ESBs; however, this is an inflexible and impractical solution for

large scale enterprises.

3.2.2 Broker ESB

Rather than statically defining the routing mediations at each ESB, a separate ESB

called a ”broker” ESB can be deployed whose sole function is to implement the requisite

mediations to support federation. This helps to consolidate the many different mediations

that might exist in the manually configured solution described above into a single ESB.

However, this consolidation is still dependent on a systems administrator to manually define

the mediations required for each service (in this case, the number of proxies to be configured

is minimized to S). Since there is no mechanism to update the mediation metadata based

on dynamic service availability, the broker ESB solution is inflexible. The broker ESB then

becomes the architectural bottleneck, which introduces issues with scalability and fault

tolerance.

3.2.3 Centralized Registry

The final known approach is to deploy a centralized registry for the entire enter-

prise. When ESBs need to route service requests to other ESBs within the SOA, they would

consult the registry at runtime to make a forwarding decision based on the current location

of a service instance, thus addressing the manual configuration concerns raised by the pre-

vious solutions (as with the broker ESB, the number of entries in the centralized registry is

equal to the number of services S). However, centralizing all service metadata and status

into a single registry forces the registry to be the architectural bottleneck in such a federated

system, causing concerns with system performance, scalability, and fault tolerance. The cen-

tralized registry is ideal from the standpoint of the consolidation of service information, but

is infeasible in many realistic business scenarios due to business-to-business interactions,

disparate geographical locations, and limitations imposed by business structures. Today,

21

Figure 3.1: Example Topology of Multiple ESB Deployments - Hub & Spokes

Figure 3.2: Example Topology of Multiple ESB Deployments - Peer Business Divisions

manual configuration of the centralized registry is required to insert/update/delete service

metadata, which limits the flexibility of this solution.

3.3 Federation Architecture

The overarching goal of ESB federation is to provide a logically centralized (at an

appropriate scope) integration platform across different geographic and business boundaries;

that is, the topology formed by the federation of ESB deployments should align directly to

the structure of entities within an enterprise. Examples of federated ESB topologies that

align with common business structures are presented in [19].

Figure 3.1 shows the logical topology of a hub/spoke federated ESB. This type

of topology directly aligns with the Store/Branch business structure described in [19], and

forces all service routing to be done through the hub ESB deployment.

Figure 3.2 shows the logical topology of a directly-connected federated ESB. In this

22

Figure 3.3: Example Topology of Interconnected Autonomous Federations

topology, all ESB deployments are connected directly to one another, so that service requests

which are routed within the federation pass directly from the source ESB to destination ESB.

This type of topology directly aligns with the Multiple Geographies & Multiple Business

Divisions business structures described in [19].

A natural extension of the intra-federation topology is interconnecting multiple

federations, as shown in Figure 3.3. There is a practical need for interconnected federa-

tions; the need arises, for example, in business-to-business environments, in which separate

enterprises have to interact to provide a service to each other or to create a composite ser-

vice to be offered to an external customer. The same need arises even within a single but

large enterprise (e.g., in an e-government setting), when the enterprise itself is organized as

multiple, autonomous, and heterogeneous federations of ESBs.

A key concept in our proposal is the notion that the amount of service registry

23

data that is shared with a federation member is configurable via policy; we refer to this

concept as policy-based service advertisement. For example, in the hub & spoke case, it is

desirable for a spoke to share appropriate service information (as defined by policy) with

the hub ESB, and share no service information with any other spoke ESB. Policy-based

service advertisement allows different members of the federation to have different views of

hosted services at a particular federation member. We envision that certain services should

only be exposed to certain federation members, and that it may be desirable to allow or

disallow the advertisement of particular services. While certainly related, we believe that

the appropriate distribution of policy documents is an orthogonal problem to the one we

are addressing and is therefore outside the scope of this manuscript.

Our method to achieve a dynamic and scalable federation of enterprise service

buses is based upon the concept of a distributed service registry. Federation members

create a distributed service registry by sending policy-based service advertisements to peer

members. Each federation member will have its own (possibly unique) converged view of all

routable service endpoints in the federation, which it will use in making routing/forwarding

decisions. This model contrasts with the centralized registry solution described in Section

II; notably, the distributed nature of the registry allows it to overcome the scalability and

robustness concerns that exist with a centralized solution. In order to allow the federa-

tion members to distribute service state amongst themselves, protocols are needed that

implement the policy-based service advertisements in an automated fashion.

3.3.1 Related Work

There are several previous attempts that have been made towards developing fed-

erated service discovery architectures that are based on service registries. The authors of

[20] propose defining a topology for collaborative service discovery, and subsequently floods

the topology with the discovery query, and all nodes respond to the client with the results.

A similar proposal is presented in [21], where UX servers are part of a federation, and a

minimum spanning tree is found in order to flood queries if an initial lookup returns null.

However in both of these proposals, flooding is required and such systems are known not

to be scalable.

24

There are several proposals for distributed service discovery based upon distributed

hash tables (DHT) and P2P technologies. [22] proposes the integration of distributed hash

tables with UDDI registries to enable a larger distributed registry; however, they do not

consider governance issues in a cross-domain system such as theirs in the proposal. The

authors of [23] present the use of DHT as a way to enable a scalable service discovery

platform for Grid environments. In [24], a P2P network is used to interconnect registries

using a layered approach. Clients issue queries about available services (which meet their

desired semantics and QoS) to a gateway layer, that is responsible for the translation of

the query into the ontologies of the different types of ontologies supported at the different

registries within the federation, and then passes the query off to a routing layer which

transmits the query onto the appropriate registry. Finally, [25] focuses on the discovery

component of the service composition problem; they utilize Pastry as a P2P service overlay

to find services that can be used in a larger composition. A proposal for using a pub/sub

network as a way for different registries to learn about advertisements and updates to

distributed service information is presented in [26]. Finally, [27] integrates a specific proposal

for storing information about existing UDDI installs inside DNS, and then using DNS

& UDDI together to enable a distributed registry. They do not consider the caching or

replication of registry data for lookups, nor governance of cross-domain situations.

Perhaps closest to our proposal are the following three manuscripts. [28] provides

an architecture similar to ours, but requires use of the UDDI protocol, and does not discuss

the use of policy, convergence of their protocols, or restrictions made on topology. [29]

provides a solution to the multiple domain discovery problem, though it is arguably not

a scalable one due to a possible single-point-of-failure in the service broker, as well as the

dependency of full replication of registry state in the broker. The authors of [30] explicitly

consider a cross-domain service discovery, and use a P2P approach to enable lookups of

services across different domains.

Also relevant to the discussion is the concept of service naming. Service naming

refers to the ability to uniquely identify and address service instances in a service-oriented

architecture. The proposal presented in [31] involves removing the tight coupling between

naming and location that exists in the Internet today; they propose by adding two layers

(service ID and endpoint ID), an architecture that readily accepts mobility of services, data,

25

and hosts can be created. Their naming architecture is flat in nature, and propose using

DHTs to deal with scalability issues in such a system. Similar in nature to this proposal is

the work proposed in [32]. They present a two-layered naming scheme for service lookup and

routing. However, their naming scheme is based on fixed length delimiters and is therefore

less flexible than an XML-based scheme such as our own. A good overview of the service

naming literature is presented in [33].

3.4 Building An Autonomous Federation

Our proposed routing/management protocol for maintaining a distributed service

registry within a single autonomous federation is similar in nature to the Open Shortest

Path First routing protocol [34]. It is also built atop the Web Services Distributed Manage-

ment (WSDM) framework. We envision that a reliable messaging infrastructure, such as

WS-ReliableMessaging or WSRM would be utilized to ensure delivery of messages between

federation members. Also, we expect that a security mechanism, such as mutually authen-

ticated SSL, would be used to ensure communication only occurs between actual federation

members.

The intra-federation routing/management protocol has four main message types:

• Hello: This message is used to establish a connection with peers in the federation; it

also provides a mechanism to detect if a peer is currently reachable or not so that the

distributed registry can be updated appropriately.

• Database Description: Used as an acknowledgement of the Hello message, this

message is used to share the sender’s current view of the topology with the receiver;

it also contains the set of all appropriate exportable service information between the

peers.

• Service State Request: This message is sent to a peer if a federation member needs

information about a particular service.

• Service State Update: This message is sent as a response to a Service State Request

message with relevant information about the requested service, or in a ”push” model

to send updates to service metadata to federation members.

26

Figure 3.4: Message Exchange Between Two ESBs Within a Federation

In the text below and in Figures 3.4-3.9, we provide an example which describes

the semantics of the protocol (and provides examples of message format, etc), and how the

protocol can be utilized to establish and maintain the distributed service registry within an

autonomous federation.

Once peering relationships are extracted from the ESB topology (which is defined

by a system architect), and assuming appropriate policies exist to drive the policy-based

service advertisement function, our protocol can begin running at each federation member.

When an ESB member joins the federation, it sends a Hello message to all other federation

members to which it has a peering relationship - this can be seen in Figures 3.4 & 3.5.

When a federation member receives a Hello message, it consults its policies to

determine what subset of its service registry information it should share with the sender of

the Hello message. Once it has made this decision, it responds to the joining member with

a Database Description message, as seen in Figure 3.6, which contains the appropriate

service information.

The joining member acknowledges the receipt of the Database Description mes-

sage by sending a Database Description message which lists the shared services in the peering

27

<?xml version="1.0"?><amp:Hello xmlns:amp="http://www.ibm.com/schemas/appliance/management/2.0"

srcID="ESB2_ID" asdID="1"><amp:esbInfo>

<amp:ipAddress>1.2.3.4</amp:ipAddress><amp:mgmtPort>9876</amp:mgmtPort>

</amp:esbInfo><amp:helloInterval>1000</amp:helloInterval><amp:ESBsInASD>

<amp:esb esbID="ESB2_ID"/></amp:ESBsInASD>

</amp:Hello>

Figure 3.5: Example of Contents of Hello XML Message

<?xml version="1.0"?><amp:DatabaseDescription srcID="ESB1_ID" asdID="1"xmlns:amp="http://www.ibm.com/schemas/appliance/management/2.0">

<amp:ESBsInASD><amp:esb esbID="ESB1_ID"/><amp:esb esbID="ESB2_ID"/>

</amp:ESBsInASD><amp:services>

<amp:service id="A" esb="ESB1_ID"><amp:ipAddress>1.2.3.100</amp:ipAddress>

<amp:port>80</amp:port><amp:protocol type="SOAP/HTTP"><amp:url>http://1.2.3.100:80/someService/a</amp:url><amp:https>false</amp:https>

</amp:protocol></amp:service></amp:services>

</amp:DataBaseDescription>

Figure 3.6: Example of Contents of Database Description XML Message

28

<?xml version="1.0"?><amp:DatabaseDescription srcID="ESB2_ID" asdID="1"xmlns:amp="http://www.ibm.com/schemas/appliance/management/2.0">

<amp:ESBsInASD><amp:esb esbID="ESB1_ID"/><amp:esb esbID="ESB2_ID"/>

</amp:ESBsInASD><amp:services>

<amp:service id="A" esbID="ESB1_ID"><amp:ipAddress>1.2.3.100</amp:ipAddress>

<amp:port>80</amp:port><amp:protocol type="SOAP/HTTP">

<amp:url>http://1.2.3.100:80/someService/a</amp:url><amp:https>false</amp:https>

</amp:protocol></amp:service><amp:service id="B" esbID="ESB2_ID">

<amp:ipAddress>1.2.3.200</amp:ipAddress><amp:port>900</amp:port><amp:protocol type="SOAP/HTTP">

<amp:url>http://1.2.3.200:900/someService/b</amp:url><amp:https>false</amp:https>

</amp:protocol></amp:service>

</amp:services></amp:DatabaseDescription>

Figure 3.7: Example of Contents of Acknowledgement Database Description XML Message

relationship; this can be seen in Figure 3.7.

Hello messages are periodically exchanged with peers in a ”heartbeat” fashion to

ensure connectivity exists between federation members. If a particular federation member

needs information about a particular service, it sends a Service State Request message

to a peer; the peer responds with a Service State Update message with the requested

information. The Service State Update message provides an automated mechanism for

the protocol to dynamically update the distributed registry amongst federation members.

This message type could be used to enable autonomic functionality like fast-failover. In

this case, the Service State Update messages sent would cause the distributed registry

to converge to a new state, causing a new endpoint to be chosen when a routing decision

29

<?xml version="1.0"?><amp:ServiceStateUpdate srcID="ESB2_ID" asdID="1"xmlns:amp="http://www.ibm.com/schemas/appliance/management/2.0">

<amp:services><amp:service id="A" esbNodeID="ESB1_ID">

<amp:ipAddress>1.2.3.100</amp:ipAddress><amp:port>80</amp:port><amp:protocol type=SOAP/HTTP>

<amp:url>http://1.2.3.100:80/someService/a</amp:url><amp:https>false</amp:https>

</amp:protocol></amp:service><amp:service id="B" esbNodeID="ESB2_ID">

<amp:ipAddress>1.2.3.200</amp:ipAddress><amp:port>4205</amp:port><amp:protocol type=SOAP/HTTP>

<amp:url>http://1.2.3.200:900/someService/b</amp:url><amp:https>false</amp:https>

</amp:protocol></amp:service>

</amp:services></amp:ServiceStateUpdate>

Figure 3.8: Example of Contents of Service State Update XML Message

is made for a relevant service request. Figure 3.8 shows an example of a Service State

Update message being sent to a peer ESB to inform the peer that a port number is changing

for a routable service proxy.

3.4.1 Service Request Forwarding

In this section, we’ve shown examples of how the routing/management protocol is

used to synchronize the state of the distributed registry within a federation. Figure 3.9 shows

how the distributed registry enables the routing/forwarding of service requests within a

federation. When a request is received, either directly from a service requestor or forwarded

from another ESB node in the deployment, it is passed to a routing mediation. This routing

mediation determines the destination for this request by consulting the local service registry

along with its own locally defined service connections. If this is the appropriate ESB node

30

Figure 3.9: Flowchart for Routing/Forwarding Service Requests within a Federation ofEnterprise Service Buses

for the request (i.e. the service instance can be directly reached through a mediation flow

at this node), the request is passed to the mediation flow for processing and eventually

passed onto the service instance. If the service request can not be serviced (or should not

be serviced, according to policy) within this ESB deployment, the routing mediation then

consults the distributed registry for matching service instances available in the federation to

decide where to send the request. If an appropriate destination is reachable in the federation,

the request is sent to the correct ESB deployment and then forwarded onto the appropriate

ESB node which provides connectivity for the particular service being requested. Otherwise,

the request is discarded as not being serviceable within the federation.

3.5 Interconnecting Autonomous Federations

Our proposed routing/management protocol for maintaining a distributed service

registry between autonomous federations is similar in nature to the Border Gateway Proto-

col [35]. As with the intra-federation protocol, it is also built atop the WSDM framework.

Again, we envision that a reliable messaging infrastructure would be utilized to ensure de-

livery of messages between boundary nodes of federations. Also, we expect that a security

mechanism, such as mutually authenticated SSL, would be used to ensure communication

only occurs between actual boundary nodes.

The inter-federation routing/management protocol has four main message types:

• Open: This is the first message exchanged between two peers. It is used to establish

a connection with peers in the federation; it also provides a mechanism to detect if

a peer is currently reachable or not so that the distributed registry can be updated

31

appropriately. It may also be used in the case that an individual node suffers failure

in order to request a current update of the distributed registry including any changes

that occurred during the failure.

• KeepAlive: This message is used to maintain “reachibility” between peers.

• Update: This message is used to convey routing information between peers. It is

used to share the sender’s current view of the topology with the receiver, e.g., to

advertise new service availability or withdraw unavailable services. Update messages

advertise routes to individual or aggregated services. Note that the routes themselves

may be calculated to optimize some criterion, or they can be “default” routes.

• Notification: This message is sent when an “error” condition is detected. As an

example, such a message may be used to detect incompatibilities between two feder-

ations.

In the text below and in Figures 3.10-3.20, we provide an example which describes

the semantics of the protocol; we outline the essentials of the message format and show how

the protocol can be utilized to establish and maintain the distributed service registry.

Once peering relationships are extracted from the topology of interconnected fed-

erations (which is defined by a system architect), and assuming appropriate policies exist

to drive the policy-based service advertisement function, our protocol can begin running at

each federation. One node in each federation is appointed as a “boundary node” who is

responsible for establishing and maintaining the interconnection between two autonomous

federations. When the boundary node is defined, it sends a Open message to its peer

boundary node in the other autonomous federation - this can be seen in Figures 3.10 & 3.11

where the boundary node in Federation 2 sends an Open message to the boundary node

in Federation 1.

When Federation 2’s boundary node receives the Open message, it acknowledges

the message by responding to Federation 1’s boundary node with a KeepAlive message,

as seen in Figure 3.12, which contains the local ID for the boundary node and federation.

KeepAlive messages are sent periodically between boundary nodes in order to maintain

state on the reachability of peer federations.

32

Figure 3.10: Message Exchange Between Two Autonomous Federations

<?xml version="1.0"?><amp:Open srcID="border2_ID" asdID="2"

xmlns:amp="http://www.ibm.com/schemas/appliance/management/2.0"><amp:holdTime>1000</amp:holdTime>

</amp:Open>

Figure 3.11: Example of Contents of Open XML Message

<?xml version="1.0"?><amp:KeepAlive srcID="border1_ID" asdID="1"xmlns:amp="http://www.ibm.com/schemas/appliance/management/2.0"/>

Figure 3.12: Example of Contents of KeepAlive XML Message

33

<?xml version="1.0"?><amp:Update srcID="border2_ID" asdID="2"xmlns:amp="http://www.ibm.com/schemas/appliance/management/2.0">

<amp:WithdrawnServiceRoutes/><amp:AvailableServiceRoutes>

<amp:ServiceRoute serviceID="A"><amp:Origin>IGP</amp:Origin><amp:Path><amp:ASD id="1"/></amp:Path><amp:NextHop><amp:ASD id="1"/></amp:NextHop>

</amp:ServiceRoute><amp:ServiceRoute serviceID="B">

<amp:Origin>IGP</amp:Origin><amp:Path><amp:ASD id="1"/></amp:Path><amp:NextHop><amp:ASD id="1"/></amp:NextHop>

</amp:ServiceRoute></amp:AvailableServiceRoutes>

</amp:Update>

Figure 3.13: Example of Contents of Update XML Message

<?xml version="1.0"?><amp:Notification srcID="border2_ID" asdID="2"xmlns:amp="http://www.ibm.com/schemas/appliance/management/2.0">

<amp:Error>Authentication failure</amp:Error></amp:Notification>

Figure 3.14: Example of Contents of Notification XML Message

When Federation 2’s boundary node receives the KeepAlive message as an ac-

knowledgment of its Open message, bidirectional communication has been established be-

tween the federations. At this point, service routing information can be exchanged between

the two federations. This is achieved by Federation 2 sending an Update message, which

contains the available service routes from its federation, as seen in Figure 3.13.

If there is an error in the process, a Notification message is sent between boundary

nodes. An example of this is seen in Figure 3.14.

Now suppose that another federation (Federation 3) wishes to interconnect with

Federation 1. The boundary node of Federation 3 sends an Open message to the boundary

34

Figure 3.15: Message Exchange Between Three Autonomous Federations

<?xml version="1.0"?><amp:Open srcID="border3_ID" asdID="3"xmlns:amp="http://www.ibm.com/schemas/appliance/management/2.0">

<amp:holdTime>1000</amp:holdTime></amp:Open>

Figure 3.16: Example of Contents of Open XML Message

node of Federation 1, as seen in Figures 3.15 & 3.16.

As before, Federation 1 responds to the Open message by sending a KeepAlive

message to the boundary node of Federation 3. This is seen in Figure 3.17.

Figure 3.18 shows Federation 3 advertising its service routes to Federation 1 by

sending an Update message.

Now that Federation 1 has new service routing information from Federation 3, it

shares it with Federation 2 by sending Federation 2’s boundary node an Update message,

as seen in Figure 3.19.

35

<?xml version="1.0"?><amp:KeepAlive srcID="border1_ID" asdID="1"xmlns:amp="http://www.ibm.com/schemas/appliance/management/2.0"/>

Figure 3.17: Example of Contents of KeepAlive XML Message

<?xml version="1.0"?><amp:Update srcID="border3_ID" asdID="3"xmlns:amp="http://www.ibm.com/schemas/appliance/management/2.0">

<amp:WithdrawnServiceRoutes/><amp:AvailableServiceRoutes>

<amp:ServiceRoute serviceID="C"><amp:Origin>IGP</amp:Origin><amp:Path><amp:ASD id="3"/></amp:Path><amp:NextHop><amp:ASD id="3"/></amp:NextHop>

</amp:ServiceRoute></amp:AvailableServiceRoutes>

</amp:Update>

Figure 3.18: Example of Contents of Update XML Message

<?xml version="1.0"?><amp:Update srcID="border1_ID" asdID="1"xmlns:amp="http://www.ibm.com/schemas/appliance/management/2.0">

<amp:WithdrawnServiceRoutes/><amp:AvailableServiceRoutes>

<amp:ServiceRoute serviceID="C"><amp:Origin>IGP</amp:Origin><amp:Path>

<amp:ASD id="1"/><amp:ASD id="3"/>

</amp:Path><amp:NextHop><amp:ASD id="1"/></amp:NextHop>

</amp:ServiceRoute></amp:AvailableServiceRoutes>

</amp:Update>

Figure 3.19: Example of Contents of Update XML Message

36

Figure 3.20: Flowchart for Routing/Forwarding Service Requests Between Autnomous Fed-erations of Enterprise Service Buses

3.5.1 Service Request Forwarding

In this section, we’ve shown examples of how the routing/management protocol

is used to synchronize the state of the distributed registry between interconnected feder-

ations of ESBs. Figure 3.20 below shows how the distributed registry enables the rout-

ing/forwarding of service requests amongst federations. When a request is received, either

directly from a service requestor or forwarded from another ESB node in the deployment,

it is passed to a routing mediation. This routing mediation determines the destination for

this request by consulting the local service registry along with its own locally defined service

connections. If this is the appropriate ESB node for the request (i.e. the service instance

can be directly reached through a mediation flow at this node), the request is passed to the

mediation flow for processing and eventually passed onto the service instance. If the service

request can not be serviced (or should not be serviced, according to policy) within this

ESB deployment, the routing mediation then consults the distributed registry for match-

ing service instances available in the federation to decide where to send the request. If

an appropriate destination is reachable in the federation, the request is sent to the correct

ESB deployment and then forwarded onto the appropriate ESB node which provides con-

nectivity for the particular service being requested. Otherwise, the routing mediation then

consults the distributed registry for matching service instances available in interconnected

federations to decide where to send the request. If a suitable match is found, the request

is forwarded to the boundary node for the desired federation, and routing proceeds as in

the intra-federation case. If this entire process fails, the request is discarded as not being

serviceable.

37

3.6 Conclusions

In this chapter, we proposed a novel method for enabling federations of enterprise

service buses via distributed service registry. Rapid adoption of SOA is causing the size

of ESB deployments to grow, and business-to-business interconnections are becoming more

frequent. We utilize modified versions of Internet protocols to maintain a distributed service

registry which are known to be robust and scalable. Policy-based service advertisements

allow different members of the federation to have different views of available services at a

particular federation member; this allows any desirable topology for the federation.

38

Chapter 4

An Autonomic Service Delivery

Platform

4.1 Introduction

The overarching goal of adopting service oriented architectures (SOA) is to allo-

cate an organization’s computing resources such that they are directly aligned with core

business processes. Implemented correctly, SOAs provide a framework that reuses existing

elements of an IT infrastructure while reducing total cost of ownership and providing a more

flexible and robust environment for the integration of IT and business processes. Services in

SOAs are coarse-grained, discoverable software entities that exist as single instances and in-

teract with consumers, applications and other services via a loosely coupled, message-based

communication model. These properties enable the flexibility of SOAs because they re-

move dependencies on implementation specifics by relying on interactions between services

through standardized interfaces.

The use of standardized interfaces also supports service virtualization, which allows

entities to provide alternate interfaces to the same service instance. This further allows

value-added functionality to be inserted into the flow of a service invocation in a manner

39

transparent to the consumer; similar concepts are being adopted in next-generation IMS

and telecommunication networks. Service virtualization can also provide overload protection

and security benefits, as intermediaries are able to enforce admission control policies and

prevent denial-of-service attacks from reaching an actual service instance.

Loose coupling and service virtualization enable a dynamic and flexible integration

infrastructure where different service providers, each of which are perfect substitutes for one

another, can be chosen at runtime to fulfill service requests. The service selection prob-

lem has been well-addressed in service engineering literature and in dynamic supply chain

management. In both of these research areas, transportation costs between the consumer

and the provider should be considered because they can play a large part in the consumer’s

perception of the overall performance of the service invocation. Dynamic service selection

enables service-oriented supply chain environments to become more agile to changing eco-

nomic and environmental conditions [36]. In general, service systems seek to gain efficiency

by adapting autonomically to changes in the marketplace [37]. With these points in mind,

we postulate that a mapping exists between the electronic services management required in

SOAs and the more tangible supply chain management practices adopted by corporations

today.

In this paper, we propose a novel service delivery platform that optimally routes

service requests from consumers to providers through a network of cooperative intermedi-

aries. The intermediaries will select the “best” service provider for the request based on

weighted criteria such as relative importance of requests (as defined by business policy)

and current congestion observed in the intermediaries and the providers themselves. The

platform seeks to provide optimal flow control and routing of service requests that adapts

autonomically to current conditions observed in the service-oriented environment. This ap-

proach is novel in its goal to effectively maximize the value derived from the underlying

IT resources in a manner proportional to the goals of the business [38]. An instantiation

of such a service delivery platform delivers the promises of SOAs by enabling a dynamic

and robust integration infrastructure that we believe is applicable to both middleware and

next-generation telecommunication systems.

To build the platform, we apply a cross-disciplinary research approach, drawing

insight from the diverse areas of dynamic supply chain management, service engineering,

40

network economics, application-layer networking, and distributed systems to enable an auto-

nomic service delivery platform based on the concept of a service-oriented network. Service-

oriented networking, an emerging paradigm that enables network devices to operate at the

application-layer with features such as offloading, protocol integration, and content-based

routing, is key to instantiating our service delivery platform [39].

The remainder of the chapter is structured as follows: in the following section,

we explicitly propose our service delivery platform and the function it enables. We also

discuss how methodologies from diverse research areas can be integrated to create such a

platform, and we provide a brief review of related literature in service-oriented brokered

architectures, service selection algorithms, and dynamic supply chain management. In

Section 4.3, we present an overview of the analytic framework that is used to provide the

optimal routing and flow control in the platform. In Section 4.4, we provide a simple

example and simulation results which show how service-differentiated results are obtained

by the solution to the optimization problem.

4.2 Architecture of Service Delivery Platform

4.2.1 Overview

In this section, we propose our autonomic service delivery platform that explicitly

links the value extracted from IT resources to the business processes they support within an

enterprise. The platform is composed of service consumers, service-oriented intermediaries,

and service providers. The platform provides:

• A fully distributed, content-based, and optimal routing infrastructure

• Flexible and optimal selection of service providers that can be based on various system-

level goals (e.g. end-to-end delay, proximity, etc.)

• Optimal flow control of service requests

The novelty of our proposal arises from the integration of several well-established

theoretical and practical techniques from networking, microeconomics, and service-oriented

41

Consumer

Network of Intermediaries

LogicalDestination

Provider #1

Provider #2

Consumer

Figure 4.1: Example of Service-Oriented Network Topology with Multiple Service Providers

computing that together form a fully-distributed service delivery platform. The core compo-

nent that enables the service delivery platform is a utility-based cooperative service routing

protocol. The objective of this protocol is to route requests such that the weighted “social

welfare” of the system is maximized. It disseminates current pricing and utility informa-

tion amongst service intermediaries in the service delivery platform to cause the system to

optimally forward and rate limit service requests. The system administrator defines the

requisite utility functions on a per class-of-consumer basis, rather than inferring them from

consumers who can be untruthful in their appraisal of services. In this way, we avoid the

selfish nature of consumers and subsequently the “tragedy of the commons” that can result

from such a situation.

4.2.2 Key Assumptions

To build our service delivery platform, we make several key assumptions:

• We reuse a graph-based formulation proposed in [40], as illustrated in Figure 4.1.

In this model, we add a logical destination node to the topology that is connected

to all possible providers of a semantically equivalent service over zero-cost virtual

links. We also assume that a semantic matching algorithm exists a priori that can

be used to select available paths through the network topology to fulfill a consumer’s

42

request. These assumptions allow us to directly apply existing optimal multipath

routing algorithms to our architecture and use pricing information as the final decision

variable to make a forwarding decision for a given request.

• We assume that consumers only submit their service request to a single intermediary.

This delegates the service selection decision to an intermediary with current system

state to make an optimal forwarding decision.

• Service providers advertise relevant metrics to all intermediaries that act as a “last

hop” in the service-oriented network before the provider. The intermediaries that

receive metrics from a provider will determine the current price for the service and

propagate that price throughout the network. This limits the scope for distribution

of metrics from service providers to the delivery platform.

• Since the platform assumes global knowledge of per-service utility functions and

trusted relationships between intermediaries such that all nodes cooperate to opti-

mally achieve common goals, it is assumed that the delivery platform exists within a

single autonomous system.

4.2.3 Methodologies Integrated in the Platform

The service delivery platform is based on the integration of several key method-

ologies such as content-based routing, optimal routing and flow control theory, network

economics, and congestion pricing. In the subsections below, we give a brief overview of

relevant issues related to each the methodologies in our service delivery platform.

Content-Based Routing

While previously discouraged because it violates the networking end-to-end prin-

ciple, the idea of using network intermediaries to provide value-added application-aware

function in the network fabric has recently been embraced [41]. Similar to active and over-

lay networks in its objective, service-oriented networking challenges the previous assumption

that implementing application-awareness in the network fabric is too costly and complex

43

[39]. Due to advances in hardware, software, and networking technologies, intermediaries

are able to understand data encoded in XML and legacy formats, act upon that content to

enforce QoS or security policies, transform the data into an alternate representation, and/or

make content-based routing decisions.

We directly leverage the content-based routing function provided by a service-

oriented network to enable request forwarding in our service delivery platform. Content-

based routing algorithms typically apply rules against some portion of a service request

(header or content) to extract attributes. These attributes are used to semantically match

the service request to possible providers in the service-oriented network topology.

In conventional network layer routers and switches, the amount of resources per

packet is approximately constant; this greatly simplifies capacity planning and network de-

sign. However, we argue that the resources required per request in application-layer devices

is not constant, and furthermore, that building accurate characterizations of application-

layer workload is a difficult and possibly intractable problem. Instead of trying to develop

such a model, we believe that a measurement-driven, autonomic approach to resource allo-

cation based on metrics such as CPU or memory utilization is a more elegant and feasible

solution. This logic can also be applied to resource allocation problems of service providers,

as done in [42], to define the effective capacity of a resource, as defined as cj in the formu-

lation shown in Section III.

Optimal Routing & Flow Control

In addition to considering the content of requests, our service delivery platform

also incorporates the observed state of the system into its optimal routing algorithm. In the

seminal paper [43], a distributed algorithm to an optimal minimum delay routing problem is

presented. The algorithm populates routing tables with weights that represent the fraction

of incoming traffic that should be forwarded to the neighboring nodes in the network. The

solution reveals that these weights are a function of the measured marginal delay on the

link to each neighbor. An extension of this work is presented in [44], where the restrictions

in [43] of quasi-stationary traffic, synchronization of nodes, and knowledge of the aggregate

traffic demand at each node are removed. It is also shown how a near-optimal multipath

44

routing algorithm can be implemented in a distance-vector framework while maintaining

loop-free routes at every instant.

In addition to using optimal routing, we must ensure that the rate of incoming

requests to a particular node in our service delivery platform is throttled appropriately.

We can achieve this by integrating optimal flow control into our architecture. A proposed

method for integrating a utility-maximization problem and optimal flow control is presented

in [45], where the optimal routing and flow control problems are solved simultaneously while

observing capacity constraints. The issue of fairness in such an algorithm is addressed in

[46]. By definition, a strictly utility-based algorithm will converge to a Pareto-optimal

equilibrium, which is logically equivalent to the concept of max-min fairness. However, we

believe that a flow control mechanism that implements per-service-weighted proportional

fairness is more appropriate for our platform.

The integration of a distributed, loop-free, and optimal multipath routing and

flow control algorithm is essential to the robustness and scalability of our service delivery

platform. Since forwarding costs are determined by the sum of the congestion price of the

intermediary in question and the price as advertised by the next hop (an intermediary or a

provider), we exploit the additive path cost property of the underlying economic framework

to build the requisite service routing protocol.

Network Economics

Microeconomics offers a well-developed theory on the subject of rational choice

in multi-agent environments; utility functions and price are natural ways to express the

common tradeoffs in such systems. Microeconomic models have been extensively applied

to many different engineering problems; for example, network economics are used in [40]

as a method to solve dynamic supply chain management problems. The solutions that are

yielded from these methods have many desirable properties, such as provable convergence

to a Pareto-optimal equilibrium, in which no other solution exists that could increase the

benefit of a user without reducing the benefit of another user. A comprehensive review of

how economic theory can be applied to various networking problems is found in [47].

In our architecture, we incorporate the economic concept of social welfare maxi-

mization when formulating our optimization problem for the platform, as seen in (4.1) in

45

Section 4.3. A key distinction of our work, as compared to prior attempts in the literature,

is that our formulation does not rely on the perceived or advertised utility from consumers;

rather, we explicitly link the utility of services to the benefit that a corporation derives

from providing the IT infrastructure. The benefits of this distinction are two-fold; first, it

allows us to avoid restrictive assumptions about the explicit knowledge and/or validity of

the utility functions for the system. Second, it delivers a link between IT resources and the

benefits that are derived from them, which is the premise for adopting SOAs.

Congestion Pricing

The law of supply and demand states that as the available quantity of a resource

decreases, the unit price should increase to reflect the scarcity of the resource. Congestion is

defined in economics as a negative market externality, which occurs when a participant in a

market can make a decision which adversely affects other participants in the market without

penalty. By integrating the current level of congestion observed into the total price paid

to obtain a service, we “internalize the externality” and successfully manage the tradeoff

between idle resources and degradation of service [47].

Congestion pricing was first proposed in [48] as a basis for welfare economics and

has been subsequently been applied to many engineering disciplines [49, 50]. The use of

congestion-pricing resources has been investigated extensively in the networking literature

in an attempt to address resource allocation problems [51]. We apply the concept of con-

gestion pricing to balance the current state of the underlying network conditions and the

performance characteristics of service providers and network intermediaries in order to op-

timally route requests [47]. This is represented by the term f(xs, γf , zf ) in (4.1), shown in

Section 4.3.

The notion of “split-edge” pricing was proposed in [52]. In this model, prices are

determined locally and only reflect prices from onward networks and providers in providing

the service; however, pricing information is consolidated at each step, whether it be an

intermediate broker or the actual provider. Split-edge pricing is analogous to additive path

cost in next-hop routing algorithms, such as a distributed Bellman-Ford algorithm, where

knowledge of the full topology and paths through the network are not required in order

46

to make minimum cost routing decisions. We leverage split-edge pricing in the distributed

solution to the optimization problem described in the next section.

We believe that the combination of “split-edge” and congestion pricing provides an

intuitive and scalable method to provide congestion control in our service delivery platform.

Our architecture is flexible in such a way that it is configurable for administrators to set

congestion-based prices for invoking transport services, the services of an intermediary, and

the desired service at a particular provider, or any subset of the prices therein. A description

of how to set a congestion price for networked applications is presented in [53], and a realistic

system built on this premise is proposed in [54].

An inclusive overview of pricing schemes for networks is provided in [55]. However,

the majority of the work in this area is theoretical in nature, and does not discuss issues

with practical implementations of such systems.

4.2.4 Related Work in Service Systems

There are several previous attempts that have been made towards developing bro-

kered architectures that connect service consumers to service providers. Several proposals

have been made to create “service overlay networks” with the goal of applying advances in

overlay network research to the services layer. In [56], an open service market architecture

is presented that aims to balance load across multiple service providers by using a network

of proxies configured by an external centralized “trader” that computes the optimal routes

for service requests. This architecture does not consider the current state of the proxies

when making routing decisions. The authors of [57] propose a management overlay for

Web Services based on interconnected service intermediaries, but do not address the service

selection or routing problems.

Several previous efforts have focused on using overlay methodologies to provide

better end-to-end quality of service for requests in the network, by provisioning bandwidth

or selecting the best path through the network based on available bandwidth [58, 59, 60].

The integration of bandwidth together with other QoS metrics into optimizations in a

service overlay network is presented in [61]. There have also been attempts to develop a

service overlay network based upon network economics [62]. While the overarching goals of

47

the work in this area are similar to ours, the work assumes that nodes of the service overlay

network are inherently selfish and non-cooperative; this distinction has a dramatic effect

on the underlying economic framework they create, which makes their work inapplicable to

the problems we address. A useful review of brokered service-oriented systems is shown in

[63].

Service selection algorithms utilize rational decision making processes that are used

to decide which service instance to invoke according to some predefined criteria. A common

component of such algorithms is the concept of a QoS registry [64]. A multi-agent approach

to distributed service selection is proposed in [65]; however, the underlying transportation

costs of the network are not considered in the model. A network-sensitive service selection

algorithm is proposed in [66], but it does not incorporate the current state of the service

providers or the intermediaries in the selection decision.

The concepts of brokered architectures and service selection are also addressed

in the supply chain management literature. There is an increasing amount of literature

discussing the application of multi-agent systems to dynamic supply chain management

problems [67]. Transportation and handling costs in a graph-theoretic framework are inte-

grated with traditional supply chain analysis in [40] and the references therein. A combined

service selection and service pricing framework for supply chain managers is discussed in

[68]. Distributed pricing issues in supply chains are addressed in [69].

4.3 Analytic Framework of Service Delivery Platform

The analytic foundation for our service delivery platform comes from the merger of

the key methodologies described in the previous section and the concept of network utility

maximization [70].

In this section, we reuse the notation and closely follow the derivation as presented

in [71, 72].

Consider a service-oriented network with resources that consist of intermediaries

and providers, denoted by J = 1, 2, . . . , J . Let cj be the capacity of resource j ∈ J and

c = [c1, c2, . . . , cJ ]T. Let S = 1, 2, . . . , S be the set of sources (consumers). Each source s has

Ks available loop-free paths from the source to the logical destination node corresponding

48

to the semantic service that is being consumed by a source. Let Hs be a J × Ks 0 − 1

matrix which describes the mapping of resources on paths for particular sources; that is,

Hsji =

1, if path i of source s uses resource j

0, otherwise

Let Hs be the set of all columns of Hs that represent all available paths to source s under

single-path routing. Define the J ×K matrix H as

H =[H1, H2, . . . ,HS

]where K :=

∑sK

sH defines the topology of the network.

Let ws be a Ks × 1 vector where the ith entry represents the fraction of s’s flow

on its ith path such that

wsi ≥ 0 ∀i, and 1Tws = 1

where 1 is a vector of an appropriate dimension with the value 1 in every entry. We allow

wsi ∈ [0, 1] for multipath routing. Collect the vectors ws, s = 1, . . . , S into a K × S block

diagonal matrix W . Let W be the set of all such matrices corresponding to multipath

routing as

W ={W |W = diag

(w1, . . . , wS

)∈ [0, 1]K×S ,1Tws = 1

}As mentioned above, H defines the set of loop-free paths available to each source,

and also represents the network topology. W defines how the sources split the load across

the multiple paths. Their product defines a J × S routing matrix R = HW that specifies

the fraction of s’s flow at each resource j. The set of all multipath routing matrices is:

R = {R |R = HW,W ∈ W}

A multipath routing matrix in R is one with entries in the range [0, 1]:

Rjs =

> 0, if resource j is in a path of source s

= 0, otherwise.

The path of source s is denoted by rs = [R1s, . . . , RJs]T, the sth column of the routing

matrix R.

49

We wish to consider the following optimization problem:

maxR∈R

maxx≥0

∑s∈S

Us(xs)−∑f∈Fs

f(xs, γf , zf )

(4.1)

s.t. Rx ≤ c (4.2)

(4.1) optimizes “social welfare” by maximizing utility over both source rates and routes.

However, (4.1) is not a convex problem because the feasible set specified by Rx ≤ c is

generally not convex. We now transform the problem by defining the Ks × 1 vectors ys in

terms of the scalar xs, and the Ks × 1 vectors ws as the new variables:

ys = xsws (4.3)

The mapping from (xs, ws) to ys is one-to-one; the inverse of (4.3) is xs = 1Tys and

ws = ys/xs.

Now we change the variables in (4.1) and (4.2) from (W,x) to y, by substituting

xs = 1Tys and Rx = HWx = Hy, obtaining the equivalent problem:

maxy≥0

∑s∈S

Us

(1Tys

)−∑f∈Fs

f(1Tys, γf , zf )

(4.4)

s.t. Hy ≤ c. (4.5)

Provided the functions Us(·) and f(·) are strictly concave, this is a strictly concave problem

with a linear constraint, and therefore has no duality gap [73].

4.3.1 Solution of Problem

To find a distributed algorithm that solves (4) & (5), we inspect the problem

through its Lagrangian dual. We form the following Lagrangian:

L(y, p) =∑s∈S

Us

(1Tys

)−∑f∈Fs

f(1Tys, γf , zf )

− J∑j=1

pj(Hy − c) (4.6)

where p =[p1, p2, . . . , pJ

]T is a J × 1 vector of Lagrange multipliers associated with the

capacity constraint on resource j. Letting psi =∑J

j=1Hsjip

j and ps = [ps1, . . . , psKs ]. We

50

continue by formulating the objective function of the dual problem as:

D(p) = maxy≥0

L(y, p)

= maxy≥0

∑s∈S

Us

(1Tys

)−∑f∈Fs

f(1Tys, γf , zf )− psys

+J∑

j=1

pjc(4.7)

Since D(p) is separable in s, we can swap the order of the maximization and the summation,

forming the following equivalent equation:

D(p) =∑s∈S

maxy≥0

Us

(1Tys

)−∑f∈Fs

f(1Tys, γf , zf )− psys

+J∑

j=1

pjc

=∑s∈S

Bs(ys, ps) +J∑

j=1

pjc

(4.8)

where Bs(ys, ps) = maxy≥0

[Us

(1Tys

)−∑

f∈Fsf(1Tys, γf , zf )− psy

s].

The dual problem of (4) & (5) corresponds to minimizing D over the dual variables

p, i.e.

minp≥0

D(p)

Since the objective function of the primal problem (4) & (5) is strictly concave, the dual

problem is always differentiable. The gradient of D is:

δD

δpj= cj −

∑s∈S

Ks∑i=1

Hsjiy∗si

where y∗si comes from the solution of Bs(ys, ps).

Using gradient descent iterations on the dual variables (herein referred to as con-

gestion prices) yields the following equation:

pj(t+ 1) =

[pj(t) + βj

(cj −

∑s∈S

Ks∑i=1

Hsjiy

si (t)

)]+

(4.9)

where ysi (t) is the solution of the following optimization problem at time t:

ysi (t+ 1) = max

ysi≥0

Us

(1Tys

)−∑f∈Fs

f(1Tys, γf , zf )− ysi

J∑j=1

pj(t)Hsji

(4.10)

The joint solution of (4.9) and (4.10) completes the distributed algorithm that

solves (4.1). The resources update the rates of each source ysi by based on explicit feedback

51

from downstream resources via congestion prices pj . Each resource maximizes the utility

for source s while balancing the price of placing load on a path i. The path price is the

product of the source rate with the price per load for path i (computing by summing pj over

all resources in the path). The assignment of the rates ysi at the resources determines the

total traffic that traverses each resource. The resulting load through each resource serves

as implicit feedback that is used to compute the congestion price pj .

Convergence of Gradient Descent Algorithm

Convergence of this algorithm is presented in [71, 74, 75], as it is classified as

a separable, strictly concave nonlinear optimization problem with linear constraints; the

convergence of a gradient projection algorithm applied to such a problem is well-known for

sufficiently small step sizes βj > 0.

4.4 Simulation

In this section, we provide results from an experiment which exhibits the basic

characteristics of the autonomic service delivery platform.

4.4.1 Overview

Suppose we have an e-commerce website, where we have two services. The first

service is called Browse, which provides our consumers the ability to search our inventory

for items they might desire to purchase. The other service is called Order, where consumers

complete their purchases and subsequently the order is processed and shipped to the con-

sumer. While traffic destined for the “Browse” service is essential to the business (without it

we can not process any Order traffic), traffic destined to the Order service is more valuable

to the business since this actually drives revenue for our business.

In developing a mapping from this business model to our service delivery platform,

first we must choose utility functions for our services; these functions directly connect

the value derived from the service to the underlying importance in the service selection

52

and routing algorithms which deliver requests from consumer to provider. We select the

following function for the Browse service traffic:

Us(xs, z) = x1/5s

Theorem 4.4.1. The function x1/5s is concave ∀x ≥ 0.

Proof of Theorem 4.4.1. To show that x1/5s is concave in the domain <+, we must show

that its second derivative, d2

d2xx

1/5s is negative for all values in the domain.

d2

d2xx1/5

s =d

dx

(d

dxx1/5

s

)=

d

dx

(1

5(x4/5s )

)= − 4

25(x9/5s )

∀x ∈ <+, d2

d2xx

1/5s is negative as shown above. Therefore, x1/5

s is concave in the domain

<+.

This is useful from a business perspective because the true value derived from the

Browse traffic is small regardless of the actual amount of traffic supported.

Next, we choose a utility function for our Order traffic. Since this service is more

valuable than the Browse service (due to its direct link to revenue), we chose a utility

function that, while being concave, grows much faster than the Browse utility function.

We select the following function for the Order service traffic:

Us(xs, z) = x4/5s

Theorem 4.4.2. The function x4/5s is concave ∀x ≥ 0.

Proof of Theorem 4.4.2. To show that x4/5s is concave in the domain <+, we must show

that its second derivative, d2

d2xx

4/5s is negative for all values in the domain.

d2

d2xx4/5

s =d

dx

(d

dxx4/5

s

)=

d

dx

(4

5(x1/5s )

)= − 4

25(x6/5s )

∀x ∈ <+, d2

d2xx

4/5s is negative as shown above. Therefore, x4/5

s is concave in the domain

<+.

53

0 100 200 300 400 500 600 700 800 900 10000

50

100

150

200

250

x s

Us(x

s)

Utility as a Function of Load

xs0.8

xs0.2

Figure 4.2: Plot of Utility Functions for Browse and Order Services

A graphical representation of these utility functions is shown in Figure 4.2.

We define a sample topology to analyze the problem and the solution. There are

three intermediaries, and only one provider of each service. Customers of each type of

service enter the SON through each intermediary. The topology is displayed in Figure 4.3.

To showcase the ability of the service delivery platform to adapt to the offered

load for each service based on their value to the system, we use the following input traffic,

as shown in Figure 4.4.

The goal of this experiment is to show the ability of the SDP to adapt the amount

of service requests permitted into the network based on their relative value to the system

as a whole.

54

Browse Service

Order Service

Consumers

Node 1

Consumers

Node 3

Consumers

Node 2

Figure 4.3: Logical Diagram of Topology of the Service-Oriented Network

0 50 100 150 200 250 300 350 400 450 5000

50

100

150

200

250

300

350

time

x s,i

Total BrowseBrowse

Node 1

BrowseNode 2

BrowseNode 3

Total OrderOrder

Node 1

OrderNode 2

OrderNode 3

Figure 4.4: Plot of Offered Load for Browse and Order Services

55

0 50 100 150 200 250 300 350 400 450 50050

100

150

200

250

300

time

requ

ests

/sec

Total Throughput of Service Requests

Browse TrafficOrder Traffic

Figure 4.5: Plot of Supported Load for Browse and Order Services

4.4.2 Results

As seen in Figure 4.4, at time 200 the Order traffic increases beyond the capacity

of the nodes. We expect that the Browse traffic should be reduced in order to support the

more valuable Order traffic.

Figure 4.5 shows the total amount of requests per second carried by the SDP. At

time 200 the Order traffic increases beyond the capacity of the nodes. As we expected,

the Browse traffic is reduced in order to support the more valuable Order traffic, since the

utility of carrying the Order traffic is greater than that of the Browse traffic.

Figure 4.6 displays the amount of flow placed on each path through the SDP. Only

the Browse traffic offered by Source 2 is decreased between time 200 and time 300, because

it is offered directly to Node 2, which is received greater aggregate load than it can handle.

All other browse traffic does not flow through Node 2, and all other nodes have sufficient

capacity to handle all requests being processed through them.

Finally, Figure 4.7 shows the total utility gained by forwarding service requests

through the SDP. As expected, the utility received during the time interval of increased

Order requests grew due to the forwarding of added Order traffic.

56

0 50 100 150 200 250 300 350 400 450 5000

50

100

150

time

requ

ests

/sec

Per Source Flow Throughput

Source 1 BrowseSource 2 BrowseSource 3 BrowseSource 1 OrderSource 2 OrderSource 3 Order

Figure 4.6: Plot of Per-Flow Supported for Browse and Order Services

0 50 100 150 200 250 300 350 400 450 50040

50

60

70

80

90

100

time

Agg

rega

te U

tility

Aggregate Utility vs. Time

Figure 4.7: Total Utility Gained from Browse and Order Services

57

4.5 Conclusions

In this chapter, we proposed a novel autonomic service delivery platform for

service-oriented network environments. The framework of the platform is based on the

methodologies of content-based routing, network economics, congestion pricing, and opti-

mal routing and flow control. With a direct link to the business value derived from a service,

the service delivery platform maximizes the value derived from underlying IT resources. We

believe that our architecture provides exciting new multidisciplinary research opportunities

in service engineering.

Some future issues to address include investigating efficient methods to estimate

the derivatives of the congestion prices f(xs, γf , zf ) in (4.1). Another concern exists with

the assumptions regarding the concavity of utility functions; currently this assumption is

required in order for welfare economic systems to converge. Further work similar to [76]

is necessary to relax this assumption and broaden the applicability of our work to real-

time services. Finally, we believe that further investigation into the interactions between

autonomous systems could have important effects in business-to-business interactions in

such an instantiation of our distributed service delivery platform.

58

Chapter 5

Conclusions

5.1 Conclusions

In this report, we formally proposed Service-Oriented Networking as an emerging

middleware and telecommunications architecture. We discussed the challenges both in

building SON devices as well as in interconnecting the devices together to form a true

networked system. We continued by discussing large-scale service-oriented networks by

explicitly describing a use case for SON, federations of enterprise service buses. We described

how federations can be enabled by a distributed service registry, and provided details and

examples of two protocols, based upon Internet routing protocols, that enable a robust,

scalable and dynamic infrastructure. Finally, we presented our autonomic service delivery

platform. The goal of this platform is to optimally route requests from service consumers to

providers. We provided details of the underlying utility-based analytical framework, as well

as results from an initial experiment which shows the ability of the framework to optimally

route and throttle load under resource constraints.

SON provides exciting new multidisciplinary research opportunities in service-

oriented computing, hardware, software, and networking. The desire for large scale feder-

ated service-oriented systems is growing rapidly; our work is some of the initial contributions

in this area. Our autonomic service delivery platform provides a direct link from business

59

value of a service to its priority in the service-oriented network; it is also the first known

work to apply the concepts of network utility maximization and multipath routing to the

services layer. It is noteworthy that similar cross-layer, utility-oriented algorithms are being

proposed as the approach for NSF’s Future Internet Design (FIND) initiative, which is a

clean-slate approach to redesign the Internet.

5.2 Remaining Work

In this section, we provide an overview of five main areas that we feel are the best

opportunities to make significant contributions and continue the research presented in this

report towards completion of the PhD.

5.2.1 Relaxing Assumptions on Concavity of Utility & Cost Functions

In formulating the optimization problem, we have made the assumption that the

utility and cost functions are concave and continuous. In realistic economic models, utility

functions that are concave are said to be elastic; that is, consumers are limited by the law

of diminishing returns. However, in order for our service delivery platform to support all

types of services, we must investigate the limitations of this assumption. It is important to

analyze many different types of utility and cost functions, and how their properties change

the solution to the optimization problem. Figure 5.1 shows an example of several relevant

types of utility functions.

5.2.2 Multipath XML-Based Service Routing Protocols

In order to implement the distributed optimization algorithms in a real instantia-

tion of the service delivery platform, a mechanism to disseminate relevant load and pricing

information amongst nodes in the SDP. In this light, we propose adapting existing multipath

routing algorithms in the literature, such as [44, 77] to share relevant routing information.

[44] takes a distance vector approach to solving the multipath routing problem while main-

taining loop-free paths from every source to destination at every instant. It relies on the

60

Figure 5.1: Various Types of Utility Functions

concept of diffusing computations, which is also utilized in the popular single-path rout-

ing protocol EIGRP. [77] is a link-state approach to the same problem. In utilizing these

algorithms, it would be beneficial to create an XML-based version of both protocols, and

compare their relative overheads and convergence properties.

5.2.3 Minimizing Optimization Computations using Wavelet-Based Traf-

fic Prediction

In order to effectively manage service traffic in a SON, it is important to minimize

the impact of statistics collection and management functionality on the core function of

a service intermediary. A method to minimize the amount of computational resources

required by the solution method of the optimization method we proposed in Chapter 4 is to

utilize traffic prediction as a trigger to re-run the solution algorithm. If the aggregate input

rate of service requests is relatively constant, the solution will not be significantly different

for minor variations in the input rate. Therefore, it could be seen as a tradeoff to accept

a minimally sub-optimal solution for a decreased amount of optimization computations.

Figure 5.2 gives an example of how thresholds are set and when an optimization algorithm

61

Figure 5.2: Using Traffic Prediction Algorithms to Minimze Optimization Calculations

would be run to generate a new solution. In order to implement such a system, a change-

detection algorithm would be applied to relevant metrics (such as the aggregate input rate

of a particular service), and the optimization algorithm would be triggered to run when

a threshold is reached. Wavelets are a well known change-detection methodology that we

could utilize in instantiating this idea.

5.2.4 Measurement of Effective Capacity of Resources

As seen in Equation 4.5, the capacity of all SDP nodes are needed in order to

compute the optimal rates and routes for service requests to take in the SDP. The capacity

is assumed to be in units of requests per second; however, in general, the capacities of

intermediaries and providers are not defined in requests per second. Rather, they are

typically defined in terms of available CPU cycles and memory. In certain cases, a mapping

is needed to convert units to solve the optimization problem. An example of such a mapping

is presented in [42]; however, they use simple linear regression to make this mapping. More

sophisticated statistical techniques such as response surface modeling and metamodeling

may yield better mappings and subsequently better results to the optimization algorithm.

62

5.2.5 Investigate Alternative Solution Methods

In Section 4.3.1, we presented a solution to the optimization problem posed in

Equations 4.4 and 4.5. This solution was based on a well-known method called gradient

descent algorithm. However, this is not the only solution method known for nonlinear

constrained convex optimization problems. Another example of a solution methodology is

sequential quadratic programming [78], where several algorithms which iteratively compute

solutions to quadratic programs which converge to the same optimal solution of the original

nonlinear constrained optimization problem. We believe that more investigation into effi-

cient solution methods to the network utility maximization problem proposed in Chapter 4

is a useful extension of our research.

63

Bibliography

[1] M. Endrei, J. Ang, A. Arsanjani, S. Chua, P. Comte, P. Krogdahl, M. Luo, and T. Newl-

ing, Patterns: Service-Oriented Architecture and Web Services. IBM Redbooks, April

2004.

[2] M. P. Singh and M. N. Huhns, Service-Oriented Computing: Semantics, Processes,

Agents. John Wiley & Sons, Ltd., 2005.

[3] M. N. Huhns and M. P. Singh, “Service-oriented computing: Key concepts and princi-

ples,” in Proceedings of IEEE Internet Computing, vol. 9, Jan-Feb 2005, pp. 75–81.

[4] D. L. Tennenhouse, J. M. Smith, W. D. Sincoskie, D. J. Wetherall, and G. J. Minden,

“A Survey of Active Network Research,” IEEE Commun. Mag., vol. 35, no. 1, pp.

80–86, 1997.

[5] A. T. Campbell, H. G. D. Meer, M. E. Kounavis, K. Miki, J. B. Vicente, and D. Villela,

“A survey of programmable networks,” SIGCOMM Comput. Commun. Rev., vol. 29,

no. 2, pp. 7–23, 1999.

[6] D. Wetherall, J. Guttag, and D. Tennenhouse, “ANTS: A Toolkit for Building and Dy-

namically Deploying Network Protocols,” in Proceedings of IEEE Open Architectures

and Network Programming, 1998, pp. 117–129.

[7] D. Wetherall, “Active network vision and reality: Lessons from a capsule-based sys-

tem,” in SOSP ’99: Proceedings of the Seventeenth ACM Symposium on Operating

Systems Principles (SOSP), 1999.

64

[8] J. T. Moore, M. W. Hicks, and S. Nettles, “Practical programmable packets,” in Pro-

ceedings of IEEE INFOCOM 2001, 2001, pp. 41–50.

[9] S. Banerjee, B. Bhattacharjee, and C. Kommareddy, “Scalable application layer mul-

ticast,” in SIGCOMM ’02: Proceedings of the 2002 conference on Applications, tech-

nologies, architectures, and protocols for computer communications. New York, NY,

USA: ACM Press, 2002, pp. 205–217.

[10] XPath, World Wide Web Consortium, http://www.w3.org/TR/xpath.

[11] XSLT, World Wide Web Consortium, http://www.w3.org/TR/xslt.

[12] G. Cuomo, “IBM SOA “on the edge,” in SIGMOD ’05: Proceedings of the 2005 ACM

SIGMOD International Conference on Management of Data, 2005, pp. 840–843.

[13] Application Oriented Networking, Cisco Systems, 2005, http://www.cisco.com/en/US/

products/ps6455/index.html.

[14] DataPower, IBM, 2006, http://www-306.ibm.com/software/integration/datapower/.

[15] G. Zhang, “Building a Scalable Native XML Database Engine on Infrastructure for a

Relational Database,” in Proceedings of Second International Workshop on XQuery Im-

plementation, Experience and Perspectives, in cooperation with ACM SIGMOD/PODS,

2005.

[16] M. Welsh, D. Culler, and E. Brewer, “SEDA: An architecture for well-conditioned,

scalable internet services,” in SOSP ’01: Proceedings of the Eighteenth ACM Sympo-

sium on Operating Systems Principles, 2001, pp. 230–243.

[17] M. Welsh and D. Culler, “Adaptive overload control for busy internet servers,” in

Proceedings of the 4th USENIX Conference on Internet Technologies and Systems

(USITS’03), 2003.

[18] M.-T. Schmidt, B. Hutchinson, P. Lambros, and R. Phippen, “The Enterprise Service

Bus: Making service-oriented architecture real,” IBM Systems Journal, vol. 44, 2005.

65

[19] C. Nott and M. Stockton, “Choose an ESB topology to fit your business model,” in

IBM developerWorks, 2006.

[20] P. Rompothon and T. Senivongse, “A Query Federation of UDDI Registries,” in 1st In-

ternational Symposium on Information and Communication Technologies, ACM, 2003.

[21] Z. Chen, C. Liang-Tien, B. Silverajan, and L. Bu-Sung, “UX - An Architecture Pro-

viding QoS-Aware and Federated Support for UDDI,” in Proceedings of IEEE Inter-

national Conference on Web Services, 2003.

[22] L. Yin, H. Zingli, Z. Futai, and M. Fanyuan, “eDSR: A Decentralized Service Registry

for e-Commerce,” in IEEE International Conference on e-Business Engineering, 2005.

[23] S. Banerjee, S. Basu, S. Garg, S. Garg, S.-J. Lee, P. Mullan, and P. Sharma, “Scal-

able Grid Service Discovery Based on UDDI,” in Proceedings of the 3rd international

workshop on Middleware for grid computing, 2005, pp. 1–6.

[24] T. Pilioura, G.-D. Kapos, and A. Tsalgatidou, “Seamless Federation of Heterogeneous

Service Registries,” in Proceedings of 5th International Conference on E-Commerce

and Web Technologies, 2004, pp. 86–95.

[25] X. Gu, K. Nahrstedt, and B. Yu, “SpiderNet: An Integrated Peer-to-Peer Service

Composition Framework,” in IEEE International Symposium on High Performance

Distributed Computing, 2004.

[26] L. Baresi and M. Miraz, “A Distributed Approach for the Federation of Heterogeneous

Registries,” in Proceedings of ICSOC, 2006.

[27] M. Giordano, “DNS-Based Discovery System in Service Oriented Programming,” in

Proceedings of Advances in Grid Computing - EGC, 2005, pp. 840–850.

[28] A. Jagatheesan and S. Helal, “Sangam: Universal Interop Protocols for E-Service Bro-

kering Communities using Private UDDI Nodes,” in IEEE Symposium on Computers

and Communications, 2003.

[29] T. Koponen and T. Virtanen, “A Service Discovery: A Service Broker Approach,” in

37th Hawaii International Conference on System Sciences, 2004.

66

[30] N. Limam, J. Ziembicki, R. Ahmed, Y. Iraqi, D. T. Li, R. Boutaba, and F. Cuervo,

“OSDA: Open Service Discovery Architecture for Cross-domain Service Discovery,” in

2nd International Workshop on Next Generation Networking Middleware, 2005.

[31] M. Walfish, H. Balakrishnan, S. Shenker, K. Lakshminarayanan, S. Ratnasamy, and

I. Stoica, “A Layered Naming Architecture for the Internet,” in Proceedings of ACM

SIGCOMM, 2004.

[32] J. Chandrashekar, Z.-L. Zhang, Z. Duan, and Y. T. Hou, “Service Oriented Internet,”

in Proceedings of International Conference on Service-Oriented Computing, 2003.

[33] R. Ahmed, R. Boutaba, F. Cuervo, Y. Iraqi, T. Li, N. Limam, J. Xiao, and J. Ziembicki,

“Service Naming in Large-Scale and Multi-Domain Networks,” IEEE Communications

Surveys & Tutorials, vol. 7, no. 3, pp. 38–54, 2005.

[34] J. Moy, “OSPF Version 2,” RFC2328, April 1998.

[35] Y. Rekhter, T. Li, and S. Hares, “A Border Gateway Protocol 4 (BGP-4),” RFC4271,

January 2006.

[36] H. Bastiaansen and P. Hermans, “Managing Agility through Service Orientation in an

Open Telecommunication Value Chain,” IEEE Commun. Mag., pp. 86–93, October

2006.

[37] J. Spohrer, P. Maglio, J. Bailey, and D. Gruhl, “Steps Toward a Science of Service

Systems,” IEEE Computer, pp. 71–77, 2007.

[38] G. Tesauro, D. M. Chess, W. E. Walsh, R. Das, A. Segal, I. Whalley, J. O. Kephart,

and S. R. White, “A Multi-Agent Systems Approach to Autonomic Computing,” in

Proceedings of the 3rd International Joint Conference on Autonomous Agents and Mul-

tiagent Systems (AAMAS), 2004, pp. 464–471.

[39] R. D. Callaway, A. Rodriguez, M. Devetsikiotis, and G. Cuomo, “Challenges in Service-

Oriented Networking,” in Proceedings of IEEE GLOBECOM, 2006.

[40] A. Nagurney and J. Dong, Supernetworks. Edward Elgar Publishing, 2002.

67

[41] M. Walfish, J. Stribling, M. Krohn, H. Balakrishnan, R. Morris, and S. Shenker, “Mid-

dleboxes No Longer Considered Harmful,” in Proceedings of 6th Symposium on Oper-

ating Systems Design and Implementation, 2004, pp. 215–230.

[42] G. Pacifici, W. Segmuller, M. Spreitzer, and A. Tantawi, “Dynamic Estimation of CPU

Demand of Web Traffic,” in 1st Int’l Conf. on Performance Evaluation Methodologies

and Tools, October 2006.

[43] R. G. Gallager, “A Minimum Delay Routing Algorithm Using Distributed Computa-

tion,” IEEE Trans. Commun., vol. 23, pp. 73–85, 1977.

[44] S. Vutukury and J. Garcia-Luna-Aceves, “MDVA: A Distance-Vector Multipath Rout-

ing Protocol,” in Proceedings of IEEE INFOCOM, 2001.

[45] F. Kelly, “Charging and Rate Control for Elastic Traffic,” in Proceedings of European

Transactions on Telecommunications, vol. 8, January 1997, pp. 33–37.

[46] W.-H. Wan, M. Palaniswami, and S. H. Low, “Application-Oriented Flow Control:

Fundamentals, Algorithms, and Fairness,” IEEE/ACM Trans. Networking, vol. 14,

no. 6, pp. 1282–1291, December 2006.

[47] C. Courcoubetis and R. Weber, Pricing Communication Networks. John Wiley &

Sons Ltd., 2003.

[48] A. Pigou, The Economics of Welfare. Macmillan, London, 1920.

[49] J. G. Wardrop, “Some Theoretical Aspects of Road Traffic Research,” in Proceedings

of the Institute of Civil Engineers, 1952.

[50] H. Yang and H.-J. Huang, Mathematical and Economic Theory of Road Pricing. El-

sevier, 2005.

[51] I. C. Paschalidis and J. N. Tsitsiklis, “Congestion-Dependent Pricing of Network Ser-

vices,” IEEE/ACM Trans. Networking, pp. 171–184, 2000.

[52] S. Shenker, D. Clark, D. Estrin, and S. Herzog, “Pricing in Computer Networks: Re-

shaping the Research Agenda,” ACM SIGCOMM Computer Communication Review,

vol. 26, no. 2, April 1996.

68

[53] H. R. Varian and J. K. MacKie-Mason, “Pricing Congestible Network Resources,”

IEEE J. Select. Areas Commun., September 1995.

[54] G. Pacifici, W. Segmuller, M. Spreitzer, M. Steinder, A. Tantawi, and A. Youssef,

“Managing the Response Time for Multi-tiered Web Applications,” IBM T.J. Watson

Research Center, Yorktown, NY, Tech. Rep. RC23651, 2005.

[55] M. Falkner, M. Devetsikiotis, and I. Lambadaris, “An Overview of Pricing Concepts

for Broadband IP Networks,” in IEEE Communications Surveys, 2000, pp. 2–13.

[56] D. Thißen, “Load Balancing for the Management of Service Performance in Open

Service Markets: a Customer-Oriented Approach,” in Proceedings of ACM Symposium

on Applied Computing, 2002.

[57] V. Machiraju, A. Sahai, and A. van Moorsel, “Web Services Management Network:

An Overlay Network for Federated Service Management,” Hewlett-Packard, Tech. Rep.

HPL-2002-234, 2002.

[58] Z. Duan, Z.-L. Zhang, and Y. T. Hou, “Service Overlay Networks: SLAs, QoS, and

Bandwidth Provisioning,” IEEE/ACM Trans. Networking, 2003.

[59] Z. Li and P. Mohapatra, “QRON: QoS-Aware Routing in Overlay Networks,” IEEE J.

Select. Areas Commun., 2004.

[60] D. Xu and K. Nahrstedt, “Finding Service Paths in a Media Service Proxy Network,” in

Proceedings of the ACM/SPIE Conference on Multimedia Computing and Networking,

2002.

[61] X. Gu, K. Nahrstedt, R. Chang, and C. Ward, “QoS-Assured Service Composition in

Managed Service Overlay Networks,” in Proceedings of IEEE ICDCS, 2003.

[62] W. Wang and B. Li, “Market-Based Self-Optimization for Autonomic Service Overlay

Networks,” IEEE J. Select. Areas Commun., 2005.

[63] L. Grit, “Broker Architectures for Service-oriented Systems,” Master’s thesis, Duke

University, 2005.

69

[64] Y. Liu, A. Ngu, and L. Zeng, “QoS Computation and Policing in Dynamic Web Service

Selection,” in Proceedings of WWW2004, 2004.

[65] E. M. Maximillien and M. P. Singh, “Multiagent System for Dynamic Web Services

Selection,” in Proceedings of the AAMAS Workshop on Service-Oriented Computing

and Agent-Based Engineering, 2005.

[66] A.-C. Huang and P. Steenkiste, “Network Sensitive Service Discovery,” in Journal of

Grid Computing, 2004, pp. 309–326.

[67] B. Chaib-draa and J. P. Muller, Eds., Multiagent-Based Supply Chain Management.

Springer, 2006.

[68] P. M. Markopoulos and L. H. Ungar, “Shopbots and Pricebots in Electronic Service

Markets,” in Game Theory and Decision Theory in Agent-Based Systems. Kluwer

Academic Publishers, 2001.

[69] P. B. Luh, M. Ni, H. Chen, and L. S. Thakur, “Price-Based Approach for Activity

Coordination in a Supply Network,” IEEE Trans. Robot. Automat., vol. 19, no. 2, pp.

335–346, April 2003.

[70] M. Chiang, S. H. Low, A. R. Calderbank, and J. C. Doyle, “Layering as Optimiza-

tion Decomposition: A Mathematical Theory of Network Architectures,” Proc. IEEE,

vol. 95, no. 1, pp. 255–312, January 2007.

[71] J. He, M. Bresler, M. Chiang, and J. Rexford, “Towards Robust Multi-Layer Traffic

Engineering: Optimization of Congestion Control and Routing,” IEEE J. Select. Areas

Commun., vol. 25, no. 5, June 2007.

[72] J. Wang, L. Li, S. H. Low, and J. C. Doyle, “Cross-Layer Optimization in TCP/IP

networks,” IEEE/ACM Trans. Networking, vol. 13, no. 3, June 2005.

[73] D. P. Bertsekas, A. Nedic, and A. E. Ozdaglar, Convex Analysis and Optimization.

Athena Scientific, 2003.

[74] D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed Computation: Numerical

Methods. Prentice Hall, 1989.

70

[75] D. P. Bertsekas, Network Optimization: Continuous and Discrete Models. Athena

Scientific, 1998.

[76] P. Hande, S. Zhang, and M. Chiang, “Distributed rate allocation for inelastic flows,”

IEEE/ACM Trans. Networking, February 2008, To appear.

[77] S. Vutukury and J. Garcia-Luna-Aceves, “MPATH: A Loop-free Multipath Routing

Algorithm,” Microprocessors and Microsystems, vol. 24, no. 6, pp. 319–327, October

2000.

[78] P. T. Boggs and J. W. Tolle, “Sequential Quadratic Programming,” in Acta Numerica,

1995, pp. 1–51.