Towards developing next-generation distributed software A society-inspired methodology

Palette - WP0

European and Chinese Cooperation on Grid

Towards developing next-generation distributed softwareA society-inspired methodology

National University of Defense Technology, ChinaBo Ding

[email protected]

Second EchoGrid Workshop Beijing 29 & 30 October 2007

Outline
Challenges aheadKey concepts of our methodologyAutonomic Unit and CommonwealthFrom concepts to real-life programmingA reference software architectureA prototype of software infrastructureConclusion & future work
*


Grid technologies: state-of-the-art
Large-scale resource sharing and virtualizationApplicationsFramework and InfrastructureArchitectureToolsImpact on practice of science and industryECHOGRID: Towards a Shared EU & Chinese Vision for Grid Research PerspectivesOther related research projects:China 973 program Research on aggregation and collaboration mechanisms in the virtual computing environmentChina 863 program Research on self-adaptive software platform for pervasive computing
*


Distributed software is becoming more and more complexRunning on millions of computing nodes, sensors and actuatorsBeing sprayed everywhere and highly interconnectedTypical instances which is turning into reality:Internet-based large-scale computing systemPervasive computing application
A gradual revolution
*


Characteristics & Related work
Characteristics of next-generation distributed softwareUbiquitySeamless integration of the physical and computing spacesDecentralizationHuge complex systemEach node is inherently autonomousContinuous GrowthRequirements, functions and physical boundary are dynamically changing in the whole lifetime of the softwareHow to develop such kind of system has been a great challengeA new focus in the research communitySoftware-intensive systemsUltra-Large-Scale systems (CMU/SEI)Spray computer

*


Challenges of development
Some principles of developmentEvolution, instead of waterfall modelCooperation, instead of centralized controlActive adaptation instead of being passively invokedMethodology have to be proposed and software infrastructure have to be developedTo fulfill the gap between our ideal and traditional software engineering technologies

*


Outline
*


Inspiration from social behaviors
Society is a typical huge open complex systemEach individual is autonomicHe continuously and actively adapts to the environmentEach individual belongs to one or more organizationsHe enjoy rights as well as undertake duties, for instance, having to abide by the organizations rulesThe progress of the whole society is driven by the cooperation of active individuals, whose behavior is regulated by certain organizations rules.A brief overview of our methodologyResources are abstracted to Autonomic Units individualsAutonomic Units can dynamically join/quit Commonwealth organizationVarious task targets are achieved by the cooperation between regulated Autonomic Units.
*


Autonomic Unit and Commonwealth
Autonomic Unit (AU)Software abstraction of concrete resources, who has the capability toJoin: Becoming a member of Commonwealth, being able to abide by Commonwealth's rulesAdapt : actively adapt to the physical and computing environmentCommonwealth (CW)Consisting of Autonomic Units having the same interests or targetsEnacting behavior rules to ensure interoperation and encourage equal cooperation
*
Abstraction
Autonomic Units
Software Space


Model of AU & CW
Autonomic Unit

Commonwealth

*
Behavior-driven Engine
ContextCollectors
BehaviorExecutors
Physical and computing space
Autonomic Unit
Base-level entities
Meta-level entities
Commonwealth Join/Quit Mechanism
Abstraction of sensors, which collects computing and physical contexts
Concrete business logic, whose execution may change internal states or the envrionment
Actively driving the behavior based on the change of concerned contexts
Dynamically join and quit a Commonwealth, ensuring AU being regulated by the rules of the Commonwealth
Initiator AU
Common AU

A special AU to initiate a Commonwealth
AUs dynamically join/quit the Commonwealth
Commonwealth
Resource Discovery
To help initiators to invite appropriate AUs under certain circumstances


How to address the challenges?
? Evolution, instead of waterfall modelBehavior dynamically regulation instead of static deploymentThe developers can specify and modify the rules of Commonwealth on lineAU gets the newest rules by joining CommonwealthDeveloper-driven system dynamic evolution? Cooperation, instead of centralized controlEncourage cooperation by Commonwealth rules? Active adaptation, instead of being passively invokedContext-driven and reflection-based adaptationIn brief, Join/Adapt capability partially addressed the challenges to the development of next-generation distributed software!

*


Outline
Challenges aheadKey concepts of our methodologyAutonomic Unit and Commonwealth From concepts to real-life programmingA reference software architectureA prototype of software infrastructureConclusion & future work
*


Component-version Autonomic Unit
To integrate AU concept with maturing software engineering technology!Component-based designDownload-based joiningRule-based adapting
*
BehaviorComponent
Behavior-drivenengine
Policies
AUShell (Container)
BehaviorComponent
BehaviorComponent
A set of adaptation rules, in the shape of when to do what
To translate and execute the policies
ContextComponent
ContextComponent
Encapsulation of context providers
Encapsulation of business logic
Physical and computing space
AU Core


Dig deep into Join
Question: How to ensure AUs behavior regulated by the Commonwealth?Answer: Download-based JoiningAn Autonomic Unit is divided into two parts:AU Shell: Container, including behavior-driven engineAU Core: Commonwealth-related objects, composed of policies and componentsAU Shell is pre-deployed and loaded in the start-up process of each resourceAU Core is dynamically downloaded when an AU joins a Commonwealth

Concrete joining model: Actively joining model, Inviting Model, and Mixed Model

*
Container(Behavior-driven engine)
PolicesBehavior componentsContext components


Question: How to actively adapt to the environment? Answer: Rule-based AdaptationFrom the perspective of reflectioncontext component meta data providerpolice meta-level protocolbehavior-driven engine meta-level computing entitybehavior component base-level computing entity.

ComponentManagement
Dig deep into Adapt
*
RequestDispatch
Comp
Network Message
Management Request
Behavior-driven engine
Policies
Context
Comp
Invocation/management
Traditional Container
Our AUModel


A brief overview of our software architecture
Concept view

A complete light-weight component model is proposed.Dynamically loading, inversion of control, language independenceRun-time viewInitiate: Commonwealth is initiated by an initiator AU, and other AUs dynamically join/quit AU based on their interests and the demandCooperation: AUs are cooperated driven by the context, including the change of physical environment, computing resources and network state.Development viewTo be continue in the next slide
*
component
Autonomic Unit
Common Wealth


How to develop?
Step1: To define component interfaces by our UCDL languageStep2: To implement components

Step3: To assemble CommonwealthAssemble components in a graphical tool based on GMEDivide assembled components into AU CoresIf necessary, specify policies in the AU Cores

*


Design of software infrastructure
*


Protocol Framework

Abstraction of Communication

Routing/Fowarding

ProtocolSwitching

NetworkStateMonitor

XML Parser

Meta-level interaction

AU Context Management

Binding

AU LifecycleManagement

Policy Execution

ComponentManagement

Initiator

Human-machineInteractionAdaptation

Context-awareData Storage

Event Service

Resource Discovery

AU Shell

Self-adaptiveCommunication

CommonServices

UCDLCompiler

ComponentPackager

AssmblingTool

ManagementTool

Run-time Environment

Development/ManagementTools

Protocol Plugins

ComponentLibrary

Infrastructure prototype and demo applications

*


Outline
*


Conclusion & Future work
Our first step towards next generation distributed softwareProposed an abstract model to partially cope with challenges aheadIntegrated our ideal with maturing software engineering technologyFuture workDynamic system theories and their application in our methodologyEmergent Behavior, Digital evolution, Control TheoriesSocio-Economic theories to encourage equally cooperatingGame rules, Mechanism design, Credit countOther possible self-adaption and behavior regulating mechanismsFurther research on lightweight component model, policy and QoS integration

*


Thank you for your attention!
*


*
Good morning, everybody!Its a pleasure to have the opportunity to introduce our recent work here. Today I would like to present our initial attempts towards developing next-generation distributed software, especially our society-inspired methodology and software infrastructure.
Ill give this talk in five parts: At first, well start with a brief review of the challenges to the development of next-generation distributed software. And then Ill present our methodology, focusing on some key concepts such as Autonomic Unit and Commonwealth. The third part of my presentation concerns how to integrate our idea with existing maturing software technologies, including a reference software architecture and a software infrastructure prototype we developed to support this architecture. If time permits, Ill run a simple demo on my notebook. And the last part is a brief conclusion and introduction of our future work.
*
*
As we know, the main purposes of the Grid computing technologies are concerning about large-scale resource sharing and virtualization. During the past decade, many research works have been carried out within this community regarding to Grid applications, frameworks and infrastructures, architecture and tools. They have had tremendous impacts on the practices of science and industry as well. However, because of the diversity of the world we are facing today, the Grid people have also had a lot of distinct characteristics in their research works. For example, different viewpoint, different focuses, different modeling, different methodology of implementation, to name a few. Is this situation what we want? I think the answer is yes and no. Anyway, for the time being, trying to move towards a shared vision for Grid research is the main topic, which is also the reason of this workshop. In the following slides, I would like to introduce our vision on methodology and infrastructure within the context of large-scale distributed software, or Grid, for short.Please note that this work is also partially supported by some other related projects. I listed the most important two of them here.The first one is funded by The National Basic Research Program of China (also called 973 Program), its main target is about aggregation and collaboration mechanisms in the virtual computing environment, which is also an very important topic in Grid.The second one is funded by The Chinese High-tech program, its feature is on the self-adaptation and pervasive aspects.
Okay, lets start off with challenges ahead. Over the past few years, we have witnessed a rapid growth in size and complexity of distributed software, which will likely continue well into the foreseeable future. A software system may run on thousands of computing nodes, sensors and actuators. Software entities will be sprayed every corner of our lives and highly interconnected by various networks. This is not science fiction. They are coming into our reallife. Pervasive computing applications and Internet-based large-scale computing systems are typical instances, such as pervasive health, intelligent traffic and military information system.
*
The growth in size and complexity is a gradual revolution. Quantitative changes leads to a fundamental change. In another word, those systems presented some characteristics that their predecessors dont have, including: Ubiquitous, that is, seamless integration of physical and computing space, to provide services anytime, anywhere and invisibly;Decentralization, on the one hand, the system is too complex to be managed and controlled centrally; on the other hand, the nodes of such system may belong to different domain, and thus they are inherently autonomous.Continuous growth, that is, the requirements, functions as well as physical boundary of the system cant be determined in advance. They are dynamically changing in the whole lifetime of the software.Because of those new characteristics, how to develop such kind of distributed software has been a great challenge and aroused the interests of the research community. For example, the research on software-intensive system, being referred to as Systems in which software interacts with other software, systems, devices, sensors and with people; In July 2006, founded by U.S. Department of Defense, Carnegie Mellon University published the report Ultra-large-scale systems, presented a research agenda for Ultra-large-scale systems. Somebody believes that this report is as important as the software engineering report published in 1968.
*
Lets consider challenges ahead in more detail. Because next-generation distributed software is ubiquitous, decentralized and continuously growing, it is obvious that several fundamental changes in the development should be taken place: Firstly, in order to fit in with the continuous growth of the software, traditional water-fall model should be abandoned and evolution in the whole lifetime should be encouraged. Secondly, because of the decentralization of such systems, traditional centralized control and management is nearly impossible, we should encourage equally cooperation between software entities instead. Thirdly, for each software entity distributed on the computing node, we should encourage them to actively adapt to the physical and computing environment instead of merely being passively invoked.However, the above-mentioned changes are just our ideal. It is obvious that there is still a huge gap between it and existing software technologies. In the following slides, Ill introduce our initial attempts to fulfill this gap.
*
The second part of my presentation is about our methodology. However, Ill just lay emphasis on some key concepts, partly because of the time limit, partly because the research is still under way.
*
Next-generation distributed software is supposed to be huge open complex system. Lets take a look at a real huge open complex system at first at first, the human society. In a society, each individual is autonomic, continuously and actively adapting to the change of the environment. Each individual belongs to one or more organizations. The organization have clear targets, and as a member of this organization, individuals enjoy rights as well as undertake duties, for instance, having to abide by the organizations rule. For example, if a man want to gain higher education, he must become a student of a college and learn the rules of the college, including communicating with teachers and classmates in an appropriate manner, gaining sufficient credit, going to the lab on time, and so on. The progress of the whole society is driven by the cooperation of active individuals whose behavior is regulated by certain organizations rules.To some extent, our methodology is a simulation of human society. In our model, devices, who possesses various computing resources, are abstracted into Autonomic Units, an analogy of individuals tin he society. Autonomic Units can dynamic join or quit Commonwealth, an aggregation of Autonomic Units who have same interests and targets. Commonwealth is an analogy of the organization. Various targets are achieved by the cooperation between Automatic Units, whose behavior are regulated by certain Commonwealth. This is just a brief overview and Ill elaborate on those concepts in the next two slides.
*
Autonomic Unit, as we have said, is the abstraction of device. This abstraction provide at least two important capabilities: join and adapt. The former is referred to as its capability of joining a Commonwealth and abide by its behavior rules; the later means that it can actively adapt to the change of physical and computing space. Commonwealth is an aggregation of Autonomic Units who dynamically join and quit based on their own interest and the demand. It enact some behavior rules to ensure interoperation and encourage equal cooperation by the means such as credit count and peer to peer resource sharing. The following animation illustrates those concepts in more detail.Various resources is abstracted into a software entity named Autonomic Unit with join/adapt capability. When Autonomic Unit joins a Commonwealth, it becomes active. This is the software space. There are already several inactive Autonomic Unit, here inactive means that they dont join any Commonwealth at this time. When a task is given, an Autonomic Unit may initiate a corresponding Commonwealth, and other Autonomic Units may join this Commonwealth dynamically. As a result, the boundary of Commonwealth is not fixed. Another commonwealth may be organized in the same style.
*
This is the internal structure of Autonomic Unit. As shown in this diagram, it consists of four parts: Context Collector, Behavior executor, Behavior driven Engine and Commonwealth join mechanism.This is the structure of Commonwealth. A commonwealth consists of two kinds of Autonomic Units: a special Automatic Unit can act as an initiator, and other Autonomic Units dynamically join/quit. Besides, a resource discovery service is provided to facilitate the initiator to found appropriate AUs under certain circumstance.
*
Okay, this is an outline of our methodology. lets go back to the question: how does our methodology address those challenges to the development of next-generation distributed software? We have present three challenges earlier: evolution, cooperation and active adaptation. The evolution in our methodology is implemented by the Join capability, that is, behavior dynamically regulation instead of static deployment. The developers can specify and modify the rules of Commonwealth on line, and Autonomic Units can get the newest rules by joining Commonwealth. In summary, on the macro level, this is a developer-driven system dynamic evolution. With regard to cooperation, it is obvious that our methodology is based on a naturally decentralized architecture and Ive mentioned that we can encourage cooperation by Commonwealth rules. In order to implement active adaptation, we introduced Context-driven and reflection-based adaptation, as shown in the concept model diagram of Autonomic Unit.
*
Up to now, we mainly focused on abstract concepts and models. If we cant integrate them with existing software engineering technologies, it is just an empty talk. Therefore, well propose a reference software architecture based on maturing component technology and introduce a prototype of software infrastructure we developed to support such an architecture.
*
This is the component version Autonomic Unit, an implementation of the concept model we proposed earlier. In this implementation, context collectors and behavior executors are encapsulated as components, which we called context component and behavior component. And a new entity, policy, is introduced to ensure rule-based adaptation. Policy is a set of adaptation rules in the shape of when to do what and can be dynamically modified by human or other software. And the Behavior-driven engine is embodied as a fixture to translate and execute policies.You may have noticed the word downloaded-based joining and Rule-based adapting. Yes, this is concrete the implementation of join/adapt capability. Ill explain them at length in detail.
*
This slide explains the join mechanism in detail. As what Ive said, AU is divided into two parts, AU shell and AU Core. The Shell is general-purpose objects in an Autonomic Unit, mainly the Container including behavior-driven engine. It is fully implemented in the software infrastructure and pre-deployed on each resource. The Core is composed of Commonwealth-related objects, including policies, context components and behavior components. It is dynamically download when an AU joins a Commonwealth. Dynamic download is the analogy of learning rules in human society, and by this mean, we ensure AUs behavior regulated by the Commonwealth. Weve introduced several concrete joining models in the reference software architecture, including actively joining model, inviting model and mixed model.. However, in the interest of time, I wont go enter into details.
*
Okay, Lets see how to implement the adapt capability? How to actively adapt to the environment? The answer is rule-based adaptation. From the perspective of reflection, .In summary, we enhanced the meta-level function of containers in traditional component model. This is a comparison of traditional distributed component model and our Autonomic Unit Model. As shown, traditional container is just a passive mediator of network messages and management requests. On the contrary, the container in our component-based Autonomic Unit model is able to actively manipulate the base-level components according to current policies. Components and Container are integrated into an organic whole.
*
Okay, its time for a brief view of the whole reference software architecture. We originally prepared to introduce it from three aspects: concept view, which explains key concepts of our architecture; run-time view, which explains the running mechanisms of software; and development view, which explains how to develop step by step. However, in fact, in previous slides we have explained concept view and run-time view. The only thing I have to add is that the component model is a light-weight model we proposed, which characterized by dynamically loading, inversion of control and program language independence. However , I think this is not the focal point of todays presentation. Since todays topic is programming paradigm, Ill discuss our development view in more detail.
*
The development of a Commonwealth can be divided into three steps. The first two step is similar to traditional component development, as shown in this slide. The first part of the third step is still similar to traditional component composition, in an integrated graphical tool we developed based on GME. The main difference is that after assembling a Commonwealth, they have to divide the Commonwealth into AUCores and indicate some necessary properties of those Cores. And then the programmer can decide whether or not to add policies to an AUCore. Please note that policy is optional. If you chose to not to specify any policy, the development process is very similar to traditional component-based application development. We intentionally kept this compatibility with traditional development method, hoping our method can be widely accepted by the programmers.

*
We have outlined the reference software architecture, and now well take a glance of our software infrastructure to support the running and development of this architecture. This is the design blueprint of our software infrastructure, which consists of Self-adaptive communication, Autonomic Unit Shell, Common services as well as development and management tools. Up to now, we have implemented a minimal set of this blueprint, which can verify and demonstrate our idea.
*
This is some Snapshots of the user interface of our infrastructure. And we developed several applications based on this infrastructure, to verify and demonstrate our idea. This is one of them, a fire alarm demo. The left diagram is its assemble view, and the right part is its run-time snapshot.
*
Okay, The conclusion and our future work. This is Our first step towards next generation distributed software. In brief, we
*

Documents

Towards developing next-generation distributed software A society-inspired methodology