Design of Distributed Real-Time Systems

Ramani Arunachalam

Case Study: MARS

● MARS (Maintainable Real-time system)– Distributed, fault-tolerant, hard real-time– Objectives

● Guaranteed timeliness● Testability● Maintainability● Fault-tolerance● Systematic software development

– Time-triggered architecture

Objectives● Guaranteed timeliness

– Based on resource adequacy at peak load– Statistical assurances not enough

● Testability– Architecture should support testability of timeliness

● Maintainability– Needed to remedy hardware faults, design errors and

respond to change requests– Localized consequences -> minimized effort

Objectives

● Fault Tolerance– Redundancy– On-line maintenance

● Systematic software development– No 'trial and error' integration– OS guarantees predictable temporal behaviour

State View

● Time Triggered observation of states– Observe RT entities at predefined intervals

● Intelligent input output– Observation grid– Intelligent sensor

● Preprocesses raw data from input device ● observes at finer granularity called Perception granularity

State View

● Intelligent actuator – Post-processes data from computer system before

sending to output device● State Messages

– Produced at observation points– Minimal synchronization requirement – No need for buffer management– Unidirectional (from RT entity)

Structure● Clusters

– Autonomous subsystems– Disjoint name spaces– State message exchanges– Composed of Fault-tolerant units (FTUs)– Real-time communication channel (TDMA)

● FTU– Composed of replicated components– Active and shadow components

FTU FTU

Structure● Component

– Smallest replaceable unit– Fail-silent (Correct results or none)– Termination upon failure

● Task Execution– Task : Software inside component– Starts at predefined time– Proceeds without any communication or

synchronization– Execution time is deterministic

Operation● Results of periodic tasks sent as state messages● Execution time of communication is also

predefined● A Real-time transaction is a progression of

processing and communication actions between a stimulus from and a response to the environment.

● Static scheduling (at compile time!)● At run-time, no surprises● Modes (operating, emergency)

Fault-tolerance● Two levels of redundancy● Active redundancy at FTU level

– If a component fails, standby becomes active● Time redundancy at component level

– Every task is executed twice and results compared● TDMA monitor

– Monitors temporal behaviour– Controls the output from component

● Distributed clock synchronization

Fault-tolerance● Replica determinism

– All replicated components perform the same state changes at the same point in time

– Prohibit reading of local time– All replicas should agree when to change mode

● Component reintegration– i-state, h-state– Reintegration point: when size of h-state is small– New component gets the h-state at this point

Summary● Maintenance

– Failed component doesn't affect FTU– On-line reintegration after repair– Change in software

● Does it fit in current schedule?● Otherwise, new mode with new schedule

● Summary– Strict separation of functionality, timeliness and

dependability.– Designed for temporal behaviour, testing simplified.

Delta-4 XPA● Objectives

– “A real-time system is not assured to meet deadlines outside operational envelope”

– Bounded-demand school● operational envelope is predictable● Impractical assumption for complex systems

– Unbounded-demand school● Complete definition of operational envelope is not possible● Graceful degradation if it falls outside the envelope

– XPA implements hard real-time but falls into best-effort behaviour when required.

DELTASE

Group management Layer

Time and Group communication

Abstract network layer(physical + MAC+ firmware)

Architecture● Network infrastructure

– FDDI supports urgent traffic, built-in fault tolerance– Token bus/ring has media redundancy for availability

● Time– Internal time maintained by distributed time server– Clocks synchronized to tens of microseconds– External time – one of the standard time

● Group communication– Services from atomic multicast to datagram– Very fast services of varying reliability

Architecture● Group communication

– Distributed replication management● BestEffortN – guarantee delivery to N elements● BestEffortTo - guarantee delivery to named elements● AtLeastN, atLeastTo – guaranteed service even when sender

fails● Group management

– Distributed Group manager object– Management and distribution of groups of objects– Incorporates knowledge of various modes of

replication

Architecture

● Application support environment (Deltase)– Client-server and producer-consumer interactions– Apps written using deltase or converted using

preprocessors● Timeliness

– What to do under overload conditions?● Static off-line scheduling – too many possibilities● On-line scheduling – can find feasible schedules if not

overload.

Timeliness● Scheduling policy uses “precedence”

– Combination of priority and earliest-deadline– Few priority classes to avoid unfairness– Within priority class, earliest-deadline-first.

● Design-time and run-time timeliness– Targetline : instant chosen by designer for provision of

service– Liveline and deadline: earliest and latest time at which

service may be provided– Violation of these detected at runtime and design-time

actions defined.

Preemption

● Leader-follower model for replication– Decisions made by a privileged replica i.e. Leader– Preemption point

● Point at which an interrupt will be served– High precedence msg arrives for a process not running

currently● Increase the process's precedence to that of msg● Causes the process to be scheduled● These actions propogated to followers● Followers perform identical operations

Desynchronization

● Followers must not be too apart from leaders● Followers too fast

– Reach the preemption point before leader– remain blocked until leader notifies

● Followers too slow– Leader timestamps notifications– If follower didn't execute the action by T+t(desync)

● Desynchonization event raised● Another follower takes over

Summary

● Communication support using groups– Oriented to distributed computing

● Tradeoffs between QOS and efficiency– Group mgr uses atomic multicast for orderly delivery– Leader-follower uses reliable, non-ordered delivery

● Group management service– Executes leader-follower, detects replica failure– Clone the replica at another node.