Dynamic Software Update of Resource-Constrained ...dl.ifip.org/db/conf/iess/iess2007/FelserKKS07.pdf · DYNAMIC SOFTWARE UPDATE OF RESOURCE-CONSTRAINED DISTRIBUTED EMBEDDED SYSTEMS

DYNAMIC SOFTWARE UPDATE OFRESOURCE-CONSTRAINEDDISTRIBUTED EMBEDDED SYSTEMS

Meik Felser, Rudiger Kapitza,Jurgen Kleinoder, Wolfgang Schroder-PreikschatFriedrich-Alexander University of Erlangen–NurembergDept. of Computer Sciences 4 (Distributed Systems and Operating Systems)Martensstr. 1, 91058 Erlangen, GermanyE-Mail: { felser, kapitza, jk, wosch} @cs.fau.de

Abstract: Changing demands, software evolution, and bug fixes require the possibility toupdate applications as well as system software of embedded devices. Systemsthat perform updates of resource-constrained nodes are available, but most ap-proaches require a complete restart of the node after installing or updating soft-ware. Restarting the node results in the loss of important system state, such asrouting information or sensor calibration values. Rebuilding this informationrequires time and energy.

In this paper we present an online state-preserving update system for resource-constrained nodes. A remote incremental linking approach is used to generatenode-specific and execution-state dependent code. Compiler-generated symbol,relocation, and debugging information is used to determine whether a dynamicupdate of the running system is possible and how it can be achieved.

Keywords: dynamic update, managed nodes, ELF, relocations

1. INTRODUCTIONThe number of microcontrollers used in today’s embedded systems is in-

creasing. In more complex systems, a network of several different microcon-trollers is used, thereby forming a heterogeneous system of larger and smallernodes. Especially, we have to face distributed embedded systems with a mix-ture of very resource-constrained devices on the one hand and more powerfuldevices on the other hand.

An example of such a network is a sensor network [1], a large number ofsmall, non-expensive, and very resource-constrained nodes equipped with sen-sors to collect environmental data and able to communicate. Usually the sensor

388 M. Felser, R. Kapitza, J. Kleinoder, W. Schroder-Preikschat

nodes are supported and managed by a larger, more powerful node, the basestation.

Once initially set up, selective software updates will become necessary dueto bug fixes, improved functions, extensions, and parameter changes. For ex-ample, if the calibration process did not deliver a set of parameters that yieldsthe desired results, software modifications become necessary. Replacing parts,such as the sampling or pre-processing algorithm, or adding additional soft-ware to use another sensor will be the consequence. Furthermore, with an es-timated lifetime of several years, a future-proof system must be able to replaceany part of the system such as the scheduling system or the communicationprotocol.

When updating the software of a node it is desirable to do this with as lit-tle impact as possible. Considering the limited resources of sensor nodes, theupdate process itself must not use a lot of resources. Furthermore, different up-dates affect different parts of the system and updates of the application shouldnot affect system services. It is not acceptable that the addition of a sensordriver results in the restart of the complete node. When restarting the node,state information of applications and the operating system library is lost. Mul-tihop communication protocols, for example, store routing information to rep-resent the network topology. After a restart such information is lost and has tobe rebuilt. The routing of the network has to be redetermined, which does notonly affect the local node but the communication of other nodes, too.

Our goal is to update sensor nodes dynamically thereby preserving appli-cation and system state. During the update process the nodes are stopped butno restart is needed after the update is complete. The advantage is a consider-able shorter off-line time and a better performance resulting from a very quickrecovery after the update because the system state does not have to be rebuiltunnecessarily.

We describe an update infrastructure that allows dynamic reprogrammingof nodes based on the binary code of the application. Most parts of our updatesystem are located at a more powerful node, the base station for example. Weuse compiler-generated information, such as symbol tables, relocation tables,and debugging information, to identify situations in which a safe update ispossible. In rare cases, the execution state might prevent a dynamic update.Such situations are signalled to the administrator so that additional actions canbe taken, such as a redesign of the application to increase the updatability.

The following section gives an overview about related approaches for up-dating nodes in sensor networks. Section3 outlines the architecture of oursystem. After that the object file analysis is described followed by section5presenting the procedure for updating or replacing code. Section6 concludesthe paper and gives a short overview of the current prototype.

Dynamic Software Update of Resource-Const. Dist. Embedded Systems 389

2. RELATED WORKThere are several approaches for dynamically updating software. In the

following we will discuss some existing approaches for updating sensor nodes.TinyOS [7] provides XNP [2, 10] to update a sensor node. This approach

only supports very coarse-grained updates because the complete memory im-age is replaced. It has a huge impact if only some parameters are to be adjusted.Deluge [8] provides similar possibilities and uses Trickle [13] as multihop dis-semination protocol to distribute the update to multiple sensor nodes. Bothapproaches need to restart the sensor node, because a new memory image isinstalled.

FlexCup [15] is the update system of TinyCubus [16], a framework to con-figure nodes in a role-based way. Components are transmitted as relocatablebinary images including symbol tables and relocation information. A linkercomponent on each node integrates the new components into one image. Thesystem is restarted after the update to eliminate dangling pointers. Our ap-proach does not need the linker component on each node because that part ofthe process is performed remotely.

Koshy and Pandey [11] present an approach similar to ours. They mod-ify the development toolchain to create an incremental linker that is used toprepare the updated code on the base station. Thus, they can control wheremodified functions are to be placed on the node and reduce modifications re-sulting from moved code. They do not consider the current state of the systemand perform a restart after the updated code is written into flash memory.

There are projects that support updating of a running system. To find all ref-erences to the code they often use an indirection layer to access the updatablecode. One example is SOS [5], an operating system for sensor nodes that isbuilt of modules. Modules can be updated, removed, and replaced at run time.The modules are transmitted and installed in a relocatable binary image for-mat. Module code is position-independent by using only relative jumps, thuslimiting the maximum size of a module to 4 Kbytes, the maximum distanceof relative jumps on the target platform, the AVR ATmega128. Furthermore,references to functions or data outside of the module are implemented via anindirection table or are not allowed, respectively. Contiki [3] works similarand also provides a framework for dynamically loading libraries. The librariescontain relocation information that enables the installation of the code. Accessto the libraries is provided indirectly via stubs. Our approach does not relyon an indirection table; we identify and modify the references directly. Withthis approach, we can even update code of the operating system library or thekernel.


3. ARCHITECTURETo achieve a minimal update infrastructure on the resource-constrained node,

most parts of the update preparations are executed by a remote node. Thatnode is called themanagerand is usually more powerful and equipped withseveral megabytes of memory. The manager node is responsible for updatingand managing the software installed on the resource-limited nodes (Fig.1).

SensorNode

Initial base system

Application v1Manager

Application v2

install

replaceupdate

Figure 1. Duties and responsibilities of themanager.

As we will discuss in section5 some situations preserve the automatic up-date of the running system. The information automatically extracted is not suf-ficient to decide whether the update of the running system can be performed. Inthese cases the administrator, who runs and supervises the system, is informed.He can the take the decision or use the information to redesign the applicationin a way that the manager can decide the situation automatically the next time.

Figure2 shows a schematic overview of the manager node. The next para-graphs give an overview about the most important elements.

3.1 Repository ManagerTherepository manageris responsible for managing and analysing the code.

The code is provided as binary object files. Software that is already installedand currently running on a sensor node is called the initial image of a node. Therepository manager determines which functions, more precisely which sym-bols, are available in the compiled and linked ELF [22] object file of the initialimage. Software that is to be updated or installed on a node has to be providedasrelocatableELF object files.

TheELFExtractorloads these binary files and analyzes the contents of eachinput file to identify individual functions and data objects. This results in afine-grained modularization of the application. After that the repository man-ager extracts the dependencies between the identified objects from the reloca-


*.O

Application v1

*.O

Initial state

Functions

01010

00011

01010

10001

00100

01010

00011

01010

10001

00100

01010

00011

01010

10001

00100

foo()

foo()

bar()func()

foobar()

Dependencies

virtual

Memoryimage

foo()

bar()

func()

0x0420

0x051b

0x0550

Repository

FunctionsSymbols

Dependencies

updatemerge

extract initialize

Linkedcode

Sensor Node

Application

OS Library

Bo

ot

Lo

ad

er

create

allocateupdate

link

Editscript

Install

Bootloader MemoryManagerLinkerImage

Manager

ELFExtractor RepositoryManager

Figure 2. Architecture of the manager node.

tion information and builds the dependency graph to determine which data isneeded by a function and which other functions are called. More details aboutthe modularization process are given in section4.

When the administrator changes a function the system identifies which func-tions and data in the dependency graph are affected and where these parts areinstalled in the network. From this analysis a set of differences for each nodecan be calculated. This set of changed functions and their dependencies presentthe basis of semi-automatic policies that are used to determine whether the up-date operation can be performed. In case of conflicts the administrator getsinvolved to control the update operation. This is addressed in detail in sec-tion 5.

3.2 Image Manager and Memory ManagerThe image managercreates and manages a memory image model that rep-

resents the usage of the memory in the device. At first this memory model isinitialized with the initial state of the node. The model is then used for keepingtrack of the currently installed software layout. Based on this information thememory manager, a part of the cross linker, determines a location where toput modified code on the node. For example, an updated function should belocated at the same position as the original function to minimize the numberof references that need to be updated. The virtual image includes a list of ap-plication modules that are currently installed on the node including their exactposition.


The image manager also records changes the linker performs to update theapplication. Based on this information it calculates a change set after the newfunctions were linked into the virtual image. This set contains the differencesfor each updated function and the code of all new functions. Deleted functionsare not in the change set. The system ensures that these functions are not usedanymore and the memory manager reuses their space to store new functions.The changes can then be transmitted to the node in form of edit commandsas they are generated by common diff algorithms, such as UNIX diff [ 9] orXdelta [14].

3.3 BootloaderA small bootloader resides on each node and is able to receive and process

commands from the manager. Its main task is installing new code but it alsoprovides the manager with essential information about the current state of thenode. As described in section5 the current state influences whether an updateof the running system is possible or not.

When a sensor node receives an edit command, the application transferscontrol to the bootloader which either extracts the requested data or modifiesthe SRAM and flash memory of the sensor node.

The actual transfer of code blocks can be done via several protocols. Ei-ther direct single-hop communication or more complex multihop protocols likeTrickle [13] or MOAP [20] are an option.

4. MODULARISATION AND DEPENDENCYANALYSIS

Functions and data objects are identified by analysing the relocatable ELFobject files. The analysis is solely based on the information provided in thisELF files. The developer does not need to use special constructs or annotationsto support the analysis.

ELF files are organized in sections. Some sections contain symbol tablesand relocation tables, other sections contain the relocatable binary data. Asimple example is shown in figure3. To determine the dependencies, we usethe symbol and relocation information an ELF object file provides. For eachrelocation the position inside the associated binary section is given at whichthe final address of a symbol should be inserted. The final address of a symbolis resolved by the linker as soon as the location of the associated section insidethe target’s address space is determined [12].

To extract a function we look at the symbols associated with the.textsection. These symbols give us the name, the start offset, and the size of thecode. Thus we know where each function starts and how much space it allo-cates. To determine the dependencies of a function we take advantage of the


(a) source code (b) ELF file contents

Figure 3. Example of the data contained in an ELF file and its relationship

relocation table. Each relocation that affects the code of the function representsa dependency.

We instruct the compiler to place each function in a separate section in theobject file. This way we ensure that each function is visible and each depen-dency is represented by a relocation embedded in the ELF file. Otherwise callsof local (static ) functions might not be visible in the relocation table. Inthe example in figure3 both functions are in the same section. Thus the com-piler knows the distance between the two functions and the call fromfoo tohello could be done via a relative call without a relocation entry. Ifhellois additionally static, the compiler might even suppress the symbol.

We do not necessarily have to apply the same approach for the data objectsof a program, because they are never placed in the same section as the func-tions. Thus, the symbols, even of local data, are available because the compilerneeds them to insert references to data. Nevertheless, we instruct the compilerto generate separate sections because this makes it easier to identify each singledata object. Otherwise the compiler creates one symbol for several local vari-ables and just uses different addends and a platform-specific analysis would benecessary to separate such data objects.

5. DYNAMIC CODE UPDATEReplacing or updating code in a running system requires knowledge about

all references to the old code and their substitution with references to the newcode. We also have to adjust the target addresses of jumps if a function ismoved to another address. We further have to take special care if the functionis modified while it is used by a thread of control. An overview of problemsand possible solutions when updating software dynamically is given in [6].Here we address three problem classes:


references to a function as it occurs in function calls

active functions that are currently executed by the target

functions on the call stack (the function called a subfunction that is cur-rently active)

update of global variables

In the following subsections we examine and discuss these problems.

5.1 References to FunctionsThe first question we have to address is whether the updated function starts

at the same address as the old one. If the new code has the same size or issmaller than the original function it fits into the old space and can start atthe same address. This is the preferable situation because we do not needto change any references. To allow even larger updates fit into the space ofa function the memory manager may allocate extra space [18] when initiallyplacing a function. In resource-constrained environments this approach has tobe applied carefully, because the additional space is wasted as long as it is notused by a replaced function.

If the modified function is larger and does not fit into the space of the oldfunction it has to be installed at a different location. Then we have to identifyall references to this function and update them.

References to the start of a function are used in conjunction with calls. Or-dinary function calls can be detected by the use of symbol and relocation infor-mation. The relocation table for a function that calls another function containsan entry specifying the location where the address of the target function mustbe inserted (fig.4). We use this information to find and update the reference tothe function.

Figure 4. The relocation information shows thatfoo needshello . If we updatehelloand thereby move it to an new position we need to modifyfoo as well.

.


Before patching the location with the new address, we have to determinehow the address is used. It may either be used in a call operation or it may bestored in a variable to do an indirect call at a later time. We can easily patchthe call operations with the new address. In all other cases it can not be exactlydetermined when and by which function the address will be used. In the worstcase the address of the function is stored in a global variable for use by otherfunctions. A combined code and data flow analysis could be able to detect thiscase but we do not think that the cost for this highly architecture-dependantoperation is justified.

Our approach is to identify whether there is enough information to correctlyfind and patch all references. As a consequence we cannot patch calls viafunction pointers and, thus, forbid the use of function pointers, at least forfunctions that should be updated. Nevertheless, function pointers are allowedin well-known code, such as interrupt vector tables and the operating systemscheduler. A system-specific layer is used to identify code and data referencesin this code.

This is not a strong limitation as function pointers are very rarely used in em-bedded applications1. Furthermore, function pointers are an additional sourceof errors especially for inexperienced programmers. For the same reason otherprogramming paradigms for embedded systems, such as the Misra C rules [17],forbid the use of function pointers at all.

If our system detects the address of the updated function in any other than acall operation it indicates an insecure situation to the administrator who has todecide whether the function should be updated nevertheless.

5.2 Active FunctionBefore actually updating a function we have to make sure that it is not cur-

rently in use. The consequences of such an update are unpredictable. A func-tion is in use if it is interrupted by the update process, as shown in figure5, orif a thread was executing the function before it was suspended. In some event-driven systems, such as TinyOS [7], this situation does not occur as they donot offer a thread concept. Just event handlers can interrupt the normal controlflow. Thus the update process gets scheduled as a task when no other task isactive. Other systems, such as Contiki [3] or Nut/OS [4] offer a thread concept.

To identify active functions, the manager asks the bootloader at which ad-dress the system would resume operation when exiting the loader. If the sys-tem supports multiple threads the bootloader returns the current position of all

1An analysis of typical TinyOS applications showed that usually no function pointers are used. Neverthelesspointers to functions are used by the TinyOS scheduler and implicitly in the interrupt vector table.


Figure 5. A function is interrupted and updated while it is active. Resuming the executionat the interrupted address may result in a crash or unpredictable behaviour.

threads. The update manager checks whether the update would affect one ofthese addresses and indicates an update problem.

A simple but promising approach to overcome this situation is to resumethe system and retry the update a few moments later. If we are updating asensor reading function it is likely that the function is not in use the next timewe try an update. To avoid the periodic check, we can improve this approachby modifying the return address in a way that the bootloader is called as soonas the function returns. We then resume normal operation and wait until thefunction returns into the bootloader. If the function is still in use after a timeoutor several retries, we inform the administrator about the failed update. Then hecan authorize a reset after the update.

The success of this approach depends on the modularity of the application.Sensor net applications developed with some sort of framework, like TinyOS,are often designed very modular. Furthermore we inform the administratorabout the reasons of a failed update. He can use this information to improvethe application’s modularity with regard to the next update.

5.3 Functions on the Call StackBeside functions that are directly in use by a thread, functions may also be

on the call stack. That is, the function that is to be replaced called anotherfunction and this function is currently active. When the called function returnsit uses the return address on the stack to resume operation in the caller. If wereplace the code of the caller this return address might become invalid. Figure6 shows an example.foo callshello , this may be a call from a sensor cali-bration function to the function that actually gets the sensor readings. Duringthe execution ofhello the functionfoo is updated.hello is not influenced


directly but the code offoo is modified and the return address that is used toresumefoo afterhello returns may be incorrect or invalid.

Figure 6. A function on the call stack is updated. When the flow of control returns to theupdated function the resume address may be invalid.

We do not want to forbid the update of such functions in general, becauseminor changes, such as bug fixes, may alter the addresses but not the function-ality of the code. Thus, updates are allowed if they do not alter the structure ofthe function. The new code must have the same calls to the same subfunctionsas the old code and the same local variables. In our example above, we canfix a bug in the calibration algorithmfoo as long as we still call the readingfunction hello and do not insert or delete local variables. We then assumethat the return addressA of the n-th call to a specific function from the old codecan be replaced with the equivalent return addressA′ in the new code.

To accomplish this we have to know the stack layout and position. Withthis information we can walk through the stack and detect each function onthe stack. The return addresses can then be adapted. The information howto obtain the stacks is encoded in the bootloader. The manager transfers amapping from old to new return addresses if the loader requests it.

The debug information can give us information about the local variables ofa function. Thus we can perform an offline check whether new local variableswere added or removed.

If the structure of the function is modified we can retry the update sev-eral times. The function is possibly not in use anymore at a future retry. Butthe deeper the function is on the call stack the smaller is the probability thatthe function is not in use. If the retries are not successful, we inform the


administrator about the failed update. Again, he can authorize a reset and usethis information for the development of future versions of the application.

5.4 Update of VariablesUpdate of local variables is not supported as mentioned in the previous sec-

tions. Updates of global variables are allowed under certain conditions. If afunction references a new global variable, this variable is simply allocated onthe node. In the opposite case, that a variable is no longer used by an up-dated function we can not reliably determine whether the variable is used bysome other code. Therefore we ask the administrator if the variable can befreed if we do not detect any references anymore. If a variable with the samename but different type is introduced we have several possibilities. If the oldvariable is still used by some code, we generate a warning because this situ-ation most likely leads to inconsistencies and the administrator has to decidewhether to continue with the update or not. If the new variable replaces the oldthe current state should be transformed. Without further information from theadministrator this is not possible. An automatic transformation is subject tothe same restrictions as, for example, the object (de-)serialization in Java [21].

6. CONCLUSIONS AND CURRENT STATUSThe presented work builds a cornerstone of our long-term objective to pro-

vide support for robust and efficient software management in heterogeneoussensor networks [19]. It enables the management of software installed onresource-constrained nodes by providing the ability to update and replace codein the running system. This is achieved by incrementally link new code tothe existing application and to identify unused code. Necessary configurationinformation is stored at a larger node that prepares the update. This managernode also checks whether an update is possible in the running system at all. Incritical situations, the administrator gets involved who can control and decidein which way the update has to be performed.

The current prototype of the manager software is implemented in Java. Itcan load and analyze relocatable ELF object files for x86, Hitachi H8 and AVRCPUs. Up to now, the prototype communicates and updates single nodes but itwill be extended to handle groups of equal or similar nodes.

REFERENCES

[1] I. F. Akyildiz, W. Su, Y. Sankarasubramaniam, and E. Cayirci. Wireless sensor networks:A survey.Computer Networks, 38(4):393–422, Mar. 2002.

[2] Crossbow Technology, Inc.Mote In-Network Programming User Reference, version20030315 edition, 2003.


[3] A. Dunkels, B. Gronvall, and T. Voigt. Contiki - a lightweight and flexible operatingsystem for tiny networked sensors. In29th IEEE Int. Conf. on Local Computer Networks(LCN’04), pages 455–462, Nov. 2004.

[4] egnite Software GmbH.Ethernut Software Manuals, Nov. 2005.

[5] C.-C. Han, R. Kumar, R. Shea, E. Kohler, and M. Srivastava. A dynamic operatingsystem for sensor nodes. In3rd Int. Conf. on Mobile Systems, Applications, and Services(Mobisys ’05), pages 163–176, June 2005.

[6] M. Hicks. Dynamic Software Updating. PhD thesis, University of Pennsylvania, Depart-ment of Computer and Information Science, Aug. 2001.

[7] J. Hill, R. Szewczyk, A. Woo, S. Hollar, D. Culler, and K. Pister. System architecturedirections for networked sensors. In9th Int. Conf. on Architectural Support for Program-ming Languages and Operating Systems (ASPLOS-IX), pages 93–104, Nov. 2000.

[8] J. W. Hui and D. Culler. The dynamic behavior of a data dissemination protocol fornetwork programming at scale. In2nd Int. Conf. on Embedded Networked Sensor Systems(SenSys’04), pages 81–94, Nov. 2004.

[9] J. W. Hunt and M. D. McIlroy. An algorithm for differential file comparison. TechnicalReport 41, Bell Telephone Laboratories, 1976.

[10] J. Jeong, S. Kim, and A. Broad. Network reprogramming. Technical report, Universityof California at Berkeley, Aug. 2003.

[11] J. Koshy and R. Pandey. Remote incremental linking for energy-efficient reprogrammingof sensor networks. In2nd Europ. W’shop on Wireless Sensor Networks (EWSN 2005),pages 354–365, 2005.

[12] J. R. Levine.Linkers and Loaders. Morgan Kaufmann, San Francisco, CA, USA, Oct.1999.

[13] P. Levis, N. Patel, D. Culler, and S. Shenker. Trickle: A self-regulating algorithm for codepropagation and maintenance in wireless sensor networks. In1st Symp. on NetworkedSystem Design and Implementation (NSDI ’04), pages 15–28, Mar. 2004.

[14] J. P. MacDonald. File system support for delta compression. Master’s thesis, Departmentof Electrical Engineering and Computer Sciences, University of California at Berkeley,2000.

[15] P. J. Marron, M. Gauger, A. Lachenmann, D. Minder, O. Saukh, and K. Rothermel. Flex-cup: A flexible and efficient code update mechanism for sensor networks. In3rd Europ.W’shop on Wireless Sensor Networks (EWSN 2006), volume 3868 ofLNCS, pages 212–227. Springer, Feb. 2006.

[16] P. J. Marron, A. Lachenmann, D. Minder, J. Hahner, R. Sauter, and K. Rothermel. Tiny-Cubus: A Flexible and Adaptive Framework for Sensor Networks. In2nd Europ. W’shopon Wireless Sensor Networks (EWSN 2005), pages 278–289, Jan. 2005.

[17] MISRA. MISRA-C: 2004 - Guidelines for the use of the C language in critical systems,Oct. 14, 2004.

[18] R. W. Quong and M. A. Linton. Linking programs incrementally.ACM Transactions onProgramming Languages and Systems (TOPLAS), 13(1):1–20, Jan. 1991.

[19] W. Schroder-Preikschat, R. Kapitza, J. Kleinoder, M. Felser, K. Karmeier, T. H. Labella,and F. Dressler. Robust and efficient software management in sensor networks. In2ndIEEE/ACM International Workshop on Software for Sensor Networks (SensorWare 2007),Jan. 2007.


[20] T. Stathopoulos, J. Heidemann, and D. Estrin. A remote code update mechanism forwireless sensor networks. Technical Report CENS–TR–30, University of California, LosAngeles, Center for Embedded Networked Computing, Nov. 2003.

[21] Sun Microsystems.Java Object Serialization Specification, 2001.

[22] TIS Committee. Tool Interface Standard (TIS) Executable and Linking Format (ELF)Specification, version 1.2 edition, May 1995.

Documents

Dynamic Software Update of Resource-Constrained ...dl.ifip.org/db/conf/iess/iess2007/FelserKKS07.pdf · DYNAMIC SOFTWARE UPDATE OF RESOURCE-CONSTRAINED DISTRIBUTED EMBEDDED SYSTEMS