Reducing Obsolescence of Linux-Based ATEs with Virtualization

Casey Weltzin and Sarah Schlonsky

I n automatic test applications ranging from U.S. De-partment of Defense (DoD) projects to semiconductor manufacturing, automated test equipment (ATE) and test

program sets (TPS) represent a significant investment. While much of the investment in test takes the form of capital expen-ditures, another significant portion that cannot be ignored is the time spent integrating, validating, and upgrading soft-ware and developing driver support to accommodate new ATE hardware. Without ongoing hardware upgrades and as-sociated software maintenance, ATE systems can quickly face obsolescence – resulting in large replacement costs.

The threat of obsolescence is especially evident for Linux® based ATEs. While open-source software and community-based development can reduce up-front costs and speed the introduction of new features, the existence of hundreds of dis-tinct Linux distributions has forced ATE hardware vendors to choose between offering full driver support for only a few distributions or more limited support across distributions. In addition, if ATE designers must upgrade the Linux distribu-tion on a system, many different pieces of software including hardware drivers must be supported with the new distribu-tion in order to avoid obsolescence.

While developers working with Linux-based ATEs may have the option to develop custom drivers that can be up-graded in the case of a distribution change, this can represent a substantial development cost compared to taking advantage of vendor-supplied drivers. At a minimum, an abstraction layer must be created and updated to ensure driver compati-bility with changing Linux kernels.

Virtualization technology poses one solution that ad-dresses these pain points. By running two or more operating systems (OSs) in parallel on a single controller, engineers can create ATE systems that combine the advantages of open-source operating systems such as Linux with other OSs that support a wide variety of commercial off-the-shelf (COTS) hardware to reduce the risk of obsolescence. In addition,

virtualization can enable ATE designers to reuse a wide vari-ety of existing code, therefore preventing the need to rewrite code sections when older systems become obsolete.

This paper presents an overview of virtualization technol-ogy and explains how it can be used with Linux-based ATEs to maximize code reuse, enable support for a wide variety of COTS hardware devices, and ultimately reduce the risk of ob-solescence. Tradeoffs of virtualization, including potential performance impacts, are also discussed.

Virtualization BackgroundThe term virtualization refers to abstraction of underlying com-puter hardware resources from the OSs running on top of the hardware. Essentially, virtualization technology enables two or more operating systems to run simultaneously on a single controller. While this capability has been used for years in the information technology industry to reduce the size and power usage of server centers, virtualization is being increasingly used in both embedded electronics and ATEs to increase the capabili-ties of a single system without expanding hardware costs.

The key component needed to ensure the correct operation of simultaneous OSs on one computer is a piece of software called a hypervisor or virtual machine monitor (VMM). If multiple OSs are allowed to independently execute instruc-tions that affect a shared system state, called privileged instructions, they can potentially corrupt data and affect each other’s operation. The primary role of the VMM is to take con-trol when OSs attempt to execute such privileged instructions and ensure that access to shared system resources is protected to avoid any problems. VMMs may also play a role in the boot process of virtualized systems, inter-OS communication, and in directing incoming hardware interrupts.

Two major architectures exist for making use of a VMM in a computing system: hosted and bare-metal. In short, hosted VMMs run on top of a host OS such as Windows™ or Linux, while bare-metal VMMs run directly on the underlying

This paper was first presented at the IEEE AUTOTESTCON 2010 (© 2010 IEEE, Proc. AUTOTESTCON 2010, used with permission, [7]). It has been slightly edited and formatted for the Magazine.

8 IEEE Instrumentation & Measurement Magazine August20111094-6969/11/$25.00©2011IEEE

Reducing Obsolescence of Linux-Based ATEs with Virtualization

Casey Weltzin and Sarah Schlonsky

hardware without dependence on a host OS (see Fig. 1.) While a variety of useful hosted virtualization solutions are available on the market today (e.g. VMware Workstation), the remainder of this paper concentrates on bare-metal VMMs, as they offer the I/O hardware control needed for ATEs, as well as other useful capabilities such as real-time operating system (RTOS) support.

There are three major ways that execution may actually be transferred from an OS to a VMM. One technique, called bi-nary translation, replaces privileged instructions during code execution with calls to the VMM where necessary. In contrast to this software-centric approach, hardware-assist techniques do not rely on any modification of instruction code; instead the CPU directly calls the VMM when OSs attempt to execute certain privileged instructions. Finally, where OS source code is available (e.g. the Linux OS) developers can simply modify OSs to call directly into the VMM when needed. These calls are called hypercalls and give ATE designers the most con-trol over exactly when the VMM is called at the expense of additional development work. Although a variety of virtual-ization architectures and underlying techniques can be used in ATEs, the end goal remains the same: running multiple OSs in parallel on one controller.

Reusing Existing Code on Virtualized SystemsOne concrete benefit of virtualization is the ability to reuse and execute code written for a variety of operating systems and versions on a single controller. By running multiple OSs in parallel, ATEs can take advantage of code written for different versions of a Linux OS, or even for different OSs al-together – such as a Linux OS and another RTOS. Running multiple OSs in parallel can greatly reduce the risk of obso-lescence for Linux-based ATEs, as application software or driver changes that mandate the move to a certain OS do not force legacy code to be abandoned, upgraded, or reworked. Instead, legacy code can be run on its native operating sys-tem while new software can run on a different OS (see Fig. 2.)

With a variety of applications running on separate OSs in a virtualized system, it is also important that these appli-cations are able to communicate with each other across OSs. To accomplish this goal, VMMs typically provide a variety of

inter-OS communication methods, including emulated Ether-net or Serial connections and high-throughput shared RAM. Depending on the communication method used, program-mers may be able to reuse standard protocols and APIs such as TCP/IP in their applications or may need to use new APIs that exploit special capabilities of the virtualized system.

Accessing COTS Hardware via a Vendor-Supported OSIn addition to enabling code reuse, virtualization can also help provide an additional layer of abstraction to help prevent the obsolescence of ATEs due to changing Linux versions, which can break hardware support for I/O and other devices. In-stead of running just one OS, consider running both a Linux OS with support for legacy applications alongside a specific vendor-supported OS (either a Linux OS, general-purpose OS, or RTOS) on top of a VMM. With this ATE architecture, design-ers can retain the flexibility of an open source Linux OS, while ensuring that a broad variety of COTS hardware will be sup-ported in the present and future on the vendor-supported OS.

Using the inter-OS communication methods discussed in the previous section, applications running on Linux can make requests and retrieve data from the vendor-supported OS via an abstraction layer. If the Linux OS is upgraded on the ATE system, then existing applications can still take advantage of the available communication methods to call into the ven-dor-supported OS and interact with I/O devices – therefore preventing additional driver development work and poten-tial obsolescence. In effect, upgrade work is minimized to verifying that the communication drivers (e.g. shared mem-ory) function properly on the upgraded Linux OS.

Note that while a two-OS scenario is used as the basis for discussion in this case, scenarios with three or more OSs running simultaneously are also possible using virtualiza-tion technology. For example, one OS can be used to run ATE application code, another to run legacy code written for a

Fig. 1. Bare-metal virtual machine monitor (VMM) software enables running multiple operating systems simultaneously on one computer.

Fig. 2. In a virtualized system, Linux can run alongside a vendor-supported OS to reduce the risk of obsolescence due to kernel changes.

August2011 IEEE Instrumentation & Measurement Magazine 9

different OS or kernel version, and a third vendor-supported OS can provide access to I/O devices. In the case of frag-mented hardware vendor support across OSs or if hardware from multiple vendors is present in a system, running even more OSs may be advantageous.

Combining Real-Time OS (RTOS) and General-Purpose OS (GPOS) CapabilitiesOver time, as the requirements for a given ATE system change, deterministic real-time performance may be needed for certain tasks. In the past, this has necessitated the purchase of an ad-ditional ATE system or the reformatting of an existing system – another example of obsolescence.

Virtualization technology can also help address this chal-lenge. Some bare-metal VMMs are capable of running both general-purpose OSs and real-time OSs in parallel on one controller while maintaining determinism (e.g., the NI Real-Time Hypervisor). With this configuration, it is not necessary to scrap an existing ATE design because low-jitter perfor-mance is needed, and existing code does not need to be ported to run on the RTOS of choice. Additionally, footprint and power usage are reduced relative to using a multi-controller system. In this way, virtualization can help create standard test and simulation systems based on Linux that can meet the needs of multiple tests in an organization over a long period of time.

Caveats Associated with Virtualization SoftwareAs with any technology, virtualization has some drawbacks that should be evaluated when considering its adoption in ATEs. First, because virtualization introduces an extra layer of abstraction between underlying system hardware and the OS layer, there is also additional complexity for designers to deal with including configuring partitioning hardware devices between OSs, interfacing with inter-OS communication meth-ods, and determining how the reboot of one OS will affect the entire system.

Additionally, the reader should note that the VMM inter-vention needed to support multiple OSs running on a single controller is not free in the performance sense. Each call to the VMM (called a VM exit) necessitates backing up CPU state in-formation, which is a form of context switch; this can require tens to hundreds of thousands of CPU instructions that also introduces some jitter into real time systems. The opposite is also true: A context switch must also be made when exiting the VMM and resuming execution of a guest OS.

Although this performance impact does increase jitter on real-time OSs, in practice this jitter can still remain bounded so that the real-time portion of a virtualized system remains deterministic. Therefore, the practical impact of this overhead is a reduction in the maximum deterministic loop rate of ap-plications running on a virtualized system. Applications that do not result in frequent calls to the VMM (those that do not heavily utilize interrupts) typically see near-native perfor-mance, while those that do see performance degradation can

use techniques like polling rather than interrupts to mitigate this effect.

ConclusionToday’s ATE systems must increasingly incorporate open source OSs such as Linux, existing applications written in a variety of languages, and COTS hardware from a variety of vendors – each with its own defined list of supported OS dis-tributions and versions. Each additional software or hardware requirement can, therefore, quickly lead to substantial mainte-nance or a costly obsolescence of the entire system.

Virtualization has proven to be an effective strategy to com- bat this obsolescence. By running multiple OSs on the same hardware, the ATE designer can compensate for requirement changes such as new COTS hardware with the addition of new OSs and abstraction layers, all while maintaining compatibil-ity with the existing code base. As an added benefit, virtualized ATEs are capable of supporting both general-purpose and real-time OSs, which can help prevent obsolescence in situations where a high degree of reliability and very precise timing are needed.

Ultimately, using virtualization for Linux-based ATEs can help test engineers build flexible, scalable ATE platforms that have longer lifecycles than traditional ATEs. By taking advan-tage of commercially-available VMM software, organizations that rely on ATE can reduce their risk of obsolescence and save significant time and cost in the long-term.

AcknowledgementLinux® is the registered trademark of Linus Torvalds in the U.S. and other countries.

References[1] S. Delgado and C. Weltzin, “Using virtualization to reduce the cost

of test,” Proc. IEEE AUTOTESTCON 2009, pp. 439-442.

[2] P. Weaver and M. Ford, “ATE obsolescence solutions:costs and

benefits,” Proc. IEEE AUTOTESTCON 2003, pp. 623-628.

[3] A. Ribière, “Using virtualization to improve durability and

portability of industrial applications,” Proc. IEEE Industrial

Conference on Industrial Informatics 2008, pp. 1545-1550.

[4] R. L. Marion, “Mitigating COTS obsolecense in military test,”

Proc. IEEE AUTOTESTCON 2001, pp. 746-756.

[5] C. Weltzin and S. Schlonsky, “Reducing obsolescence of Linux-based

ATEs with virtualization,” Proc. IEEE AUTOTESTCON 2010, pp. 1-3.

Casey Weltzin (casey.weltzin@ni.com) is a Product Manager for Embedded Software at National Instruments. He focuses on multicore, virtualization, and FPGA technologies as they apply to test, measurement, and control applications. Weltzin received his Bachelor’s degree in Electrical Engineering from the University of Wisconsin-Madison.

Sarah Schlonsky is a Product Manager for PXI Embedded Controllers and VXI at National Instruments. She holds a bach-elor’s degree in computer engineering from the University of Michigan-Ann Arbor.

10 IEEE Instrumentation & Measurement Magazine August2011