Originally Published MEM Fall 2008

OPERATING SYSTEMS

(click to enlarge) Figure 1. A high-capacity blood analyzer from Abbott Labs.

The underlying process technologies employed in medical diagnostics applications continue to evolve and drive the need to add different features or processing steps to diagnostic test equipment. At the same time, economic forces drive equipment manufacturers to increase processing capacity and speed. Consequently, the software that supports the operation of these machines continues to become more complex.

Consider a high-capacity blood analyzer (see Figure 1). Before a blood donation enters the blood supply, it must be tested for evidence of exposure to viruses that might cause disease, such as hepatitis, HIV, and other retroviruses. The typical screening process involves making several assays on each blood sample. High-capacity blood analyzers, such as those used by the Red Cross, consolidate and automate this testing into a single instrument. Doing so reduces the errors associated with manual handling of samples and improves efficiency and throughput of the process.

In one high-capacity blood analyzer, every blood sample can go through up to six screening tests at once. According to standard protocol, when a sample reacts positively to a test (indicating that it may be carrying a harmful virus), it must be retested two more times. If two of three results for a sample are reactive, then the test is considered to be positive. The complete testing process for each sample takes approximately one hour from start to finish, and at any point in time, samples in process may be on their first, second, or third test cycle. Therefore, in order to meet productivity goals, a large number of samples must be processed and tracked simultaneously. The need to maintain the integrity of each sample test is critical. Managing hundreds of concurrent tasks reliably is a challenge that can be met by a real-time operating system (RTOS). Time-deterministic tasks managed by the RTOS in this application include:

• Bar code scanner interface (to read sample containers).
• An x-y-table and sample transport motion system.
• Pipette control to manage the fluid dispensing.
• Sample scheduling.
• Resource and reagent monitoring.
• Heater control.

An RTOS and its real-time applications are designed to execute according to predictable and repeatable timings, enabling the control and status of the device's physical systems to be managed on a predictable schedule. This minimizes the chance of an event being missed, such as the completion of a test or a sensor input signifying a change in system status. In contrast, a general-purpose operating system (GPOS) excels at supporting human-directed functions and enterprise-level applications that don't happen on a predetermined schedule and can cause random delays in processing. In this regard, RTOSs are referred to as being deterministic and GPOSs are nondeterministic.

Many medical devices can benefit from both types of operating environments. They can use an RTOS to manage concurrent machine-control processes, reducing the potential for errors and simultaneously use a GPOS, such as Windows, to provide human interface functionality and database functions.

Definition of a Real-Time System

In the most general sense, a real-time system is any system that consumes or produces data as they occur. In other words, events and actions in a real-time system occur in synchronism with the production or consumption of their associated real-time data. Therefore, a real-time system can be defined as one whose correctness depends not only on the logical results of the computation, but also on the time at which the results of that computation are produced. This general definition applies to a wide variety of systems including closed-loop (feedback) control systems, multimedia delivery systems, online transaction processing systems, telecommunication switching systems, automated manufacturing systems, medical diagnostic equipment, and process control systems.

Real-time systems can be divided into two categories: hard real-time systems and soft real-time systems. A hard real-time system must, without fail, respond to and operate on its data and events within a predetermined time window and finish within a bounded period of time. In contrast, a soft real-time system must, on average, respond to and operate on most data and events in a timely fashion, but will not be considered to fail if some deadlines are missed or some data are dropped.

A motion controller is an example of a system that depends on hard real-time conditions to ensure that the quality of its service stays within acceptable system limits. Without a hard real-time guarantee, many systems would either become unstable or exhibit an unacceptable quality of service.

A video playback system is a soft real-time system. If video frames are dropped during playback, the end-user may see occasional noise in the image; a few lost frames affects the quality of the displayed picture, but the usefulness of the picture is usually not lost. Obviously, the acceptable level of degradation in the quality of the video playback system depends on the nature of the information being displayed and the expectations of the end-user. Employing a hard real-time system would certainly improve the quality of the system and could even increase the performance of the playback system.

"Real-Fast" Does Not Equal "Real-Time"

It is important to note that a real-fast platform does not automatically translate into a real-time system. A hard real-time system responds predictably to its events. One can count on real-time tasks to start and complete within a bounded period of time. Such a system is deterministic because its behavior follows a predictable cause and effect relationship.

A bounded response to events is the key to defining a hard real-time system. Real-time systems require determinism to ensure predictable behavior. Without determinism, systems cannot be called real time. And without bounded determinism, systems cannot be classified as hard real time. GPOSs, such as Windows and Linux, are not deterministic; unfortunately, some designers believe that having a fast (or multicore) processor ensures determinism.

Fast processors, memory, and peripherals affect the aggregate performance of a system, but they don't affect a system's bounded determinism. Given the way that a GPOS assigns priorities and manages tasks, a faster processor will not change the worst-case event response time. Misplaced priorities assigned to software tasks (especially drivers) usually result in missed events, not slow hardware.

Sufficiently fast hardware is a condition that must be met to build a real-time system. But more important are the services provided by the underlying RTOS, and the determinism of those services, that run on that hardware. The types of services and features typically provided by an RTOS include the following:

• A deterministic scheduling policy.
• Control over the priority levels of every element in the system.
• Bounded event latencies and support routines.
• Multiple threads per process (aka lightweight threads).
• Multitasking of both threads and processes.
• Precise thread and process synchronization mechanisms.

GPOSs cannot ensure the first three items on the list above and, therefore, are inadequate platforms, on their own, on which to build reliable hard real-time systems.

Reconciling Compatibility with New Features

So how can medical devices gain the advantage of the human-directed resources of an OS such as Windows along with a measure of real-time determinism? It is possible to combine these two requirements by using an RTOS that runs alongside Windows on a single hardware platform.

To accomplish this, developers can benefit from the latest advances in multicore processor technology. Multicore processors respond to the need for combining new features with legacy code in multiple operating environments in the same system. A highly integrated system can be constructed with real-time software components and human-directed elements running on separate cores.

The preservation of legacy software code often influences decisions made in embedded system design. This is particularly critical for long life cycle applications such as medical devices for which stringent agency certification or approval is required. The risks involved in recoding proven time-critical software can be huge. Besides the significant cost of FDA recertification, subtle incompatibilities can erode customer confidence and can destroy chances of meeting market windows. For example, the American Association of Blood Banks states that approximately 8 million volunteers donate about 15 million units of whole blood every year; confidence in test results is of utmost importance. Many embedded system developers need to maintain compatibility at all costs, even while adding competitive new features to their products.

Virtualization Is the Solution

Special software is needed to make this real-time environment work. The software that manages the hosting of multiple operating environments must support virtualization of the processor's hardware interfaces, so that multiple operating environments can share the processor's I/O without conflict. In this context, the concept of virtualization refers to the use of software to allow a piece of hardware to run multiple OS images at the same time. The technology­ got its start on mainframes decades ago, allowing administrators to avoid wasting expensive processing power, and it's used widely in servers today. However, the application of virtualization to embedded, real-time computing environments is relatively new.

The challenge of constructing a virtual machine environment that enables multiple operating environments to share a single I/O structure is more complicated than it seems. Historically, virtual machine management software has emulated the entire underlying machine, including the I/O devices, giving each guest operating environment what it thinks is complete control over the hardware. The virtual machine manager (VMM) decides which guest OS should have access to the I/O, and the VMM handles all the I/O transactions on the guest's behalf.

The problem with this is that the VMM needs to evolve quickly to offer the functionality of the underlying silicon as it evolves. For example, new video cards are being developed all the time, and the latest ones might not work in an emulated I/O environment that was developed to be compatible with older interfaces. Even if the VMM can support all the features of contemporary I/O interfaces, using a VMM still imposes a performance penalty that the guest OS would not have if it were interacting directly with the silicon. Therefore, a different approach to VMM is needed in order to support the latest I/O hardware enhancements and yield maximum performance in deterministic processing environments.

(click to enlarge) Figure 2. Multicore CPUs can host multiple guest operating systems, including legacy software environments.

To address this problem, designers need to look for VMM software that facilitates building virtualized multi-OS systems by partitioning the chip's interfaces. Such systems assign the proper devices directly to the I/O tasks that control them. Figure 2 shows such a system. In this system, the VMM doesn't emulate the I/O interfaces of the underlying machine. Instead, it controls access to the I/O by ensuring that only authorized operating environments can access specific performance-critical I/O devices. For example, as shown in the diagram, the VMM ensures that the main operator display is only accessible to the GPOS, in this case Windows.

This notion of applying I/O exclusively to a specific virtual machine is also essential to guaranteeing real-time responsiveness. A real-time virtual machine needs to have direct physical access to its dedicated I/O. Without exclusive physical assignment of pertinent I/O, one runs the risk of waiting indeterminately for access to key devices. If another virtual machine has access to an I/O device, because the I/O is multiplexed, the wait can be significant. Even if only one virtual machine ever accesses a specific I/O device, when a request is made to access that I/O, a VMM that virtualizes I/O must translate this virtual machine request into real I/O accesses to the physical hardware. This translation step is an unnecessary and time-consuming process.

Likewise, real-time response-critical hardware would be restricted to be accessed only by the RTOS running on the system, and other legacy I/O interfaces would be restricted to being accessed by legacy application software.

Virtualization: Intel Architecture Processors

The most popular multicore processors are those that support the Intel processor architecture because this architecture can run PC-compatible software; platforms using this architecture are the most prevalent in the market. Intel Architecture (IA) processors provide special hardware facilities that enable the construction of real-time virtual machine management software. Specifically, they support four distinct privilege levels of application execution named rings 0 through 3. Ring 0 executes at the highest privilege level, and ring 3 at the lowest. In typical practice, only two levels are ever used: rings 0 and 3, referred to, respectively, as supervisor-mode and user-mode. Operating systems that utilize these execution modes are referred to as protected-mode operating systems; the OS executes instructions in supervisor-mode and its applications execute in user-mode. Drivers might execute either in supervisor- or user-mode, depending on the architecture of the OS and the nature of the driver.

On an IA processor, protected-mode guest OSs typically expect to run in supervisor-mode with full access to the underlying CPU registers and data structures. Therefore, one of the most difficult jobs a VMM must perform is to intercept guest OS instructions that modify supervisor-mode registers and data structures. The VMM must then simulate the effect of those instructions inside the virtual machine on which the guest OS is running.

Intel introduced hardware support for solving this problem when it came out with Intel Virtualization Technology (Intel VT) a couple of years ago. This hardware support added an overarching operating mode to the processing units, called VMX root, where VMM software executes with final control of the CPU hardware. Now, a VMM can intercept key supervisor-mode operations executed by any software operating outside of VMX root without requiring a priori knowledge of the guest OS binaries or internals. Using this hardware assist for virtualization, one can build a VMM that hosts protected-mode operating systems executing in ring 0 without giving up control of key CPU resources. Also, Intel VT provides a way for the VMM to implement virtual interrupts, an essential feature for hosting a real-time guest OS.

Low-interrupt latency, direct access to specialized I/O, and the assurance that a VMM won't "time slice away" the determinism and priority of real-time tasks are all key requirements of a real-time virtual machine. The combination of multicore CPUs and Intel VT are an ideal platform on which to move beyond multi-OS real-time systems that use static virtualization and build a real-time VMM.

In addition to being well supported with tools and application software, PC-compatible hardware yields the benefit of being able to run the development software environment directly on the target hardware. This saves development time and effort compared with using separate development workstations. So, migrating the system processor to the latest IA hardware platforms and software technology saves costs and provides access to the latest industry-standard hardware resources. Such resources include up-to-date interfaces for communication and I/O, such as USB. Additional advantages include the ability to incorporate new communication protocols for interaction with external systems and to adapt more-sophisticated data-reporting methods.

Conclusion

Medical device manufacturers can reap system cost benefits and add new features to their products by upgrading their systems to incorporate a computing platform that uses one of the new multicore processor chips. Using a multicore CPU, it is possible to consolidate complex devices into tightly coupled systems that have different on-chip processor cores performing different functions. In the past, they may have used multiple loosely coupled discrete processing units to accomplish those functions.

For example, real-time tasks can be run under an RTOS on one core of a multicore chip, and enhanced human interface functions can be hosted on a GPOS on another core. And, with the proper operating environment, legacy code from older applications may be run with minimal or no modification on a new hardware platform.

Using processors that provide the hardware support to enable VMM software makes it all work in a way that accomplishes real-time responsiveness and preserves investment in legacy software.

Paul Fischer is senior technical marketing engineer for TenAsys Corp. (Beaverton, OR).