Complex imaging systems such as MRI machines may use dozens of specialized processors that need to communicate with each other. iSTOCKPHOTO

Modern medicine is a miracle of technology. Diagnostic tests can identify the presence of complex biological markers with remarkable sensitivity. Exotic sensing modalities and sophisticated mathematical techniques for separating information from noise allow us to view both the function and structure of organs and disease within the body and with continuously decreasing levels of collateral harm. Exponentially expanding knowledge of the genetic codes, protein interactions, and cellular functions that contribute to or inhibit disease—propelled by a corresponding exponential expansion in the technological tools available to observe, engineer, and synthesize the chemical building blocks that underlie those processes—open up vast new possibilities for tailored therapies. And at the root of it all, the cornerstone of that vast technological advancement is the availability of ever-expanding computing power.

Data processing that once required rooms full of equipment can now be performed in a single rack or on a desktop. Quantities of data that once would have been beyond the imagination are now transferred, processed, and stored as a matter of routine. Where the management of data was once centralized, it is now distributed. The data not only travel outward into a few peripheral processors but even further outward into sensors, actuators, user interfaces, and communication channels themselves, and at any and all points in between. Even a simple medical device—say an ECG monitor—may now use several processors within the box for managing data acquisition, waveform analysis, communications, and user interactions. A complex system such as a CT scanner may employ dozens of specialized processors distributed across several rooms. This article describes a protocol for managing command and control communications between systems, which enables them to talk to each other. 

Systems have become complex, sometimes breathtakingly so, and the push toward complexity is accelerating as our expectations about what ought to be possible expand ahead of the technologies themselves. And as those expectations about what is possible expand, so, too, do our expectations about the rate at which those possibilities should be realized. Product life cycles have compressed even as products themselves have become more complex. This is the challenge of modern medical product development: how do we manage both the increasing complexity of our distributed processing systems and the imperative to build them more quickly and less expensively?

The answer, in part, is to have common building blocks for constructing these distributed elements and common tools for managing the interactions among them. The building blocks must increase the modularity of system components and tools to encapsulate some of the complexity into manageable, validated forms.

A fundamental concern in any distributed processing system is synchronization of the various processing components. One building block and tool could be a common method for managing command and control communications between them, i.e., a common language that enables the various modules to talk to each other. Such a language would increase modularity and allow for relatively easy prototyping, to the extent that any component that spoke this language could easily be connected to any other. It would also minimize effort expended in defining and implementing basic component interactions.

If properly designed, such a building block would also mitigate complexity by managing some of the basic system configuration and monitoring tasks, allowing the system developer to focus more effort on the unique aspects of the system that create value and less on the basic infrastructure. The common communication protocol described in this article is designed to provide such a building block, and offers the specification and a reference implementation as a template upon which others may build.

Requirements of a Common Protocol

Several protocols exist throughout industry for managing some of the basic infrastructure of module interactions. The Simple Network Management Protocol (SNMP), for instance, is used in network management systems to distribute basic configuration information and to alert remote servers of events that require attention.¹ The Intelligent Platform Management Interface (IPMI) (and its cousin, the Remote Management Control Protocol (RMCP)) is widely used in the telecommunications and network server world to remotely identify and configure components on a dynamically changing network.² MODBUS is a de facto industry standard for low-level communications among small electronic devices, but has evolved over time such that its packet structure is different on different types of physical channels.³

What these all have in common is a focus on low-level infrastructure functions—for instance, monitoring of individual sensors and reporting of alarms—with very limited support for high-level logical management of complex components.
Support for such low-level module and component management is important for developing distributed processing systems, especially for medical systems, for which many of the individual components are likely to implement hardware-oriented functions such as sensor readout. But a generalized protocol for application-level command and control of complex distributed systems should provide support for complex black box module interactions, which depend only on knowledge of the logical system interfaces, along with those low-level configuration and monitoring functions that are closely tied to internal module functions.

In addition, it is unlikely that a single physical channel (say Ethernet) will be acceptable as a communication standard across all systems and applications, from individual sensor elements to million-dollar imaging platforms. The industry’s experience with trying to forge consensus on the Medical Information Bus is a telling example—large and small device developers fundamentally disagreed on the level of performance (and cost) that a common standard should define. In the real world, multiple physical channels, including RS-232 and RS-485, USB, CAN Bus, Ethernet, and a range of others, will be used in various systems and even within individual systems. Hence, a generalized protocol should support operation on a wide variety of physical channels, preferably using some common packet structure to minimize the need for a similarly wide variety of formatters and parsers.

In particular, the protocol should provide for both low-level hardware-oriented functions and high-level system-oriented functions; operate on a variety of both networked and point-to-point links; support both discrete command and control operations and large data transfers; provide a mechanism to verify message delivery for those messages that require it; and provide a mechanism to identify components on the network as they are brought on or offline.

Embedded Machine Control Protocol

Rather than assembling several existing protocols and ad hoc extensions into a quasi-standard customized implementation, building a new protocol standard from scratch allows for an integrated design. The Embedded Machine Control Protocol (EMCP) is a packet-oriented protocol designed specifically for management of distributed processing elements within an embedded environment. It was developed over the course of several years for use in a large-scale x-ray system. The system incorporated several dozen processors distributed over multiple equipment racks in several rooms and was interconnected both by modern networked communication channels and legacy point-to-point links. That initial implementation was expanded and standardized within Triple Ring Technologies as the standard protocol for use in medium- and large-system development.

The EMCP is an application-layer protocol designed to operate on top of a lower-level packet protocol such as user datagram protocols and internet protocols (known as UDP and IP, respectively) in a networked environment. In a point-to-point environment, the EMCP is designed to work on top of point-to-point (PPP) or high-level data link control (HDLC) protocols. It is assumed that the underlying protocol provides packet framing and verifies data integrity but does not necessarily guarantee message delivery. Because it is not assumed that the underlying protocol will guarantee message delivery, the EMCP provides mechanisms at the application layer to verify message delivery to the application end-point.

To support the general requirements outlined above, the EMCP provides the following features:

• Packet-oriented message transfer.
• Standardized mechanisms for module identification.
• Standardized mechanism for module control and status management.
• Standard mechanism for identifying individual messages and associating responses with queries.
• Application-level control of packet fragmentation.
• Standard I/O-oriented packet extensions to support an I/O device transaction model.
• Hooks in the header for custom protocol extensions.
• Standardized command or query language to support a logical black box transaction model.

Message Format. The EMCP was designed as a packetized protocol rather than as a stream protocol for two reasons: first, it is oriented toward transmission of discrete synchronization messages such as operating commands, events, and configuration values rather than continuous flows of data. Second, it simplifies the design of formatters and parsers because the maximum size of any particular message is defined a priori. Packets comprise a header prefixed to the message data.

The basic header for an EMCP packet without optional fields comprises 24 bytes (192 bits) arranged as 4-byte values. For comparison, 192 bits is the same size as a basic IP packet used for network transmission; UDP (64 bits) and PPP packets (64 bits) are somewhat smaller. The packet provides

• Destination and source identifiers to allow for filtering of message responses based on where they came from and to whom they were addressed. IDs can identify devices by type (e.g., manufacturer and model or standard application programming interface (API)) or by class (logical system function).
• A sequence ID to provide a unique message identifier to reassemble multipacket messages (see chunking section later in this article) and associated responses with requests.
• An error or status code to allow packets to contain both data and status information.
• An operation code (OpCode) and data length to identify the specific message function and the amount of data which follows the header.

The OpCode also includes flags indicating whether the packet is a command (no response required), a query (response required), or a response (only transmitted as the result of a query). Because these fields are present in the header, the parser itself can implement matching of responses to queries and can generate an error condition if an application-defined timeout occurs before a response is received. Hence the query-response handshake can be used as an “acknowledge” mechanism to verify message transmission all the way through to the EMCP end-point (and not merely to some layer of the protocol stack), and no support for message acknowledgement is required by the underlying protocol.

Optional Header Fields and Functions. The EMCP protocol defines optional header fields to support special multipacket, I/O, and chunk management options. Because these fields are optional, transmissions are not burdened with the overhead associated with these functions unless they are required for a particular message. Presence or absence of these optional fields is signaled by flags in the OpCode; several equivalent flags are reserved in the OpCode to allow extension of the protocol to support other optional fields and functions in the future. Because the optional fields are provided as part of the message header, the presence or absence of specific options can remain transparent to the application as features are added over time.

Chunking. The optional chunk fields are used to split large logical application messages over multiple EMCP protocol packets. Conceptually, chunking is equivalent to the fragmentation support provided in other protocols such as UDP and IP. The term chunking was chosen for descriptive purposes specifically to differentiate the fragmentation done by EMCP from that which may or may not be done at other protocol layers. Chunking is provided as part of the EMCP for two reasons. First, it means EMCP can provide support for the transmission of very large messages even if it is implemented on top of a very simple transmission protocol that does not provide fragmentation support. Second, it permits an implementation to provide for some degree of application control over the fragment size. Control over the fragment size enables optimization of the data reading and transmission process when access to data buffers within the hardware environment is relatively slow; by tuning the fragment size appropriately, the hardware access time can be balanced with the message transmission time to maximize the parallelism between them and thereby minimize overall latency in responding to a request for a large data set.

I/O Functions. The optional I/O fields are used to create access to a virtual I/O space at the far end of the EMCP channel. This virtual space is, in effect, a secondary access mechanism that models the far end of the link as a physical device accessible through reads from and writes to a register space rather than as a logical machine that executes abstract operations.

Because the I/O access mechanism operates in parallel with the logical black box transaction mechanism defined by the standard protocol, a particular EMCP end-point may implement one or both access mechanisms. The I/O fields designate the offset, port width, and element count for the transfers; the function code in the OpCode field defines the specific type of I/O operation to be performed. I/O messages may be chunked if the requested I/O transfer exceeds the EMCP packet size.

Multimessage Packets. The optional multimessage count field is used to signal that an EMCP packet contains more than one message. This functionality is provided to reduce the effective message overhead required for small messages that can be transmitted in multimessage bursts. The effect of grouping messages within a packet is to reduce the EMCP overhead by approximately 160 bits per message for messages added to the packet. This method also minimizes overhead associated with the underlying packet protocol by requiring only one protocol packet for multiple EMCP messages.

Standard Messages. The EMCP defines a set of standard message functions, designated by the OpCode and supported by predefined structures in the data fields, for common machine operations. Categories of standard functions are

• Management messages that allow for querying and announcement of module identities and states as well as for some basic module controls.
• Operational messages that access the EMCP end-point as a logical machine that executes abstract operations. The operational API is conceptually symmetrical, in that the general message protocol does not depend on one end-point or another being the master of the channel. Rather, each module publishes the operations it supports, and peers request operations of each other. Operational messages support sending of commands, transfer of logical data, and notification of events.
• I/O messages that access the EMCP end-point as a physical device mapped through an addressable register space. The I/O API is conceptually asymmetrical, in that one module on the link initiates I/O transfers and the other module responds to them—that is, one end-point is the I/O master and the other is the I/O slave. I/O messages support reading and writing data through the register API and generation of I/O interrupts.

The operational messages comprise the standard parameterizable message interface that can be used for common module implementations. The parser handles these messages in a general way and dispatches to application-installed handlers indexed by the ID values provided with the messages. In that way, an EMCP implementation provides a standard package into which customized command and data handlers may be inserted to adapt it to the needs of a particular application.

Benchmarks. The EMCP was developed to simplify the creation of large, highly distributed processing systems. It was intended to be extremely flexible so that it could be used in a variety of applications. That flexibility comes with a cost: the header itself is 24 bytes long—roughly on par with similarly complex network protocols designed for other applications—and is therefore inefficient for the transmission of small data packets, especially through relatively low-speed links.

Although Triple Ring stipulated that the EMCP should be compatible with a variety of physical channels, including RS-232, it may not be the best choice for a system that is implemented primarily using such low-speed channels. Within our own environment, we have operated the protocol using an RS-232 link at 115.2 kbaud but, as might be expected, the throughput, although reasonable for large data transfers (e.g., images), was severely diminished for small command and control packets. As a rule of thumb, EMCP throughput for packets containing a single DWORD data value is about 6–10% of the raw link bandwidth, depending on the overhead of the underlying transport protocol; for packets containing 1024-byte data blocks, the throughput approaches 75–95% of the raw link bandwidth. In applications employing large numbers of small packets, the multimessage packet mode can be used to reduce the effective EMCP overhead, providing throughput approaching 40% of the raw link capacity even for messages with DWORD data.

The primary advantage of the EMCP, of course, is not performance but standardization. To the extent that all modules may be controlled, through any type of physical channel, using a common command and control protocol application, development becomes much simpler.

Conclusion

The EMCP provides a common command and control protocol for the design of complex distributed processing systems. Use of such a common protocol facilitates modular design and reduces the complexity of system management by providing standard mechanisms for module identification and control as well as common formats for command and data transfer.

References

1.    RFC1098, “Simple Network Management Protocols (SNMP),” April 1989, Internet Engineering Steering Group.
2.    IPMI, Intelligent Management Platform Interface Specification, Second Generation v2.0, Rev. 1.0, February 12, 2004, Intel/HP/NEC/Dell.
3.    “MODBUS Over Serial Line Specification & Implementation Guide, v1.0,” December 2, 2002; available from Internet: www.modbus.org.

Augustus P. Lowell is chief systems architect at Triple Ring Technologies (Newark, CA). He can be reached at
alowell@tripleringtech.com.