Designing a High-Performance Real-Time Recording System

In today’s world of high-speed A/D converters operating in the gigahertz range, real-time signal recording has become a challenging task that requires specialized hardware and intelligent application software. When designing a real-time recorder capable of streaming sustained data to disk at rates of 2 GB/s and higher, the system developer has to consider the limitations presented by the recorder’s operating and file systems, the limitations of disk drive technology, the hardware interfaces, and the redundant array of independent disks (RAID) controller technology.

Fortunately for the application developer, serial fabrics have emerged to provide the high-speed interfaces required to move this data, disk drive and RAID host-bus adapters (HBAs) have begun to exploit serial interfaces, and finally, the emergence of solid-state drive (SSD) technology provides a performance level previously unattainable in real-time recording systems. Developing software that can take advantage of these new technologies presents a challenge that can be met by understanding some key concepts required to build a real-time recording system.

Designing for Real-Time Recording

For a recording system to be considered real-time, it must capture every sample of data provided by the front end with absolutely no loss. This needs to happen consistently to create confidence that the system will perform in the most mission-critical situations. To achieve this level of performance, the application developer must start with reliable hardware and use the facilities it provides to ensure real-time performance.

When choosing a platform, it is important to consider the constant requirement to move data faster today than could be done yesterday. With this in mind, the commercial server PC market provides an excellent platform choice since it benefits from the consumer-driven requirement to move, process, and store greater and greater amounts of data every day.

Within the server PC, microprocessor technology is constantly advancing in both processing speed and architecture. Memory interfaces can stream data at rates of 10 GB/s and higher. Serial fabrics have provided PCI Express (PCIe) Gen. 2 with 16x interfaces capable of maximum data rates of 8 GB/s. Finally, SATA II offers disk storage rates of 3 Gb/s to a single drive.

The commercial PC motherboard roadmap includes transitions to PCIe Gen. 3 capable of doubling the data rates of current technology. SATA III provides storage rates of 6 Gb/s to a single drive. The latest Intel processors are approaching 4-GHz clock rates with hex and octal cores while memory continues to get smaller, cheaper, and faster. This supplies a solid foundation to build on as new high-speed digitizers are introduced.

Figure 1. System-Level Block Diagram

Recording System Overview

Once the foundation of the recorder is established, it is essential to provide a front-end I/O device capable of streaming data in real time. As shown in Figure 1, this device typically consists of one or more high-speed A/D converters or digital interfaces that acquire data at a constant rate and a set of DMA engines that moves data off the device. In a real-time system, these DMA engines are the most critical feature of the I/O device since their design dictates how well this hardware can stream the data and maintain its real-time performance requirement.

While the DMA engines are responsible for moving data off the device, it is the PCIe engine inherent within the device that provides the path to the server PC’s system memory. It is here that the data is buffered and made available to the storage device.

The server PC’s PCIe interface is the path from the I/O device to the system’s memory. It is essential that the I/O device provide a sufficiently fast PCIe interface with the bandwidth required to maintain the data rates of the front end.

The I/O device must have the capability to stream data continuously with no software intervention, allowing the hardware to be solely responsible for the data movement. The latest high-performance I/O devices provide intelligent DMA chaining that enables the system developer to tailor the data flow to maximize performance and assure the system meets the real-time requirements.

The DMA engine should allow the user to chain large buffers of data and provide many chains in a link list. It is the application developer’s responsibility to create a large circular buffer consisting of many chains, each of considerable size. Buffering this data in such a way will allow the system to absorb any momentary latency hits that can be caused by a number of external factors.

Storing Data to Disk

Buffered data must be sent to disk at data rates that match those of the front-end I/O device. Utilizing off-the-shelf high-performance RAID HBAs allows the developer to take advantage of the inherent features and functionality provided including selectable RAID-level control and automated disk recovery facilities.

The challenge in selecting the best RAID controller lies in finding one that reliably meets the sustained streaming read/write requirements of the system. Additionally, these RAIDs provide user-selectable parameters such as cache size, caching method, stripe size, command queuing, and other settings to maximize the performance of the recording system.

In multichannel recorders, it often is beneficial to create multiple RAID drives consisting of different disks for each channel but controlled by the same RAID controller. By assigning independent logical drives to the recorder’s channels, the RAID is able to write to contiguous disk space as much as possible. This is especially important when using standard hard disk drives (HDDs) that use rotating media to store information. Jumping around between multiple files in a multichannel recorder takes much more time than streaming to contiguous disk space. This makes the multidrive approach preferable in this situation (see sidebar at end of article).

It also is important to recognize the nonlinearity of HDDs when developing a recorder. Since the density of an HDD remains constant through the disk and the rotation speed remains constant, the read and write rates of a disk fall as HDD accesses move from the outer edge of the disk to the inner edge. This is because the circumference of the outer edge of a disk is longer than the circumference of the inner edge. Since the disk rotates at the same speed for either edge and the density is the same, the disk will provide many more bytes per second on the outer edge than the inner edge.

Figure 2. HDD Tune Plot Showing the Disk Data Rate as It Changes Throughout the Physical Drive

Since disks present their logical addressing from the outer edge to the inner edge, disk read and write rates fall as a disk fills up. This can be seen in the screen plot of Figure 2.

RAIDs built on several disks provide similar nonlinearity in their data rates. Because of this, the system developer either must include enough drives in the RAID to meet the maximum data-rate requirement for the entire volume or limit the size of the drive volume to the percentage of disk space that will meet the system’s data-rate requirement.

In the case of a RAID with the performance shown in Figure 2, if the system developer were required to build a recorder that could maintain 500 MB/s, the drive volume should be formatted to use about 50% of the total disk space. This is achieved simply by formatting the drive to the appropriate size within the operating system.

The developer must leave a sufficient amount of overhead when selecting the formatted disk amount. In this case, while the system may keep up with the 500-MB/s requirement for almost 70% of the drive volume, 50% is a much safer number, provided it supplies enough storage for the application.

When dealing with non-real-time operating systems like Windows and Linux, it is important to minimize the operating system’s impact on the recording application. The amount of processor intervention in the recording software can be minimized by dedicating the data transfers to the DMA controllers and leaving the processor to manage the data flow. The developer also should elevate the priorities of real-time tasks within the application and keep background tasks at lower priorities to avoid impacting the recorder’s performance.

Combining all of these techniques will help the system developer to create a highly dependable real-time instrument capable of recording and playing data at very high sustained data rates. This sets a foundation for a quality recording system, but there are other design considerations that must be addressed.

Client-Server Architecture

When designing a real-time recorder, it is important to provide an intuitive and easy-to-use control interface. The easiest way to achieve this is through a graphical user interface (GUI). Not only should it control the recorder, but the GUI also should provide facilities to manage the data files, utilities to monitor and analyze the signals being recorded, and constant status information.

The GUI must have the capability to run either locally on the recorder or remotely on an independent PC or other device. Being able to run on a remote device allows the user the ability to put the recorder in an environment that may not be appropriate for the operator. The recorder may be positioned closer to the sensor or antenna source, placing the A/D or digital I/O interface near the signal of interest.

The best way to provide both local and remote control of the recorder is through a client-server architecture. This architecture provides a socket-based communications link between the client GUI and the server recording application, separating the real-time portion of the recorder (the server) from the non-real-time portion (the client). The client then can connect to the server whether the client sits on the server PC itself or on a PC connected to the server over Ethernet.

By sending messages to the server, the client GUI can control all aspects of the recorder. This includes the capability to start and stop recordings, set up front-end hardware parameters, monitor incoming signal information, and request status information from the server.

The interface for this messaging structure should be defined in an application programming interface (API). While the API is used by the GUI to communicate with the server, it also can be provided in a format that allows the recorder to be integrated into a larger system.

User API

By providing a user API, the recorder becomes more than a stand-alone instrument; it also serves as a development platform. This allows flexibility while providing the out-of-the-box experience of an instrument all in one product.

To define an API that can be easily integrated into a larger application, it is important to establish the API routines in a simple and straightforward manner. These routines should abstract the user from details of the message building, the socket interface, and other intricacies of the recording architecture. Error checking should be included in the recording server, allowing the user to receive error codes in response to failed messages.

The API not only should contain routines that control the interface, but also routines that obtain general status information, perform built-in-test facilities, and allow the user to view snapshots of the data prior to and during recording. Combining these facilities provides the capability to create a well-featured recorder application as part of a bigger system.

Other Design Considerations

One of the problems engineers often have to deal with after recording data in the field is offloading it to a system that will perform the analysis, post-processing, and archiving. This process should be simple and quick, minimizing the downtime experienced during offload. There are several strategies to consider here.

The use of a nonproprietary file system to record data offers the capability to instantly access recordings as standard files on the recording device itself.

By providing the disk storage as hot-swappable SATA drives, field engineers can simply swap out disks filled with mission data for fresh ones and transport only the data disks to their analysis system. By adding a RAID controller to the analysis system, similar to the one designed into the recorder, data can be accessed instantly on the analysis machine, avoiding the offload process completely. Additionally, the recorder is instantly available to collect new data as soon as the disk swap is made, greatly decreasing downtime in the field.

Data files provided to the analysis system must contain critical information related to the recording event itself. This can be accomplished by adding a small header to the beginning of each data file. The parameters stored in this header should be well defined, containing all the critical information about the recording including timestamping and I/O module settings and general information about the recording session itself. Additionally, user-settable fields must be reserved, allowing the user to add information to the file header.

The information stored in the file header can be used by system analysis and signal visualization tools. Integrating these tools into the recorder provides a robust product, one that allows field engineers the ability to verify their signal integrity prior to, during, and after a recording session.

Summary

It is important to consider all of the qualities inherent in a high-performance, user-friendly system when creating a real-time recorder. The use of a high-performance PC allows us to take advantage of the latest technology while the use of a nonproprietary file system provides us with the convenience of immediate access to the data files. A GUI offers an easy-to-use control panel while the availability of a user API supports the capability to integrate the recorder into a larger system application.

By building this platform on a well-defined client-server architecture, both of these facilities are available. Integrating this with a high-performance front-end I/O module and the latest technology offered by the commercial PC market assures that the system will provide a satisfying user experience with excellent reliability.

About the Author

Chris Tojeira is the technical director and chief architect of the real-time recording systems product line at Pentek. Mr. Tojeira has more than 15 years experience in the electronics industry and previously held positions as strategic account engineer and applications engineering manager at Pentek. He holds a B.S.E.E. from Rutgers University. Pentek, One Park Way, Upper Saddle River, NJ 07458, 201-818-5900, [email protected]