Advanced driver-assistance system (ADAS) complexity is increasing exponentially as more sensors and algorithms are combined to provide greater functionality and safety. Many tools, including FPGAs, are being employed to address these issues.
I talked with Willard Tu, Senior Director of Automotive at Xilinx, about the pain points and best practices for developing ADAS and autonomous drive systems. In particular, we discuss Xilinx’s Automotive (XA) devices and its Adaptive Compute Acceleration Platform (ACAP).
What are the unique demands from advanced automotive systems and applications addressed by Xilinx’s Automotive (XA) devices? Can you provide a brief background on the company’s history in the space?
Xilinx has been working with automotive customers and partners for over 20 years, beginning with the use of our devices for infotainment and driver information systems. In the early days, Xilinx FPGAs were used primarily to address robust, low-latency connectivity and data (audio/video) processing. In 2011, we launched our first system-on-chip (SoC) device.
Xilinx SoCs combine an automotive microcontroller architecture (e.g., hardened CPUs and peripherals) with a scalable set of programmable logic. This moved Xilinx devices to the center of automotive ECU designs and enabled developers to create unprecedented optimized designs through efficient partitioning between embedded software and customized hardware acceleration.
The device architecture created unique value in ADAS and automated-driving (AD) applications where developers rely on innovative approaches for sensor processing to differentiate their product performance and features. These Xilinx advantages are being extended with the launch of our ACAP devices, which add specialized elements for on-chip data movement and neural-network processing as we usher in AI processing to automotive applications.
Today, we see a greater demand for higher programmable-logic performance and capacity, as well as I/O capabilities to enable high-speed data aggregation, pre-processing, and distribution (DAPD) and compute acceleration to support L2 to L4 ADAS and AD applications. In this application space, customers demand both device security and functional safety. Fortunately, Xilinx has a long history with those requirements from other markets we serve (e.g., A&D and ISM) and our automotive devices can leverage that heritage.
In addition, customers value the adaptability and scalability provided by the Xilinx Automotive portfolio of devices. Adaptability is critical as algorithms and methodologies for ADAS/AD continue to be innovated, while scalability is a key enabler to creating cost-effective platforms that need to meet an ever-varying set of feature bundles.
From forward-camera and surround-view systems, to 4D imaging radar and LiDAR sensors, to in-cabin monitoring solutions for hands-free, gesture-based controls and alertness monitoring, Xilinx solutions provide extremely reliable real-time performance at low power consumption, which is essential.
Whether reacting to dangers on the road or ensuring that comfort features run smoothly as expected, automakers need to meet strict demands for computation latency, performance, power efficiency, and adaptability for both distributed edge sensors and centralized domain controllers to optimally handle a growing number of complex and diverse applications. The XA devices address these demands.
What unique problems are automakers, tier 1s, and robotaxi developers facing with autonomous-vehicle development that Xilinx Versal/ACAP and traditional FPGA technology can address? How does FPGA technology address these challenges better than current GPU and CPU offerings?
Latency is a critical factor for enabling autonomous vehicles, particularly for safety. Overcoming latency issues is one of the bigger challenges faced by automakers, tier 1s, and robotaxi developers. The ability to create optimized parallel-processing pipelines within programmable logic, also referred to as FPGA fabric, that run independently and simultaneously with their own dedicated logic and memory resources leads to high performance with extremely low latency, thus addressing this challenge.
In addition, the ability to parallelize operations in the programmable logic means clock rates can be reduced when compared to GPUs and CPUs. This leads to performance efficiencies in supporting increasingly complex automotive systems, and it offers significantly lower power consumption versus GPUs and CPUs. That means greater reliability and performance while staying within automotive thermal envelopes.
Beyond latency is safety. To address safety-critical applications, XA devices offer certified “safety islands” and can process data from multiple sensors and cameras for real-time responses to other vehicles, pedestrians, animals, and unexpected road hazards. The sensor data can be analyzed and fused for effective redundancy and the programmable logic offers the unique ability to create application/function-specific diagnostic circuits and cross monitors to enable automotive developers to meet ASIL requirements—no other technology on the market can do that.
To satisfy the need for increased computational performance due to the adoption of AI processing for various automotive applications, Xilinx expanded the 16-nm Zynq UltraScale+ MPSoC portfolio by automotive-qualifying devices that offer nearly 3,000 DSP slices (multiply and accumulate units) and correspondingly scaled raw logic and memory resources. Xilinx’s new Versal ACAPs, furthermore, provide even greater flexibility with scalar, adaptable, and AI engines that deliver performance improvements up to 20X over today’s fastest FPGA implementations and 100X over today’s fastest CPU implementations (see figure).
What are the greatest pain points for developing and implementing ADAS and AD applications that customers ask you to address with new solutions?
Power consumption, latency, and functional safety are all challenges for automotive customers in ADAS/AD. Xilinx continues to innovate and deliver in these areas.
One of our autonomous-vehicle partners, Pony.ai, found that switching to XA FPGA devices resolved performance issues for its sensor-fusion system, which previously saw an output discrepancy that measured nearly half a car’s length—a lag that would prove extremely dangerous in real-world scenarios. For our customers developing ADAS and AD solutions, the wall they hit is not having enough compute power and low latency to manage multiple complex computations that are critical to ensuring that their vehicles are as safe or, ideally, safer than manual human alternatives. Our devices provide reliable performance and the ability to perform greater and multiple complex computations closer to the action to remove the latency barrier for the most critical applications.
In another example, when designing the MBUX Interior Assistant, Daimler engineers were faced with a unique challenge. It had to design the entire MBUX Interior Assistant computing subsystem within the roof of the vehicle, which is a very thermally constrained environment. Daimler engineers benchmarked many powerful computing platforms, but Xilinx was chosen because it had the best performance-per-watt and lowest latency, combined with automotive-grade qualification.
Are XA devices being implemented in end-user applications or primarily being used for R&D today?
Our devices are used across the board for both development and in production. Cumulatively, Xilinx has shipped more than 170 million devices globally for automotive use, with 70 million used for production ADAS systems. Xilinx works with over 200 automotive companies, comprised of major tier 1s, OEMs, and startups globally.
The myth that programmable-logic/FPGA technology is only used for development and prototyping has been proven wrong many times over as Xilinx devices are in their third generation of forward-camera and ADAS central modules. Since the 90-nm process node, Xilinx innovation has created very cost-competitive solutions in the automotive space.
Our latest devices in the 16-nm portfolio, the recently announced XA Zynq UltraScale+ MPSoC 7EV and 11EG, will continue to enable ADAS and AD applications through to production, with these particular devices developed specifically to handle the heavy demands of centralized domain controllers.
What specific applications do the new Zynq UltraScale+ MPSoC 7EV and 11EG each enable, and what advantages do they provide for the automotive industry over other solutions? What hardware specs/features are most critical to accelerating deployments for autonomous vehicles?
The XAZU7EV and XAZU11EG were both developed in response to customer demands for DAPD and to accelerate compute for L2+ and L4 ADAS and AD applications. As the number of camera, radar, and LiDAR sensors expand and the data from each sensor becomes more dense (e.g., camera resolutions moving from 1 to 8 Mpixels and beyond), the demand for dedicated independent processing pipelines is increasing.
At the same time, ASSP/GPU vendors attempt to throw more CPUs at the problem and “thread the needle” in defining the appropriate number and type of interfaces, guaranteeing freedom-of-interference (a foundation of functional safety) and managing unprecedented software architecture complexity, which is a huge challenge. Xilinx’s introduction of the XAZU7EV and XAZU11EG devices is just another step in our ability to provide scalability in I/O and support for fully independent, yet simultaneous, sensor processing pipelines. This makes these devices extremely effective in the DAPD role.
The larger device of the two newest additions, the XAZU11EG, provides over 650,000 programmable logic cells and close to 3,000 DSP slices—2.5X that of the previous largest device. The DSP slices can be arranged in a “systolic array” of multiply and accumulate units that form the heart of Xilinx Deep Learning Processing Units (DPUs) for neural-network computations.
The XAZU7EV features a video-codec unit for h.264/h.265 encode and decode, and the XAZU11EG features 32 12.5-Gb/s transceivers and four PCIe Gen3x16 blocks. Between the two new XA devices, we’re enabling automotive customers to move forward with DAPD and compute acceleration within strict power envelopes to put AD vehicles into scalable production.
What are the safety requirements to implement ADAS and AD applications, and at what level do the new devices perform? What are the most notable performance specifications for the XA 7EV and 11EG enabling this?
The entire XA Zynq UltraScale+ MPSoC portfolio is qualified according to AEC-Q100 test specifications. The devices integrate both Xilinx programmable logic and a feature-rich 64-bit quad-core Arm Cortex-A53 and dual-core Arm Cortex-R5-based processing system that’s certified to ASIL-C level in the low-power domain.
The programmable-logic domain is unique from a functional-safety perspective, as it allows developers to create application-specific diagnostic and cross-monitoring circuits that enable them to meet their specific functional-safety goals. No other technology enables this. The XAZU7EV and XAZU11EG MPSoCs are scalable extensions of the other XA MPSoC devices in the family.
Can you provide real-world examples of how the new 7EV and 11EG enable new applications, advance existing ones, or a mix of both?
In addition to providing Xilinx’s entry into centralized domain controllers for the automotive industry with the highest performance to date, the XAZU7EV and XAZU11EG enable faster development and deployment for testing and production vehicles while leveraging FPGA power, efficiency, and reliability to manage critical DAPD processes. For automotive customers continuing to develop solutions toward the holy grail of L5 fully autonomous vehicles, the added performance in tandem with Xilinx’s developer tools will unlock new capabilities that can accelerate their evolution.
For automotive applications, ensuring safety at every stage of the technologies’ evolution is critical for passengers as well as other vehicles and pedestrians on the road. Ensuring fast, precise responses, especially for ADAS and AD—whether reacting to a drowsy driver behind the wheel, another vehicle driving recklessly, a cyclist in your blind spot, or an individual crossing a poorly lit street—can mean the difference in saving a life.
Xilinx recently launched the Vitis Unified Software Platform for developers. Does this platform apply for XA devices as well? If so, what advantages does it offer in conjunction with the new 7EV and 11EG devices?
All XA devices are supported by Vitis and Vitis AI. It allows developers to work in more popular languages like Python and C++ and provides highly optimized domain-specific accelerated libraries, including for vision and image, alongside common libraries such as for math and linear algebra. As a result, it streamlines the development process and enables more developers outside of hardware development to work on these projects.
Our full line of XA devices can be programmed with Vitis. And by leveraging our comprehensive developer libraries, the broader developer community can effectively build and implement automotive applications including the most demanding for ADAS and AD faster.
Willard Tu is a senior director at Xilinx, where he leads global business development, product planning and marketing strategies for the company’s automotive business.
Tu has spent over two decades at the axis of the semiconductor, automotive and computing industries. He was previously at Arm, where he evangelized CPU IP and developed ecosystems to support Arm's growth in automotive. At NEC Electronics (now Renesas), Tu led the North American Automotive sales and marketing teams, growing sales to over $150 million.