Technology Editor Cabe Atwell and I got a chance to ride with Ambarella’s General Manager, Alberto Broggi, (watch the video above) to check out the company’s latest software running on Ambarella’s CV3 family of system-on-chips (see figure). The car was the same one we took a ride in at the 2024 Consumer Electronics Show (CES), watch “Advanced AI SoC Powers Self-Driving Car.”
This time around, we got to see generative AI in action with a vision language model (VLM), which is a form of large language model (LLM) that runs on a single CV3 chip. It was tied into the cameras and sensors. The CV3 provided multi-sensor perception, fusion, and path-planning support for the advanced driver-assistance system (ADAS).
In the demonstration, the VLM identifies objects and their state, such as a red light reporting the details in text like a conventional LLM. Typically, this information would be provided to the path-planning support to provide feedback to the driver.
The CV3 includes a dozen Arm Cortex-A78AE and six R52 cores. The latter operate in lockstep pairs. It integrates an advanced image signal processor (ISP) and a dense stereo and optical flow engine to help with sensor fusion. The ISP is designed to handle low-light conditions. It can drive displays directly with an onboard GPU. The CVflow AI neural vector processor (NVP) engine handles the VLM support.
