Low-Power Vision Processing for Physical AI

Physical artificial intelligence (AI) typically equates to robotics or some variation thereof. One of the key components of such a system is vision processing using large language models (LLMs) and vision language models (VLMs) running on a local system.

I talked with Yaniv Sulkes, Hailo VP of Physical AI at Hailo, about physical AI issues, including the need to handle models on the edge rather than the cloud and doing so without eating a lot of power. The video (see above) focuses on physical AI in general with a short mention of Hailo’s hardware (Fig. 1).

Data streams from optical, LiDAR, and radar sources can be processed by LLMs and VLMs like the Hailo-8 and Hailo-10 accelerators. The Hailo-10H has 40 TOPS of INT4 performance suitable for handling multiple streams and models. Hailo also provides a standalone system-on-chip (SoC), the Hailo-15. The DSP core handles image signal processing (ISP) and, through the use of AI models, it can work in low-light environments.

As with most AI accelerators, Hailo provides a compiler and HailoRT runtime as part of its software development kit (SDK). The company’s Model Zoo (Fig. 2) is a collection of AI models that developers can enhance in addition to using their own custom models.

Hailo hardware and software support many of the commonly used AI frameworks and interchange standards, including TensorFlow, PyTorch, Keras, and ONNX.