Broadening the Scope of Advanced Awareness and Machine Vision

This article is part of our 2023 Electronic Design Technology Forecast issue.

What you’ll learn:

• The differences between MIPI CSI-2 v3.0 and v4.0.
• How the new features in CSI-2 v4.0 lower cost barriers for next-generation systems.
• How to shape ongoing innovation of the CSI-2 specification.

MIPI Camera Serial Interface 2 (MIPI CSI-2) has been designed to support both simple and advanced applications in a dizzying array of product spaces—mobile, augmented and virtual reality, drones, Internet of Things (IoT), medical devices, industrial systems, automobiles, and client devices such as tablets, notebooks and all-in-ones.

As the de facto high-speed protocol for the transmission of still and video images from image sensors to application processors, CSI-2 supports the gamut of use cases demanding high performance, low power, and low electromagnetic interference (EMI). Thanks to its flexibility and continuing evolution since its introduction in 2005, CSI-2 today is regarded as the world’s most widely embedded camera and imaging interface.

The release of CSI-2 version 4.0 in 2022 delivered a major update to the specification. This new version introduced the Always-On Sentinel Conduit (AOSC) feature for simplified, low-power environmental monitoring; multi-pixel compression (MPC); and RAW-28 color depth for superior image quality in high-performance, safety-critical applications.

With this latest update in the ongoing innovation of the scalable interface, the new version vastly expanded developers’ options and capabilities in building next-generation artificial intelligence and machine-vision systems for enhanced human-device interaction.

Extending CSI-2's Mobile-Phone Success to New Platforms

When the MIPI Camera Working Group was formed in 2004 to address the lack of a standardized camera interface specification for mobile devices, the market was fragmented with multiple incompatible proprietary serial interfaces. Complexity and high costs throttled mobile innovation.

Quickly, though, the booming global mobile market began to rely on CSI-2. Thus, the MIPI Camera Working Group's focus evolved to enable greater capabilities for machine awareness across a much broader category of platforms than only phones. CSI-2 v3.0, released in September 2019, delivered key new features to enhance contextual awareness across highly diverse systems:

• Unified Serial Link (USL) for encapsulating connections between an image sensor module and application processor, and reducing the number of wires needed in IoT, automotive, and client products for productivity and content creation.
• Smart Region of Interest (SROI) for analyzing images, inferencing algorithms, and making better deductions in application spaces such as the Industrial IoT (IIoT).
• RAW-24 for representing individual image pixels with 24-bit precision in application spaces such as automotive that demand the highest image quality.

Even as CSI-2 v3.0 was being released to developers, efforts already were well underway in the MIPI Camera Working Group on v4.0 of the interface. The innovation of three crucial new features would build on CSI-2's capabilities to bring sight to AI and support advanced machine-vision applications in any application space.

CSI-2 v4.0 also is a milestone in that it’s the first version of the popular interface to support transmission of CSI-2 image frames over the low-cost, low-pin-count MIPI I3C/I3C Basic two-wire interface.

For system designers, this conveys new flexibility with CSI-2, which is typically implemented for shorter-reach applications over the MIPI C-PHY or MIPI D-PHY physical-layer interfaces. However, it also can be implemented over the long-reach MIPI A-PHY SerDes interface, or other MIPI Board-approved PHY (see figure).

AOSC for Environmental Monitoring

The Always-On Sentinel Conduit (AOSC) feature introduced in CSI-2 v4.0 supports always-on machine-vision systems. A combination of ultra-low-power image sensors and a video signal processor (VSP)—either a separate device or a component integrated within a larger system-on-chip (SoC)—is enabled to continuously monitor the surrounding environment and wake a higher-power, host central processing unit (CPU) as significant events occur.

Operating over as few as two wires, CSI-2 v4.0 with AOSC slashes the cost and complexity for ultra-low-power awareness and inferencing solutions and machine-vision applications. Laptop/tablet-based face monitoring and video surveillance are among the example use cases.

For applications not requiring extremely high resolution, image frames can be cost-efficiently streamed from image sensors to VSPs over a low-power MIPI I3C bus in a highly efficient manner, with scaling options to add I3C lanes and bandwidth as defined by that two-wire general-purpose input/output (GPIO) solution. Or the AOSC engine can be used as a triggering wakeup mechanism that could then subsequently use the more robust MIPI PHYs to enable more advanced vision analytics and camera imaging applications. Whole hybrid architecture solutions could manifest from this approach.

The introduction of AOSC in CSI-2 v4.0 renders dozens of use cases for platforms in the consumer, commercial, and infrastructure space more feasible.

Multi-Pixel Compression

The multi-pixel compression (MPC) feature in CSI-2 v4.0 is designed to accommodate the geometric increase in resolution of image-sensor photodiodes being implemented in mobile, IoT, automotive, and other devices today. The latest generation of tetracell and nonacell image sensors with multi-pixel color filter arrays (CFAs), for example, facilitate video capture on mainstream mobile devices of a cinematographic quality that rivals commercial moviemaking.

MPC in CSI-2 v4.0 is intended to help alleviate the extraordinary bandwidth requirements of image capture with the leading-edge sensors by compressing multi-pixel and standard Bayer CFA images at greater efficiency and quality than otherwise possible. It does so, in part, by leveraging the statistical probabilities of neighboring pixels within a given image sharing similar characteristics.

This dramatically reduces the number of pixels that must be processed by components throughout the system pipeline. By keeping the processing overhead to be incurred across the system in check, the CSI-2 v4.0 MPC feature also helps solve the physics problem of staying within allowable thermal dissipation margins.

In these ways, the introduction of MPC in CSI-2 v4.0 is intended to make the MIPI specification much more valuable to designers of premium imaging applications with the most challenging requirements around power-signal-to-noise ratio (PSNR) and dynamic range (DR).

RAW-28 Color Depth for Safety-Critical Applications

RAW-28 color depth is another new feature in CSI-2 v4.0 that’s designed to support unprecedented image quality and superior PSNR and DR. In particular, RAW-28 pixel encoding is targeted for the next generation of high-DR image sensors in automotive applications, such as advanced driver-assistance systems (ADAS), which directly relate to human safety.

In ADAS applications, a computer essentially tries to discern and properly react to what’s in front of a vehicle. RAW-28 is a method for gathering more and better information for that process.

Previous versions of CSI-2 introduced RAW-20 and then RAW-24, and RAW-28 is the new state of the art. With CSI-2, the MIPI Camera Working Group continues to leverage the advances of high-DR extraction techniques in the world’s most sophisticated automotive sensors.

Leveraging and Shaping MIPI CSI-2

All three of the key new features in CSI-2 v4.0 are “normative optional.” System designers aren’t bound to implementing AOSC, MPC, or RAW-28 in utilizing CSI-2. They remain free to strategically pick and choose those specific features that are most beneficial to their designs. The specification also defines the optimal transport mode for the features.

MIPI Alliance members can download CSI-2 and access more information about the specification—including articles, conference presentations, webinars, whitepapers, and other resources for developers—on the MIPI website. CSI-2 v4.0.1, published in November 2022, introduced minor editorial clarifications to the specification, including terminology revisions that are more inclusive and more accurately reflect the functions of technical devices.

The MIPI Camera Working Group strives to continually advance its specifications and develop new solutions that address emerging requirements in the industry to aid system integration and achieve even higher levels of performance. The group continues to focus on bringing better sight to artificial intelligence and minimizing human-device friction in any use case.

The MIPI Alliance also launched a Camera Imaging and Vision Interest Group, open to MIPI Alliance members, to gather more input from global industry.

Read more articles in our 2023 Electronic Design Technology Forecast issue