What you’ll learn:
- What is event-based vision?
- How do event-based vision systems work?
- Why are event-based vision systems useful?
There’s more than one way to digitize an image. The conventional approach using CMOS sensors captures an entire frame. Another way is to track changes to individual pixels and deliver that information instead. This is called an event-based vision system. I talked with Dr. Luca Verre, Founder of Prophesee, about this technology and the company’s implementation (watch the video above).
Converting Event-Based Vision Capture to Frames
Prophesee’s camera provides a data stream that includes a pixel location and color values when a change is detected. This information can be used directly with tools like Brainchip’s neomorphic computing platform based on spiking neural networks (SNNs). SNNs are event-based machine-learning (ML) models that provide artificial-intelligence (AI) functionality similar to other neural-network ML models. Typically, an SNN used for image processing must have frame-based data converted to an event-based stream while it can use event-based camera data directly.
On the flip side, many ML models are built around frame-based image data. An event-based data stream can be used to generate a frame-based image (Fig. 1). Essentially, changes would be made to a frame buffer as new event data is received.
Low-Power Object Tracking Using Event-Based Vision Systems
Event-based cameras can report movement (Fig. 2) using very little power. A system is able to determine movement, including direction, which helps with things like gesture recognition. Detecting change, and hence movement, while ignoring the rest of the scene can reduce the overhead of processing the scene.
The eye-tracking demonstration places a camera in the glasses that track the eye movement (Fig. 3). In this example, the yellow dot in the laptop display shows where the user is looking at their forefinger. The augmented-reality (AR) glasses also had a forward-facing camera that generated the image on the laptop.
High-Speed Tracking Using Event-Based Vision
One application in which the event-based technology particularly shines involves fast-moving recognition items. A frame-based camera usually operates at a frame rate of 30 or 60 frames/s. This is insufficient when it comes to accurate tracking of fast-moving objects like the water whirlpool in Figure 4.
The display on the screen tracks the fast-moving particles at a faster rate than a frame-based system. Of course, tracking too many objects could generate more data than a frame-based system, but even such a system would have a tough time extracting the object information.
Each type of image-capture system has advantages and disadvantages. Their functionality can overlap, but an event-based approach often is better for tracking many types of objects or state changes with better response time and using less power.
