Members can download this article in PDF format.
What you’ll learn:
- Why are automotive thermal cameras finally ready for primetime use in vehicles worldwide?
- How can automotive thermal cameras see in the dark?
- How do thermal cameras and AI technology work together for safe nighttime driving operation?
Public concern over rising pedestrian deaths at night has focused the attention of automotive manufacturers and regulators on the failure of automotive pedestrian automatic emergency braking (PAEB) systems to work when the sun goes down.
During the day, the current combination of video cameras and radar works well. However, at night when the cameras can’t see, radar alone has insufficient spatial resolution to locate and identify pedestrians. LiDAR has been proposed as an adjunct, but its low repetition rates and difficulties seeing through fog and rain can limit the effectiveness, and it can’t be used for object classification.
Decades of experience in surveillance and defense have shown that thermal imaging in the infrared should be able to address all of the current system shortcomings. Until now, though, the cost, bulk, complexity, and operating constraints of current cameras has prevented adoption of this technology in automobiles. We’re here to shatter those myths.
1. Thermal cameras detect temperature.
Specially designed thermal cameras can be calibrated to show absolute temperature. However, what the sensors in thermal cameras really detect is electromagnetic energy in the mid- or long-wave infrared bands, not temperature.
For most applications like surveillance, inspection of heat leaks, firefighting, and locating people in the dark, the benefit from thermal cameras comes from their ability to see small differences in these energy levels in common environments. Those differences are detectable by thermal infrared sensors because the energy emitted by various objects changes enough with temperature so that the sensors can form thermal images.
2. If the air temperature outside is 98.6°F, thermal cameras can’t see people.
The infrared bands used for thermal imaging were selected precisely because the atmosphere is transparent to electromagnetic radiation in those bands. Certainly, the molecules in the atmosphere emit radiation just like solid objects, but the density of gases is so low that the atmospheric contribution to the image is insignificant.
A person standing on a beach with nothing but sky behind stands out just as in a visible image. In the unlikely event that the person stands in front of a building with a wall at a uniform 98.6°F, just the small variations in body temperature are sufficient to provide a definitive image (Fig. 1). In fact, the skin temperature is likely to be significantly below 98.6 due to evaporation of the perspiration produced in reaction to proximity to a hot wall.
More interesting is the situation in cold environments where people are likely to be bundled up so much that the outer surface of the clothing takes on the air temperature. It’s common in winter scenes for only the hands, feet, and faces of well-bundled people to be clearly visible in thermal images. However, even then, enough of the clothing will be above the ambient temperate for typical thermal cameras to outline the person because some heat will escape through the clothing.
3. “Thermal camera” is just another name for “infrared camera.”
Generic “infrared” cameras detect radiation in some part of the infrared band that runs from roughly 900 nm to 15 µm. However, the thermal cameras developed for L2+ use cases and L3 autonomous vehicles are usually infrared cameras that detect in the 3- to 5-µm (MWIR) or 7- to 14-µm (LWIR) bands. They’re also calibrated to report their images in terms of temperature variations. Both of those bands can successfully generate detailed images showing thermal variation.
4. Thermal cameras can see in the dark.
Humans can see only when light is available in the visible band. Thermal cameras, on the other hand, can see infrared. So, at night, when humans see darkness, the cameras can detect objects from their own emissions of thermal radiation (Fig. 2). People appear bright while most surrounding objects, which are at lower temperatures, appear darker.
Other warm objects like car engines and smokestacks can also be seen by the camera. People and other living objects can be sorted from the other warm objects by the viewer or by appropriate computer software. If there are no warm objects, as may be the case in, for instance, surveillance of a parking lot after everyone has gone home, then it may not be possible to discern any objects because the thermal emission is uniform. To a thermal camera, this is truly dark.
5. Thermal cameras need really low operating temperatures.
Years ago, that was true. Now, though, modern thermal cameras can use a long-validated technology called a microbolometer (Fig. 3). It operates by absorbing the thermal radiation and sensing the resulting temperature change in the absorber using a temperature-variable resistor.
To isolate the sensing element from the mounting substrate, a platform on long legs is used and the elements are operated in a vacuum. An array of elements is sequentially scanned to read the various resistor values representative of the thermal image.
Microbolometers produce sufficient signal variation with changing temperature to enable image formation at speeds useful for video imaging while operating at room temperature, thus meeting the requirements for automotive use.
6. Thermal cameras are big and bulky.
This is another myth that was once factual due to the cooling requirements of earlier sensors. The detection method used in microbolometers, in contrast, runs at room temperature.
However, microbolometers are still subject to output variations resulting from changes in the temperature of the detector itself. To counteract it, small thermoelectric coolers may be used to stabilize the device temperature.
To correct variations over time, some thermal cameras have been provided with mechanical shutters to close off the incoming scene periodically to allow for recalibration. Critically, the shutter stops the data flow, disrupting the continuous operation of the car’s computer and its automated braking system.
A solution requiring only precalibration implemented through a new digital adjustment technique will soon be available to provide uninterrupted, fully corrected data from a small, light camera.
7. Two thermal cameras are needed to measure distance.
Traditionally, distance was calculated using the stereoscopic technique—two views of a scene using a pair of cameras spaced apart. While this may be practical for daytime imaging in cars with multiple visible cameras for other purposes, providing multiple thermal cameras for nighttime imaging can be avoided by using the power of computation already available in the vehicle.
One of the contributions of machine learning is the development of neural-network configurations and algorithms that can extract distance information from single images. These techniques, which extract distance information from small disparities in the image that change with distance, were originally developed to add distance measurement to visible cameras. Now, they have been successfully adapted to attach accurate distance labels to objects detected by single thermal cameras.
8. Thermal cameras can see through anything, even glass.
Due to their molecular structure, all materials have bands where electromagnetic wave transmission is blocked. Water, for instance, will not pass much outside the ultraviolet and visible bands (probably not a coincidence compared to what we can see). Common window glass blocks everything outside 300-2000 nm, explaining why cars get hot inside in summer with their windows closed. Neither of these ubiquitous materials is compatible with thermal imaging.
To operate effectively, thermal-camera lenses must be made of silicon or germanium, or a chalcogenide material, or one of the few crystalline materials like zinc selenide that pass the right bands but don’t absorb water. For a while, car manufacturers expected that thermal cameras would need to be mounted on the roof or grille. However, recently, methods have emerged that can incorporate infrared-transparent sections into windshields, which means thermal cameras can move inside, safe from the environment.
9. Thermal cameras capture too much data for a car’s computer to handle.
Luckily, adding neural networks to the computer mitigates this problem. In the new generation of automotive computer architectures, raw camera data in digital form travels to a central computer surrounded by dedicated processors.
The neural network can analyze the thermal image to find the pedestrians, surround them with rectangular containers called bounding boxes, and then label them with location in the field of view and distance. Scene information that doesn’t contain objects of interest is discarded.
The remaining small data packet, often in the form of Robot Operating System (ROS) messages, is all that needs to pass to the next stage, where the pedestrian data is converted into commands for action by the car. The main computer only sees the data reduced to tens of bytes from the multi-megabytes in the original image (Fig. 4).
10. Thermal cameras can only offer low resolution.
Some thermal cameras used for industrial boiler monitoring, fever detection, and other low-resolution tasks, have low-resolution (up to perhaps 120 × 160) sensors, but obviously these aren’t suitable for pedestrian location. The question then becomes, “What resolution is needed for appropriate imaging in cars at night?”
Careful analysis of the resolution required to provide reliable recognition of pedestrians at safe distances with curb-to-curb coverage indicates that HD megapixel sensors are needed. Since these can now be readily fabricated, that answers the resolution question. But then the next myth arises:
11. Thermal cameras are too expensive to be used in cars.
Microbolometer technology has already yielded megapixel sensors for defense use, but these would be too expensive for automotive applications. Even traditional microbolometers would be too costly because of the way they’re manufactured and controlled.
Now, a new design strategy can produce HD-class microbolometer sensors with integrated digital processing, automatic temperature compensation, necessary thermal sensitivity, and a size compatible with low-cost optics. Thus, production of cameras for any car becomes affordable in a variety of resolutions.
Owl Autonomous Imaging is currently working on this thermal sensor. We predict that vehicles entering production in 2026 will have thermal cameras augmenting standard visible light RGB sensors and radar, significantly benefiting the safety of pedestrians and the vehicle’s occupants.
For more details, check out these videos:
