Automakers Heed Warnings

Spurred by legislation aimed at preventing children's injuries and deaths from backover accidents, which occur at an ave-rage rate of 50 per week, NHTSA (the National Highway Traffic Safety Administration) is developing a rearward visibility performance standard.

“Speculation is that (NHTSA) could recommend a performance standard requiring light vehicles to feature a back-view camera,” according to Jon Cropley, manager of IMS Research's Automotive Group. Cropley noted that such a ruling would result in new business for suppliers of back-view cameras and image sensors.

Cropley estimates that the automotive market for camera-based driver-assistance systems will climb from $941 million in 2010 to $2.6 billion in 2014. “Helping to fuel this growth is the automotive industry's evolution from simple camera systems that provide images to the driver to more sophisticated collision warning systems that can notify the driver of potential crash situations.”

The possibility exists that NHTSA could someday require light vehicles to include lane departure warning (LDW) and/or forward collision warning (FCW) systems. The agency has discussed those and other crash avoidance technologies with automakers and suppliers while considering how best to update its New Vehicle Assessment Program (NCAP), but consensus is lacking on which technologies are most effective.

To address that issue, the Insurance Institute for Highway Safety (IIHS) identified five crash prevention technologies and then examined crashes that occurred between 2002 and 2006 to determine the number of crashes that were relevant to each technology (Figure 1, bottom of page). The IIHS concluded that FCW and LDW systems have more potential to avoid or mitigate crashes — at least fatal ones — than emergency brake assist, blind-spot detection or adaptive headlights.

According to the IIHS, at least 18 automotive brands currently offer one or more of the five crash prevention technologies the institute considered (Figure 2, bottom of page). It cited Volvo as the winner among manufacturers since Volvo puts some version of all five technologies on some of its models.

Market trends in vision technology include an increase in integration, the emergence of safety applications for cameras, such as forward collision warning, and the emergence of vehicle interior cameras for driver monitoring, according to David Goff, manager of product management and technical support at Omron Automotive Electronics.

“Multiple applications are being integrated onto a single camera. Lane departure warning and automatic high-beam headlamp control represent the first example of this type of integration. We're also seeing the integration of multiple sensors in a single unit; for example, the integration of camera and lidar sensors into a single interior package.”

Automakers, system developers, and component suppliers see clear opportunity for differentiation in vision-based advanced driver assistance systems (ADAS). What is not so clear is which features will have the greatest appeal.

“Lane departure warning continues to lead current applications, but we expect additional interest in medium-range forward collision warning applications due to recent NHTSA focus on active safety technologies in their NCAP standards,” said Bill Shogren, chief engineer of advanced safety systems at Delphi Electronics & Safety division. “Pedestrian and traffic sign recognition are also gaining interest.”

“There is no mandatory requirement for vision technology. The moment that comes, it will be just a matter of time before vision technology will be as prevalent as AM/FM,” said Sandeep Kishore, senior vice president and global head of manufacturing at HCL Technologies.

HCL uses two rear-view and two side-view cameras in a reverse view aid system. The firm also offers traffic sign and license plate recognition, and motion detection functions. Kishore said HCL is rolling out an automatic high beam controller with a tier one supplier in North America.

Hella KGaA Hueck & Co. is providing General Motors' European 2009 Opel Insignia with a camera system, Opel Eye, which combines traffic sign recognition (TSR) with LDW. The camera is mounted at the base of the rearview mirror. The TSR system reads speed-limit signs, no passing signs and other traffic restriction markers, and displays a symbol on the vehicle's dashboard. The system saves sign images for short periods of time. The LDW system sounds an alarm if the driver unintentionally crosses a lane marker.

The Opel Eye includes a high-resolution camera with a wide-angle lens, and a processor (Figure 3). The camera, about the size of a cell phone, operates at 30 frames per second (fps), thus it detects and reads signs several times from a distance of approximately 100 yards. Images are filtered and analyzed by two signal processors.

Continental Automotive Systems is supplying a driver assistance system for the BMW 7 series that uses a high data rate (HDR) CMOS camera for simultaneous speed limit monitoring (SLM), intelligent headlamp control (IHC) and LDW. The SLM function interprets various types of speed limit signs, and verifies the information with data from the navigation system before displaying it. The current speed limit is displayed in the BMW 7 series' instrument cluster and in the optional head-up display on the windshield.

Dean McConnell, Continental's director of occupant safety and driver assistance systems, said that another automaker has awarded Continental a contract to develop a system similar to BMW's for MY 2011 launch in North America and Europe. McConnell said Continental is looking at recognition of other road signs, but the problem is complicated by the fact that signs in Europe, as well as in North America, come in different sizes, shapes, and colors.

For Integrated Vehicle Based Safety System (IVBSS) testing sponsored by the U.S. Department of Transportation, Visteon Corp. integrated its radar-based multifunction system, which includes FCW, lane change merge warning, blind spot detection, and navigation/map-based curve speed warning, with Takata Corporation's vision-based LDW system.

Visteon and its development partner, Smart Eye AB, have been awarded a field operations test for a driver monitoring system in Sweden. The system can provide warnings based on the driver's head position, gaze direction, blink rate, and eye closure percentage.

“We're seeing a focus on advanced cockpit electronic products to enhance the driving experience,” said Tim Tiernan, senior manager of advanced cockpit electronics at Visteon. “This can help make drivers better drivers and improve the value proposition for our customers and consumers. There is also a focus on evolving rear view cameras from view-only products to smart or image-processed systems.

Robert Bosch LLC currently supplies night vision systems for Mercedes Benz S Class and CL Class vehicles. Infrared high-beam headlights illuminate an area of more than 150 meters in front of the vehicle. The beam is invisible to the human eye, so it does not blind oncoming traffic. A video camera converts the incoming IR signal to an image that the driver can see.

Bosch is currently developing Night Vision Plus (Figure 4), which will be able to recognize whether pedestrians in the displayed image are standing or moving. The system can highlight pedestrians or bicycles in color to draw the driver's attention to them. The system can also recognize predefined road signs in the image.

“If the driver is not sure what is 100 or 120 meters in front of them, he or she can glance down for a couple of seconds and then go back to normal driving behavior,” said Dieter Hoetzer, Bosch's product manager for automotive radar and camera products. “It's an additional HMI (human-machine interface), but it's not a distraction.”

Hoetzer and others say that European automakers are pushing vision technology more aggressively than their counterparts in North America. “Lane departure warning and forward collision warning are more in the eye of American OEMs,” he said. “NHTSA is expected to come out with a new NCAP five-star rating that will give specific language to new technology. Lane departure and forward collision warning test procedures are in place. Because of NCAP, OEMs are still interested in new technology, but we're seeing a lot of push toward low-cost applications — reduced functionalities with low-cost hardware, such as a standalone sensor with all functions in one box in the front of the vehicle, communicating via CAN.”

“North America is focused on NHTSA NCAP compliance and rearward visibility applications, while Europe looks for AHC (adaptive headlight control) and TSR features on top of a base of LDW,” said Delphi's Shogren. “Asian countries are focused on ‘around view’ systems that apply to more congested environments, and there is sporadic interest in China toward status symbol-type features.”

Shogren added that the automotive camera costs are coming down as more systems move into production. “Camera components are beginning to become a commodity,” he said.

NEW CMOS CAMERAS

STMicroelectronics, Aptina, and Melexis launched CMOS cameras last fall for automotive vision applications.

ST's VL5510 complements the vision-processor family that the firm developed with Mobileye. It has a 1024 × 512-pixel format with 5.6 × 5.6-micron pixels, and is suited for wide-angle products, according to the company. The sensor offers high sensitivity (7.14V/lux), low dark current (33aA/pixel at 25°C), and high QE (Quantum Efficiency) at near infrared. It also integrates anti-dark sun and defect pixel correction algorithms, and delivers an embedded image histogram along with the image.

Aptina's MT9V126 is a multi-function automotive imaging SOC said to feature an exceptionally small automotive-grade (AEC-Q100) package, low-light sensitivity (0.5 Lux), on-chip lens distortion correction, perspective correction, and a dynamic graphic overlay capability. Aptina said the pixel design in the MT9V126 eliminates structured noise and row noise, resulting in clearer details and better overall picture quality. Operating from -40° to 105°C and supported by a royalty-free form factor automotive reference design, the chip lowers system cost by eliminating the need for a DSP/FPGA and a distortion correction lens. Development is simplified because tuning is done with Host System Commands versus direct register writes.

Aptina automotive segment director Curtis Stith said that ex-tended capabilities like the chip's dynamic graphic overlay engine, and lens distortion and perspective correction “bring high-end features to the mainstream.” The MT9V126 is scheduled for production by the end of Q2 2009.

Melexis launched its MLX75307 CMOS image sensor for automotive front vision applications, including high beam assist, and night vision. The MLX75307 features a high dynamic range pixel matrix capable of extending both input and output dynamic ranges. It also features a low light response boost that increases image brightness and contrast in low light conditions. Melexis said the sensor is well suited for night vision, tunnel entrance/exit, and application fusion.

Based on the EMVA1288 standard, the 12-bit performance of the MLX75307 has been demonstrated to detect NIR light irradiance below 500 photons/pixel at 150 fps — the equivalent of detecting a candle at more than 250 meters. Production of the MLX75307 sensor is scheduled for 2009.

“Camera-based systems are semiconductor-based and therefore subject to ongoing cost reductions,” said Inayat Khajasha senior automotive product marketing manager at OmniVision. “As technologies improve and manufacturing processes mature, camera system costs can be reduced while performance and capabilities improve.”

OmniVision announced last fall that Valeo is integrating OmniVision sensors into automotive safety and driver assistance programs, including rearview cameras, parking assistance systems, side view and surround view systems.

“With the automotive industry in difficult economic times, car manufacturers seem to be increasing their focus on equipping lower-end cars with high-end features to help drive sales,” Khajasha suggested. “Vision and display-based systems rank high in the luxury features set, but may soon become standard in many mid- to lower-end cars.”

Key technology trends in automotive vision systems include color HDR for applications such as sign detection, traffic light recognition, or white and yellow line distinction for lane departure warning, according to Khajasha. “Night vision, enabled by near-infrared (NIR) technology, has also gained acceptance.” OmniVision sensors include an NIR capability, and Khajasha said they function equally well in day and night vision applications, eliminating the need for separate solutions.

“Next year and beyond we see megapixel resolution taking on an important role in the more advanced, multi-camera vision and sensing systems,” Khajasha said. “Megapixel resolution is critical for extreme wide angle (>160°) applications, such as 360° view multi-camera systems, where distortion correction and image stitching is required. VGA resolutions simply cannot address distortion correction beyond 130 degree angles, resulting in stretching and blurring the image, which can seriously compromise driver and passenger safety.”

“Cameras need more signal-processing capability,” said Continental's McConnell. “They combine an inexpensive optical sensor and sophisticated signal processing and compression, whereas radar requires more sophisticated sensors and antennas, but less processing hardware and simpler algorithms to process data. The economies are different, but there is a need for both.”

McConnell added, “Customers are looking at the cost effectiveness of comfort and safety. We're pushing microcontroller suppliers to increase speed and memory. We're always looking to optimize combinations of microcontrollers and signal processors; balancing features and size for scalability.”

Eight tier one ADAS systems currently in production include DSP processors from Texas Instruments, according to Brooke Williams, TI's marketing manager for automotive vision products. He said production systems doubled from three to six last year and design-in sockets increased by seven. Production of those systems will be staggered from 2009 through 2013.

Autoliv selected a TI DaVinci processor for a night vision system with pedestrian detection. Williams said that system is in production.

Williams said TI's TMS230DM643x and TMS320DM64x SoCs are popular for vision-based applications. Both leverage the TMS320C64x+ digital signal processor (DSP) core, for which TI recently released a Vision Library of functions for ADAS applications. Williams said the VLIB includes more than 40 royalty-free software kernels that execute background modeling and subtraction, object feature extraction, tracking, recognition and low-level pixel processing.

Renesas Technology America has vision applications in development, according to Paul Kanan, marketing manager for the firm's automotive business unit. He said a single SH74504 or SH74513 microcontroller with up to 2 MB of embedded flash memory would be suitable for a lane departure warning application. “As you start adding functions like traffic sign recognition or object detection the embedded flash may not be sufficient,” he said. “The SH77650 is better suited for higher-end applications. It uses off-board flash and SRAM, but it has an SH4 core and a built-in image recognition accelerator, and it runs at up to 300 MHz or about 540 MIPS.”

Kanan said Renesas is receiving numerous requests for multi-function systems. “Customers seem to at least want to have a basic lane departure warning system, but engineers are asking what it will take to add object detection or other applications. We're hearing from our customers that OEMs are continually asking for traffic sign recognition, headlamp control and other applications, all in the same box.

Last fall, NEC Electronics introduced the IMAPCAR2 image processors in four configurations. The scalable processors execute up to 270 giga (billion) operations per second (GOPS) and can be used to detect nearby objects such as vehicles, pedestrians and lane markers in real time for obstacle-detection, forward-collision warning, lane-departure warning, and other automotive safety applications that require intensive computing, including systems that combine multiple applications with different computation requirements.

The IMAPCAR2 processors are successors to the 100 GOPS IMAPCAR, launched in August 2006. The new models offer an increase from 8 to 16 bits in the data-processing capabilities of the 128 processing engines, as well as an increase from four to six instructions in the variable-length instruction word (VLIW) to enable faster processing speeds. While the IMAPCAR offered only SIMD operation, in which all 128 processing engines were focused on the same operation, the IMAPCAR2 processors feature a multicore architecture that enables high-speed parallel processing of multiple algorithms simultaneously.

“There are challenges in where to place a camera if you want one camera to handle multiple applications,” said Nathan John, system LSI manager for NEC Electronics America's Automotive Strategic Business Unit. “For lane tracking the camera should be placed as high up in the vehicle as possible so it can get a better angle when it looks down the road. We've noticed a difference based on just a few inches of height. For applications like object detection, or park assist, it's better to place the camera lower down. Otherwise, design engineers will choose a camera resolution and a sample rate based on the most demanding application.”

John added that running multiple applications presents a challenge for processors. “If customers want to combine different applications to leverage their initial investment, that plays to our strength,” he said, referring to the multicore architecture and parallel processing capabilities of NEC's IMAPCAR2 processors.

Camera-based safety systems are effective, but car buyers are not sufficiently aware of how they work and why consumers should be willing to pay for safety features, according to John.

Lane tracking (lane departure warning) is one of the more popular camera-based applications, but it's still available largely on higher-end vehicles, according to John. Rear-view cameras in lower-end versions that leave object recognition to the driver, and more sophisticated models that include image processing.

“We've had good success in pedestrian detection,” said John. “It's the most computationally intensive application that anyone has focused on. “It's difficult because the characteristics of the image change quickly. You can't count on the shape being the same. Algorithms have to be more sophisticated, and the more sophisticated the algorithm, the more processing power that's needed.”

ABOUT THE AUTHOR

John Day writes regularly about automotive electronics and other technology topics. He is based in Michigan and can be reached by e-mail at [email protected].

Figure 1. The Insurance Institute for Highway Safety compared analyzed crash data to see which mitigation technologies were most relevant.

Source: Insurance Institute for Highway Safety

Figure 2. Insurance Institute for Highway Safety research indicates that crash mitigation technologies are more common on luxury vehicles.

Source: Insurance Institute for Highway Safety