To perform the higher level of processing for sensor fusion, Xilinx has both its high-end Virtex and its cost-effective Spartan series of FPGAs. “As we have gone down the process node curve, we have also taken a lot of the features that used to be only in the Virtex devices and brought those down into the Spartan devices,” says Zoratti. “It’s the Spartan-level devices that are cost-effective for automotive.”
Spartan devices qualified for automotive applications provide an enabling technology for sensor fusion. “When you talk about system enablers, it’s the amount of processing power, or you can think of it as MIPS per penny that you can get out of the silicon now that is also driving this interest in fusion because it is now becoming more and more feasible,” says Zoratti.
Sensor fusion of radar and camera sensing is one of the rapidly developing automotive applications. In his presentation at the PReVENT/ProFusion2 Fusion Forum Workshop8 in 2006, David Schwartz of Delphi provided a generic depiction of a fusion system for vehicle active safety (Fig. 2). In this architecture, sensor fusion of radar enhanced with vision adds precision and confidence for false-alarm reduction and high-resolution position as well as object classification for vehicle/non-vehicle and more.
Jim Grothe, marketing manager for Automotive Sensors at Freescale Semiconductor, has a couple of theories on sensor fusion. “The confusion in the marketplace is probably, to somewhat, on purpose. You know how marketers like to confuse things,” he quips. “At the end of the day, I think there is a spectrum of definitions.”
Grothe explains how the transition from a single-axis sensor to multiple axes and then multiple types of sensors co-packaged provides a type of sensor fusion. This is fusing of the sensors themselves at the packaging level and provides a parallel path for sensor fusion (Fig. 3). This direct sensor fusion is most commonly associated with microelectromechanical-systems (MEMS) sensors made using semiconductor processing techniques.
The next layer is just beginning in automotive systems today. The merging or fusing of multiple systems into one results in reusing the same sense information from one application for another. While this seems similar to multiplexing, Grothe says that it really isn’t. The merging of electronics stability control (ESC) and airbags explains why.
“You could have two low-g axes accelerometers, two angular sensors for the ESC system, and then you have another two or three axes of acceleration for the airbag. That’s seven degrees of freedom that you would require,” says Grothe. “That’s if they were separate systems. By fusing them, you could go back down to three.”
Some sensor parameters change as the result of fusion. For merging the ESC with an airbag system, the transducer has to be sensitive to a wider range of operation—the low-g range for the ESC system as well as the medium-g range required for the airbag system. “The design of our mass and the spring constants and the processing circuitry of the signal from the transducer itself have to be tuned appropriately across a wider bandwidth as well as wider dynamic range from the sensor,” explains Grothe.
Claire Jackoski, marketing manager of consumer and industrial sensors at Freescale, notes that there are both similarities and differences to automotive for sensor fusion in consumer applications. Today, cell phones provide one of the highest uses of inertial sensors. “We are seeing that the demand for multiple sensors doing multiple activities is kind of the path that they are taking,” says Jackoski. The sensors could be magnetic sensors plus accelerometers for six degrees of freedom or gyroscopes plus accelerometers or pressure sensors plus an accelerometer. “The two sensors together can give you a new solution,” she says.
Communication among sensors helps to make the decisions. “The fusion comes from not only the potential of a packaging exercise but a communications exercise that the algorithms start talking to one another to give heading and speed rather than simply providing the usual X, Y, Z output or a north or magnetic field direction,” Jackoski says. The progression from simple sensing to this multi-sensor environment has taken sensors to the direct level of sensor fusion shown in Figure 3.
LET THE MORE COMPLEX GAMES BEGIN
With sensor fusion, games like the Nintendo Wii are providing more sophisticated interaction. In the initial systems, decisions were made in the control unit regarding the player’s movements based on accelerometer and optics inputs and how the game would respond. Now games are becoming more complex.
“What they want to do now is capture a different motion. They capture angular acceleration motion with the gyros and linear motion with accelerometers,” says Jackoski. “Together that becomes six degrees of freedom of motion.” Games aren’t the only area where motion control is advancing thanks to sensor fusion. The technique in Figure 4 can be applied to gaming, medical, and other advanced motion applications.
Xsens is one of the companies taking advantage of the capability of advanced motion sensing. Its MVN inertial motion capture suit (Fig. 5) is a system for full-body human motion capture based on inertial sensors, biomechanical models, and sensor fusion algorithms. The motion capture technique was used in the recent movie Alice in Wonderland, avoiding the use of cameras to capture motion information and saving time and money.
Another company, 24eight, has developed a motion control suit for medical applications that employs MEMS sensor fusion. MEMS sensor fusion (Fig. 6) is also part of security and energy applications. However, the local processing of biometric data including the ability to perform edge processing is applicable to gaming and personal computing as well as medical systems.