AAA Testing Finds Level 2 ADAS Has “Some Type of Issue” Every 8 Miles

According to the National Highway Traffic Safety Administration (NHTSA), there were approximately 6,734,000 police-reported motor vehicle crashes across the U.S in 2018, resulting in 2,710,000 injuries and 36,560 fatalities. Analysis from the AAA Foundation estimated that if driver-assistance technologies were installed on all vehicles, they could have helped prevent or mitigate roughly 40% of all crashes involving passenger vehicles.

To reduce the number of injuries and fatalities, “building blocks” of autonomy are already being deployed within vehicles available to the general public. For the 2020 model year, advanced driver-assistance systems (ADAS) are standard on 10% of new vehicles sold in the United States. Examples include adaptive cruise control, lane-keeping assistance, automatic emergency braking, parking assistance, and more.

Active ADAS, which integrates both longitudinal and lateral motion control, is the most advanced semi-autonomous vehicle technology available to the consumer. These systems are known as SAE Level 2 driver support features. With Level 2 driving support, the driver is expected to maintain situational awareness at all times for the specific purpose of responding to objects and/or events that may require the driver to assume full control immediately.

But would closed-course and real-world highway testing of ADAS show that it meets expectations? Research conducted by the AAA Foundation for Traffic Safety evaluated the potential impact of Level 2 automation-equipped vehicles on driving behavior.

Overall Test Results

AAA automotive researchers found that over the course of 4,000 miles of real-world driving, vehicles equipped with active driving-assistance systems experienced some type of issue every eight miles on average. Researchers noted instances of trouble with the systems keeping the vehicles tested in their lane and coming too close to other vehicles or guardrails. Lane-keeping accounted for 73% of noted events during the tests.  

AAA also found that systems combining vehicle acceleration with braking and steering often disengage with little notice—almost instantly handing control back to the driver. That could lead to a dangerous situation if a driver isn’t paying sufficient attention to the task of driving or has become too dependent on the system.

“AAA has repeatedly found that active driving-assistance systems do not perform consistently, especially in real-word scenarios,” said Greg Brannon, director of automotive engineering and industry relations. “Manufacturers need to work toward more dependable technology, including improving lane-keeping assistance and providing more adequate alerts.”

Test Parameters

AAA selected the following vehicles for testing:

• 2019 BMW X7 with “Active Driving Assistant Professional”
• 2019 Cadillac CT6 with “Super Cruise”
• 2020 Kia Telluride with “Highway Driving Assist”
• 2019 Ford Edge with “Ford Co-Pilot360”
• 2020 Subaru Outback with “EyeSight”

The odometer reading of all test vehicles was between 1,000 and 7,000 miles at the start of testing.

In the AAA test, each vehicle was outfitted with an Oxford Technical Solutions (OxTS) RT3000 v2 to capture vehicle kinematic information and an OxTS RT-Range Hunter to process vehicle-to-lane and vehicle-to-vehicle measurements relative to the vehicle under test. The RT3000 interfaced with a site-installed base station to incorporate real-time kinematics (RTK) technology. The RT-Range Hunter interfaced with targets via extended LAN (XLAN) communications. Test vehicles were equipped with a CAN interface to capture data from OxTS instrumentation. Vehicle kinematics and range data were captured at a rate of 100 Hz and time-synced with pedal force measurements and video.

Each vehicle also was equipped with a brake-pedal force sensor to verify that no braking intervention was applied during closed-course testing. Each vehicle employed a camera facing the instrument cluster to monitor the activation state of the ADAS system. Another camera was mounted to each side of the vehicle to monitor positioning relative to lane markers. Video from all cameras was captured at a rate of 45 Hz.

Closed-Course Testing

Closed-course testing took place on surface streets at the GoMentum Station proving ground in Concord, Calif. All testing was conducted on a dry asphalt surface free of visible moisture. The surface was straight and flat, free of potholes and other irregularities that could cause significant variations in the trajectory of the test vehicle.

The testing area was approximately 0.4 miles long and consisted of a two-lane roadway divided down the middle by a dashed white line. The width of each dashed white line segment was 7 inches with a uniform spacing of 17 feet 2 inches between segments. Before testing, all test lanes were mapped using OxTS Lane Survey and Map Creation software.

While AAA’s closed-course testing found that the systems performed mostly as expected, they were particularly challenged when approaching a simulated disabled vehicle placed halfway within the testing lane in the lateral direction. When encountering this test scenario, in aggregate, a collision occurred 66% of the time and the average impact speed was 25 mph.

In a simulated stop-and-go scenario, each evaluated system made no contact with the lead vehicle during any test run, for lead vehicle deceleration rates up to and including 0.6G. This “indicates good longitudinal performance in the context of deceleration rates typically observed in the naturalistic (actual road) environment.”

AAA concluded: “While systems generally performed according to expectations, closed-course evaluation illustrated all systems are incapable of consistently reacting to a scenario of a disabled vehicle partially within the roadway. All available ADA systems are not capable of sustained operation without constant driver supervision. Drivers must be prepared to intervene anytime ADA systems are engaged.”

Road Testing

During “naturalistic” evaluation—test vehicles were evaluated on U.S Highway 101 and Interstate 5 between the greater Los Angeles and San Francisco metropolitan areas—in total, approximately 4,000 miles were traversed among five test vehicles. Specifically, 521 events were noted; this translates to approximately one event every eight miles.

In aggregate, 73% of noted events related to the lane-keeping functionality of the ADAS system. The majority of events classified as “other” related to system disengagement or failure to engage. This finding is similar to earlier work in 2018 which observed 88% of events in the naturalistic setting related to the lane-keeping functionality of evaluated systems.

Due to the frequency and unpredictability of lane-keeping events, all test drivers expressed some level of distrust for each system. In addition, all test drivers reached a general consensus that combining adaptive-cruise and lane-keeping functionalities in a single system didn’t consistently enhance the driving experience. Indeed, some test drivers reported evaluated systems could sometimes increase perceived driver workload because in many cases constant monitoring and intervention was required.

Conclusions

Overall, according to AAA, the evaluation showed a greater impact on various driver behavioral measures when ADAS was active compared to when these systems weren’t active but available. Eye-glance metrics indicated drivers had their eyes off-road and on non-driving-related tasks more frequently and for longer durations when ADAS was active compared to any other system status.

Unfortunately, in both closed-course and naturalistic environments “clear examples of critical performance demerits that may result in a serious collision were observed.”

But while nothing contained within the AAA report “is intended to directly imply evaluated ADA systems are inherently unsafe,” AAA recommends that “currently available ADA systems are not capable of sustained vehicle operation without constant driver supervision; it is imperative the driver maintain situational awareness at all times.”