Google’s Self-Driving Software Still Learning

During a 14-month period, Google’s test fleet of self-driving cars in California experienced 272 detected technology failures that required the driver to immediately take over control of the vehicle, according to a new company report.

Such disengagements from the self-driving mode occur when software detects an autonomous technology issue that might affect safe vehicle operation, Google noted in the report submitted to the California Department of Motor Vehicles. Moreover, during the same 14-month period, there were 69 events during which a Google test driver decided to disengage the autonomous mode and take immediate manual control of the vehicle to ensure safety.

The Google report covers the period from Sept. 24, 2014, through Nov. 30, 2015.

Google is among the 11 companies that have a California DMV permit for self-driving car testing on public roads. California state regulations require that all these companies submit an annual report summarizing the autonomous technology disengagements that occurred during testing. Seven manufacturers — Volkswagen/Audi, Mercedes-Benz, Google, Delphi Automotive, Tesla Motors, Bosch and Nissan — were required to submit their first disengagement report by Jan. 1.

By the end of November 2015, Google had already operated its self-driving cars in autonomous mode for more than 1.3 million miles, according to the report. A total of 424,331 of those miles were traveled on public roads, primarily in suburban communities in and around Mountain View, Calif.

As required by the DMV, the Google report focuses on two disengagement scenarios: when software detects a failure of the autonomous technology, prompting the disengagement and transfer of control to the driver; and when the test driver initiates the disengagement for safety reasons and takes control.

During what’s known as “immediate manual control” disengagements, software triggers distinct audio and visual signals to alert the driver to take control. Trained for such events, Google test drivers averaged a response time of 0.84 seconds for all measurable events during the reporting period, according to Google.

Further development and refinement of the company’s self-driving software have resulted in a decrease in this type of disengagement, Google reported, despite the growing number of miles driven each month.

“The number of autonomous miles we are driving between immediate manual control disengagements is increasing steadily over time,” the report said. “The rate of this type of disengagement has dropped significantly from 785 miles per disengagement in the fourth quarter of 2014 to 5,318 miles per disengagement in the fourth quarter of 2015.”

Disengagements from the self-driving mode — whether initiated by software or by a test driver — underscore the continuing need for a licensed driver behind the wheel during testing. 

The California DMV on Dec. 16 released draft autonomous vehicle regulations requiring the presence of a licensed driver inside all self-driving cars — at least for the immediate future. “The department will address the unique safety, performance, and equipment requirements associated with fully autonomous vehicles in subsequent regulatory packages,” the department said in a statement released at the time.

Meanwhile, these disengagements serve a significant purpose. They help Google gather data to refine its evolving autonomous vehicle technology, the company noted, and that’s why disengagement thresholds are set conservatively.

Disengagements, in essence, help the software get smarter.

“Our objective is not to minimize disengages; rather, it is to gather, while operating safely, as much data as possible to enable us to improve our self-driving system,” Google said in the report. "Our self-driving system runs thousands of checks on itself every second. Immediate manual control disengages are triggered primarily when we detect a communication failure between the primary and secondary (back-up) self-driving systems (for example, a broken wire); when we detect anomalies in sensor readings related to our acceleration or position in the world (accelerometers or GPS); or when we detect anomalies in the monitoring of key functions like steering and braking.” 

In cases where the test driver perceives a safety risk and assumes control of the car, the autonomous system automatically records the circumstances leading up to the disengagement, the report explained. That information, along with with feedback from the driver, helps Google to identify potential safety concerns and to refine the software accordingly. A simulator program, developed by Google engineers, can also generate thousands of variations of the event to permit analysis of slightly different scenarios.

“Each of these events is carefully studied to root out the underlying issue or family of issues, and our software is then refined,” Google said in the report. “The revised software is tested extensively, in simulation, on closed courses and on public roads with our test drivers. Even with the vast majority of our autonomous miles being driven in complex city street environments, we only record a few safe operation disengagements each month.”

“Safe operation disengagements” are the ones in which the test driver initiates the transfer of control, rather than the software.

Of the 69 events during which the test driver opted to trigger disengagement, 13 likely would have resulted in a crash if the driver hadn’t assumed manual control, Google engineers determined after event analysis. But such driver-initiated, crash-averting disengagements are occurring less frequently.

“From April 2015 to November 2015, our cars self-drove more than 230,000 miles without a single such event,” Google reported.

To download the disengagement reports from Google as well as those from Bosch, Delphi Automotive Systems, Nissan, Mercedes-Benz, Tesla Motors and Volkswagen, click here.