In an attempt to prove that his robot drivers are safer than humans, Waymo simulated dozens of fatal, real-world accidents that occurred in Arizona over nearly a decade. The Google spinoff found that replacing any of the vehicles in a two-car accident with its robot-driven minivans would almost eliminate all deaths, according to data it published today.
The results are intended to reinforce Waymo’s case that autonomous vehicles operate more safely than those powered by humans. With millions of people dying in car accidents around the world each year, AV operators are increasingly relying on this safety case to encourage regulators to pass laws that allow more autonomous vehicles on the roads.
But that case was difficult to prove, thanks to the very limited number of autonomous vehicles that operate on public roads today. To provide more statistical support for his argument, Waymo turned to counterfactuals, or “what if?” scenarios, designed to show how your robotic vehicles would react in real-world situations.
Last year, the company published 6.1 million miles of driving data in 2019 and 2020, including 18 accidents and 29 near misses. In those incidents where their security operators took control of the vehicle to prevent an accident, Waymo engineers simulated what would have happened if the driver had not turned off the vehicle’s autonomous steering system to generate a counterfactual. The company also made some of its data available to academic researchers.
That work on counterfactuals continues in this latest data release. Through a third party, Waymo collected information about each fatal accident that occurred in Chandler, Arizona, a suburban community outside Phoenix, between 2008 and 2017. Focusing only on accidents that occurred within its operational design domain, or approximately 100- Area of square miles in which the company allows its cars to travel, Waymo identified 72 accidents to reconstruct in simulation to determine how its autonomous system would respond in similar situations.
Some of these accidents involved one vehicle, while the majority involved two. For collisions with two vehicles, Waymo carried out separate experiments simulating his autonomous vehicles in the role of each vehicle – first replacing the vehicle that initiated the collision and then replacing the vehicle that responded to the actions of the other vehicle. For collisions with a vehicle, Waymo simulated only a single vehicle. This left him with 91 simulations in total.
The company reconstructed these accidents, systematically aligning the vehicle’s trajectory to ensure that its Waymo vehicles are exposed to a situation similar to that of the actual fatal accident. Waymo used the same simulation platform he uses to train and evaluate his autonomous vehicles on virtual roads in normal operations.
The results show that Waymo’s autonomous vehicles would have “avoided or mitigated” 88 out of a total of 91 simulations, said Trent Victor, Waymo’s director of safety research and best practices. In addition, for accidents that were mitigated, Waymo’s vehicles would have reduced the likelihood of serious injury by a factor of 1.3 to 15 times, Victor said.
“This means that, even if you have not avoided the accident completely, you have taken steps to reduce the severity of the impact,” said Victor. “If the severity were reduced, the driver would be less likely to die.”
When Waymo replaced the vehicle in response to the instigator of the collision, it found that it “completely avoided” 82 percent of simulated collisions – the vast majority of which without the need for sudden braking or evasive action. In another 10 percent as a respondent, the urgent maneuvers of the Waymo vehicle helped to mitigate the severity of the accident. All of these accidents occurred at an intersection where the other vehicle turned left or cut the Waymo vehicle in a straight line, leaving little time to react, the company said.
“We are not saying that we will eliminate all fatalities, but we say that the best way to reduce the chances of a serious injury would be to do an evasive maneuver when possible,” said Victor. “And in all of these collision simulations, the Waymo driver did evasive maneuvers.”
There were three incidents in which a person died after being hit by another vehicle. In the simulation, Waymo was not able to avoid these accidents when his vehicle was in the rear. “Specifically in rear-end collisions, there is not much that the rescuer role can do,” said Matthew Schwall, Waymo’s head of field security. “Therefore, the Waymo pilot really suffers from the same challenge as humans in these situations, that it is difficult to forecast well in advance to be able to make an evasive maneuver.”
Twenty simulated collisions involved a pedestrian or cyclist being hit by a driver. Waymo’s autonomous vehicles avoided 100% of these simulated collisions, the company said.
There is no standard approach to assess AV security. A recent RAND study found that, in the absence of a structure, customers are more likely to trust the government – although US regulators seem happy to allow the private sector to dictate what is safe. In this vacuum, Waymo hopes that, by releasing this data, policymakers, researchers and even other companies will be able to start taking on the task of developing a universal framework.
To be sure, Waymo did not make its findings subject to peer-reviewed analysis for publication in an academic or scientific journal, although it would be open to publishing it in the future, a spokesman said. The simulations were not carried out independently of the company, nor were they reviewed by third parties for verification before being released by the company.
The company shared its findings with a select group of academic experts to get their reactions. Daniel McGehee, director of the National Advanced Driving Simulator Laboratories at the University of Iowa, said Waymo is taking safety and transparency analysis “to a new level”.
Jonas Bargman, associate professor of vehicle safety at Chalmers University of Technology in Sweden, said Waymo’s use of simulation-based assessment is at the “scientific cutting edge”, but only one of the components needed to assess the safety of automated vehicles.
After reviewing the report, Bargman concluded that the “Waymo simulation platform is highly sophisticated”, citing the company’s 3D sensor-level perception models. He highlighted the use of hypothetical counterfactual scenarios as “highly relevant to the introduction of Waymo’s autonomous vehicles to the public”.