CoChat
Sign Up
Cited by 57Traffic and Road SafetyAutonomous Vehicle Technology and SafetyHuman-Automation Interaction and Safety

The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index; 2 of the 17 sources its claims cite were checked for support.

Comparative safety performance of autonomous- and human drivers: A real-world case study of the Waymo Driver

Open access
Di Lillo, Luigi, Gode, Tilia, Zhou, Xilin, Atzei, Margherita, Chen, Ruoshu, Victor, Trent·2024·Heliyon
DOI 10.1016/j.heliyon.2024.e34379PMC PMC11305169
Open this paper in CoChat — free account

Abstract

After several years of public road testing, the commercial deployment of fully autonomous vehicles-or Automated Driving Systems (ADS)-is poised to scale substantially following significant technological advancements and recent regulatory approvals. However, the fundamental question of whether an ADS is safer than its human counterparts remain largely unsolved due to several challenges in establishing an appropriate real-world safety comparison method. As scaling ensues, the lack of an established method can contribute to misinterpretations or uncertainties regarding ADS safety and impede the continuous and consistent assessment of ADS performance. This study introduces three research developments to define a robust and replicable safety comparison method to address this critical methodological gap. First, we introduce the use of liability insurance claims data to measure the comparative safety between ADS and human drivers. Second, we use Swiss Re insurance claims data to establish the first zip code- and responsibility-calibrated human performance benchmark, composed of over 600,000 private passenger vehicle claims and 125 billion miles of driving exposure. Third, we perform a case study by applying the developed baseline to evaluate the safety impact of the Waymo Driver. We find that when benchmarked against zip code-calibrated human baselines, the Waymo Driver significantly improves safety towards other road users. The comparison method established in this study can be replicated for other regions or ADS deployments to aid the decision-making of ADS safety stakeholders such as regulators, and instill trust in the general public.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

This study introduces liability insurance claims data as a method to compare automated driving system (ADS) and human driver safety, using Swiss Re claims to build zip code- and responsibility-calibrated human baselines. In a Waymo case study, rider-only and testing-operations driving showed significantly lower liability claim frequencies than human baselines, while manual driving showed mixed and partly inconclusive results.

Key findings

  • Manual driving: bodily injury claims frequency was 45% lower than baseline (0.55 vs 1.01 claims per million miles) but 95% CIs overlapped, so evidence was insignificant/inconclusive; property damage claims were significantly reduced by 34% (2.22 vs 3.34 claims per million miles).
  • Testing operations (TO): bodily injury claims were significantly reduced by 92% (0.09 vs 1.09 claims per million miles); property damage claims were significantly reduced by 95% (0.17 vs 3.17 claims per million miles).
  • Rider-only (RO): bodily injury claims were reduced by 100%, or zero claims (0.00 vs 1.11 claims per million miles), with non-overlapping 95% CIs; property damage claims were significantly reduced by 76% (0.78 vs 3.26 claims per million miles).
  • TO + RO combined: bodily injury claims were significantly reduced by 93% (0.08 vs 1.09 claims per million miles); property damage claims were significantly reduced by 93% (0.23 vs 3.17 claims per million miles).

Methods

  • The study used third-party liability insurance claims data from Swiss Re and Waymo, split into property damage liability and bodily injury liability coverages. The human baseline comprised over 600,000 claims and over 125 billion miles of exposure, summed across over 25 million vehicle insurance policy years, for claims occurring between 2016 and 2021.
  • Waymo data covered miles driven and claims reported from January 1, 2018, to August 1, 2023, organized into manual, testing operations (TO), rider-only (RO), and TO + RO categories.
  • Human baselines were calibrated by mileage and zip code, with separate baselines for each Waymo mileage category. Exposure years were converted to miles using Federal Highway Administration and U.S. Census data; significance was assessed by non-overlapping 95% confidence intervals, using normal approximation for the human baseline and the Poisson Exact Method for Waymo operations.

Limitations

  • The paper states that the human baseline uses the zip code of the registered vehicle address rather than the collision location, which may include claims occurring outside the target region; the authors expect this impact to be small to negligible. It also states that the human database includes freeway miles and claims while Waymo’s operational design domain largely excludes freeway driving, potentially making the baseline more conservative (lower rate) than a roadway-matched baseline.
  • The paper states that annual per-vehicle average mileage is estimated from aggregate data and carries estimation uncertainty; this could reasonably affect the RO and Manual comparisons, so the authors chose state-based VMT estimates to yield a conservative baseline.
  • The paper states that causality effects and ADS–HDV interactions in mixed traffic are not accounted for, though it reasons such effects would be weak given the relatively small number of ADS compared with HDV and the lack of drivers’ emotional descriptions in claims reports. The text also makes evident that the liability claims data cannot be publicly shared due to commercial sensitivity, and that injury severity levels are not differentiated because claims data include cost severity rather than injury severity.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.heliyon.2024.e34379

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 of 17 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Liability claims data uniquely capture information regarding crash or injury causation contributions because collision responsibility is directly determined during the liability claims adjudication process.

    “Liability claims data uniquely capture information regarding crash or injury causation contributions2 because collision responsibility is directly determined during the liability claims adjudication process [12].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [12] Braver, 2004, Are older drivers actually at higher risk of involvement in collisions resulting in deaths or non-fatal injuries among their passengers and other road users?, Inj. Prev. : journal of the International Society for Child and Adolescent Injury Prevention source

      The Insurance claims section (page unavailable) says bodily injury liability claims involve a driver deemed at fault and that insurance data address driver fault more directly than police-reported crashes. It does not say that responsibility is determined during claims adjudication, or establish that liability claims data uniquely capture causation contributions. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
  • Findingown data

    The human performance benchmark used in this study comprises over 600,000 private passenger vehicle claims and 125 billion miles of driving exposure, calibrated by zip code and responsibility.

    “we use Swiss Re insurance claims data to establish the first zip code- and responsibility-calibrated human performance benchmark, composed of over 600,000 private passenger vehicle claims and 125 billion miles of driving exposure.”
    • Also stated in: § Methods · plain
      “For this study 600,000 claims and over 125 billion miles of exposure were used (i.e., post-zip code and responsibility calibration filtering), summed across over 25 million vehicle insurance policy years.11”
    Quote in the paperSubstantial claim
  • Findingown data

    When Waymo vehicles were driven fully manually, property damage claims frequency was significantly reduced by 34% compared with the Swiss Re human driver baseline (2.22 vs 3.34 claims per million miles).

    “Property damage claims frequency was significantly reduced by 34 % (2.22 vs 3.34 claims per million miles). This result was confirmed by non-overlapping confidence intervals (ManualPDL 95 % CI [1.52, 3.13], Baseline 95 % CI [3.33, 3.36]).”
    Quote in the paperSubstantial claim
  • Findingown data

    While driving without a human behind the steering wheel in rider-only mode, property damage claims frequency was significantly reduced by 76% compared with the Swiss Re human driver baseline (0.78 vs 3.26 claims per million miles).

    “Property damage claims frequency was significantly reduced by 76 % (0.78 vs 3.26 claims per million miles), as indicated by non-overlapping 95 % CIs (ROPD 95 % CI [0.16, 2.27], Baseline 95 % CI [3.24, 3.27]).”
    Quote in the paperSubstantial claim
  • Findingown data

    While driving 35,228,320 miles in testing operations mode, property damage claims frequency was significantly reduced by 95% compared with the Swiss Re human driver baseline (0.17 vs 3.17 claims per million miles).

    “Property damage claims frequency was significantly reduced by 95 % (0.17 vs 3.17 claims per million miles), TOPD 95 % CI [0.06, 0.37], Baseline 95 % CI [3.16, 3.18].”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim. 19
  • It may state two separate claims as one. 10

  • Probably not a substantial claim.

    Premisecitation

    Most injury collisions occur within a small radius from residency.

    “Previous studies have shown that most injury collisions occur within a small radius from residency [18]”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [18] Haas, 2015, Close to home: an analysis of the relationship between location of residence and location of injury, J. Trauma Acute Care Surg. source

      The Discussion reports that three quarters of motor vehicle collisions involving patients who met trauma-center triage criteria occurred within 10 miles of home. This supports the proximity claim for that study population, but the citing sentence extends it to injury collisions generally. (LLM, conf=0.83)

      Experimental — accuracy not yet measured

  • It may state two separate claims as one.

    Findingown data

    While driving without a human behind the steering wheel in rider-only mode for 3,868,506 miles, the Waymo Driver reduced bodily injury claims frequency by 100% (zero claims) compared with the Swiss Re human driver baseline (0.00 vs 1.11 claims per million miles), and the difference was statistically significant.

    “While driving without a human behind the steering wheel in RO mode for 3,868,506 miles, the Waymo Driver reduced bodily injury claims frequency by 100 %, or zero claims, (0.00 vs 1.11 claims per million miles). The difference is statistically significant, as indicated by the non-overlapping confidence intervals (ROBI 95 % CI [0.000, 0.95], Baseline 95 % CI [1.10, 1.12]).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationreasoning

    The selected baseline population is likely the population that may use Waymo services instead of driving themselves.

    “In addition, the selected baseline population is likely the population that may use Waymo services instead of driving themselves.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Interpretationreasoning

    The benefits of the selected baseline population include its size and robustness, which lends itself to narrower confidence intervals.

    “The benefits of this population include its size and robustness, which lends itself to narrower confidence intervals.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationreasoning

    For estimation uncertainty to result in a false positive, the human driving mileage estimates would have to have been underestimated, which would lead to an overestimated baseline.

    “For this estimation uncertainty to result in a “false positive,” that is, a significant demonstration of positive safety impact which otherwise would be found inconclusive, our human driving mileage estimates would have to have been underestimated (which would lead to an overestimated baseline).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.

    Interpretationnone given

    The introduced methodology overcomes many of the existing challenges facing ADS and human driver crash rate comparisons and can be applied within the industry to assess the safety performance of additional ADS deployments.

    “The introduced methodology overcomes many of the existing challenges facing ADS and human driver crash rate comparisons and can be applied within the industry to assess the safety performance of additional ADS deployments.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Real-world safety impact studies are yet to include responsibility-calibrated comparisons.

  • It may state two separate claims as one.

    Study design and conductreasoning

    The annual per-vehicle average mileage used in the case study is an estimate based on aggregate mileage and population data, and this estimation uncertainty can impact the point estimates of the human benchmark.

    “In addition, the annual per-vehicle average mileage used in the case study is an estimate based on aggregate mileage and population data. This estimated average mileage naturally exhibits a degree of estimation uncertainty that can impact the point estimates of the human benchmark.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.

    Premisecitation

    Valid safety comparisons must select an appropriate safety performance metric, overcome differences in collision reporting standards between ADS and human-driven vehicles, account for the underreporting of human driving collisions, set a safety-relevant collision severity threshold, use human driver data specific to the region in which the ADS operates while maintaining robustness, and apply a统计方法

    “Valid safety comparisons must select an appropriate safety performance metric, overcome differences in collision reporting standards between ADS and human-driven vehicles (HDV), account for the underreporting of human driving collisions, and set a safety-relevant collision severity threshold. Furthermore, they should use human driver data specific to the region in which the ADS operates while”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [1] Blincoe, 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [2] Victor

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [3] Blanco, 2016 source

  • Probably not a substantial claim.

    Study design and conductown data

    The human driver baselines were calibrated using both mileage (driving exposure) and zip codes (geographic region).

    “The baseline was calibrated using both mileage (driving exposure) and zip codes (geographic region).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The garaging zip code is representative of a zip code's risk exposure.

  • It may state two separate claims as one.

    Study design and conductreasoning

    As a precaution against false positives, the authors chose mileage estimates that would yield a lower baseline frequency, so the estimation is expected to yield a conservative baseline.

    “As a precaution against such false positives, we chose to use mileage estimates which would yield a lower baseline frequency (i.e., choosing state-based aggregate data over urbanized-region-based aggregate data). As a result, we expect our estimation to yield a conservative baseline.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. It may state two separate claims as one.

    Premisecitation

    Insurance data capture motor incidents that result in harm or the need for financial recovery, including lower-severity collisions and injuries, but excluding minor events that produce no damage or harm.

    “Unlike police-reported crashes that underreport lower-severity collisions and naturalistic driving data that report any contact event, insurance data capture motor incidents that result in harm or the need for financial recovery, including lower-severity collisions and injuries, but excluding minor events that produce no damage or harm.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [10] Isaksson-Hellman, 2018, An evaluation of the real-world safety effect of a lane change driver support system and characteristics of lane change crashes based on insurance claims data, Traffic Inj. Prev. source

  • Probably not a substantial claim.

    Premisenone given

    The fundamental question of whether an ADS is safer than its human counterparts remains largely unsolved due to several challenges in establishing an appropriate real-world safety comparison method.

    “However, the fundamental question of whether an ADS is safer than its human counterparts remain largely unsolved due to several challenges in establishing an appropriate real-world safety comparison method.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Real-world safety impact studies are yet to include responsibility-calibrated comparisons.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Interpretationnone given

    The use of the zip code of the registered vehicle address, as opposed to the collision location, is a limitation of this study.

    “Although the use of the zip code of the registered vehicle address, as opposed to the collision, is a limitation of this study”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Premisenone given

    Improperly controlled variations across collision datasets can lead to inflated or deflated collision statistics.

    “Improperly controlled variations across collision datasets can lead to inflated or deflated collision statistics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Premisereasoning

    A responsibility-calibrated comparison requires a human baseline that reports the frequency in which a human driver contributes to a safety-relevant incident.

    “A responsibility-calibrated comparison requires a human baseline that reports the number of instances (i.e., frequency) in which a human driver contributes to a safety-relevant incident.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Real-world safety impact studies are yet to include responsibility-calibrated comparisons.

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given

    Well-designed safety comparisons can shape evidence-based public policies, inform insurance risk assessments, increase public transparency, and set future safety standards for ADS deployments.

    “Well-designed safety comparisons can shape evidence-based public policies, inform insurance risk assessments, increase public transparency, and set future safety standards for ADS deployments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Premisecitation

    The Waymo Driver does not require a human driver behind the wheel when in Rider-Only operation.

    “and does not require a human driver behind the wheel when in Rider-Only (RO), “driverless” operation.3”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper may give no reason why it holds.

    Interpretationnone given

    The use of the zip code of the registered vehicle address is expected to have a small-to-negligible impact on the frequency estimate.

    “we expect this to have a small-to-negligible impact on the frequency estimate.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The garaging zip code is representative of a zip code's risk exposure.

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.

    Interpretationnone given

    Establishing scientifically robust safety comparison methods is critical for an accurate, continuous, and consistent assessment of the performance of ADS deployments as scaling ensues.

    “Additionally, establishing scientifically robust safety comparison methods is critical for an accurate, continuous, and consistent assessment of the performance of ADS deployments as scaling ensues.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.

    Interpretationreasoning

    The lack of an established method can contribute to misinterpretations or uncertainties regarding ADS safety and impede the continuous and consistent assessment of ADS performance.

    “As scaling ensues, the lack of an established method can contribute to misinterpretations or uncertainties regarding ADS safety and impede the continuous and consistent assessment of ADS performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. It may be too vague to check against evidence.

    Premisecitation

    Safety comparison research should also consider crash responsibility, as the level of contribution to a crash is an important factor in assessing a driver's safety performance.

    “Safety comparison research should also consider crash responsibility, as the level of contribution to a crash is an important factor in assessing a driver's safety performance [7].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] Klauer, 2009

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationreasoning

    The comparison method established in this study can be replicated for other regions or ADS deployments to aid the decision-making of ADS safety stakeholders such as regulators, and instill trust in the general public.

    “The comparison method established in this study can be replicated for other regions or ADS deployments to aid the decision-making of ADS safety stakeholders such as regulators, and instill trust in the general public.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim.

    Premisecitation

    The Waymo Driver is a level 4 Automated Driving System as defined in SAE J3016.

    “The Waymo Driver, the core of the Waymo One service, is a level 4 Automated Driving System (ADS) as defined in SAE J3016 (SAE 2021)”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites SAE 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

23 of 23

  • The paper relies on this without support.

    Real-world safety impact studies are yet to include responsibility-calibrated comparisons.

    “However, real-world safety impact studies are yet to include responsibility-calibrated comparisons.”

  • The paper relies on this without support.

    The garaging zip code is representative of a zip code's risk exposure.

    “it is common practice in actuarial science and insurance ratemaking that the garaging zip code is representative of a zip code's risk exposure.”
2 of 2

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. 2023, Highway Statistics Series
  2. 2023, Traffic Volume Trends

Cited by

Sorted by citations

Related

Metadata

Authors
Di Lillo, L., Gode, T., Zhou, X., Atzei, M., Chen, R., & Victor, T.
Affiliations
  • Casualty Actuarial Society (US)
  • Swiss Re (Switzerland) (CH)
ORCID
Venue
Heliyon
Publisher
Elsevier BV
Published
2024-07
Volume
10
Issue
14
Pages
e34379
ISSN
2405-8440 (ISSN-L)
DOI
10.1016/j.heliyon.2024.e34379
PMC
PMC11305169
OpenAlex
W4400624502
Semantic Scholar
213bd8991c9e0a838103ef58e549098eb9a46492
Cited by
45
References
11
Citations by year
2026: 23 · 2025: 18 · 2024: 4
FWCI
11.86
Type
journal article
Language
en
Open access
Licence
Locations
  • Heliyon (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by-nc-nd)
  • ETH Zürich Research Collection
Topics
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Traffic and Road Safety
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
Keywords
autonomous driving, autonomous vehicle safety, safety benchmarking, road user safety, safety assessment methodology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-22, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-22 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review