The area under the precision‐recall curve as a performance metric for rare binary events

Abstract

Species distribution models are used to study biogeographic patterns and guide decision‐making. The variable quality of these models makes it critical to assess whether a model's outputs are suitable for the intended use, but commonly used evaluation approaches are inappropriate for many ecological contexts. In particular, unrealistically high performance assessments have been associated with models for rare species and predictions over large geographic extents. We evaluated the area under the precision‐recall curve (AUC‐PR) as a performance metric for rare binary events, focusing on the assessment of species distribution models. Precision is the probability that a species is present given a predicted presence, while recall (more commonly called sensitivity) is the probability the model predicts presence in locations where the species has been observed. We simulated species at three levels of prevalence, compared AUC‐PR and the area under the receiver operating characteristic curve (AUC‐ROC) when the geographic extent of predictions was increased and assessed how well each metric reflected a model's utility to guide surveys for new populations. AUC‐PR was robust to species rarity and, unlike AUC‐ROC, not affected by an increasing geographic extent. The major advantages of AUC‐PR arise because it does not incorporate correctly predicted absences and is therefore less prone to exaggerate model performance for unbalanced datasets. AUC‐PR and precision were useful indicators of a model's utility for guiding surveys. We show that AUC‐PR has important advantages for evaluating models of rare species, and its benefits in the context of unbalanced binary responses will make it applicable for other ecological studies. By not considering the true negative quadrant of the confusion matrix, AUC‐PR ameliorates issues that arise when the geographic extent is increased beyond the species’ range or when a large number of background points are used when absence information is unavailable. However, no single metric captures all aspects of performance nor provides an absolute index that can be compared across datasets. Our results indicate AUC‐PR and precision can provide useful and intuitive metrics for evaluating a model's utility for guiding sampling, and can complement other metrics to help delineate a model's appropriate use.