The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Metrics for evaluating performance of prognostic techniques
Abstract
Prognostics is an emerging concept in condition based maintenance (CBM) of critical systems. Along with developing the fundamentals of being able to confidently predict Remaining Useful Life (RUL), the technology calls for fielded applications as it inches towards maturation. This requires a stringent performance evaluation so that the significance of the concept can be fully exploited. Currently, prognostics concepts lack standard definitions and suffer from ambiguous and inconsistent interpretations. This lack of standards is in part due to the varied end-user requirements for different applications, time scales, available information, domain dynamics, etc. to name a few issues. Instead, the research community has used a variety of metrics based largely on convenience with respect to their respective requirements. Very little attention has been focused on establishing a common ground to compare different efforts. This paper surveys the metrics that are already used for prognostics in a variety of domains including medicine, nuclear, automotive, aerospace, and electronics. It also considers other domains that involve prediction-related tasks, such as weather and finance. Differences and similarities between these domains and health maintenance have been analyzed to help understand what performance evaluation methods may or may not be borrowed. Further, these metrics have been categorized in several ways that may be useful in deciding upon a suitable subset for a specific application. Some important prognostic concepts have been defined using a notational framework that enables interpretation of different metrics coherently. Last, but not the least, a list of metrics has been suggested to assess critical aspects of RUL predictions before they are fielded in real applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Nonparametric Estimation from Incomplete Observations
Kaplan, E. L., & Meier, P. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_25 · Cited by 45,181
Nonparametric Estimation from Incomplete Observations
Kaplan, E. L., & Meier, P. (1958) · Journal of the American Statistical Association · doi:10.1080/01621459.1958.10501452 · Cited by 39,158
Another look at measures of forecast accuracy
Hyndman, R. J., & Koehler, A. B. (2006) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2006.03.001 · Cited by 5,757
Diebold, F., & Mariano, R. (1994) · National Bureau of Economic Research · doi:10.3386/t0169 · Cited by 5,185
Diebold, F. X., & Mariano, R. S. (1995) · Journal of Business and Economic Statistics · doi:10.2307/1392185 · Cited by 2,340
The Statistical Evaluation of Medical Tests for Classification and Prediction
Pepe, M. S. (2003) · doi:10.1093/oso/9780198509844.001.0001 · Cited by 2,198
Data Structures and Algorithms
(2003) · Series on software engineering and knowledge engineering · doi:10.1142/9789812791245 · Cited by 2,117
Understanding robust and exploratory data analysis
Hoaglin, D. C., Mosteller, F. C., & Tukey, J. W. (1983) · Wiley eBooks · Cited by 1,906
The Statistical Evaluation of Medical Tests for Classification and Prediction
(2005) · Technometrics · doi:10.1198/tech.2005.s278 · Cited by 1,899
The M3-Competition: results, conclusions and implications
Makridakis, S., & Hibon, M. (2000) · International Journal of Forecasting · doi:10.1016/s0169-2070(00)00057-1 · Cited by 1,695
The accuracy of extrapolation (time series) methods: Results of a forecasting competition
Makridakis, S., Andersen, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., Newton, J., Parzen, E., & Winkler, R. (1982) · Journal of Forecasting · doi:10.1002/for.3980010202 · Cited by 1,462
Data Structures and Algorithms
Wirth, N. (1984) · Scientific American · doi:10.1038/scientificamerican0984-60 · Cited by 1,418
Error measures for generalizing about forecasting methods: Empirical comparisons
Armstrong, J. S., & Collopy, F. (1992) · International Journal of Forecasting · doi:10.1016/0169-2070(92)90008-w · Cited by 1,396
Forecast Verification: A Practitioner's Guide in Atmospheric Science
(2011) · Wiley eBooks · doi:10.1002/9781119960003 · Cited by 1,267
Ruist, E., & Theil, H. (1967) · The Swedish Journal of Economics · doi:10.2307/3439127 · Cited by 1,207
Understanding Robust and Exploratory Data Analysis.
Collett, D., Hoaglin, D. C., Mosteller, F., & Tukey, J. W. (1984) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2981753 · Cited by 1,154
Intelligent Fault Diagnosis and Prognosis for Engineering Systems
Vachtsevanos, G., Lewis, F., Roemer, M., Hess, A., & Wu, B. (2006) · Wiley eBooks · doi:10.1002/9780470117842 · Cited by 1,134
Palmer, T. N., Alessandri, A., Andersen, U., Cantelaube, P., Davey, M., Délécluse, P., Déqué, M., Díez, E., Doblas-Reyes, F. J., Feddersen, H., et al. (2004) · Bulletin of the American Meteorological Society · doi:10.1175/bams-85-6-853 · Cited by 959
The state-of-the-art in short-term prediction of wind power. A literature overview
Giebel, G., Draxl, C., Brownsword, R. A., Kariniotakis, G., & Denhard, M. (2011) · Cited by 539
Lee, K. L., & Mark, D. B. (1996) · Cited by 488
Standardizing the Performance Evaluation of Short-Term Wind Power Prediction Models
Madsen, H., Pinson, P., Kariniotakis, G., Nielsen, H. A., & Nielsen, T. S. (2005) · Wind Engineering · doi:10.1260/030952405776234599 · Cited by 343
Djurdjanovic, D., Lee, J., & Ni, J. (2003) · Advanced Engineering Informatics · doi:10.1016/j.aei.2004.07.005 · Cited by 305
A Survey of Data-Driven Prognostics
Schwabacher, M. (2005) · Infotech@Aerospace · doi:10.2514/6.2005-7002 · Cited by 280
Self-citationA Survey of Artificial Intelligence for Prognostics.
Schwabacher, M., & Goebel, K. (2007) · National Conference on Artificial Intelligence · Cited by 224
Self-citationFault prognosis using dynamic wavelet neural networks
Vachtsevanos, G., & Wang, P. (2002) · doi:10.1109/autest.2001.949467 · Cited by 149
Carbone, R., & Armstrong, J. S. (1982) · Journal of Forecasting · doi:10.1002/for.3980010207 · Cited by 138
Model-based prognostic techniques [maintenance applications]
Luo, J., Namburu, S. M., Pattipati, K. R., Qiao, L., Kawamoto, M., & Chigusa, S. (2004) · IEEE AUTOTESTCON (IEEE Systems Readiness Technology Conference) · doi:10.1109/autest.2003.1243596 · Cited by 124
Prognostics implementation of electronics under vibration loading
Gu, J., Barker, D., & PECHT, M. G. (2007) · Microelectronics Reliability · doi:10.1016/j.microrel.2007.02.015 · Cited by 116
Prognostic Models in Medicine: AI and Statistical Approaches
Lucas, P., & Abu–Hanna, A. (2001) · Methods of Information in Medicine · doi:10.1055/s-0038-1634456 · Cited by 107
Intelligent systems in the automotive industry: applications and trends
Gusikhin, O., Rychtyckyj, N., & Filev, D. (2007) · Knowledge and Information Systems · doi:10.1007/s10115-006-0063-1 · Cited by 73
Error measures and the choice of a forecast method
Ahlburg, D. A. (1992) · International Journal of Forecasting · doi:10.1016/0169-2070(92)90010-7 · Cited by 65
A commentary on error measures
Chatfield, C. (1992) · International Journal of Forecasting · doi:10.1016/0169-2070(92)90011-w · Cited by 40
Harrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4<361::aid-sim168>3.0.co;2-4
Cited by
Damage propagation modeling for aircraft engine run-to-failure simulation
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008) · 2008 International Conference on Prognostics and Health Management · doi:10.1109/phm.2008.4711414 · Cited by 1,954
Self-citationMultiobjective Deep Belief Networks Ensemble for Remaining Useful Life Estimation in Prognostics
Zhang, C., Lim, P., Qin, A. K., & Tan, K. C. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582798 · Cited by 870
Machine learning pipeline for battery state-of-health estimation
Roman, D., Saxena, S., Robu, V., Pecht, M., & Flynn, D. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00312-3 · Cited by 660
Patil, M. A., Tagade, P., Hariharan, K. S., Kolake, S. M., Song, T., Yeo, T., & Doo, S. (2015) · Applied Energy · doi:10.1016/j.apenergy.2015.08.119 · Cited by 626
Botchkarev, A. (2019) · Interdisciplinary Journal of Information, Knowledge, and Management · doi:10.28945/4184 · Cited by 616
Metadata
- Authors
- Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, M.
- Venue
- 2008 International Conference on Prognostics and Health Management
- Publisher
- IEEE
- Published
- 2008-10
- Pages
- 1-17
- OpenAlex
- W2155848441
- Cited by
- 518
- References
- 33
- FWCI
- 56.25
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.