The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Metrics for evaluating performance of prognostic techniques
Abstract
Prognostics is an emerging concept in condition based maintenance (CBM) of critical systems. Along with developing the fundamentals of being able to confidently predict Remaining Useful Life (RUL), the technology calls for fielded applications as it inches towards maturation. This requires a stringent performance evaluation so that the significance of the concept can be fully exploited. Currently, prognostics concepts lack standard definitions and suffer from ambiguous and inconsistent interpretations. This lack of standards is in part due to the varied end-user requirements for different applications, time scales, available information, domain dynamics, etc. to name a few issues. Instead, the research community has used a variety of metrics based largely on convenience with respect to their respective requirements. Very little attention has been focused on establishing a common ground to compare different efforts. This paper surveys the metrics that are already used for prognostics in a variety of domains including medicine, nuclear, automotive, aerospace, and electronics. It also considers other domains that involve prediction-related tasks, such as weather and finance. Differences and similarities between these domains and health maintenance have been analyzed to help understand what performance evaluation methods may or may not be borrowed. Further, these metrics have been categorized in several ways that may be useful in deciding upon a suitable subset for a specific application. Some important prognostic concepts have been defined using a notational framework that enables interpretation of different metrics coherently. Last, but not the least, a list of metrics has been suggested to assess critical aspects of RUL predictions before they are fielded in real applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Nonparametric Estimation from Incomplete Observations
Kaplan, E. L., & Meier, P. (1992) · Springer Series in Statistics · doi:10.1007/978-1-4612-4380-9_25 · Cited by 45,181
Nonparametric Estimation from Incomplete Observations
Kaplan, E. L., & Meier, P. (1958) · Journal of the American Statistical Association · doi:10.1080/01621459.1958.10501452 · Cited by 39,158
Another look at measures of forecast accuracy
Hyndman, R. J., & Koehler, A. B. (2006) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2006.03.001 · Cited by 5,757
Diebold, F., & Mariano, R. (1994) · National Bureau of Economic Research · doi:10.3386/t0169 · Cited by 5,185
Diebold, F. X., & Mariano, R. S. (1995) · Journal of Business and Economic Statistics · doi:10.2307/1392185 · Cited by 2,340
The Statistical Evaluation of Medical Tests for Classification and Prediction
Pepe, M. S. (2003) · doi:10.1093/oso/9780198509844.001.0001 · Cited by 2,198
Data Structures and Algorithms
(2003) · Series on software engineering and knowledge engineering · doi:10.1142/9789812791245 · Cited by 2,117
Understanding robust and exploratory data analysis
Hoaglin, D. C., Mosteller, F. C., & Tukey, J. W. (1983) · Wiley eBooks · Cited by 1,906
The Statistical Evaluation of Medical Tests for Classification and Prediction
(2005) · Technometrics · doi:10.1198/tech.2005.s278 · Cited by 1,899
The M3-Competition: results, conclusions and implications
Makridakis, S., & Hibon, M. (2000) · International Journal of Forecasting · doi:10.1016/s0169-2070(00)00057-1 · Cited by 1,695
Cited by
Damage propagation modeling for aircraft engine run-to-failure simulation
Saxena, A., Goebel, K., Simon, D., & Eklund, N. (2008) · 2008 International Conference on Prognostics and Health Management · doi:10.1109/phm.2008.4711414 · Cited by 1,954
Self-citationMultiobjective Deep Belief Networks Ensemble for Remaining Useful Life Estimation in Prognostics
Zhang, C., Lim, P., Qin, A. K., & Tan, K. C. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582798 · Cited by 870
Machine learning pipeline for battery state-of-health estimation
Roman, D., Saxena, S., Robu, V., Pecht, M., & Flynn, D. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00312-3 · Cited by 660
Patil, M. A., Tagade, P., Hariharan, K. S., Kolake, S. M., Song, T., Yeo, T., & Doo, S. (2015) · Applied Energy · doi:10.1016/j.apenergy.2015.08.119 · Cited by 626
Botchkarev, A. (2019) · Interdisciplinary Journal of Information, Knowledge, and Management · doi:10.28945/4184 · Cited by 616
Liao, L., & Kottig, F. (2014) · IEEE Transactions on Reliability · doi:10.1109/tr.2014.2299152 · Cited by 563
Metrics for Offline Evaluation of Prognostic Performance
Saxena, A., Celaya, J., Saha, B., Saha, S., & Goebel, K. (2021) · International Journal of Prognostics and Health Management · doi:10.36001/ijphm.2010.v1i1.1336 · Cited by 449
Self-citationHealth assessment and life prediction of cutting tools based on support vector regression
Benkedjouh, T., Medjaher, K., Zerhouni, N., & Rechak, S. (2015) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-013-0774-6 · Cited by 386
A Data-Driven Failure Prognostics Method Based on Mixture of Gaussians Hidden Markov Models
Tobon-Mejia, D. A., Medjaher, K., Zerhouni, N., & Tripot, G. (2012) · IEEE Transactions on Reliability · doi:10.1109/tr.2012.2194177 · Cited by 363
Cloud-enabled prognosis for manufacturing
Gao, R., Wang, L., Teti, R., Dornfeld, D., Kumara, S., Mori, M., & Helu, M. (2015) · CIRP Annals · doi:10.1016/j.cirp.2015.05.011 · Cited by 351
Liu, K., Gebraeel, N. Z., & Shi, J. (2013) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2013.2250282 · Cited by 332
Machine learning applications in minerals processing: A review
McCoy, J. T., & Auret, L. (2019) · Minerals Engineering · doi:10.1016/j.mineng.2018.12.004 · Cited by 317
Pan, Z., Meng, Z., Chen, Z., Gao, W., & Shi, Y. (2020) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2020.106899 · Cited by 296
Prognostics of PEM fuel cell in a particle filtering framework
Jouin, M., Gouriveau, R., Hissel, D., Péra, M.-C., & Zerhouni, N. (2014) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2013.10.054 · Cited by 290
Discovering Prognostic Features Using Genetic Programming in Remaining Useful Life Prediction
Liao, L. (2014) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2013.2270212 · Cited by 274
Liao, L., & Köttig, F. (2016) · Applied Soft Computing · doi:10.1016/j.asoc.2016.03.013 · Cited by 258
Bidirectional handshaking LSTM for remaining useful life prediction
Elsheikh, A., Yacout, S., & Ouali, M.-S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2018.09.076 · Cited by 256
Javed, K., Gouriveau, R., & Zerhouni, N. (2017) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2017.01.050 · Cited by 217
“From a survey, [22, 60] classified performance metrics into four classes.”
BackgroundPrognostics: a literature review
Elattar, H. M., Elminir, H. K., & Riad, A. M. (2016) · Complex & Intelligent Systems · doi:10.1007/s40747-016-0019-3 · Cited by 181
“• They can be used in comparison between different algorithms in a standardized way [6].”
BackgroundMethodInfluentialAfridi, Y. S., Ahmad, K., & Hassan, L. (2022) · International Journal of Energy Research · doi:10.1002/er.7100 · Cited by 140
Zheng, X., Fan, W., Chen, C., & Peng, Z. (2024) · IEEE Transactions on Reliability · doi:10.1109/tr.2024.3359212 · Cited by 48
Fault Prognostics for the Predictive Maintenance of Wind Turbines: State of the Art
Abid, K., Sayed Mouchaweh, M., & Cornez, L. (2019) · Communications in Computer and Information Science · doi:10.1007/978-3-030-14880-5_10 · Cited by 18
Metadata
- Authors
- Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., & Schwabacher, M.
- Venue
- 2008 International Conference on Prognostics and Health Management
- Publisher
- IEEE
- Published
- 2008-10
- Pages
- 1-17
- OpenAlex
- W2155848441
- Cited by
- 518
- References
- 33
- FWCI
- 56.25
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.