A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings
Open accessAbstract
Artificial Intelligence (AI) is one of the hottest topics in our society, especially when it comes to solving data-analysis problems. Industry are conducting their digital shifts, and AI is becoming a cornerstone technology for making decisions out of the huge amount of (sensors-based) data available in the production floor. However, such technology may be disappointing when deployed in real conditions. Despite good theoretical performances and high accuracy when trained and tested in isolation, a Machine-Learning (M-L) model may provide degraded performances in real conditions. One reason may be fragility in treating properly unexpected or perturbed data. The objective of the paper is therefore to study the robustness of seven M-L and Deep-Learning (D-L) algorithms, when classifying univariate time-series under perturbations. A systematic approach is proposed for artificially injecting perturbations in the data and for evaluating the robustness of the models. This approach focuses on two perturbations that are likely to happen during data collection. Our experimental study, conducted on twenty sensors’ datasets from the public University of California Riverside (UCR) repository, shows a great disparity of the models’ robustness under data quality degradation. Those results are used to analyse whether the impact of such robustness can be predictable—thanks to decision trees—which would prevent us from testing all perturbations scenarios. Our study shows that building such a predictor is not straightforward and suggests that such a systematic approach needs to be used for evaluating AI models’ robustness.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Another dominant topic in the DL papers is completeness, which the ML community almost exclusively refers to as missing data.
“Another dominant topic is completeness to which the ML community almost exclusively refers to as missing data119,125–128,133–135,182.”
From The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review (2024) · cites this paper as [134]
Section 3 describes missing time-series data points as a “dropping perturbation” and discusses filling missing values. The paper does not say that completeness is a dominant topic in deep-learning papers or that the ML community almost exclusively calls it “missing data”; its literature review in Section 2 instead focuses on robustness under perturbations. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Internet of Things: A survey
Atzori, L., Iera, A., & Morabito, G. (2010) · Computer Networks · doi:10.1016/j.comnet.2010.05.010 · Cited by 15,613 · see citing works
Internet of Things (IoT): A vision, architectural elements, and future directions
Gubbi, J., Buyya, R., Marusic, S., & Palaniswami, M. (2013) · Future Generation Computer Systems · doi:10.1016/j.future.2013.01.010 · Cited by 12,293 · see citing works
Beyond Accuracy: What Data Quality Means to Data Consumers
Wang, R. Y., & Strong, D. M. (1996) · Journal of Management Information Systems · doi:10.1080/07421222.1996.11518099 · Cited by 4,672 · see citing works
Deep learning for time series classification: a review
Ismail Fawaz, H., Forestier, G., Weber, J., Idoumghar, L., & Muller, P.-A. (2019) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-019-00619-1 · Cited by 3,342 · see citing works
Deep learning for smart manufacturing: Methods and applications
Wang, J., Ma, Y., Zhang, L., Gao, R. X., & Wu, D. (2018) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2018.01.003 · Cited by 1,740 · see citing works
Dau, H. A., Bagnall, A., Kamgar, K., Yeh, C.-C. M., Zhu, Y., Gharghabi, S., Ratanamahatana, C. A., & Keogh, E. (2019) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2019.1911747 · Cited by 1,128 · see citing works
Adler, J., & Parmryd, I. (2010) · Cytometry Part A · doi:10.1002/cyto.a.20896 · Cited by 1,097 · see citing works
Machine Learning for Internet of Things Data Analysis: A Survey
Mahdavinejad, M. S., Rezvan, M., Barekatain, M., Adibi, P., Barnaghi, P. M., & Sheth, A. (2017) · Surrey Research Insight Open Access (The University of Surrey) · Cited by 1,008 · see citing works
10 CHALLENGING PROBLEMS IN DATA MINING RESEARCH
Yang, Q., & Wu, X. (2006) · International Journal of Information Technology & Decision Making · doi:10.1142/s0219622006002258 · Cited by 939 · see citing works
Machine learning for internet of things data analysis: a survey
Mahdavinejad, M. S., Rezvan, M., Barekatain, M., Adibi, P., Barnaghi, P., & Sheth, A. P. (2018) · Digital Communications and Networks · doi:10.1016/j.dcan.2017.10.002 · Cited by 921 · see citing works
Machine Learning Testing: Survey, Landscapes and Horizons
Zhang, J. M., Harman, M., Ma, L., & Liu, Y. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2962027 · Cited by 897 · see citing works
Industrial artificial intelligence for industry 4.0-based manufacturing systems
Lee, J., Davari, H., Singh, J., & Pandhare, V. (2018) · Manufacturing Letters · doi:10.1016/j.mfglet.2018.09.002 · Cited by 870 · see citing works
On the Importance of the Pearson Correlation Coefficient in Noise Reduction
Benesty, J., Chen, J., & Huang, Y. A. (2008) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2008.919072 · Cited by 492 · see citing works
Artificial intelligence in Internet of things
Ghosh, A. K., Chakraborty, D., & Law, A. (2018) · CAAI Transactions on Intelligence Technology · doi:10.1049/trit.2018.1008 · Cited by 370 · see citing works
A review of machine learning for the optimization of production processes
Weichert, D., Link, P., Stoll, A., Stefan Rüping, A., Ihlenfeldt, S., & Wrobel, S. (2019) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-019-03988-5 · Cited by 364 · see citing works
Fast and Accurate Time Series Classification with WEASEL
Schäfer, P., & Leser, U. (2017) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3132847.3132980 · Cited by 308 · see citing works
Automating large-scale data quality verification
Schelter, S., Lange, D., Schmidt, P. S., Celikel, M., Bießmann, F., & Grafberger, A. (2018) · Proceedings of the VLDB Endowment · doi:10.14778/3229863.3229867 · Cited by 247 · see citing works
Wireless requirements and challenges in Industry 4.0
Varghese, A., & Tandur, D. (2014) · International Conference on Contemporary Computing and Informatics (IC3I) · doi:10.1109/ic3i.2014.7019732 · Cited by 227 · see citing works
Internet of Things for Industrial Automation -- Challenges and Technical Solutions
Breivold, H. P., & Sandström, K. (2015) · IEEE International Conference on Data Science and Data Intensive Systems · doi:10.1109/dsdis.2015.11 · Cited by 189 · see citing works
A time-series classification approach based on change detection for rapid land cover mapping
Yan, J., Wang, L., Song, W., Chen, Y., Chen, X., & Deng, Z. (2019) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2019.10.003 · Cited by 172 · see citing works
Deep learning in the automotive industry: Applications and tools
Luckow, A., Cook, M., Ashcraft, N., Weill, E., Djerekarov, E., & Vorster, B. (2016) · 2016 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2016.7841045 · Cited by 164 · see citing works
Adversarial Attacks on Deep Neural Networks for Time Series Classification
Fawaz, H. I., Forestier, G., Weber, J., Idoumghar, L., & Müller, P.-A. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851936 · Cited by 158 · see citing works
Robust predictive model for evaluating breast cancer survivability
Park, K., Ali, A., Kim, D., An, Y., Kim, M., & Shin, H. (2013) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2013.06.013 · Cited by 130 · see citing works
Chen, R. J., Janković, F., Marinsek, N., Foschini, L., Kourtis, L. C., Signorini, A., Pugh, M., Shen, J., Yaari, R., Maljkovic, V., et al. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330690 · Cited by 119 · see citing works
Mikalsen, K. Ø., Bianchi, F., Soguero-Ruíz, C., & Jenssen, R. (2017) · Pattern Recognition · doi:10.1016/j.patcog.2017.11.030 · Cited by 104 · see citing works
How to Access Factory Floor Information Using Internet Technologies and Gateways
Sauter, T., & Lobashov, M. (2011) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2011.2166788 · Cited by 82 · see citing works
Challenges for Quality of Data in Smart Cities
Barnaghi, P., Bermúdez-Edo, M., & Tönjes, R. (2015) · Journal of Data and Information Quality · doi:10.1145/2747881 · Cited by 64 · see citing works
Non-linear classifiers applied to EEG analysis for epilepsy seizure detection
del Rincón, J. M., Santofimia, M. J., García, X., Barba, J., Romero, F., Navas, P., & López, J. C. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.05.052 · Cited by 64 · see citing works
McDonald, A. D., Sasangohar, F., Jatav, A., & Rao, A. H. (2019) · IISE Transactions on Healthcare Systems Engineering · doi:10.1080/24725579.2019.1583703 · Cited by 49 · see citing works
Analysis of the requirements for offering industrie 4.0 applications as a cloud service
Khan, W. A., Wiśniewski, Ł., Lang, D., & Jasperneite, J. (2017) · IEEE International Symposium on Industrial Electronics (ISIE) · doi:10.1109/isie.2017.8001413 · Cited by 33 · see citing works
Boukerche, A., Kanhere, S. S., & Bellavista, P. (2018) · doi:10.1145/3242102 · Cited by 33 · see citing works
Instance selection for time series classification based on immune binary particle swarm optimization
Zhai, T., & He, Z. (2013) · Knowledge-Based Systems · doi:10.1016/j.knosys.2013.04.021 · Cited by 26 · see citing works
Scalable Classification of Univariate and Multivariate Time Series
Karimi-Bidhendi, S., Munshi, F., & Munshi, A. (2018) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2018.8621889 · Cited by 25 · see citing works
Huang, H., & Lin, S. (2018) · ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems (MSWiM) · doi:10.1145/3242102.3242119 · Cited by 24 · see citing works
Nancy, J. Y., Khanna, N. H., & Kannan, A. (2017) · Expert Systems with Applications · doi:10.1016/j.eswa.2017.01.056 · Cited by 18 · see citing works
Oregi, I., Ser, J. D., Pérez, A., & Lozano, J. A. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.04.030 · Cited by 13 · see citing works
Online Learning for Time Series Prediction of AR Model with Missing Data
Yang, H., Pan, Z., & Tao, Q. (2019) · Neural Processing Letters · doi:10.1007/s11063-019-10007-x · Cited by 12 · see citing works
Quasi View-Independent Human Motion Recognition in Subspaces
Tanisaro, P., Mahner, F. P., & Heidemann, G. (2017) · International Conference on Machine Learning and Computing (ICMLC) · doi:10.1145/3055635.3056577 · Cited by 6 · see citing works
O-MI/O-DF vs. MQTT: A performance analysis
Benedick, P.-L., Robert, J., Traon, Y. L., & Sylvain Kubler, A. (2018) · IEEE International Conference on Industrial Cyber-Physical Systems (ICPS) · doi:10.1109/icphys.2018.8387652 · Cited by 4 · see citing works
Self-citationTRIDENT: A Three-Steps Strategy to Digitise an Industrial System for Stepping into Industry 4.0
Benedick, P.-L., Robert, J., & Traon, Y. L. (2019) · Annual Conference of the IEEE Industrial Electronics Society (IECON) · doi:10.1109/iecon.2019.8927010 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- [25]Gamboa J.C.B. Deep learning for time-series analysis arXiv 2017 1701.01887
- [26]IEEE IEEE Standard Glossary of Software Engineering Terminology IEEE Std 610.12-1990 IEEE Piscataway, NJ, USA 1990 1 84
- [42]Tsipras D. Santurkar S. Engstrom L. Turner A. Madry A. Robustness may be at odds with accuracy arXiv 2018 1805.12152
- [43]Pearson K. VII Mathematical contributions to the theory of evolution.—III. Regression, heredity, and panmixia Philosophical Transactions of the Royal Society of London Series A, Containing Papers of a Mathematical or Physical Character Royal Society of London for Improving Natural Knowledge London, UK 1896 253 318
Cited by
Sorted by citations
The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review
Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01196-4 · Cited by 169 · see citing works
Smart Viniculture: Applying Artificial Intelligence for Improved Winemaking and Risk Management
Izquierdo‐Bueno, I., Moraga, J., Cantoral, J. M., Carbú, M., Garrido, C., & González‐Rodríguez, V. E. (2024) · Applied Sciences · doi:10.3390/app142210277 · Cited by 61 · see citing works
Yudhistira, B., Adi, P., Mulyani, R., Chang, C., Gavahian, M., & Hsieh, C. (2024) · Comprehensive Reviews in Food Science and Food Safety · doi:10.1111/1541-4337.13413 · Cited by 48 · see citing works
Adversarial Machine Learning in Industry: A Systematic Literature Review
Jedrzejewski, F. V., Thode, L., Fischbach, J., Gorschek, T., Méndez, D., & Lavesson, N. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103988 · Cited by 44 · see citing works
Bigonah, M., Jamshidi, F., Pant, A., Poudel, S., Nallapareddy, S. R., Langroudi, A. C., & Marghitu, D. (2025) · IEEE Access · doi:10.1109/access.2025.3550891 · Cited by 19 · see citing works
Immersive Agricultural Education
Bigonah, M., Jamshidi, F., & Marghitu, D. (2024) · Advances in early childhood and K-12 education · doi:10.4018/979-8-3693-1710-5.ch002 · Cited by 18 · see citing works
A systematic literature review on data quality assessment
Reda, O., Benabdellah, N. C., & Zellou, A. (2023) · Bulletin of Electrical Engineering and Informatics · doi:10.11591/eei.v12i6.5667 · Cited by 14 · see citing works
Evaluation criteria for artificial intelligence
Montrosse‐Moorhead, B. (2023) · New Directions for Evaluation · doi:10.1002/ev.20566 · Cited by 14 · see citing works
Oh, J.-Y., Yoon, Y. T., & Sohn, J.-M. (2024) · Energies · doi:10.3390/en17133344 · Cited by 8 · see citing works
Why it Remains Challenging to Assess Artificial Intelligence
Brecker, K., Lins, S., & Sunyaev, A. (2023) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2023.641 · Cited by 7 · see citing works
Advancing Quality Metrics in Statistical Data Analysis
Zhgun, T. V., Александровна, Ж. А., & Prowse, D. (2023) · International Conference Management of Large-Scale System Development (MLSD) · doi:10.1109/mlsd58227.2023.10303915 · Cited by 1 · see citing works
Predicción con modelo ARIMA en series temporales de Salmonella spp en Chile entre 2014-2022
Ayala-Aldana, N., Monleón-Getino, T., Canela‐Soler, J., & Contreras, E. R. (2023) · Ciencia Latina Revista Científica Multidisciplinar · doi:10.37811/cl_rcm.v7i1.4484
- Authors
- Benedick, P.-L., Robert, J., & Le Traon, Y.
- Affiliations
- University of Luxembourg (LU)
- Cebi Luxembourg S.A, 30 rue J.F. Kennedy, L-7327 Steinsel, Luxembourg
- Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg, 6 Rue Richard Coudenhove-Kalergi, L-1359 Luxembourg, Luxembourg
- Corresponding author
- Benedick, Paul-Lou
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2021-09-15
- Volume
- 21
- Issue
- 18
- Pages
- 6195
- ISSN
- 1424-8220 (ISSN-L)
- PMC
- PMC8469892
- OpenAlex
- W3200248201
- Cited by
- 12
- References
- 42
- Citations by year
- 2025: 1 · 2024: 6 · 2023: 5
- FWCI
- 0.97
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Sensors (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Open Repository and Bibliography (University of Luxembourg) (submittedVersion, cc-by-nc-sa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Fonds National de la Recherche Luxembourg (BRIDGES19/IS/13706587)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Engineering › Control and Systems Engineering › Fault Detection and Control Systems
- Keywords
- deep learning, machine learning, decision tree, industrial data
- MeSH terms
- Machine Learning, Algorithms, Artificial Intelligence, Industry, Technology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
