Data Evaluation and Enhancement for Quality Improvement of Machine Learning
The DOI matches its Crossref record; 57 of the 58 references it cites were found in a registry or an index.
Abstract
Poor data quality has a direct impact on the performance of the machine learning system that is built on the data. As a demonstrated effective approach for data quality improvement, transfer learning has been widely used to improve machine learning quality. However, the “quality improvement” brought by transfer learning was rarely rigorously validated, and some of the quality improvement results were misleading. This article first exposed the hidden quality problem in the datasets used to build a machine learning system for normalizing medical concepts in social media text. The system was claimed to have achieved the best performance compared to existing work on a machine learning task. However, the results of our experiments showed that the “best performance” was due to the poor quality of the datasets and the defective validation process. To address the data quality issue and build a high-performance medical concept normalization system, we developed a transfer-learning-based strategy for data quality enhancement and system performance improvement. The results of the experiments showed a strong correlation between the quality of the datasets and the performance of the machine learning system. The results also demonstrated that a rigorous evaluation of data quality is necessary for guiding the quality improvement of machine learning. Therefore, we propose a data quality evaluation framework that includes the quality criteria and their corresponding evaluation approaches. The data validation process, the performance improvement strategy, and the data quality evaluation framework discussed in this article can be used for machine learning researchers and practitioners to build high-performance machine learning systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Chen et al. [19] suggest that the thresholds can be based on the expected distribution of a targeted population for relevant features (e.g. the usage frequency of a phrase, or the number of individuals with a particular skin tone).”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [19]
In the citing paper
“For example, duplicate entries in a dataset can produce an overlap between the datasets used to train and evaluate a model, which can cause the performance metrics to be exaggerated [19].”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [19]
In the citing paper
“In cases like this, reusing and tuning a trained model helps to improve model performance only if the tuning is done using a dataset that contains task-specific data entries [19].”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [19]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“Therefore,thestate-of-the-artlanguagemodels,suchasBERT orGPT-3[41],weretrainedwithbillionsofwords.”
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
A Comprehensive Survey on Transfer Learning
Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., & He, Q. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2020.3004555 · Cited by 6,678 · cited 1×
“In order to show the performance of different transfer learning models, Zhuang et al. conductedanexperimentalstudyon20representativetransfer learning models, and its results demonstrated that appropriate transfer learning models should be carefully selected for different applications [62].”
BackgroundShin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528162 · Cited by 5,759
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Beyond Accuracy: What Data Quality Means to Data Consumers
Wang, R. Y., & Strong, D. M. (1996) · Journal of Management Information Systems · doi:10.1080/07421222.1996.11518099 · Cited by 4,672
The unified medical language system (UMLS): integrating biomedical terminology
Bodenreider, O. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh061 · Cited by 4,356
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 8×
“ULMFit [28] is a well-known fine-tuning technique that has been successfully used for the implementation of different text”
BackgroundMethodInfluentialA general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018) · Science · doi:10.1126/science.aar6404 · Cited by 3,562 · cited 1×
“winning the world champions in Go game playing [2], beating human experts on recognizing objects in images, and significantly improving the quality of voice recognition, natural lan-”
Background(2018) · Cited by 3,510
Cluster failure: why fMRI inferences for spatial extent have inflated false-positive rates
Eklund, A., Nichols, T. E., & Knutsson, H. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1602413113 · Cited by 3,487 · cited 2×
“The problem is due to the inadequate validation of the statistical methods with real data in the fMRI studies [9].”
BackgroundMethodA Survey on Deep Transfer Learning
Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., & Liu, C. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01424-7_27 · Cited by 3,165
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484
The management and control of quality
Evans, J. R., & Lindsay, W. M. (1989) · Cited by 1,633
How to Fine-Tune BERT for Text Classification?
Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32381-3_16 · Cited by 1,534
Self-taught learning: transfer learning from unlabeled data
Raina, R., Battle, A., Lee, H., Packer, B., & Ng, A. Y. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273592 · Cited by 1,510
Torrey, L., & Shavlik, J. (2010) · Handbook of Research on Machine Learning Applications and Trends · doi:10.4018/978-1-60566-766-9.ch011 · Cited by 1,503
A comparative study of fine-tuning deep learning models for plant disease identification
Too, E. C., Yujian, L., Njuki, S., & Yingchun, L. (2019) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2018.03.032 · Cited by 1,487
Anchoring Data Quality Dimensions in Ontological Foundations
Wand, Y., & Wang, R. Y. (1996) · Communications of the ACM · doi:10.1145/240455.240479 · Cited by 1,479
Do Better ImageNet Models Transfer Better?
Kornblith, S., Shlens, J., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00277 · Cited by 1,325
Methodologies for data quality assessment and improvement
Batini, C., Cappiello, C., Francalanci, C., & Maurino, A. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541883 · Cited by 1,288
CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D. Y., Bagul, A., Langlotz, C. P., Shpanskaya, K. S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05225 · Cited by 1,219
Regularizing and Optimizing LSTM Language Models
Merity, S., Keskar, N. S., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.02182 · Cited by 697
Truth Discovery with Multiple Conflicting Information Providers on the Web
Yin, X., Han, J., & Yu, P. S. (2008) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2007.190745 · Cited by 648
DNorm: disease name normalization with pairwise learning to rank
Leaman, R., Islamaj Doğan, R., & Lu, Z. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt474 · Cited by 484
Fine-Tuning Convolutional Neural Networks for Biomedical Image Analysis: Actively and Incrementally
Zhou, Z., Shin, J., Zhang, L., Gurudu, S., Gotway, M., & Liang, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.506 · Cited by 415
Deep Learning is Robust to Massive Label Noise
Rolnick, D., Veit, A., Belongie, S., & Shavit, N. (2017) · arXiv · doi:10.48550/arxiv.1705.10694 · Cited by 402
Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., & Allahbakhsh, M. (2019) · ACM Computing Surveys · doi:10.1145/3148148 · Cited by 313
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition
Krause, J., Sapp, B., Howard, A., Zhou, H., Toshev, A., Duerig, T., Philbin, J., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_19 · Cited by 296
Lu, Z., Sreekumar, G., Goodman, E., Banzhaf, W., Deb, K., & Boddeti, V. N. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3052758 · Cited by 177
Classification Of Breast Cancer Histology Images Using ALEXNET
Nawaz, W., Ahmed, S., Tahir, A., & Khan, H. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93000-8_99 · Cited by 135
Normalising Medical Concepts in Social Media Texts by Learning Semantic Representation
Limsopatham, N., & Collier, N. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1096 · Cited by 133
Using rule-based natural language processing to improve disease normalization in biomedical text
Kang, N., Singh, B., Afzal, Z., van Mulligen, E. M., & Kors, J. A. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001173 · Cited by 131
Knowledge-based trust: estimating the trustworthiness of web sources
Dong, X. L., Gabrilovich, E., Murphy, K., Dang, V., Horn, W., Lugaresi, C., Sun, S., & Zhang, W. (2015) · Proceedings of the VLDB Endowment · doi:10.14778/2777598.2777603 · Cited by 129
CNN-based ranking for biomedical entity normalization
Li, H., Chen, Q., Tang, B., Wang, X., Xu, H., Wang, B., & Huang, D. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1805-7 · Cited by 117
Giannoulatou, E., Park, S. H., Humphreys, D. T., & Ho, J. W. K. (2014) · BMC Bioinformatics · doi:10.1186/1471-2105-15-s16-s15 · Cited by 108
Big Data Validation and Quality Assurance -- Issuses, Challenges, and Needs
Gao, J., Xie, C., & Tao, C. (2016) · 2016 IEEE Symposium on Service-Oriented System Engineering (SOSE) · doi:10.1109/sose.2016.63 · Cited by 98
On the Meaningfulness of “Big Data Quality” (Invited Paper)
Firmani, D., Mecella, M., Scannapieco, M., & Batini, C. (2016) · Data Science and Engineering · doi:10.1007/s41019-015-0004-7 · Cited by 84
A Practical Framework for Evaluating the Quality of Knowledge Graph
Chen, H., Cao, G., Chen, J., & Ding, J. (2019) · Communications in computer and information science · doi:10.1007/978-981-15-1956-7_10 · Cited by 73
Self-citationAI-GAN: Attack-Inspired Generation of Adversarial Examples
Bai, T., Zhao, J., Zhu, J., Han, S., Chen, J., Li, B., & Kot, A. (2021) · 2021 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip42928.2021.9506278 · Cited by 58
Self-citationRisk-based data validation in machine learning-based software systems
Foidl, H., & Felderer, M. (2019) · doi:10.1145/3340482.3342743 · Cited by 54
Multi-task Character-Level Attentional Networks for Medical Concept Normalization
Niu, J., Yang, Y., Zhang, S., Sun, Z., & Zhang, W. (2018) · Neural Processing Letters · doi:10.1007/s11063-018-9873-x · Cited by 53
A Case Study of the Augmentation and Evaluation of Training Data for Deep Learning
Ding, J., Li, X., Kang, X., & Gudivada, V. N. (2019) · Journal of Data and Information Quality · doi:10.1145/3317573 · Cited by 48
Self-citationSáez, J. A., Krawczyk, B., & Woźniak, M. (2016) · Applied Artificial Intelligence · doi:10.1080/08839514.2016.1193719 · Cited by 44
Analysis of cellular objects through diffraction images acquired by flow cytometry
Zhang, J., Feng, Y., Moran, M. S., Lu, J. Q., Yang, L. V., Sa, Y., Zhang, N., Dong, L., & Hu, X. (2013) · Optics Express · doi:10.1364/oe.21.024819 · Cited by 37
Medical Concept Normalization for Online User-Generated Texts
Lee, K., Hasan, S. A., Farri, O., Choudhary, A., & Agrawal, A. (2017) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi.2017.59 · Cited by 37
Multi-Task Medical Concept Normalization Using Multi-View Convolutional Neural Network
Luo, Y., Song, G., Li, P., & Qi, Z. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12060 · Cited by 25
An Inference Method for Disease Name Normalization.
Islamaj, R., & Lu, Z. (2012) · National Conference on Artificial Intelligence · Cited by 24
Hybrid Recommender System based on Deep Learning Model
Su, C., & Huang, D. (2020) · International Journal of Performability Engineering · doi:10.23940/ijpe.20.01.p13.118129 · Cited by 11
Robust and distributed web-scale near-dup document conflation in microsoft academic service
Wu, C., & Song, Y. (2015) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2015.7364059 · Cited by 10
Self-reinforcing Unsupervised Matching
Lu, J., Li, L., & Zhang, C. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3061945 · Cited by 7
Distributed Deep Transfer Learning by Basic Probability AssignmentPreprint
Shahriari, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.07437 · Cited by 3
Research works printed but not linked: 1
- belousov, 0, Using an ensemble of generalised linear and deep learning models in the SMM4H 2017 medical concept normalisation task, Proc 2nd Workshop Social Media Mining Health Appl, 54
Cited by
Soft Tissue Feature Tracking Based on Deep Matching Network
Lu, S., Liu, S., Hou, P., Yang, B., Liu, M., Yin, L., & Zheng, W. (2023) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2023.025217 · Cited by 109
Li, H., Zheng, H., Yue, T., Xie, Z., Yu, S. Z., Zhou, J. C., Kapri, T., Wang, Y., Cao, Z., Zhao, H., et al. (2024) · Nature Energy · doi:10.1038/s41560-024-01670-z · Cited by 133
Mehdary, A., Chehri, A., Jakimi, A., & Saadane, R. (2024) · Sensors · doi:10.3390/s24041230 · Cited by 107
A Survey of Data Quality Requirements That Matter in ML Development Pipelines
Priestley, M., O’donnell, F., & Simperl, E. (2023) · Journal of Data and Information Quality · doi:10.1145/3592616 · Cited by 86
Shaker, B., Lee, J., Lee, Y., Yu, M., Lee, H., Lee, E., Kang, H., Oh, K., Kim, H. W., & Na, D. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad577 · Cited by 90
Metadata
- Authors
- Chen, H., Chen, J., & Ding, J.
- Affiliations
- University of North Texas (US)
- Venue
- IEEE Transactions on Reliability
- Published
- 2021-06
- Volume
- 70
- Issue
- 2
- Pages
- 831-847
- ISSN
- 0018-9529 (ISSN-L)
- 1558-1721
- OpenAlex
- W3159579291
- Semantic Scholar
- 591160ceaa7d65ee9c65186b6907a1f82a419902
- Cited by
- 166
- References
- 58
- Citations by year
- 2026: 24 · 2025: 51 · 2024: 41 · 2023: 28 · 2022: 18 · 2021: 4
- FWCI
- 13.84
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Reliability (publishedVersion)
- Funders
- National Science Foundation (1852249)
- National Security Agency (H98230-20-1-0417)
- State Key Laboratory for Novel Software Technology in Nanjing University, China (KFKT2019A19)
- Topics
- Physical Sciences › Computer Science › Information Systems › Data Quality and Management
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- data quality, transfer learning, social media text, data validation, performance improvement, dataset quality, validation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.