A Self-Attention-Based Imputation Technique for Enhancing Tabular Data Quality
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Recently, data-driven decision-making has attracted great interest; this requires high-quality datasets. However, real-world datasets often feature missing values for unknown or intentional reasons, rendering data-driven decision-making inaccurate. If a machine learning model is trained using incomplete datasets with missing values, the inferred results may be biased. In this case, a commonly used technique is the missing value imputation (MVI), which fills missing data with possible values estimated based on observed values. Various data imputation methods using machine learning, statistical inference, and relational database theories have been developed. Among them, conventional machine learning based imputation methods that handle tabular data can deal with only numerical columns or are time-consuming and cumbersome because they create an individualized predictive model for each column. Therefore, we have developed a novel imputational neural network that we term the Denoising Self-Attention Network (DSAN). Our proposed DSAN can deal with tabular datasets containing both numerical and categorical columns; it considers discretized numerical values as categorical values for embedding and self-attention layers. Furthermore, the DSAN learns robust feature expression vectors by combining self-attention and denoising techniques, and can predict multiple, appropriate substituted values simultaneously (via multi-task learning). To verify the validity of the method, we performed data imputation experiments after arbitrarily generating missing values for several real-world tabular datasets. We evaluated both imputational and downstream task performances, and we have seen that the DSAN outperformed the other models, especially in terms of category variable imputation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We used the residual connection approach [27] to enable deep layer learning.”
MethodVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“We employed multi-head attention [21]; this linearly projects queries, keys, and values h times but with different parametric weights, and (in parallel) runs the attention function h times.”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“The attention mechanism employed in machine translation [16] has been successfully ported to natural language processing tasks [17,18] and computer vision [19].”
MethodRUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · cited 1×
“Then, we use a denoising technique [24] to learn robust feature representation parameters for incomplete inputs.”
MethodMissForest—Non-parametric missing value imputation for mixed-type data
Stekhoven, D. J., & Bühlmann, P. (2012) · Bioinformatics · doi:10.1093/bioinformatics/btr597 · Cited by 6,156 · cited 2×
“MVI methods employing generative models include Generative Adversarial Imputation Nets (GAIN) [3] and denoising autoencoder (DAE) for multiple imputations [4]; MVI methods that feature predictive models include MissForest [5] and Datawig [6].”
MethodAn Overview of Multi-Task Learning in Deep Neural Networks
Ruder, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.05098 · Cited by 2,265 · cited 1×
“The DSAN features hard parameter sharing [22]; the shared-layer module learns the shared parameters by sharing the hidden layers among all tasks.”
BackgroundDeep Neural Networks and Tabular Data: A Survey
Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., & Kasneci, G. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3229161 · Cited by 1,025 · cited 1×
“The representational learning of a deep neural network is used to extract features of tabular data that aid predictive imputation [6,12,13].”
MethodUnderstanding Data Augmentation for Classification: When to Warp?
Wong, S. C., Gatt, A., Stamatescu, V., & McDonnell, M. D. (2016) · 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta.2016.7797091 · Cited by 1,007 · cited 1×
“This improves performance; the technique is similar to data augmentation [25].”
MethodAn analysis of four missing data treatment methods for supervised learning
Batista, G. E. A. P. A., & Monard, M. C. (2003) · Applied Artificial Intelligence · doi:10.1080/713827181 · Cited by 895 · cited 1×
“If a machine learning model is trained using incomplete datasets, the inferred results may be biased [1].”
MethodA Bayesian missing value estimation method for gene expression profile data
Oba, S., Sato, M.-A., Takemasa, I., Monden, M., Matsubara, K.-I., & Ishii, S. (2003) · Bioinformatics · doi:10.1093/bioinformatics/btg287 · Cited by 753 · cited 1×
“For numerical variables, we calculated the normalized root mean squared errors (NRMSEs) [29]; for categorical variables, we derived error rates.”
MethodMissing value imputation: a review and analysis of the literature (2006–2017)
Lin, W.-C., & Tsai, C.-F. (2020) · Artificial Intelligence Review · doi:10.1007/s10462-019-09709-4 · Cited by 612 · cited 1×
“MVI seeks to replace the missing values with substituted values estimated based on the observed values [2].”
BackgroundGAIN: Missing Data Imputation using Generative Adversarial Nets
Yoon, J., Jordon, J., & van der Schaar, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.02920 · Cited by 570 · cited 1×
“MVI methods employing generative models include Generative Adversarial Imputation Nets (GAIN) [3] and denoising autoencoder (DAE) for multiple imputations [4]; MVI methods that feature predictive models include MissForest [5] and Datawig [6].”
MethodHoloClean: holistic data repairs with probabilistic inference
Rekatsinas, T., Chu, X., Ilyas, I. F., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3137628.3137631 · Cited by 480
Ba, J. L., Kiros, J., & Hinton, G. E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.06450 · Cited by 440 · cited 1×
“To compensate for this, numerical values that are not discretized are learned in parallel via a fully connected (FC) unit consisting of an FC layer, Layer Normalization [26], the ReLU function, and Dropout.”
BackgroundMIDA: Multiple Imputation Using Denoising Autoencoders
Gondara, L., & Wang, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93040-4_21 · Cited by 292
TabTransformer: Tabular Data Modeling Using Contextual Embeddings
Huang, X., Khetan, A., Cvitkovic, M. W., & Karnin, Z. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06678 · Cited by 200 · cited 3×
“TabTransformer [20] used the encoder of a transformer [21] and a self-attention mechanism to map categorical features into a contextual embedment.”
BackgroundMethodSAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C. B., & Goldstein, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01342 · Cited by 118 · cited 1×
“Meanwhile, predictive-model-based MVI methods train models for each variable, and can thus deal with tabular data [7].”
Method"Deep" Learning for Missing Value Imputationin Tables with Non-Numerical Data
Bießmann, F., Salinas, D., Schelter, S., Schmidt, P. S., & Lange, D. (2018) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3269206.3272005 · Cited by 109 · cited 2×
“MVI methods employing generative models include Generative Adversarial Imputation Nets (GAIN) [3] and denoising autoencoder (DAE) for multiple imputations [4]; MVI methods that feature predictive models include MissForest [5] and Datawig [6].”
MethodEnriching Data Imputation under Similarity Rule Constraints
Song, S., Sun, Y., Zhang, A., Chen, L., & Wang, J. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2883103 · Cited by 70 · cited 1×
“Similarly, Song et al. [10] showed that data imputation is possible with valid candidate values (called neighbors) that can be identified by a similarity relationship.”
BackgroundMissing data imputation for traffic congestion data based on joint matrix factorization
Jia, X., Dong, X., Chen, M., & Yu, X. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107114 · Cited by 57 · cited 1×
“Furthermore, Jia et al. [11] devised tensor-based data imputation models to extract latent features from a traffic dataset and predict neighbors to fill in the missing values via tensor factorization.”
BackgroundHoloClean: Holistic Data Repairs with Probabilistic InferencePreprint
Rekatsinas, T., Chu, X., Ilyas, I. F., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.00820 · Cited by 54 · cited 1×
“For this, there has been a study that probabilistically calculates the correct value for erroneous data through statistical inference after detecting erroneous data using integrity constraints [8].”
BackgroundAutoencoder-based multi-task learning for imputation and classification of incomplete data
Lai, X., Wu, X., & Zhang, L. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106838 · Cited by 28 · cited 1×
“MTL can simultaneously handle regression and classification tasks; thus, an imputational model can combine a regression task and a classification task [14,15].”
MethodEfficient Multiple Imputation for Diverse Data in Python and R: MIDASpy and rMIDAS
Lall, R., & Robinson, T. S. (2023) · Journal of Statistical Software · doi:10.18637/jss.v107.i09 · Cited by 21 · cited 1×
“MTL can simultaneously handle regression and classification tasks; thus, an imputational model can combine a regression task and a classification task [14,15].”
MethodRENUVER: A Missing Value Imputation Algorithm based on Relaxed Functional Dependencies
Breve, B., Caruccio, L., Deufemia, V., & Polese, G. (2022) · Movebank · doi:10.5441/002/edbt.2022.05 · Cited by 18 · cited 1×
“[9] proposed a method to impute missing values by identifying candidate values that ensure data integrity using relaxed functional dependencies.”
MethodEnd-to-end Multi-task Learning of Missing Value Imputation and Forecasting in Time-Series Data
Kim, J.-H., Kim, T., Choi, J., & Choo, J. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412112 · Cited by 14 · cited 1×
“The representational learning of a deep neural network is used to extract features of tabular data that aid predictive imputation [6,12,13].”
MethodMIDA: Multiple Imputation using Denoising AutoencodersPreprint
Gondara, L., & Wang, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.02737 · Cited by 10 · cited 1×
“MVI methods employing generative models include Generative Adversarial Imputation Nets (GAIN) [3] and denoising autoencoder (DAE) for multiple imputations [4]; MVI methods that feature predictive models include MissForest [5] and Datawig [6].”
MethodDeep Neural Networks and Tabular Data: A Survey
Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., & Kasneci, G. (2021) · arXiv · doi:10.48550/arxiv.2110.01889 · cited 1×
“The representational learning of a deep neural network is used to extract features of tabular data that aid predictive imputation [6,12,13].”
Method
Research works printed but not linked: 2
- Arik, S.Ö., and Pfister, T. (March, January 22). Tabnet: Attentive interpretable tabular learning. Proceedings of the AAAI Conference on Artificial Intelligence (Virtual Conference), Virtual.
- Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
Cited by
Data Quality in the Age of AI: A Review of Governance, Ethics, and the FAIR Principles
Guillen-Aguinaga, M., Aguinaga-Ontoso, E., Guillen-Aguinaga, L., Guillen-Grima, F., & Aguinaga-Ontoso, I. (2025) · Data · doi:10.3390/data10120201 · Cited by 37
“Advanced methods, including deep learning models such as the Denoising Self-Attention Network (DSAN), enhance the imputation of mixed-type tabular data by leveraging self-attention and multitask learning to predict substitutes for both numerical and categorical fields [197].”
Attention-Based Imputation of Missing Values in Electronic Health Records Tabular Data
Kowsar, I., Rabbani, S. B., & Samad, M. D. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00030 · Cited by 24
“A recent study proposes a denoising self-attention network (DSAN) to impute missing values in numerical and category columns [21].”
BackgroundKowsar, I., Rabbani, S. B., Hou, Y., & Samad, M. D. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113506 · Cited by 17
“GAIN[12] Diffputer[22] DSAN[11] MIDASpy[15] Between-feature[40] Between-sample[ GAIN[12] Diffputer[22] DSAN[11] MIDASpy[15] Between-feature[40] Between-sample[ with 30% missing values, respectively.”
InfluentialHarmonisation of Indonesia's One Poverty Data Policy Oriented Towards Sustainable Development Goals
Ardiansyah, A., Ilyas, A., & Haeranah, H. (2023) · Journal of Law and Sustainable Development · doi:10.55908/sdgs.v11i11.1413 · Cited by 5
Al-Ghiffari, M. F., Arief, M., & Pramesti, D. (2026) · International Conference on Electronics Representation and Algorithm (ICERA) · doi:10.1109/icera72709.2026.11666637
“Prior studies on missing-value handling show that imputation and native missing-value strategies can behave differently across tabular datasets [13], [14].”
Metadata
- Authors
- Lee, D.-H., & Kim, H.-J.
- Affiliations
- University of Seoul (KR)
- School of Electrical and Computer Engineering, University of Seoul, 163 Seoulsiripdaero, Seoul 02504, Republic of Korea
- Venue
- Data
- Publisher
- MDPI AG
- Published
- 2023-06-04
- Volume
- 8
- Issue
- 6
- Pages
- 102
- ISSN
- 2306-5729 (ISSN-L)
- OpenAlex
- W4379472973
- Cited by
- 5
- References
- 20
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 1 · 2023: 1
- FWCI
- 1.02
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Data (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2022R1A2C1011937, IITP-2023-2018-08-01417)
- MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (NRF-2022R1A2C1011937, IITP-2023-2018-08-01417)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Information Systems › Data Quality and Management
- Keywords
- missing data imputation, tabular data, self-attention, multi-task learning, data quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.