A Survey of Handwritten Character Recognition with MNIST and EMNIST
Open accessThe DOI matches its Crossref record; 84 of the 84 references it cites were found in a registry or an index.
Abstract
This paper summarizes the top state-of-the-art contributions reported on the MNIST dataset for handwritten digit recognition. This dataset has been extensively used to validate novel techniques in computer vision, and in recent years, many authors have explored the performance of convolutional neural networks (CNNs) and other deep learning techniques over this dataset. To the best of our knowledge, this paper is the first exhaustive and updated review of this dataset; there are some online rankings, but they are outdated, and most published papers survey only closely related works, omitting most of the literature. This paper makes a distinction between those works using some kind of data augmentation and works using the original dataset out-of-the-box. Also, works using CNNs are reported separately; as they are becoming the state-of-the-art approach for solving this problem. Nowadays, a significant amount of works have attained a test error rate smaller than 1% on this dataset; which is becoming non-challenging. By mid-2017, a new dataset was introduced: EMNIST, which involves both digits and letters, with a larger amount of data acquired from a database different than MNIST’s. In this paper, EMNIST is explained and some results are surveyed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although the images are not of a real-world environment, they can be an appropriate alternative considering that the main objective of this study was to explore the possibility of deep learning for the objective quantification of the visual perception of the substituted signal.”
From Deep Learning-Based Optimization of Visual–Auditory Sensory Substitution (2023) · cites this paper as [31]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 7×
“For instance, Belongie et al. [5] achieved 0.63% and Keysers et al. [6] achieved 0.54% and 0.52% using K-NN, Kégl and Busa-Fekete [7] achieved 0.87% using boosted stumps on Haar features, LeCun et al. [4] achieved 0.8% and Decoste and Schölkopf [8] attained results from 0.56 to 0.68% using SVM.”
BackgroundMethodInfluentialExtreme learning machine: Theory and applications
Huang, G.-B., Zhu, Q.-Y., & Siew, C.-K. (2006) · Neurocomputing · doi:10.1016/j.neucom.2005.12.126 · Cited by 13,285 · cited 1×
“In 2015, McDonnell et al. [21] proposed an approach using the ‘extreme learning machine’ (ELM) [22] algorithm to train shallow non-convolutional neural networks, and they agree on the fact that data augmentation with distortions can reduce error rates even further.”
MethodShape matching and object recognition using shape contexts
Belongie, S., Malik, J., & Puzicha, J. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.993558 · Cited by 6,306 · cited 2×
“For instance, Belongie et al. [5] achieved 0.63% and Keysers et al. [6] achieved 0.54% and 0.52% using K-NN, Kégl and Busa-Fekete [7] achieved 0.87% using boosted stumps on Haar features, LeCun et al. [4] achieved 0.8% and Decoste and Schölkopf [8] attained results from 0.56 to 0.68% using SVM.”
MethodInfluentialDynamic Routing Between Capsules
Sabour, S., Frosst, N., & Hinton, G. E. (2017) · arXiv · doi:10.48550/arxiv.1710.09829 · Cited by 4,167 · cited 1×
“An interesting work using capsule layers (a concept recently introduced by Sabour et al. [76]) is TextCaps [77], a CNN with 3 convolutional layers and two capsule layers, which obtain accuracies of 95.36% in EMNIST Letters and 99.79% in EMNIST Digits.”
MethodInfluentialMulti-column deep neural networks for image classification
Ciresan, D., Meier, U., & Schmidhuber, J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248110 · Cited by 3,720
Best practices for convolutional neural networks applied to visual document analysis
Simard, P. Y., Steinkraus, D., & Platt, J. C. (2005) · Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings. · doi:10.1109/icdar.2003.1227801 · Cited by 2,741 · cited 2×
“87% NN 2-layer 800 hidden units, MSE loss [9] 0.”
BackgroundSalakhutdinov, R., & Hinton, G. E. (2009) · International Conference on Artificial Intelligence and Statistics · Cited by 2,605
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553453 · Cited by 2,412 · cited 2×
“Lee et al. [54] described convolutional deep belief networks with probabilistic max-pooling achieving an error rate of 0.82%.”
BackgroundInfluentialWhat is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · cited 3×
“…[39] 0.47% CNN (3 conv, 1 dense), stochastic pooling [25] 0.49% CNN (2 conv, 1 dense, relu) with dropout [24] 0.52% CNN, unsup pretraining [17] 0.53% CNN (2 conv, 1 dense, relu) with DropConnect [24] 0.57% SparseNet + SVM [15] 0.59% CNN (2 conv, 1 dense), unsup pretraining [16] 0.60%…”
MethodInfluential(2013) · Cited by 2,056
EMNIST: Extending MNIST to handwritten letters
Cohen, G., Afshar, S., Tapson, J., & van Schaik, A. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966217 · Cited by 1,651
Invariant Scattering Convolution Networks
Bruna, J., & Mallat, S. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.230 · Cited by 1,633 · cited 2×
“Bruna and Mallat [41] designed a network that performed wavelet transform convolutions to the input and a generative PCA (principal component analysis) classifier to obtain an error of 0.43%.”
BackgroundInfluentialPCANet: A Simple Deep Learning Baseline for Image Classification?
Chan, T.-H., Jia, K., Gao, S., Lu, J., Zeng, Z., & Ma, Y. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2475625 · Cited by 1,362
Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.06228 · Cited by 1,246 · cited 2×
“[38] used so-called convolutional “highway" networks (inspired by LSTM recurrent networks) to achieve an error rate of 0.”
Method(2011) · doi:10.5591/978-1-57735-516-8/ijcai11-210 · Cited by 1,229
Efficient Learning of Sparse Representations with an Energy-Based Model
Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0147 · Cited by 1,106
Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition
Ranzato, M., Huang, F. J., Boureau, Y.-L., & LeCun, Y. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383157 · Cited by 1,086 · cited 2×
“Ranzato et al. [40] used convolutional layers for feature extraction and two fully connected NN for classification, attaining an error rate of 0.62%.”
BackgroundInfluentialDeep, Big, Simple Neural Nets for Handwritten Digit Recognition
Cireşan, D. C., Meier, U., Gambardella, L. M., & Schmidhuber, J. (2010) · Neural Computation · doi:10.1162/neco_a_00052 · Cited by 1,078
On optimization methods for deep learning
Ngiam, J., Coates, A., Lahiri, A., Prochnow, B., Le, Q. V., & Ng, A. Y. (2011) · International Conference on Machine Learning · Cited by 1,076
Lee, C., Xie, S., Gallagher, P., Zhang, Z., & Zhuowen Tu, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.5185 · Cited by 993 · cited 2×
“In 2015, Lee et al. [27] described “deeply supervised nets”, an approach by which they introduce a classifier (SVM or softmax) at hidden layers, reporting a result of 0.39% without data augmentation.”
BackgroundMethodInfluentialRecurrent convolutional neural network for object recognition
Liang, M., & Hu, X. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298958 · Cited by 976 · cited 2×
“…CNNs [66] 0.25% Committees of 7 neuroevolved CNNs [64] 0.28% CNN with gated pooling function [30] 0.29% Inception-Recurrent CNN + LSUV + EVE [60] 0.29% Recurrent CNN [31] 0.31% CNN with norm. layers and piecewise linear activation units [32] 0.31% CNN (5 conv, 3 dense) with full training [45] 0.32%”
MethodInfluentialGoodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1302.4389 · Cited by 935 · cited 1×
“Goodfellow et al. [26] described the maxout model averaging technique, attaining a test error rate of 0.45% without data augmentation.”
MethodInfluentialTask-Driven Dictionary Learning
Mairal, J., Bach, F., & Ponce, J. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.156 · Cited by 823 · cited 2×
“…2-layer 800 hidden units [11] 0.39% RNN [48] 0.45% K-NN (P2DHMDM) [6] 0.52% COSFIRE [49] 0.52% K-NN (IDM) [6] 0.54% Task-driven dictionary learning [51] 0.54% Virtual SVM, deg-9 poly, 2-pixel jit [8] 0.56% RF-C-ELM, 15,000 hidden units [21] 0.57% PCANet (LDANet-2) [50] 0.62% K-NN (shape context)…”
MethodInfluentialStochastic Pooling for Regularization of Deep Convolutional Neural Networks
Zeiler, M. D., & Fergus, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3557 · Cited by 755
Extreme Learning Machines [Trends & Controversies]
Cambria, E., Huang, G.-B., Kasun, L. L. C., Zhou, H., Vong, C. M., Lin, J., Yin, J., Cai, Z., Liu, Q., Li, K., et al. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.140 · Cited by 710
Training Invariant Support Vector Machines
Decoste, D., & Schölkopf, B. (2002) · Machine Learning · doi:10.1023/a:1012454411458 · Cited by 588
Convolutional Neural Network Committees for Handwritten Character Classification
Ciresan, D. C., Meier, U., Gambardella, L. M., & Schmidhuber, J. (2011) · 2011 International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2011.229 · Cited by 536 · cited 1×
“[19] have used this database to test the performance of committees of CNNs, attaining accuracies of 88.”
MethodEMNIST: an extension of MNIST to handwritten letters
Cohen, G., Afshar, S., Tapson, J. C., & André, V. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.05373 · Cited by 419 · cited 3×
“As a result, Cohen et al. [67] introduced in April 2017 the Extended MNIST database (EMNIST), consisting on both handwritten digits and letters, and sharing the same structure than the MNIST database.”
MethodResultDesigning Neural Network Architectures using Reinforcement Learning
Baker, B., Gupta, O., Naik, N., & Raskar, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.02167 · Cited by 372 · cited 3×
“Finally, it is worth mentioning some techniques that automatically optimize the topology of CNNS: MetaQNN [61], which relies on reinforcement learning for the design of CNNs architectures, has reported an error rate on MNIST of 0.44%, and 0.32% when using an ensemble of the best found neural…”
MethodInfluentialSupervised translation-invariant sparse coding
Yang, J., Yu, K., & Huang, T. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539958 · Cited by 355 · cited 2×
“Finally, Yang et al. [56] used supervised translation-invariant sparse coding with a linear SVM attaining an error rate of 0.84%.”
InfluentialGraham, B. (2014) · arXiv · doi:10.48550/arxiv.1412.6071 · Cited by 337 · cited 2×
“MetaQNN (ensemble) [61] 0.32% Fractional max-pooling CNN with random overlapping [34] 0.32% CNN with competitive multi-scale conv. filters [33] 0.33% CNN neuroevolved with GE [63] 0.37% Fast-learning shallow CNN [35] 0.37% CNN FitNet with LSUV initialization and SVM [59] 0.38% Deeply supervised CNN…”
MethodInfluentialA trainable feature extractor for handwritten digit recognition
Lauer, F., Suen, C. Y., & Bloch, G. (2007) · Pattern Recognition · doi:10.1016/j.patcog.2006.10.011 · Cited by 319 · cited 3×
“Lauer et al. [14] attained between 0.54% and 0.83% using a trainable feature extractor along with SVMs, and Labusch et al. [15] reported an error rate of 0.59% using a similar technique consisting on a CNN to extract sparse features and SVMs for classification.”
BackgroundMethodInfluentialDeformation Models for Image Recognition
Keysers, D., Deselaers, T., Gollan, C., & Ney, H. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1153 · Cited by 255
Event-Driven Random Back-Propagation: Enabling Neuromorphic Deep Learning Machines
Neftci, E. O., Augustine, C., Paul, S., & Detorakis, G. (2017) · Frontiers in Neuroscience · doi:10.3389/fnins.2017.00324 · Cited by 237 · cited 1×
“As an example, the dataset has also been used by Neftci et al. [81] for training neuromorphic deep neural networks and by Shu et al. [82] for the sake of pairwise classification.”
MethodBeyond spatial pyramids: Receptive field learning for pooled image features
Jia, Y., Huang, C., & Darrell, T. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248076 · Cited by 234 · cited 1×
“[52] explored the receptive field of pooling, and obtained an error of 0.”
BackgroundAutomatic recognition of handwritten numerical strings: a recognition and verification strategy
Oliveira, L. S., Sabourin, R., Bortolozzi, F., & Suen, C. Y. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1046154 · Cited by 218
ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
Visin, F., Kastner, K., Cho, K., Matteucci, M., Courville, A. C., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00393 · Cited by 218 · cited 2×
“Test Error Rate NN 6-layer 5,700 hidden units [12] 0.35% MSRV C-SVDDNet [46] 0.35% Committee of 25 NN 2-layer 800 hidden units [11] 0.39% RNN [48] 0.45% K-NN (P2DHMDM) [6] 0.52% COSFIRE [49] 0.52% K-NN (IDM) [6] 0.54% Task-driven dictionary learning [51] 0.54% Virtual SVM, deg-9 poly, 2-pixel jit…”
BackgroundInfluentialYang, Z., Moczulski, M., Denil, M., De Freitas, N., Song, L., & Wang, Z. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.173 · Cited by 215 · cited 2×
“Finally, Yang et al. [44] developed a new transform named Adaptative Fastfood to reparameterize the fully connected layers, obtaining a test error rate of 0.71%.”
MethodInfluentialMishkin, D., & Matas, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06422 · Cited by 192 · cited 15×
“37% CNN FitNet with LSUV initialization and SVM [59] 0.”
MethodInfluentialEvolutionary convolutional neural networks: An application to handwriting recognition
Baldominos, A., Saez, Y., & Isasi, P. (2018) · Neurocomputing · doi:10.1016/j.neucom.2017.12.049 · Cited by 175 · cited 1×
“[63] presented a work in 2018 where the topology of the network is evolved using grammatical evolution, attaining a test error rate of 0.”
BackgroundSelf-citationBochinski, E., Senst, T., & Sikora, T. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8297018 · Cited by 168 · cited 3×
“24% Committees of evolved CNNs (CEA-CNN) [65] 0.”
BackgroundInfluentialMairal, J., Koniusz, P., Harchaoui, Z., & Schmid, C. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.3332 · Cited by 158 · cited 2×
“…shallow CNN [35] 0.37% CNN FitNet with LSUV initialization and SVM [59] 0.38% Deeply supervised CNN [27] 0.39% Convolutional kernel networks [36] 0.39% CNN with Multi-loss regularization [37] 0.42% MetaQNN [61] 0.44% CNN (3 conv maxout, 1 dense) with dropout [17] 0.45% Convolutional highway…”
MethodInfluentialDeep convex net: a scalable architecture for speech pattern classification
Deng, L., & Yu, D. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-607 · Cited by 150
Trainable COSFIRE Filters for Keypoint Detection and Pattern Recognition
Azzopardi, G., & Azzopardi, N. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.106 · Cited by 145 · cited 2×
“…units [12] 0.35% MSRV C-SVDDNet [46] 0.35% Committee of 25 NN 2-layer 800 hidden units [11] 0.39% RNN [48] 0.45% K-NN (P2DHMDM) [6] 0.52% COSFIRE [49] 0.52% K-NN (IDM) [6] 0.54% Task-driven dictionary learning [51] 0.54% Virtual SVM, deg-9 poly, 2-pixel jit [8] 0.56% RF-C-ELM, 15,000 hidden…”
MethodInfluentialMulti-Prediction Deep Boltzmann Machines
Goodfellow, I., Mirza, M., Courville, A. C., & Bengio, Y. (2013) · neural information processing systems · Cited by 120
Deep convolutional neural networks as generic feature extractors
Hertel, L., Barth, E., Kaster, T., & Martinetz, T. (2015) · 2015 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280683 · Cited by 113 · cited 3×
“31% CNN (5 conv, 3 dense) with full training [45] 0.”
MethodInfluentialGeneralizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree
Lee, C., Gallagher, P. W., & Zhuowen Tu, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.08985 · Cited by 107 · cited 2×
“[30] when using gated pooling functions in CNNs (0.”
BackgroundMethodAPAC: Augmented PAttern Classification with Neural Networks
Sato, I., Nishimura, H., & Yokoi, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.03229 · Cited by 105 · cited 1×
“[28] explore and optimize data augmentation, and attain a test error rate of 0.”
BackgroundEnhanced image classification with a fast-learning shallow convolutional neural network
McDonnell, M. D., & Vladusich, T. (2015) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280796 · Cited by 100 · cited 2×
“Additional works where state-of-the-art results were obtained without data augmentation include those by McFonnell and Vladusich [35], who reported a test error rate of 0.”
BackgroundInfluentialBotalb, A., Moinuddin, M., Al-Saggaf, U. M., & Ali, S. S. A. (2018) · International Conference on Intelligent and Advanced Systems · doi:10.1109/icias.2018.8540626 · Cited by 100 · cited 1×
“[72] comparing a multi-layer perceptron with CNNs in EMNIST.”
MethodSimple Method for High-Performance Digit Recognition Based on Sparse Coding
Labusch, K., Barth, E., & Martinetz, T. (2008) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2005830 · Cited by 99
Multi-loss Regularized Deep Neural Network
Xu, C., Lu, C., Liang, X., Gao, J., Zheng, W., Wang, T., & Yan, S. (2016) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2015.2477937 · Cited by 93 · cited 2×
“…using a fast-learning shallow convolutional neural network, Mairal et al. [36], who achieved 0.39% using convolutional kernel networks, Xu et al. [37], who explored multi-loss regularization in CNNs obtaining an error rate of 0.42%, and Srivastava et al. [38] used so-called convolutional…”
MethodInfluentialTextCaps: Handwritten Character Recognition With Very Small Datasets
Jayasundara, V., Jayasekara, S., Jayasekara, H., Rajasegaran, J., Seneviratne, S., & Rodrigo, R. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00033 · Cited by 93 · cited 2×
“[76]) is TextCaps [77], a CNN with 3 convolutional layers and two capsule layers, which obtain accuracies of 95.”
BackgroundInfluentialMcDonnell, M. D., Tissera, M. D., Vladusich, T., André, V. S., & Tapson, J. C. (2015) · PLoS ONE · doi:10.1371/journal.pone.0134254 · Cited by 89
Boosting products of base classifiers
Kégl, B., & Busa‐Fekete, R. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553439 · Cited by 88 · cited 2×
“52% using K-NN, Kégl and Busa-Fekete [7] achieved 0.”
MethodInfluentialEDEN: Evolutionary deep networks for efficient machine learning
Dufourq, E., & Bassett, B. A. (2017) · 2017 Pattern Recognition Association of South Africa and Robotics and Mechatronics (PRASA-RobMech) · doi:10.1109/robomech.2017.8261132 · Cited by 88
Unseen Class Discovery in Open-world ClassificationPreprint
Shu, L., Xu, H., & Liu, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.05609 · Cited by 87 · cited 1×
“[82] for the sake of pairwise classification.”
A Deep Non-linear Feature Mapping for Large-Margin kNN Classification
Min, R., Stanley, D. A., Yuan, Z., Bonner, A., & Zhang, Z. (2009) · 2009 Ninth IEEE International Conference on Data Mining · doi:10.1109/icdm.2009.27 · Cited by 83
Batch-normalized Maxout Network in Network
Chang, J.-R., & Chen, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02583 · Cited by 83 · cited 2×
“A very interesting result is reported by Chang and Chen [29], where an error rate of 0.24% (the best result found as of 2017 in the state of the art without data augmentation) was obtained by using a network-in-network approach with a maxout multi-layer perceptron, an approach they named MIN.”
BackgroundResultInfluentialBetter Digit Recognition with a Committee of Simple Neural Nets
Meier, U., Ciresan, D. C., Gambardella, L. M., & Schmidhuber, J. (2011) · 2011 International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2011.252 · Cited by 69 · cited 2×
“Test Error Rate NN 6-layer 5,700 hidden units [12] 0.35% MSRV C-SVDDNet [46] 0.35% Committee of 25 NN 2-layer 800 hidden units [11] 0.39% RNN [48] 0.45% K-NN (P2DHMDM) [6] 0.52% COSFIRE [49] 0.52% K-NN (IDM) [6] 0.54% Task-driven dictionary learning [51] 0.54% Virtual SVM, deg-9 poly, 2-pixel jit…”
ResultInfluentialBangla Handwritten Digit Recognition Using Deep CNN for Large and Unbiased Dataset
Shawon, A., Rahman, M. J.-U., Mahmud, F., & Zaman, M. M. A. (2018) · International Conference on Bangla Speech and Language Processing (ICBSLP) · doi:10.1109/icbslp.2018.8554900 · Cited by 62 · cited 1×
“[85] designed a system using a CNN with six convolutional layers and two fully connected layers for Bangla handwritten digit recognition, testing it also in EMNIST and reporting accuracies of 90.”
MethodOn the importance of normalisation layers in deep learning with piecewise linear activation units
Liao, Z., & Carneiro, G. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477624 · Cited by 55 · cited 2×
“…CNNs [66] 0.25% Committees of 7 neuroevolved CNNs [64] 0.28% CNN with gated pooling function [30] 0.29% Inception-Recurrent CNN + LSUV + EVE [60] 0.29% Recurrent CNN [31] 0.31% CNN with norm. layers and piecewise linear activation units [32] 0.31% CNN (5 conv, 3 dense) with full training [45] 0.32%”
MethodInfluentialOptical Character Recognition on Bank Cheques Using 2D Convolution Neural Network
Srivastava, S., Priyadarshini, J., Gopal., S., Gupta, S., & Dayal, H. S. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-981-13-1822-1_55 · Cited by 49 · cited 1×
“Srivastava et al. [83] used EMNIST for training a optical character recognition system for bank cheques, and Sharma et al. [84] used the database for optical recognition in specialized forms filled by students, although in neither case performance on the EMNIST test set was reported.”
MethodUnsupervised feature learning with C-SVDDNet
Wang, D., & Tan, X. (2016) · Pattern Recognition · doi:10.1016/j.patcog.2016.06.001 · Cited by 48
Online and adaptive pseudoinverse solutions for ELM weights
André, V. S., & Tapson, J. C. (2014) · Neurocomputing · doi:10.1016/j.neucom.2014.01.071 · Cited by 47 · cited 1×
“In the original EMNIST paper [67], authors included a baseline using a linear classifier and another technique named OPIUM (Online Pseudo-Inverse Update Method), a classifier introduced by van Schaik and Tapson [70].”
MethodEstimating accurate multi-class probabilities with support vector machines
Milgram, J., Cheriet, M., & Sabourin, R. (2006) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2005.1556171 · Cited by 44 · cited 2×
“90% SVMs (one against all + sigmoid) [86] – – – – 98.”
Deep convolutional extreme learning machines: Filters combination and error model validation
Santos, M. M., Filho, A. G. D. S., & dos Santos, W. P. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.10.063 · Cited by 42 · cited 1×
“Another interesting work was published by dos Santos et al. [78], where they proposed the use of deep convolutional extreme learning machine, where gradient descent is not used to train the network, allowing a very fast learning stage (only 21 min using CPU, according to authors).”
BackgroundBaldominos, A., Sáez, Y., & Isasi, P. (2019) · Complexity · doi:10.1155/2019/2952304 · Cited by 42 · cited 3×
“…[29] 0.24% Committees of evolved CNNs (CEA-CNN) [65] 0.24% Genetically evolved committee of CNNs [66] 0.25% Committees of 7 neuroevolved CNNs [64] 0.28% CNN with gated pooling function [30] 0.29% Inception-Recurrent CNN + LSUV + EVE [60] 0.29% Recurrent CNN [31] 0.31% CNN with norm. layers…”
BackgroundMethodInfluentialSelf-citationSparse Activity and Sparse Connectivity in Supervised LearningPreprint
Thom, M., & Palm, G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.08367 · Cited by 41 · cited 2×
“…0.64% Virtual SVM, deg-9 poly, 1-pixel jit [8] 0.68% NN 2-layer 800 hidden units, XE loss [9] 0.70% SOAE-σ with sparse connectivity and activity [53] 0.75% SVM, deg-9 poly [4] 0.80% Product of stumps on Haar f. [7] 0.87% NN 2-layer 800 hidden units, MSE loss [9] 0.90% CNN (2 conv, 1 dense,…”
BackgroundMethodInfluentialHandwritten Digit and Letter Recognition Using Hybrid DWT-DCT with KNN and SVM Classifier
Ghadekar, P. P., Ingole, S., & Sonone, D. (2018) · International Conference on Computing Communication Control and automation · doi:10.1109/iccubea.2018.8697684 · Cited by 40 · cited 1×
“[71] discrete wavelet transform (DWT) and discrete cosine transform (DCT) were hybridized to extract features from the handwritten samples, later using K-NN and SVM for classification.”
MethodConfusion Matrix-Based Building of Hierarchical Classification
Cavalin, P., & Oliveira, L. S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-13469-3_32 · Cited by 38 · cited 4×
“62% CNN (flat; 2 conv + 1 dense) [79] – – 87.”
MethodInfluentialCompetitive Multi-scale ConvolutionPreprint
Liao, Z., & Carneiro, G. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05635 · Cited by 38 · cited 2×
“MetaQNN (ensemble) [61] 0.32% Fractional max-pooling CNN with random overlapping [34] 0.32% CNN with competitive multi-scale conv. filters [33] 0.33% CNN neuroevolved with GE [63] 0.37% Fast-learning shallow CNN [35] 0.37% CNN FitNet with LSUV initialization and SVM [59] 0.38% Deeply supervised CNN…”
MethodInfluentialAn implicit segmentation-based method for recognition of handwritten strings of characters
Cavalin, P., Britto, A. D. S., Bortolozzi, F., Sabourin, R., & Oliveira, L. S. (2006) · ACM Symposium on Applied Computing · doi:10.1145/1141277.1141468 · Cited by 37 · cited 2×
“[91] used hidden Markov models reporting an accuracy of 87% for letters (both uppercase and lowercase) and 98% for digits.”
MethodDeep Versus Wide Convolutional Neural Networks for Object Recognition on Neuromorphic System
Alom, M. Z., Josue, T., Rahman, N., Mitchell, W., Yakopcic, C., & Taha, T. M. (2018) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489635 · Cited by 36
Supervised Learning of Fuzzy ARTMAP Neural Networks Through Particle Swarm Optimisation
Granger, É., Henniges, P., Sabourin, R., & Oliveira, L. S. (2007) · Journal of Pattern Recognition Research · doi:10.13176/11.23 · Cited by 32 · cited 2×
“53% * PSO + fuzzy ARTMAP NNs [87] – – – – 96.”
MethodParallelization of digit recognition system using Deep Convolutional Neural Network on CUDA
Singh, S., Paul, A., & Arun, M. R. (2017) · International Conference on Sensing, Signal Processing and Security (ICSSS) · doi:10.1109/ssps.2017.8071623 · Cited by 30 · cited 1×
“[74] have used a CNN with three convolutional layers and two dense layers, reporting an accuracy in EMNIST Digits of 99.”
MethodUnconstrained handwritten character recognition using metaclasses of characters
Koerich, A. L., & Kalva, P. R. (2005) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2005.1530112 · Cited by 23 · cited 3×
“49% * Multi-layer perceptron [90] – – – 87.”
MethodInfluentialSparse activity and sparse connectivity in supervised learning
ThomMarkus, & PalmGünther (2013) · Journal of Machine Learning Research · doi:10.5555/2567709.2502615 · Cited by 23 · cited 2×
“…0.64% Virtual SVM, deg-9 poly, 1-pixel jit [8] 0.68% NN 2-layer 800 hidden units, XE loss [9] 0.70% SOAE-σ with sparse connectivity and activity [53] 0.75% SVM, deg-9 poly [4] 0.80% Product of stumps on Haar f. [7] 0.87% NN 2-layer 800 hidden units, MSE loss [9] 0.90% CNN (2 conv, 1 dense,…”
BackgroundMethodInfluentialModel Selection in Committees of Evolved Convolutional Neural Networks Using Genetic Algorithms
Baldominos, A., Sáez, Y., & Isasi, P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-03493-1_39 · Cited by 15
Self-citationHybrid orthogonal projection and estimation (HOPE)
Zhang, S., Jiang, H., & Dai, L.-R. (2016) · Journal of Machine Learning Research · doi:10.5555/2946645.2946682 · Cited by 15
A Deep CNN Model for Student Learning Pedagogy Detection Data Collection Using OCR
Sharma, A. S., Mridul, M. A., Marium-E-Jannat, & Islam, M. S. (2018) · International Conference on Bangla Speech and Language Processing (ICBSLP) · doi:10.1109/icbslp.2018.8554701 · Cited by 11
Using the RRT algorithm to optimize classification systems for handwritten digits and letters
Radtke, P. V. W., Sabourin, R., & Wong, T. (2008) · ACM Symposium on Applied Computing · doi:10.1145/1363686.1364103 · Cited by 8 · cited 2×
“00% * Record-to-record travel [89] – – – 93.”
MethodOn the Importance of Normalisation Layers in Deep Learning with Piecewise Linear Activation UnitsPreprint
Liao, Z., & Carneiro, G. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.00330 · Cited by 5 · cited 2×
“…CNNs [66] 0.25% Committees of 7 neuroevolved CNNs [64] 0.28% CNN with gated pooling function [30] 0.29% Inception-Recurrent CNN + LSUV + EVE [60] 0.29% Recurrent CNN [31] 0.31% CNN with norm. layers and piecewise linear activation units [32] 0.31% CNN (5 conv, 3 dense) with full training [45] 0.32%”
MethodInfluentialLarge-Margin kNN Classification Using a Deep Encoder NetworkPreprint
Min, M. R., Stanley, D. A., Yuan, Z., et al. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0906.1814 · Cited by 2
Research works printed but not linked: 5
- Kasun, 2013, Representational learning with extreme learning machine for big data, IEEE Intell. Syst., 28, 31
- Lin, M., Chen, Q., and Yan, S. (2014, January 14–16). Network In Network. Proceedings of the 2nd International Conference on Learning Representations, Banff, AB, Canada.
- Calderón, A., Roa-Valle, S., and Victorino, J. (2003, January 19–21). Handwritten digit recognition using convolutional neural networks and Gabor filters. Proceedings of the 2003 International Conference on Computational Intelligence, Cancun, Mexico.
- Yin, 2017, Markov Random Field Based Convolutional Neuralx Networks for Image Classification, IDEAL 2017: Intelligent Data Engineering and Automated Learning, Volume 10585, 387
- Mor, 2019, Handwritten text recognition: With deep learning and Android, Int. J. Eng. Adv. Technol., 8, 172
Cited by
Arabic handwriting recognition system using convolutional neural network
Altwaijry, N., & Al-Turaiki, I. (2021) · Neural Computing and Applications · doi:10.1007/s00521-020-05070-8 · Cited by 228
Sakai, A., Komatsu, M., Komatsu, R., Matsuoka, R., Yasutomi, S., Dozen, A., Shozu, K., Arakaki, T., Machino, H., Asada, K., et al. (2022) · Biomedicines · doi:10.3390/biomedicines10030551 · Cited by 68
“The splitting ratio of the data was standard for machine learning [47–49].”
MethodA Hybrid Sparrow Search Algorithm of the Hyperparameter Optimization in Deep Learning
Fan, Y., Zhang, Y., Guo, B., Luo, X., Peng, Q., & Jin, Z. (2022) · Mathematics · doi:10.3390/math10163019 · Cited by 59
Classifying Garments from Fashion-MNIST Dataset Through CNNs
Henrique, A. S., Fernandes, A. M. D. R., Lyra, R., Leithardt, V. R. Q., Correia, S. D., Crocker, P. A., & Dazzi, R. L. S. (2021) · Advances in Science Technology and Engineering Systems Journal · doi:10.25046/aj0601109 · Cited by 50
Startups and Consumer Purchase Behavior: Application of Support Vector Machine Algorithm
Ebrahimi, P., Salamzadeh, A., Soleimani, M., Khansari, S. M., Zarea, H., & Fekete‐Farkas, M. (2022) · Big Data and Cognitive Computing · doi:10.3390/bdcc6020034 · Cited by 49
“Unsupervised learning is the path to realizing “real” artificial intelligence [60].”
Background
Metadata
- Authors
- Baldominos, A., Saez, Y., & Isasi, P.
- Affiliations
- Universidad Carlos III de Madrid (ES)
- Computer Science Department, Universidad Carlos III of Madrid, 28911 Leganés, Madrid, Spain
- Corresponding author
- Baldominos, Alejandro
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2019-08-04
- Volume
- 9
- Issue
- 15
- Pages
- 3169
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W2966275995
- Cited by
- 279
- References
- 76
- Citations by year
- 2026: 20 · 2025: 39 · 2024: 64 · 2023: 54 · 2022: 41 · 2021: 36 · 2020: 24 · 2019: 1
- FWCI
- 11.41
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- e-Archivo (Carlos III University of Madrid) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Ministerio de Educación, Cultura y Deporte (FPU13/03917)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Vehicle License Plate Recognition
- Keywords
- handwritten character recognition, handwritten digit recognition, MNIST, EMNIST, convolutional neural networks, data augmentation, letter recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.


