Character-Level Language Modeling with Deeper Self-Attention
Open accessAbstract
LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model (Vaswani et al. 2017) with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses, both at intermediate network layers and intermediate sequence positions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Auxiliary losses in decoder help the model output the correct number of objects of each class.
“We found helpful to use auxiliary losses [1] in decoder during training, especially to help the model output the correct number of objects of each class.”
From End-to-End Object Detection with Transformers (2020) · cites this paper as [1]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Self-citationHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., & Janvin, C. (2003) · Journal of machine learning research · doi:10.1162/153244303322533223 · Cited by 8,257 · see citing works
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · Neural Information Processing Systems · doi:10.48550/arxiv.1509.01626 · Cited by 7,552 · see citing works
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,091 · see citing works
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879 · see citing works
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,451 · see citing works
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317 · see citing works
Language Modeling with Gated Convolutional NetworksPreprint
Dauphin, Y., Fan, A., Auli, M., & Grangier, D. (2016) · International Conference on Machine Learning · doi:10.48550/arxiv.1612.08083 · Cited by 3,097 · see citing works
Recurrent Neural Network RegularizationPreprint
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv.org · doi:10.48550/arxiv.1409.2329 · Cited by 3,086 · see citing works
LSTM neural networks for language modeling
Sundermeyer, M., Schlüter, R., & Ney, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-65 · Cited by 1,963 · see citing works
Extensions of recurrent neural network language model
Mikolov, T., Kombrink, S., Burget, L., Černocký, J., & Khudanpur, S. P. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947611 · Cited by 1,594 · see citing works
Exploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y.-H. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02410 · Cited by 1,202 · see citing works
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
Li, S., Li, W., Cook, C., Zhu, C., & Gao, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00572 · Cited by 972 · see citing works
Memory NetworksPreprint
Weston, J., Chopra, S., & Bordes, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.3916 · Cited by 819 · see citing works
A Theoretically Grounded Application of Dropout in Recurrent Neural NetworksPreprint
Gal, Y., & Zoubin Ghahramani, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.05287 · Cited by 765 · see citing works
Regularizing and Optimizing LSTM Language Models
Merity, S., Keskar, N. S., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.02182 · Cited by 697 · see citing works
Gated Feedback Recurrent Neural Networks
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.02367 · Cited by 583 · see citing works
Axiom: A Householder-Parameterized Pure Unitary RNN for Long-Range Sequence ModelingPreprint
Arjovsky, M., Shah, A., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06464 · Cited by 510 · see citing works
Neural Machine Translation in Linear Time
Kalchbrenner, N., Espeholt, L., Simonyan, K., Oord, A. V. D., Graves, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1610.10099 · Cited by 414 · see citing works
Hierarchical Multiscale Recurrent Neural Networks
Chung, J., Ahn, S., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1609.01704 · Cited by 386 · see citing works
Learning to Generate Reviews and Discovering Sentiment
Radford, A., Jozefowicz, R., & Sutskever, I. (2017) · arXiv · doi:10.48550/arxiv.1704.01444 · Cited by 330 · cited 1× · see citing works
“Character-level modeling has shown promise in many areas such as sentiment analysis [35], question answering [21] and classification [50], and is an exciting area due to its simplicity and the ability to easily adapt to other languages.”
BackgroundImproving Neural Language Models with a Continuous CachePreprint
Grave, E., Joulin, A., & Usunier, N. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1612.04426 · Cited by 316 · see citing works
Sharp nearby, fuzzy far away: How neural language models use context
Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1027 · Cited by 303 · see citing works
Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018) · arXiv · doi:10.48550/arxiv.1805.04623 · Cited by 302 · see citing works
Recurrent Highway NetworksPreprint
Zilly, J., Srivastava, R., Koutník, J., & Schmidhuber, J. (2016) · arXiv (Cornell University) · doi:10.3929/ethz-b-000233981 · Cited by 243 · see citing works
Multiplicative LSTM for sequence modellingPreprint
Krause, B., Murray, I., Renals, S., & Lu, L. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07959 · Cited by 232 · see citing works
Recurrent Batch NormalizationPreprint
Cooijmans, T., Ballas, N., Laurent, C., Gülçehre, Ç., & Courville, A. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.09025 · Cited by 218 · see citing works
Zoneout: Regularizing RNNs by Randomly Preserving Hidden ActivationsPreprint
Krueger, D., Maharaj, T., Kramár, J., Pezeshki, M., Ballas, N., Ke, N. R., Goyal, A., Bengio, Y., Courville, A. C., & Pal, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01305 · Cited by 214 · see citing works
Very Deep Convolutional Networks for Natural Language Processing.Preprint
Conneau, A., Schwenk, H., Barrault, L., & LeCun, Y. (2016) · arXiv (Cornell University) · Cited by 214 · see citing works
Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNPreprint
Li, S., Li, W., Cook, C., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04831 · Cited by 123 · see citing works
Improving GANs Using Optimal TransportPreprint
Salimans, T., Zhang, H., Radford, A., & Metaxas, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.05573 · Cited by 120 · see citing works
Architectural Complexity Measures of Recurrent Neural NetworksPreprint
Zhang, S., Wu, Y., Che, T., Lin, Z., Memisevic, R., Salakhutdinov, R., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.08210 · Cited by 114 · see citing works
Dynamic Evaluation of Neural Sequence Models
Krause, B., Kahembwe, E., Murray, I., & Renals, S. (2017) · arXiv · doi:10.48550/arxiv.1709.07432 · Cited by 90 · see citing works
Fast-Slow Recurrent Neural NetworksPreprint
Mujika, A., Meier, F., & Steger, A. (2017) · Neural Information Processing Systems · doi:10.48550/arxiv.1705.08639 · Cited by 83 · see citing works
Frustratingly Short Attention Spans in Neural Language ModelingPreprint
Daniluk, M., Rocktäschel, T., Welbl, J., & Riedel, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.04521 · Cited by 58 · see citing works
Memory Architectures in Recurrent Neural Network Language Models
Yogatama, D., Miao, Y., Melis, G., Wang, L., Kuncoro, A., Dyer, C., & Blunsom, P. (2018) · International Conference on Learning Representations · Cited by 49 · see citing works
Unbiasing Truncated Backpropagation Through TimePreprint
Tallec, C., & Ollivier, Y. (2017) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.48550/arxiv.1705.08209 · Cited by 47 · see citing works
Byte-Level Machine Reading Across Morphologically Varied Languages
Kenter, T., Jones, L., & Hewlett, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12050 · Cited by 19 · see citing works
Self-citationSparse Attentive Backtracking: Long-Range Credit Assignment in Recurrent NetworksPreprint
Ke, N. R., Goyal, A., Bilaniuk, O., Binas, J., Charlin, L., Pal, C., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02326 · Cited by 9 · see citing works
Surprisal-Driven Feedback in Recurrent NetworksPreprint
Rocki, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.06027 · Cited by 6 · see citing works
MuFuRU: The Multi-Function Recurrent UnitPreprint
Weissenborn, D., & Rocktäschel, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.03002 · Cited by 3 · see citing works
Flagged first, then by citations
6 of the 51 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Chelba, C.; Mikolov, T.; Schuster, M.; Ge, Q.; Brants, T.; Koehn, P.; and Robinson, T. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005 . · arXiv
- Lei Ba, J.; Kiros, J. R.; and Hinton, G. E. 2016. Layer Normalization. ArXiv e-prints .
- Mikolov, T.; Sutskever, I.; Deoras, A.; Le, H.-S.; Kombrink, S.; and Cernocky, J. 2012. Subword language modeling with neural networks. preprint (http://www. fit. vutbr. cz/imikolov/rnnlm/char. pdf) 8.
- Shazeer, N.; Mirhoseini, A.; Maziarz, K.; Davis, A.; Le, Q. V.; Hinton, G. E.; and Dean, J. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. CoRR abs/1701.06538. · arXiv
- Sukhbaatar, S.; Weston, J.; Fergus, R.; et al. 2015. End-to-end memory networks. In Advances in neural information processing systems , 2440-2448.
Cited by
Sorted by citations
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
Cross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586 · see citing works
Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422 · see citing works
Generating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063 · see citing works
Transformer-XL: Attentive Language Models Beyond a Fixed-Length ContextPreprint
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.ecrts.2023.7 · Cited by 860 · see citing works
Structured Denoising Diffusion Models in Discrete State-SpacesPreprint
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., & van den Berg, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03006 · Cited by 340 · see citing works
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models
Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., & Raffel, C. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00461 · Cited by 330 · see citing works
Self-citationCompressive Transformers for Long-Range Sequence Modelling
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019) · arXiv · doi:10.48550/arxiv.1911.05507 · Cited by 205 · see citing works
Code Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209 · see citing works
On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention
Lee, J., Park, S., Baek, J., Oh, S. J., Kim, S., & Lee, H. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00281 · Cited by 172 · see citing works
Smart Glass System Using Deep Learning for the Blind and Visually Impaired
Mukhiddinov, M., & Cho, J. (2021) · Electronics · doi:10.3390/electronics10222756 · Cited by 132 · see citing works
Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right
Holtzman, A., West, P., Shwartz, V., Choi, Y., & Zettlemoyer, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.564 · Cited by 127 · see citing works
Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of ViewPreprint
Lu, Y., Li, Z., He, D., Sun, Z., Dong, B., Qin, T., Wang, L., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02762 · Cited by 146 · see citing works
R-Transformer: Recurrent Neural Network Enhanced TransformerPreprint
Wang, Z., Ma, Y., Liu, Z., & Tang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.05572 · Cited by 103 · see citing works
Explicit Sparse Transformer: Concentrated Attention Through Explicit SelectionPreprint
Zhao, G., Lin, J., Zhang, Z., Ren, X., Su, Q., & Sun, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.11637 · Cited by 131 · see citing works
Efficient Nearest Neighbor Language Models
He, J., Neubig, G., & Berg-Kirkpatrick, T. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.461 · Cited by 59 · see citing works
“, 2012) or selfattentional (Vaswani et al., 2017; Al-Rfou et al., 2019) neural networks are mostly parametric, where the predictions are solely dependent on the model parameters given the input data.”
BackgroundRevisiting Monocular Satellite Pose Estimation With Transformer
Wang, Z., Zhang, Z., Sun, X., Li, Z., & Yu, Q. (2022) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2022.3161605 · Cited by 59 · see citing works
Training language models with memory augmentation
Zhong, Z., Lei, T., & Chen, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.382 · Cited by 60 · see citing works
Wang, G., Chen, H., Chen, L., Zhuang, Y., Zhang, S., Zhang, T., Dong, H., & Gao, P. (2023) · Remote Sensing · doi:10.3390/rs15071773 · Cited by 53 · see citing works
BP-Transformer: Modelling Long-Range Context via Binary PartitioningPreprint
Ye, Z., Guo, Q., Gan, Q., Qiu, X., & Zhang, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04070 · Cited by 61 · see citing works
On the Limit of English Conversational Speech Recognition
Tüske, Z., Saon, G., & Kingsbury, B. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-211 · Cited by 41 · see citing works
Practical Text Classification With Large Pre-Trained Language ModelsPreprint
Kant, N., Puri, R., Yakovenko, N., & Catanzaro, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.01207 · Cited by 42 · see citing works
WB-DETR: Transformer-Based Detector without Backbone
Liu, F., Wei, H., Zhao, W., Li, G., Peng, J., & Li, Z. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00297 · Cited by 41 · see citing works
On the localness modeling for the self-attention based end-to-end speech synthesis
Yang, S., Lu, H., Kang, S.-Y., Xue, L.-M., Xiao, J., Su, D., Xie, L., & Yu, D. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.01.034 · Cited by 32 · see citing works
Dynamic Evaluation of Transformer Language Models
Krause, B., Kahembwe, E., Murray, I., & Renals, S. (2019) · arXiv · doi:10.48550/arxiv.1904.08378 · Cited by 30 · see citing works
Relevance-Promoting Language Model for Short-Text Conversation
Li, X., Li, P., Bi, W., Liu, X., & Lam, W. P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6340 · Cited by 16 · see citing works
Dual-axial self-attention network for text classification
Zhang, X., Qiu, X., Pang, J., Liu, F., & Li, X. (2021) · Science China Information Sciences · doi:10.1007/s11432-019-2744-2 · Cited by 16 · see citing works
Character-based NMT with TransformerPreprint
Gupta, R., Besacier, L., Dymetman, M., & Gallé, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04997 · Cited by 16 · see citing works
Segatron: Segment-Aware Transformer for Language Modeling and Understanding
Bai, H., Shi, P., Lin, J., Xie, Y., Tan, L., Xiong, K., Gao, W., & Li, M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17485 · Cited by 15 · see citing works
Ali, A. M., Zhuang, H., VanZwieten, J. H., Ibrahim, A. K., & Chérubin, L. M. (2021) · Forecasting · doi:10.3390/forecast3040056 · Cited by 13 · see citing works
Privacy Policy Classification with XLNet (Short Paper)
Mustapha, M., Krasnashchok, K., Bassit, A. A., & Skhiri, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66172-4_16 · Cited by 12 · see citing works
Neural Shuffle-Exchange Networks -- Sequence Processing in O(n log n) TimePreprint
Freivalds, K., Ozoliņš, E., & Šostaks, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.07897 · Cited by 12 · see citing works
Zhao, J., & Lee, H. J. (2022) · Applied Sciences · doi:10.3390/app12136497 · Cited by 11 · see citing works
Learning Dynamic Latent Spaces for Lifelong Generative Modelling
Ye, F., & Borş, A. G. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26291 · Cited by 11 · see citing works
An Empirical Study of Efficient ASR Rescoring with TransformersPreprint
Huang, H., & Peng, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.11450 · Cited by 15 · see citing works
Lim, Y. Q., Lim, C. M., Gan, K. H., & Samsudin, N. H. (2020) · IEEE Symposium on Computer Applications & Industrial Electronics (ISCAIE) · doi:10.1109/iscaie47305.2020.9108706 · Cited by 9 · see citing works
Residual Shuffle-Exchange Networks for Fast Processing of Long Sequences
Draguns, A., Ozoliņš, E., Šostaks, A., Apinis, M., & Freivalds, K. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16890 · Cited by 8 · see citing works
Copy Mechanism and Tailored Training for character-based data-to-text generation
Roberti, M., Bonetta, G., Cancelliere, R., & Gallinari, P. (2019) · Institutional Research Information System University of Turin (University of Turin) · Cited by 7 · see citing works
Xu, C., Xu, P., Dai, Y., Su, S., Zhang, L., Zhang, J., Bai, Y., Gao, T., Xie, Q., Shang, L., et al. (2025) · Energies · doi:10.3390/en18051275 · Cited by 6 · see citing works
Classical Chinese Poetry Generation based on Transformer-XL
Zhao, J., & Lee, H. J. (2021) · International Conference on Computer Engineering and Artificial Intelligence (ICCEAI) · doi:10.1109/icceai52939.2021.00011 · Cited by 6 · see citing works
Universal Language Model Fine-Tuning with Subword Tokenization for PolishPreprint
Czapla, P., Howard, J., & Kardas, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.10222 · Cited by 6 · see citing works
BERT for Opinion Mining and Sentiment Farming
Buche, A. (2020) · Bioscience Biotechnology Research Communications · doi:10.21786/bbrc/13.14/9 · Cited by 5 · see citing works
Pay Attention when RequiredPreprint
Mandava, S., Migacz, S., & Fit-Florea, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.04534 · Cited by 5 · see citing works
Language Model Using Differentiable Neural Computer Based on Forget Gate-Based Memory Deallocation
Lee, D., Park, H.-S., Seo, S., Kim, C., Son, H., Kim, G., & Kim, J. (2021) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2021.015430 · Cited by 4 · see citing works
Speformer: An Efficient Hardware-Software Cooperative Solution for Sparse Spectral Transformer
Sun, Y., Hu, W., Liu, F., Jiang, M., Huang, F., & Xu, D. G. (2022) · IEEE International Conference on Cyber Security and Cloud Computing / IEEE International Conference on Edge Computing and Scalable Cloud (CSCloud/EdgeCom) · doi:10.1109/cscloud-edgecom54986.2022.00039 · Cited by 3 · see citing works
Enabling Perspective-Aware Ai with Contextual Scene Graph Generation
Platnick, D., Alirezaie, M., & Rahnama, H. (2024) · Information · doi:10.3390/info15120766 · Cited by 3 · see citing works
Comparative Analysis of Deep Learning Models for Web Defacement Detection Based on Textual Context
Ayunda, S. P., Qomariasih, N., Hadiprakoso, R. B., & Kabetta, H. (2023) · IEEE International Conference on Cryptography, Informatics, and Cybersecurity (ICoCICs) · doi:10.1109/icocics58778.2023.10276697 · Cited by 3 · see citing works
Lee, D., Park, H.-S., Seo, S., Son, H., Kim, G., & Kim, J. (2021) · KSII Transactions on Internet and Information Systems · doi:10.3837/tiis.2021.03.002 · Cited by 3 · see citing works
Design and Development of a Public AI Referee Assistance System Based on Harmony OS Platform
Zhao, J., Zhu, C., Leng, B., & Qi, J. (2025) · Sensors · doi:10.3390/s25072127 · Cited by 2 · see citing works
Advances in Sentiment Analysis in Deep Learning Models and Techniques
Lakshmi, E. A. P. V. (2023) · International Journal on Recent and Innovation Trends in Computing and Communication · doi:10.17762/ijritcc.v11i9.8831 · Cited by 2 · see citing works
Zhang, K., Ning, W., Zhu, Y., Li, Z., Wang, T., Jiang, W., Zeng, M., & Yang, Z. (2023) · Applied Sciences · doi:10.3390/app14010218 · Cited by 2 · see citing works
End-to-End Entity Detection with Proposer and Regressor
Wen, X., Zhou, C., Tang, H., Liang, L., Qi, H., & Jiang, Y. L. (2023) · Neural Processing Letters · doi:10.1007/s11063-023-11201-8 · Cited by 2 · see citing works
Character-Level Japanese Text Generation with Attention Mechanism for Chest Radiography Diagnosis
Sakka, K., Nakayama, K., Kimura, N., Inoue, T., Iwasawa, Y., Yamaguchi, R., Kawazoe, Y., Ohe, K., & Matsuo, Y. (2020) · Studies in computational intelligence · doi:10.1007/978-3-030-53352-6_13 · Cited by 2 · see citing works
A Combinatorial Strategy for API Completion: Deep Learning and Heuristics
Liu, Y., Yin, Y., Deng, J., Li, W., & Peng, Z. (2024) · Electronics · doi:10.3390/electronics13183669 · Cited by 1 · see citing works
GENPIA: A Genre-Conditioned Piano Music Generation System
Nguyen, Q.-V., Lai, H.-W., Nguyen, K.-D., Sun, M.-T., Hwang, W., Sakai, K., & Ku, W. (2024) · IEEE International Symposium on the Internet of Sounds (IS2) · doi:10.1109/is262782.2024.10704094 · Cited by 1 · see citing works
Sim-T: Simplify the Transformer Network by Multiplexing Technique for Speech RecognitionPreprint
Duan, Z., Wei, G., Li, S., Yang, G., Yu, X., Li, J., & Li, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4611060 · Cited by 1 · see citing works
Semantically-enabled Browsing of Large Multilingual Document Collections
Badenes-Olmedo, C. (2021) · doi:10.20868/upm.thesis.67594
Hou, W.-J., & Ceesay, B. (2021) · International Journal of Machine Learning and Computing · doi:10.18178/ijmlc.2021.11.4.1046
Medical BigBERTa: An Optimized Transformer for Long Medical DocumentPreprint
Mugisha, C., & Paik, I. (2022) · doi:10.36227/techrxiv.21673598.v1
Medical BigBERTa: An Optimized Transformer for Long Medical DocumentPreprint
Mugisha, C., & Paik, I. (2022) · doi:10.36227/techrxiv.21673598
Spatial-Temporal Decoupled Deformable Attention for Video Instance SegmentationPreprint
Zhang, Z., Shao, F., Dong, Z., Dai, Z., & Zhu, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4765637
Martinez-Ramon, M., Ajith, M., & Kurup, A. R. (2024) · Wiley eBooks · doi:10.1002/9781119861898.biblio
Xu, C., Xu, P., Dai, Y., Su, S., Zhang, L., Zhang, J., Bai, Y., Gao, T., Xie, Q., Shang, L., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4898980
BiomarkerExtractor: A Deep Learning–Based Model for Text Mining of Urine Biomarkers
Bai, M., Zhang, X., & Shao, C. (2025) · Springer eBooks · doi:10.1007/978-981-96-7886-0_7
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
- Authors
- Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L.
- Affiliations
- Google (United States) (US)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2019-07-17
- Volume
- 33
- Issue
- 01
- Pages
- 3159-3166
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2886490473
- Cited by
- 56
- References
- 51
- Citations by year
- 2025: 3 · 2024: 6 · 2023: 7 · 2022: 4 · 2021: 12 · 2020: 8 · 2019: 13 · 2018: 3
- FWCI
- 3.54
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- www.aaai.org/ (unspecified)
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- self-attention, transformer models, auxiliary loss, LSTM comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
