The DOI matches its Crossref record; 78 of the 82 references it cites were found in a registry or an index.
Muppet: Massive Multi-task Representations with Pre-Finetuning
Open accessAbstract
We propose pre-finetuning, an additional largescale learning stage between language model pre-training and fine-tuning.Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to encourage learning of representations that generalize better to many different tasks.We show that prefinetuning consistently improves performance for pretrained discriminators (e.g.RoBERTa) and generation models (e.g.BART) on a wide range of tasks (sentence prediction, commonsense reasoning, MRC, etc.), while also significantly improving sample efficiency during fine-tuning.We also show that large-scale multi-tasking is crucial; pre-finetuning can hurt performance when few tasks are used up until a critical point (usually above 15) after which performance improves linearly in the number of tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
(2014) · Cited by 33,419
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Self-citationExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Self-citationBuilding a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
Research works printed but not linked: 7
- Hector Levesque, Ernest Davis, and Leora Morgen- stern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Princi- ples of Knowledge Representation and Reasoning. Citeseer.
- Xin Li and Dan Roth. 2002. Learning question clas- sifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
- Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S. Gordon. 2011. Choice of plausible alterna- tives: An evaluation of commonsense causal reason- ing. In 2011 AAAI Spring Symposium Series.
- Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. 2015. Re- thinking the inception architecture for computer vi- sion. corr abs/1512.00567 (2015). · arXiv
- Adina Williams, Nikita Nangia, and Samuel Bowman. 2018b. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
- Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015b. Character-level convolutional networks for text classification. In NIPS.
Cited by
Sorted by citations
Items 101–185 of 185
Unveiling the Black Box of PLMs with Semantic Anchors: Towards Interpretable Neural Semantic Parsing
Nie, L., Sun, J., Wang, Y., Du, L., Shi, H., Zhang, D., Hou, L., Li, J., & Zhai, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26572 · Cited by 4
Code Compliance Assessment as a Learning Problem
Sawant, N., & Sengamedu, S. H. (2023) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip58684.2023.00046 · Cited by 5
Du, Y., & Chilton, L. B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.171 · Cited by 8
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Pagnoni, A., Fabbri, A. R., Kryściński, W., & Wu, C.-S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.713 · Cited by 8
Enabling Contextual Natural Language Search on Oil and Gas Databases
Lokhande, A., Pillai, P., & Mangsuli, P. (2023) · Abu Dhabi International Petroleum Exhibition & Conference (ADIPEC) · doi:10.2118/216349-ms · Cited by 4
Kumar, S., Bhatia, S., Aggarwal, M., & Chakraborty, T. (2024) · Natural language processing. · doi:10.1017/nlp.2024.42 · Cited by 5
Chai, Y., Chen, M., Wu, H., & Wang, S. (2023) · IEEE Access · doi:10.1109/access.2023.3299880 · Cited by 4
Wang, J., Wang, C., Wang, X., Huang, J., & Jin, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.8 · Cited by 7
Recyclable Tuning for Continual Pre-training
Qin, Y., Qian, C., Han, X., Lin, Y., Wang, H., Xie, R., Liu, Z., Sun, M., & Zhou, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.723 · Cited by 13
Vitiugin, F., & Purohit, H. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31410 · Cited by 6
PreQuEL: Quality Estimation of Machine Translation Outputs in Advance
Don-Yehiya, S., Choshen, L., & Abend, O. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.767 · Cited by 4
Identifying task groupings for multi-task learning using pointwise V-usable information
Li, Y., Miller, T., Bethard, S. J., & Savova, G. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104881 · Cited by 3
Artificial fine-tuning tasks for yes/no question answering
Dimitriadis, D., & Tsoumakas, G. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000286 · Cited by 4
Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics
Cho, H., Sankar, C., Lin, C., Sadagopan, K. R., Shayandeh, S., Çelikyılmaz, A., May, J., & Beirami, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.391 · Cited by 3
Software Entity Recognition with Noise-Robust Learning
Nguyen, T. D., Di, Y., Lee, J., Chen, M., & Zhang, T. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00203 · Cited by 3
E-code: Mastering efficient code generation through pretrained models and expert encoder group
Pan, Y., Lyu, C., Yang, Z., Li, L., Liu, Q., & Shao, X. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107602 · Cited by 4
Task-guided Disentangled Tuning for Pretrained Language Models
Zeng, J., Jiang, Y., Wu, S., Yin, Y., & Li, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.247 · Cited by 3
Goot, R. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.233 · Cited by 3
Long-Tailed Question Answering in an Open World
Dai, Y. Q., Lang, H., Zheng, Y., Huang, F., & Li, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.351 · Cited by 6
When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP
Ni, J., Jin, Z., Wang, Q., Sachan, M., & Leippold, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.412 · Cited by 4
Controlled Randomness Improves the Performance of Transformer Models
Deußer, T., Zhao, C., Krämer, W., Leonhard, D., Bauckhage, C., & Sifa, R. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00274 · Cited by 3
Zan, C., Ding, L., Shen, L., Cao, Y. K., & Liu, W. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109532 · Cited by 2
Wang, Y., Li, H., & Wei, N. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714644 · Cited by 3
Cross-prompt Pre-finetuning of Language Models for Short Answer Scoring
Funayama, H., Matsubayashi, Y., Asazuma, Y., Mizumoto, T., & Inui, K. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00474-w · Cited by 2
Asynchronous Convergence in Multi-Task Learning via Knowledge Distillation from Converged Tasks
Lu, W., Rajagopalan, S., Nigam, P., Singh, J., Sun, X., Xu, Y., Zeng, B., & Chilimbi, T. M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-industry.18 · Cited by 2
STT: Soft Template Tuning for Few-Shot Adaptation
Yu, P., Wang, W., Li, C., Zhang, R., Jin, Z., & Chen, C. (2022) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw58026.2022.00122 · Cited by 2
Pre-Finetuning with Impact Duration Awareness for Stock Movement Prediction
Chiu, C.-C., Chen, C.-C., Huang, H., & Chen, H. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715551 · Cited by 2
Learning From the Source Document: Unsupervised Abstractive Summarization
Zhuang, H., Zhang, W. E., Yang, J. Y., Ma, C., Qu, Y., & Sheng, Q. Z. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.309 · Cited by 2
Zero-Shot Classification by Logical Reasoning on Natural Language Explanations
Han, C., Pei, H., Du, X., & Ji, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.571 · Cited by 2
Awasthi, P., Cortes, C., & Mohri, M. (2024) · Annals of Mathematics and Artificial Intelligence · doi:10.1007/s10472-023-09917-3 · Cited by 2
Zhu, M., Liu, G., & Wei, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8620-6_26 · Cited by 1
Semantically Improved Adversarial Attack Based on Masked Language Model via Context Preservation
Tian, H., Wu, H., Cheung, Y., He, J., & Tian, Z. (2025) · International Conference on Dependable Systems & Networks/Proceedings · doi:10.1109/dsn64029.2025.00029 · Cited by 1
How scenarios can impact small models for QA task in education?
Tarifa, A., & Chabir, C. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.09.525 · Cited by 1
Singh, P., & Raman, B. (2026) · Knowledge and Information Systems · doi:10.1007/s10115-026-02719-z · Cited by 1
A Hybrid Automatic Text Summarization Model for Judgment Documents
Ouyang, J., Huang, W., & Liu, T. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-06767-9_31 · Cited by 1
Self-supervision and Controlling Techniques to Improve Counter Speech Generation
Saha, P. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3572991 · Cited by 1
Small is the New Big: Pre-finetuned compact models are better for Asynchronous Active Learning
Liu, D., Pavani, K., & Dasgupta, S. (2023) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2023.sustainlp-1.7 · Cited by 1
Investigating the Effect of Pre-finetuning BERT Models on NLI Involving Presuppositions
Kabbara, J., & Cheung, J. C. K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.703 · Cited by 2
Word and Character Semantic Fusion by Pretrained Language Models for Text Classification
Wu, Y., & Wan, J. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651461 · Cited by 1
Improving Text Classification Performance Through Multimodal Representation
Wu, Y., Zhang, X., & Ren, H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8511-7_23 · Cited by 1
CAML Model for Computational Linguistics
Chaudhuri, P. P., Dutta, A., Choudhury, S. P., Chowdhury, D. R., & Hazari, R. (2025) · Springer eBooks · doi:10.1007/978-981-96-1501-8_6 · Cited by 1
Let the Model Decide its Curriculum for Multitask Learning
Varshney, N., & Baral, S. M. A. C. (2022) · Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo) · doi:10.18653/v1/2022.deeplo-1.13 · Cited by 5
LINGO : Visually Debiasing Natural Language Instructions to Support Task Diversity
Arunkumar, A., Sharma, S., Agrawal, R., Chandrasekaran, S., & Bryan, C. (2023) · Computer Graphics Forum · doi:10.1111/cgf.14840 · Cited by 1
van der Goot, R. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.32 · Cited by 1
TaskWeb: Selecting Better Source Tasks for Multi-task NLP
Kim, J., Asai, A., Ilharco, G., & Hajishirzi, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.680 · Cited by 5
Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems ResearchPreprint
Gruetzemacher, R., & Paradice, D. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3944942 · Cited by 1
Wu, B., Razuvayevskaya, O., Heppell, F., Leite, J. A., Scarton, C., Bontcheva, K., & Song, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09421 · Cited by 1
When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLPPreprint
Ni, J., Jin, Z., Wang, Q., Sachan, M., & Leippold, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4456653 · Cited by 3
Enriching large language models with semantic lexicons and analogies
Zervakis, G. (2023) · theses.fr (ABES) · doi:10.70675/4f4cd0e5z5987z422az813bzc6e431d1a918 · Cited by 4
Cross-lingual Intermediate Fine-tuning improves Dialogue State TrackingPreprint
Moghe, N., Steedman, M. J., & Birch, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.13620
Building an Integrated Framework of Korean Text Summarization and Text-to-Speech
Kim, T., Kim, J., Kang, H., Kim, S., & Kang, P. (2022) · Journal of Korean Institute of Industrial Engineers · doi:10.7232/jkiie.2022.48.1.080
ConDef: Automated Context-Aware Lexicography Using Large Online Encyclopedias
Liu, H., & Sayyah, Z. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-10464-0_41
LITE: Intent-based Task Representation Learning Using Weak Supervision
Otani, N., Gamon, M., Jauhar, S. K., Yang, M., Malireddi, S. R., & Riva, O. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.172
Condon, D. M., & Cutler, A. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ywd87
Two-Level Classifier for Detecting Categories of Offensive Language
Aroyehun, S. T., Gelbukh, A. F., & Sidorov, G. (2023) · Studies in fuzziness and soft computing · doi:10.1007/978-3-031-23476-7_20
Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction
Yang, Y., Panagopoulou, A., Apidianaki, M., Yatskar, M., & Callison-Burch, C. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.45
Meta-training with Demonstration Retrieval for Efficient Few-shot Learning
Mueller, A., Narang, K., Mathias, L., Wang, Q., & Firooz, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.376 · Cited by 2
On the Compositional Generalization in Versatile Open-domain Dialogue
Fu, T., Zhao, X., Liu, L., & Yan, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.760
Pre-trained Models for Representation Learning
Lin, Y., Ding, N., Liu, Z., & Sun, M. (2023) · Springer eBooks · doi:10.1007/978-981-99-1600-9_5 · Cited by 2
BASiP: A Novel Architecture for Abstractive Text Summarization
Das, D., Madaan, J., Chanda, R., Gossain, R., Saha, T., & Bhattacharya, S. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-99-6755-1_14
Continual Dialogue State Tracking via Example-Guided Question Answering
Cho, H., Madotto, A., Lin, Z., Chandu, K. R., Kottur, S., Xu, J., May, J., & Sankar, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.235 · Cited by 2
Few-shot Unified Question Answering: Tuning Models or Prompts?
Bansal, S., Yavuz, S., Pang, B., Bhat, M., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.550 · Cited by 3
Generate to Understand for Representation in One Pre-training Stage
Xue, C., Zhong, X., & Liu, X. (2023) · International Conference on Information Systems and Computing Technology (ISCTech) · doi:10.1109/isctech60480.2023.00054
Lightweight Patch-Wise Casformer for dynamic scene deblurring
Chen, Z., Cui, G., Li, Z., & Zhao, J. (2024) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2024.104112
Leveraging Unannotated Data to Improve Zero-Shot Question Answering in the French Legal Domain
Touila, A., Louis, E., & Gharbi, H. (2023) · IEEE/ACIS International Conference on Big Data, Cloud Computing, and Data Science (BCD) · doi:10.1109/bcd57833.2023.10466348
Sotudeh, S., & Goharian, N. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3652505
Collaborative Multi-Task Representation for Natural Language Understanding
Yang, Y., Cao, D., Zeng, M., Yu, J. X., Ren, X., Tong, Y., & Wang, Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650257
Improving Zero-Shot Image Captioning Efficiency with Metropolis-Hastings
Du, D., & Wu, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-8511-7_22
Enhancing logical reasoning in language models: An investigation of the Capybara dataset
Guerrero, L. E. M., Ceballos, Y. F., & Rojas, L. D. T. (2025) · Contemporary Educational Technology · doi:10.30935/cedtech/16425
Analysis of Sampling Strategies for Multi-Task Learning in Transformer Models
Lakhotia, A., Kamalesh, A., Shantharaju, N. H., Kulkarni, P. S., & Srinivasa, G. (2025) · doi:10.1109/iceeng64546.2025.11031295
Chen, C.-C., & Takamura, H. (2025) · SpringerBriefs in intelligent systems · doi:10.1007/978-3-031-94687-5_3
A Domain-Adaptive Parameter-Efficient Fine-Tuning Method for Multi-task Learning
Liu, W., Fu, M., Cheng, S., Zhang, X., & Qu, H. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-94892-3_54
Compact and efficient language modeling for classical poetry: generation and interpretation
Cao, D., Cheng, C., & Zhang, G. (2025) · Multimedia Systems · doi:10.1007/s00530-025-01988-y · Cited by 1
Rethinking Data Use in Large Language Models
Min, S., Hajishirzi, H., & Zettlemoyer, L. (2025) · Computational Linguistics · doi:10.1162/coli.a.573
Self-citationJoshi, A., & Potika, K. (2025) · IEEE International Symposium on Technology and Society (ISTAS) · doi:10.1109/istas65609.2025.11269631
Jin, J., Gao, X., Fu, X., Qin, Z., Cheng, T., & Yan, R. (2026) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2026.105537
ET-FS: Functional Specialization Method for Multi-Task Learning in Encrypted Traffic Classification
Feng, X., Gou, G., Liu, C., Gu, Z., Dong, W., He, F., & Wang, X. (2025) · IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) · doi:10.1109/trustcom66490.2025.00088
Pegasus-copynet: a novel summarization generation framework for scientific and technological texts
Wang, S., Wang, H., Wang, X., Jiang, Y., Sun, S., Pan, X., & Ren, P. (2026) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02986-2
Tahir, N., & Ježek, K. (2025) · International Conference on Electrical, Computer and Energy Technologies (ICECET) · doi:10.1109/icecet63943.2025.11472435
Data and Knowledge Dual-Driven Text Embedding Approaches: A Survey
Chen, X., 詹佳青, Lin, J., Liu, H., Zhang, Q., Chen, J., & Chen, H. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3721883
Self-citationConstruction Method for Instruction Tuning Dataset for Domain-specific Entity Extraction Tasks
Kang, C., & Shen, C. (2026) · International Conference on Computational Intelligence and Machine Learning (CIML) · doi:10.1109/ciml69155.2026.11692762
BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing
Fries, J. A., Weber, L., Seelam, N., Altay, G., Datta, D., Garda, S., Kang, M., Su, R., Kusa, W., Cahyawijaya, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.15076 · Cited by 17
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations
Singh, A., D'Arcy, M., Cohan, A., Downey, D., & Feldman, S. (2022) · arXiv · doi:10.48550/arxiv.2211.13308 · Cited by 76
“We train our models in a multi-task setup with task-heterogeneous batching (Aghajanyan et al., 2021).”
Influential
Metadata
- Authors
- Aghajanyan, A., Gupta, A., Shrivastava, A., Chen, X., Zettlemoyer, L., & Gupta, S.
- Venue
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Published
- 2021
- Pages
- 5799-5811
- OpenAlex
- W3124687886
- Cited by
- 183
- References
- 82
- Citations by year
- 2026: 7 · 2025: 21 · 2024: 23 · 2023: 64 · 2022: 55 · 2021: 13
- FWCI
- 17.90
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- multi-task learning, representation learning, language model pretraining, fine-tuning, sample efficiency, RoBERTa, BART, commonsense reasoning, machine reading comprehension, sentence prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
