The DOI matches its Crossref record.
Message Understanding Conference-6: a brief history
Open accessAbstract
We have recently completed the sixth in a series of "Message Understanding Conferences" which are designed to promote and evaluate research in information extraction. MUC-6 introduced several innovations over prior MUCs, most notably in the range of different tasks for which evaluations were conducted. We describe some of the motivations for the new format and briefly discuss some of the results of the evaluations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Still being checked.
Cited by
Sorted by citations
A survey of named entity recognition and classification
Nadeau, D., & Sekine, S. (2007) · Lingvisticae Investigationes · doi:10.1075/li.30.1.03nad · Cited by 2,507
A Survey on Deep Learning for Named Entity Recognition
Li, J., Sun, A., Han, J., & Li, C. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2981314 · Cited by 1,801
2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text
Uzuner, Ö., South, B. R., Shen, S., & DuVall, S. L. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000203 · Cited by 1,292
Nasukawa, T., & Yi, J. (2003) · International Conference on Knowledge Capture · doi:10.1145/945645.945658 · Cited by 1,167
Large-Scale Named Entity Disambiguation Based on Wikipedia Data
Cucerzan, S. (2007) · Cited by 1,061
A shortest path dependency kernel for relation extraction
Bunescu, R., & Mooney, R. J. (2005) · doi:10.3115/1220575.1220666 · Cited by 971
Sarawagi, S. (2008) · Foundations and Trends in Databases · doi:10.1561/1900000003 · Cited by 731
GATE, a General Architecture for Text Engineering
Cunningham, H. (2002) · Computers and the Humanities · doi:10.1023/a:1014348124664 · Cited by 666
Aggarwal, C. C. (2015) · Springer eBooks · doi:10.1007/978-3-319-14142-8_13 · Cited by 625
Open domain event extraction from twitter
Ritter, A., Mausam, M., Etzioni, O., & Clark, S. (2012) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2339530.2339704 · Cited by 607
Active Hidden Markov Models for Information Extraction
Scheffer, T., Decomain, C., & Wrobel, S. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44816-0_31 · Cited by 576
Arabic Natural Language Processing: Challenges and Solutions
Farghaly, A., & Shaalan, K. (2009) · ACM Transactions on Asian Language Information Processing · doi:10.1145/1644879.1644881 · Cited by 557
A Survey on Recent Advances in Named Entity Recognition from Deep Learning modelsPreprint
Yadav, V., & Bethard, S. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.11470 · Cited by 612
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480
Extracting medication information from clinical text
Uzuner, Ö., Solti, I., & Cadag, E. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.003947 · Cited by 466
Evaluating the State-of-the-Art in Automatic De-identification
Uzuner, Ö., Luo, Y., & Szolovits, P. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2444 · Cited by 486
Test Collection Based Evaluation of Information Retrieval Systems
Sanderson, M. (2010) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000009 · Cited by 485
Information extraction: Techniques and challenges
Grishman, R. (1997) · Lecture notes in computer science · doi:10.1007/3-540-63438-x_2 · Cited by 437
Self-citationEvent Extraction by Answering (Almost) Natural Questions
Du, X., & Cardie, C. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.49 · Cited by 459
Will, G. K., King, G., Lowe, W., Abbott, T. T. M., Bond, D., Bond, J., Bradford, G., Cobb, C., Pevehouse, J., Quinn, K., et al. (2003) · Cited by 401
Identifying Patient Smoking Status from Medical Discharge Records
Uzuner, Ö., Goldstein, I., Luo, Y., & Kohane, I. S. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2408 · Cited by 359
Analysis of named entity recognition and linking for tweets
Derczynski, L., Maynard, D., Rizzo, G. F., van Erp, M., Gorrell, G., Raphaël Troncy, A., Petrak, J., & Bontcheva, K. (2014) · Information Processing & Management · doi:10.1016/j.ipm.2014.10.006 · Cited by 353
Recent Named Entity Recognition and Classification techniques: A systematic review
Goyal, A., Gupta, V., & Kumar, M. (2018) · Computer Science Review · doi:10.1016/j.cosrev.2018.06.001 · Cited by 333
A Unified Generative Framework for Various NER Subtasks
Yan, H., Gui, T., Dai, J., Guo, Q., Zhang, Z., & Qiu, X. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.451 · Cited by 299
“The term “Named Entity” was coined in the Sixth Message Understanding Conference (MUC-6) (Gr-ishman and Sundheim, 1996).”
BackgroundChapman, W. W., Nadkarni, P. M., Hirschman, L., D'Avolio, L. W., Savova, G. K., & Uzuner, O. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000465 · Cited by 296
Information retrieval and text mining technologies for chemistry
Krallinger, M., Rabal, O., Lourenço, A., Oyarzabal, J., & Valencia, A. (2017) · Chemical Reviews · doi:10.1021/acs.chemrev.6b00851 · Cited by 281
Impact of word embedding models on text analytics in deep learning environment: a review
Asudani, D. S., Nagwani, N. K., & Singh, P. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10419-1 · Cited by 274
Named Entity Recognition: Fallacies, challenges and opportunities
Marrero, M., Urbano, J., Sánchez-Cuadrado, S., Morato, J., & Gómez-Berbís, J. M. (2012) · Computer Standards & Interfaces · doi:10.1016/j.csi.2012.09.004 · Cited by 243
Recognizing Obesity and Comorbidities in Sparse Data
Uzuner, Ö. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3115 · Cited by 238
Named Entity Extraction for Knowledge Graphs: A Literature Overview
Al-Moslmi, T., Ocaña, M. G., Opdahl, A. L., & Veres, C. (2020) · IEEE Access · doi:10.1109/access.2020.2973928 · Cited by 258
Text Mining for Big Data Analysis in Financial Sector: A Literature Review
Bach, M. P., Krstić, Ž., Seljan, S., & Turulja, L. (2019) · Sustainability · doi:10.3390/su11051277 · Cited by 235
Mind the GAP: A balanced corpus of gendered ambiguous pronouns
Webster, K., Recasens, M., Axelrod, V., & Baldridge, J. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00240 · Cited by 223
Information extraction: beyond document retrieval
Gaizauskas, R., & Wilks, Y. A. (1998) · Journal of Documentation · doi:10.1108/eum0000000007162 · Cited by 219
Information Extraction: Past, Present and Future
Jakub Piskorski, B., & Yangarber, R. (2012) · Theory and applications of natural language processing · doi:10.1007/978-3-642-28569-1_2 · Cited by 200
Computational methods for Traditional Chinese Medicine: A survey
Lukman, S., He, Y., & Hui, S.-C. (2007) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2007.09.008 · Cited by 196
A Review of Relation Extraction
Yang, Y., Xun, H., & You, H. (2013) · Shuju fenxi yu zhishi faxian · doi:10.11925/infotech.1003-3513.2013.11.05 · Cited by 193
Named Entity Recognition and Classification in Historical Documents: A Survey
Ehrmann, M., Hamdi, A., Pontes, E. L., Romanello, M., & Doucet, A. (2023) · ACM Computing Surveys · doi:10.1145/3604931 · Cited by 285
Automatic paraphrase acquisition from news articles
Shinyama, Y., Sekine, S., & Sudo, K. (2002) · International Conference on Human Language Technology Research · doi:10.3115/1289189.1289218 · Cited by 188
Language, Culture, and Society
Deumert, A. (2013) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199585847.013.0030 · Cited by 184
The Oxford Handbook of the History of Linguistics
(2013) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199585847.001.0001 · Cited by 175
Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and ApproachesPreprint
Storks, S., Gao, Q., & Chai, J. Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01172 · Cited by 280
Definition, Dictionaries and Tagger for Extended Named Entity Hierarchy
Sekine, S., & Nobata, C. (2004) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3zc57kdyb4p6 · Cited by 169
Information Extraction from Text
Jiang, J. (2012) · Springer eBooks · doi:10.1007/978-1-4614-3223-4_2 · Cited by 168
A Replicable Comparison Study of NER Software: StanfordNLP, NLTK, OpenNLP, SpaCy, Gate
Schmitt, X., Sylvain Kubler, A., Robert, J., Papadakis, M., & Traon, Y. L. (2019) · International Conference on Social Networks Analysis, Management and Security (SNAMS) · doi:10.1109/snams.2019.8931850 · Cited by 162
MAVEN: A Massive General Domain Event Detection Dataset
Wang, X., Wang, Z., Han, X., Jiang, W., Han, R., Liu, Z., Li, J., Li, P., Lin, Y., & Zhou, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.129 · Cited by 180
Locating complex named entities in web text
Downey, D., Broadhead, M., & Etzioni, O. (2007) · Cited by 157
Itkonen, E. (2013) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199585847.013.0035 · Cited by 155
Named Entity Recognition in Chinese Clinical Text Using Deep Neural Network
Wu, Y., Min, J., Jianbo, L., & Xu, H. (2015) · Studies in health technology and informatics · doi:10.3233/978-1-61499-564-7-624 · Cited by 148
SystemT: An Algebraic Approach to Declarative Information Extraction
Chiticariu, L., Krishnamurthy, R., Li, Y., Raghavan, S., Reiss, F. R., & Vaithyanathan, S. (2010) · Cited by 146
Information Extraction, Automatic
Cunningham, H. (2006) · Elsevier eBooks · doi:10.1016/b0-08-044854-2/00960-3 · Cited by 146
SUMMAC: a text summarization evaluation
Mani, I., Klein, G. L., HOUSE, D., Hirschman, L., Firmin, T., & Sundheim, B. M. (2002) · Natural Language Engineering · doi:10.1017/s1351324901002741 · Cited by 143
Self-citationOn-demand information extraction
Sekine, S. (2006) · International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics (COLING/ACL) · doi:10.3115/1273073.1273167 · Cited by 140
Developing a corpus of plagiarised short answers
Clough, P. D., & Stevenson, M. (2010) · Language Resources and Evaluation · doi:10.1007/s10579-009-9112-1 · Cited by 139
CM-Builder: A Natural Language-Based CASE Tool for Object-Oriented Analysis
Harmain, H. M., & Gaizauskas, R. (2003) · Automated Software Engineering · doi:10.1023/a:1022916028950 · Cited by 139
Mixed-initiative development of language processing systems
Day, D. S., Aberdeen, J. S., Hirschman, L., Kozierok, R., Robinson, P., & Vilain, M. (1997) · Conference on Applied Natural Language Processing · doi:10.3115/974557.974608 · Cited by 139
A survey on Named Entity Recognition — datasets, tools, and methodologies
Jehangir, B., Radhakrishnan, S., & Agarwal, R. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100017 · Cited by 178
Liberal Event Extraction and Event Schema Induction
Huang, L., Cassidy, T., Feng, X., Ji, H., Voss, C. R., Han, J., & Sil, A. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1025 · Cited by 135
Fine-Grained Named Entity Recognition in Legal Documents
Leitner, E., Rehm, G., & Schneider, J. M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-33220-4_20 · Cited by 142
Weakly Supervised Extraction of Computer Security Events from Twitter
Ritter, A., Wright, E., Casey, W., & Mitchell, T. M. (2015) · International World Wide Web Conference (WWW) · doi:10.1145/2736277.2741083 · Cited by 137
Cunningham, H., Humphreys, K., Gaizauskas, R., & Wilks, Y. A. (1997) · Conference on Applied Natural Language Processing · doi:10.3115/974281.974299 · Cited by 135
Mining opinion components from unstructured reviews: A review
Khan, K., Baharudin, B., Khan, A., & Ullah, A. (2014) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2014.03.009 · Cited by 131
Extraction of Adverse Drug Effects from Clinical Records
Aramaki, E., Miura, Y., Tonoike, M., Ohkuma, T., Masuichi, H., Waki, K., & Ohe, K. (2010) · Studies in health technology and informatics · doi:10.3233/978-1-60750-588-4-739 · Cited by 130
Metrics for Evaluation of Ontology-based Information Extraction
Maynard, D., Peters, W., & Li, Y. (2006) · Cited by 128
Hirst, G. (2013) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199585847.013.0033 · Cited by 127
Biomedical named entity recognition using deep neural networks with contextual information
Cho, H., & Lee, H. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3321-4 · Cited by 129
Tsai, R. T., Sung, C.-L., Dai, H.-J., Hung, H.-C., Sung, T.-Y., & Hsu, W.-L. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-s5-s11 · Cited by 127
Generalisation in named entity recognition: A quantitative analysis
Augenstein, I., Derczynski, L., & Bontcheva, K. (2017) · Computer Speech & Language · doi:10.1016/j.csl.2017.01.012 · Cited by 125
Datasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 268
CASIE: Extracting Cybersecurity Event Information from Text
Satyapanich, T., Ferraro, F., & Finin, T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6401 · Cited by 128
A Survey on Recent Named Entity Recognition and Relationship Extraction Techniques on Clinical Texts
Bose, P., Srinivasan, S., Sleeman, W. C., Palta, J. R., Kapoor, R., & Ghosh, P. (2021) · Applied Sciences · doi:10.3390/app11188319 · Cited by 130
Landolsi, M., Hlaoua, L., & Romdhane, L. B. (2022) · Knowledge and Information Systems · doi:10.1007/s10115-022-01779-1 · Cited by 127
Knowledge Graphs: A Practical Review of the Research Landscape
Kejriwal, M. (2022) · Information · doi:10.3390/info13040161 · Cited by 131
A statistical information extraction system for Turkish
Tür, G., Hakkani-Tür, D. Z., & Oflazer, K. (2003) · Natural Language Engineering · doi:10.1017/s135132490200284x · Cited by 111
Overview of the BioCreative III Workshop
Arighi, C. N., Lu, Z., Krallinger, M., Cohen, K. B., Wilbur, W. J., Valencia, A., Hirschman, L., & Wu, C. H. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s8-s1 · Cited by 110
Information extraction for enhanced access to disease outbreak reports
Grishman, R., Huttunen, S., & Yangarber, R. (2002) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(03)00013-3 · Cited by 108
Self-citationKLUE: Korean Language Understanding EvaluationPreprint
Park, S., Moon, J., Kim, S., Cho, W. I., Han, J., Park, J., Song, C., Kim, J., Song, Y., Oh, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.09680 · Cited by 193
Word-Based Morphology from Aristotle to Modern WP (Word and Paradigm Models)
Blevins, J. P. (2013) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199585847.013.0017 · Cited by 109
Artificial intelligence: revolutionizing cardiology with large language models
Boonstra, M., Weissenbacher, D., Moore, J. H., Gonzalez‐Hernandez, G., & Asselbergs, F. W. (2023) · European Heart Journal · doi:10.1093/eurheartj/ehad838 · Cited by 115
Ontology-based information extraction and integration from heterogeneous data sources
Paul Buitelaar, C., Cimiano, P., Frank, A., Härtung, M., & Racioppa, S. (2008) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2008.07.007 · Cited by 104
Named Entity Recognition and Resolution in Legal Text
Dozier, C., Kondadadi, R., Light, M., Vachher, A., Veeramachaneni, S., & Wudali, R. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-12837-0_2 · Cited by 102
Aiding Pronoun Translation with Co-Reference Resolution
Nagard, R. L., & Koehn, P. (2010) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 100
A factuality profiler for eventualities in text
Pustejovsky, J. D., & Sauri, R. S. (2008) · Cited by 100
Event graphs for information retrieval and multi-document summarization
Glavaš, G., & Šnajder, J. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2014.04.004 · Cited by 100
Finite-state transducer cascades to extract named entities in texts
Friburger, N., & Maurel, D. (2003) · Theoretical Computer Science · doi:10.1016/j.tcs.2003.10.007 · Cited by 93
Fagin, R., Kimelfeld, B., Reiss, F., & Vansummeren, S. (2015) · Journal of the ACM · doi:10.1145/2699442 · Cited by 61
Louvan, S., & Magnini, B. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.42 · Cited by 106
An information extraction core system for real world German text processing
Neumann, G., Backofen, R., Baur, J., Becker, M., & Braun, C. (1997) · Conference on Applied Natural Language Processing · doi:10.3115/974557.974588 · Cited by 91
An Attentive Neural Architecture for Fine-grained Entity Type Classification
Shimaoka, S., Stenetorp, P., Inui, K., & Riedel, S. (2016) · Automated Knowledge Base Construction · doi:10.18653/v1/w16-1313 · Cited by 95
Nested Named Entity Recognition: A Survey
Wang, Y., Tong, H., Zhu, Z., & Li, Y. (2022) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3522593 · Cited by 103
Han, X., Gao, T., Lin, Y., Peng, H. M., Yang, Y., Xiao, C., Liu, Z., Li, P., Zhou, J., & Sun, M. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.75 · Cited by 102
Evaluating and Combining Name Entity Recognition Systems
Jiang, R., Banchs, R. E., & Li, H. (2016) · Named Entities Workshop (NEWS) · doi:10.18653/v1/w16-2703 · Cited by 89
Software Architecture for Language Engineering
Cunningham, H., & Scott, D. (2004) · Natural Language Engineering · doi:10.1017/s1351324904003481 · Cited by 85
Surdeanu, M. (2013) · Cited by 83
An Overview of Named Entity Recognition
Sun, P., Yang, X., Zhao, X., & Wang, Z. (2018) · International Conference on Asian Language Processing · doi:10.1109/ialp.2018.8629225 · Cited by 92
What is Event Knowledge Graph: A Survey
Guan, S., Cheng, X., Bai, L., Zhang, F., Li, Z., Zeng, Y., Jin, X., & Guo, J. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3180362 · Cited by 86
Malay Named Entity Recognition Based on Rule-Based Approach
Alfred, R., Leong, L. C., On, C. K., & Anthony, P. (2014) · International Journal of Machine Learning and Computing · doi:10.7763/ijmlc.2014.v4.428 · Cited by 82
CM-Builder: an automated NL-based CASE tool
Harmain, H. M., & Gaizauskas, R. (2000) · doi:10.1109/ase.2000.873649 · Cited by 82
Event coreference for information extraction
Humphreys, K., Gaizauskas, R., & Azzam, S. (1997) · doi:10.3115/1598819.1598830 · Cited by 81
Computational Modeling of Narrative
Mani, I. (2013) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-02147-3 · Cited by 80
Nadeau, D. (2007) · doi:10.20381/ruor-19854 · Cited by 78
Metadata
- Authors
- Grishman, R., & Sundheim, B.
- Affiliations
- New York University (US)
- Venue
- Proceedings of the 16th conference on Computational linguistics -
- Published
- 1996
- Volume
- 1
- Pages
- 466
- ISSN
- 1525-2477 (ISSN-L)
- OpenAlex
- W2068882115
- Cited by
- 1,408
- References
- 0
- Citations by year
- 2026: 26 · 2025: 37 · 2024: 62 · 2023: 74 · 2022: 64 · 2021: 81 · 2020: 64 · 2019: 73 · 2018: 68 · 2017: 50 · 2016: 61 · 2015: 71 · 2014: 65 · 2013: 117 · 2012: 83
- Type
- conference abstract
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Conference on Computational Linguistics (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- MUC6, information extraction, benchmark tasks, natural language processing evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
