The DOI matches its Crossref record.
Message Understanding Conference-6: a brief history
Open accessAbstract
We have recently completed the sixth in a series of "Message Understanding Conferences" which are designed to promote and evaluate research in information extraction. MUC-6 introduced several innovations over prior MUCs, most notably in the range of different tasks for which evaluations were conducted. We describe some of the motivations for the new format and briefly discuss some of the results of the evaluations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Still being checked.
Cited by
Sorted by citations
Items 101–200 of 1,000
Luo, Y., Uzuner, Ö., & Szolovits, P. (2016) · Briefings in Bioinformatics · doi:10.1093/bib/bbw001 · Cited by 80
Handbook of Research on Text and Web Mining Technologies
Song, M., & Wu, Y.-F. B. (2009) · IGI Global eBooks · doi:10.4018/978-1-59904-990-8 · Cited by 79
A search result clustering method using informatively named entities
Toda, H., & Kataoka, R. (2005) · ACM International Workshop on Web Information and Data Management (WIDM) · doi:10.1145/1097047.1097063 · Cited by 78
A software architecture for Twitter collection, search and geolocation services
Oussalah, M., Bhat, F., Challis, K., & Schnier, T. (2012) · Knowledge-Based Systems · doi:10.1016/j.knosys.2012.07.017 · Cited by 77
Exploring entity recognition and disambiguation for cultural heritage collections
van Hooland, S., Wilde, M. D., Verborgh, R., Steiner, T., & Rik Van De Walle, B. (2013) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqt067 · Cited by 77
Vossen, P. T. J. M., Agerri, R., Aldabe, I., Cybulska, A., van Erp, M., Fokkens, A. S., Laparra, E., Minard, A.-L., Aprosio, A. P., Rigau, G., et al. (2016) · Knowledge-Based Systems · doi:10.1016/j.knosys.2016.07.013 · Cited by 80
Large-Scale Learning of Relation-Extraction Rules with Distant Supervision from the Web
Krause, S., Li, H., Uszkoreit, H., & Xu, F. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35176-1_17 · Cited by 75
Discourse in Statistical Machine Translation
Hardmeier, C. (2012) · Discours · doi:10.4000/discours.8726 · Cited by 118
BoB, a best-of-breed automated text de-identification system for VHA clinical documents
Ferrández, Ó., South, B. R., Shen, S., Friedlin, J., Samore, M. H., & Meystre, S. M. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001020 · Cited by 73
HAREM: An Advanced NER Evaluation Contest for Portuguese
Santos, D., Seco, N., Cardoso, N., & Vilela, R. (2006) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/233zhxzy8zqn · Cited by 75
Magnus, M. (2013) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199585847.013.0010 · Cited by 73
Frisoni, G., Moro, G., & Carbonaro, A. (2021) · IEEE Access · doi:10.1109/access.2021.3130956 · Cited by 76
Understanding script-based stories using commonsense reasoning
Mueller, E. T. (2004) · Cognitive Systems Research · doi:10.1016/j.cogsys.2004.06.001 · Cited by 72
NERD: evaluating named entity recognition tools in the web of data
Rizzo, G. F., & Raphaël Troncy, A. (2011) · PORTO@Iris · Cited by 71
Kopliku, A., Pinel-Sauvagnat, K., & Boughanem, M. (2014) · ACM Computing Surveys · doi:10.1145/2523817 · Cited by 72
Software infrastructure for natural language processing
Cunningham, H., Humphreys, K., Gaizauskas, R., & Wilks, Y. A. (1997) · Conference on Applied Natural Language Processing · doi:10.3115/974557.974592 · Cited by 71
Adapting Open Information Extraction to Domain‐Specific Relations
Soderland, S. G., Roof, B., Qin, B., Xu, S., Mausam, M., & Etzioni, O. (2010) · AI Magazine · doi:10.1609/aimag.v31i3.2305 · Cited by 69
Information extraction from case law and retrieval of prior cases
Jackson, P. E., Al-Kofahi, K., Tyrrell, A., & Vachher, A. (2003) · Artificial Intelligence · doi:10.1016/s0004-3702(03)00106-1 · Cited by 67
Proceedings of the Australasian Language Technology Workshop 2005
Baldwin, T., Curran, J., & van Zaanen, M. M. (2005) · Cited by 66
Chung, K. S. (2013) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199585847.013.0011 · Cited by 65
Model-portability experiments for textual temporal analysis
Kolomiyets, O., Bethard, S. J., & Moens, M. (2011) · Lirias · Cited by 65
A Finnish news corpus for named entity recognition
Ruokolainen, T., Kauppinen, P., Silfverberg, M., & Lindén, K. (2019) · Language Resources and Evaluation · doi:10.1007/s10579-019-09471-7 · Cited by 71
A Novel Document-Level Relation Extraction Method Based on BERT and Entity Information
Han, X., & Wang, L. (2020) · IEEE Access · doi:10.1109/access.2020.2996642 · Cited by 65
Challenges in applying large language models to requirements engineering tasks
Norheim, J. J., Rebentisch, E., Xiao, D. Q., Draeger, L., Kerbrat, A., & Weck, O. L. D. (2024) · Design Science · doi:10.1017/dsj.2024.8 · Cited by 86
Graph-Based Named Entity Linking with Wikipedia
Hachey, B., Radford, W., & Curran, J. R. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24434-6_16 · Cited by 63
WikiNEuRal: Combined Neural and Knowledge-based Silver Data Creation for Multilingual NER
Tedeschi, S., Maiorca, V., Campolungo, N., Cecconi, F., & Roberto Navigli, A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.215 · Cited by 63
Generative Event Schema Induction with Entity Disambiguation
Nguyen, K.-H., Tannier, X., Ferret, O., & Besançon, R. (2015) · doi:10.3115/v1/p15-1019 · Cited by 61
Abdullah, M. H. A., Aziz, N., Abdulkadir, S. J., Alhussian, H., & Talpur, N. (2023) · IEEE Access · doi:10.1109/access.2023.3240898 · Cited by 67
Relation extraction for inferring access control rules from natural language artifacts
Slankas, J., Xiao, X., Williams, L. A., & Xie, T. (2014) · Annual Computer Security Applications Conference · doi:10.1145/2664243.2664280 · Cited by 60
Cerno: Light-weight tool support for semantic annotation of textual documents
Kiyavitskaya, N., Zeni, N., Cordy, J. R., Mich, L., & Mylopoulos, J. (2009) · Data & Knowledge Engineering · doi:10.1016/j.datak.2009.07.012 · Cited by 59
Liu, J., Chen, Y., & Xu, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.214 · Cited by 61
Zaghouani, W. (2012) · ACM Transactions on Asian Language Information Processing · doi:10.1145/2090176.2090178 · Cited by 58
Han, Y., & Moghaddam, M. (2020) · Journal of Mechanical Design · doi:10.1115/1.4048819 · Cited by 57
Ahmad, P. N., Shah, A. M., & Lee, K. Y. (2023) · Healthcare · doi:10.3390/healthcare11091268 · Cited by 59
NERD meets NIF: Lifting NLP Extraction Results to the Linked Data Cloud
Rizzo, G. F., Raphaël Troncy, A., Hellmann, S., & Bruemmer, M. (2012) · PORTO@Iris · Cited by 56
Mi, B., & Fan, Y. H. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10197-2 · Cited by 58
Augmenting Wikipedia with Named Entity Tags.
Dakka, W., & Cucerzan, S. (2008) · Cited by 56
Automatic reconstruction of a bacterial regulatory network using Natural Language Processing
Rodriguez-Penagos, C. G., Salgado, H., Martínez‐Flores, I., & Collado‐Vides, J. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-293 · Cited by 55
Dalianis, H., & Velupillai, S. U. (2010) · Journal of Biomedical Semantics · doi:10.1186/2041-1480-1-6 · Cited by 55
Moving beyond the Turing Test with the Allen AI Science Challenge
Schoenick, C., Clark, P., Tafjord, O., Turney, P., & Etzioni, O. (2017) · Communications of the ACM · doi:10.1145/3122814 · Cited by 54
A survey on semantic processing techniques
Mao, R., He, K., Zhang, X., Chen, G., Ni, J., Yang, Z., & Cambria, E. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101988 · Cited by 55
Named Entity Recognition Experiments on Turkish Texts
Küçük, D., & Yazıcı, A. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04957-6_45 · Cited by 54
Latent semantics in Named Entity Recognition
Konkol, M., Brychcín, T., & Konopík, M. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2014.12.015 · Cited by 58
Annotating Large Email Datasets for Named Entity Recognition with Mechanical Turk
Lawson, N., Eustice, K., Perkowitz, M., & Yetisgen-Yildiz, M. (2010) · Cited by 53
Logarithmic opinion pools for conditional random fields
Smith, A., Cohn, T., & Osborne, M. (2005) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1219840.1219843 · Cited by 53
Llanos, L. C., Deléger, L., Grouin, C., Thierry Hamon, C., Ligozat, A., & Aurélie Névéol, A. (2017) · Language Resources and Evaluation · doi:10.1007/s10579-017-9382-y · Cited by 53
A Shallow Text Processing Core Engine
Neumann, G., & Jakub Piskorski, B. (2002) · Computational Intelligence · doi:10.1111/0824-7935.00197 · Cited by 53
Modeling Noisiness to Recognize Named Entities using Multitask Neural Networks on Social Media
Aguilar, G., López-Monroy, A. P., Gonzalez, F. A., & Solorio, T. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1127 · Cited by 52
Second HAREM: Advancing the State of the Art of Named Entity Recognition in Portuguese
Freitas, C., Mota, C., Santos, D., Oliveira, H. G., & Carvalho, P. (2010) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/56zvugg5kwyo · Cited by 52
Extended Named Entity Ontology with Attribute Information
Sekine, S. (2008) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/56bcet4s8aza · Cited by 52
A Survey on Relation Extraction
Cui, M., Li, L., Wang, Z., & You, M. (2017) · Communications in computer and information science · doi:10.1007/978-981-10-7359-5_6 · Cited by 52
The Origins and the Evolution of Language
Mufwene, S. S. (2013) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199585847.013.0002 · Cited by 53
Unsupervised Approaches for Textual Semantic Annotation, A Survey
Liao, X., & Zhao, Z. (2019) · ACM Computing Surveys · doi:10.1145/3324473 · Cited by 54
A hybrid named entity recognizer for Turkish
Küçük, D., & Yazıcı, A. (2011) · Expert Systems with Applications · doi:10.1016/j.eswa.2011.08.131 · Cited by 51
Ferrández, Ó., South, B. R., Shen, S., Friedlin, J., Samore, M. H., & Meystre, S. M. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-109 · Cited by 51
Zero-Shot Open Entity Typing as Type-Compatible Grounding
Zhou, B., Khashabi, D., Tsai, C.-T., & Roth, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1231 · Cited by 58
Document-level relation extraction using evidence reasoning on RST-GRAPH
Wang, H., Qin, K., Lu, G., Yin, J., Zakari, R. Y., & Owusu, J. W. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107274 · Cited by 51
Textual Similarity based on Proper Names
Friburger, N., & Maurel, D. (2002) · Cited by 50
King, G., & Lowe, W. (2003) · International Organization · doi:10.1017/s0020818303573064 · Cited by 50
Open Entity Extraction from Web Search Query Logs
Jain, A., & Marco Pennacchiotti, B. (2010) · Cited by 50
A comparison of named entity recognition tools applied to biographical texts
Atdağ, S., & Labatut, V. (2013) · International Conference on Systems and Computer Science (ICSCS) · doi:10.1109/icconscs.2013.6632052 · Cited by 49
Named Entities in Czech: Annotating Data and Developing NE Tagger
Ševčíková, M., Žabokrtský, Z., & Krůza, O. (2007) · Springer eBooks · doi:10.1007/978-3-540-74628-7_26 · Cited by 49
Automatic extraction of citation contexts for research paper summarization
Kaplan, D., Iida, R., & Tokunaga, T. (2009) · doi:10.3115/1699750.1699764 · Cited by 49
ETER : a new metric for the evaluation of hierarchical named entity recognition
Jannet, M. A. B., Adda‐Decker, M., Galibert, O., Kahn, J., & Rosset, S. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5preynfutk8v · Cited by 48
Lexicography from Earliest Times to the Present
Hanks, P. (2013) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199585847.013.0023 · Cited by 50
APTNER: A Specific Dataset for NER Missions in Cyber Threat Intelligence Field
Wang, X., He, S., Xiong, Z., Wei, X., Jiang, Z., Chen, S., & Jiang, J. (2022) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd54268.2022.9776031 · Cited by 54
QasemiZadeh, B., & Handschuh, S. (2014) · doi:10.3115/v1/w14-4807 · Cited by 48
Urdu Named Entity Recognition and Classification System Using Artificial Neural Network
Malik, M. K. (2017) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3129290 · Cited by 48
Vernaculars and the Idea of a Standard Language
Linn, A. (2013) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199585847.013.0016 · Cited by 48
Veena, G., Kanjirangat, V., & Gupta, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120440 · Cited by 47
Review of Relation Extraction Methods: What Is New Out There?
Konstantinova, N. (2014) · Communications in computer and information science · doi:10.1007/978-3-319-12580-0_2 · Cited by 47
A survey of Named Entity Recognition in English and other Indian Languages
Gupta, V. (2010) · Cited by 47
Parsed use case descriptions as a basis for object-oriented class model generation
Elbendak, M., Vickers, P., & Rossiter, N. (2011) · Journal of Systems and Software · doi:10.1016/j.jss.2011.02.025 · Cited by 46
(2012)
Evaluation of Named Entity Extraction Systems
Marrero, M., Sánchez-Cuadrado, S., Morato, J., & Andreadakis, G. (2009) · Cited by 46
An Analysis of the Performance of Named Entity Recognition over OCRed Documents
Hamdi, A., Jean-Caurant, A., Sidère, N., Coustaty, M., & Doucet, A. (2019) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl.2019.00057 · Cited by 48
On the Declassification of Confidential Documents
Abril, D., Navarro‐Arribas, G., & Torra, V. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-22589-5_22 · Cited by 45
Jin, Y. (2014) · Acta Automatica Sinica · Cited by 45
A definition and short history of Language Engineering
Cunningham, H. (1999) · Natural Language Engineering · doi:10.1017/s1351324999002144 · Cited by 45
Tang, J., Hong, M., Zhang, D. L., & Li, J. (2008) · IGI Global eBooks · doi:10.4018/978-1-59904-373-9.ch001 · Cited by 45
Extracting Time Expressions and Named Entities with Constituent-Based Tagging Schemes
Zhong, X., Cambria, E., & Hussain, A. (2020) · Cognitive Computation · doi:10.1007/s12559-020-09714-8 · Cited by 45
De-identification of clinical notes in French: towards a protocol for reference corpus development
Grouin, C., & Aurélie Névéol, A. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.12.014 · Cited by 44
Open Knowledge Extraction Challenge
Nuzzolese, A. G., Gentile, A. L., Presutti, V., Aldo Gangemi, A., Garigliotti, D., & Roberto Navigli, A. (2015) · Communications in computer and information science · doi:10.1007/978-3-319-25518-7_1 · Cited by 48
Emami, A., Trichelair, P., Trischler, A., Suleman, K., Schulz, H., & Cheung, J. C. K. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1386 · Cited by 47
Ren, R., & Zhang, J. (2021) · Journal of Computing in Civil Engineering · doi:10.1061/(asce)cp.1943-5487.0000971 · Cited by 43
Named Entity Recognition for Indian Languages
Nayan, A., Rao, B. R. K., Singh, P., Sanyal, S., & Sanyal, R. (2008) · Cited by 43
Modelling Space and Time in Narratives about Restaurants
Mueller, E. T. (2006) · Literary and Linguistic Computing · doi:10.1093/llc/fql014 · Cited by 39
Frame-Based Ontology Population with PIKES
Corcoglioniti, F., Rospocher, M., & Aprosio, A. P. (2016) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2016.2602206 · Cited by 42
Natural Language Processing for the Semantic Web
Maynard, D., Bontcheva, K., & Augenstein, I. (2017) · Synthesis lectures on data, semantics and knowledge · doi:10.1007/978-3-031-79474-2 · Cited by 42
Graph-based clustering of extracted paraphrases for labelling crime reports
Das, P., & Das, A. K. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.05.004 · Cited by 42
Seki, K., Fujii, A., & Ishikawa, T. (2002) · International Conference on Computational Linguistics · doi:10.3115/1072228.1072306 · Cited by 42
Named Entity Recognition in Natural Language Processing: A Systematic Review
Sharma, A. B., Amrita, A., Chakraborty, S., & Kumar, S. (2021) · Advances in intelligent systems and computing · doi:10.1007/978-981-16-3346-1_66 · Cited by 41
Knowledge graph theory and structural parsing
Zhang, L. (2002) · University of Twente Research Information · Cited by 41
AutoKG: Efficient Automated Knowledge Graph Generation for Language Models
Chen, B., & Bertozzi, A. L. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386454 · Cited by 55
Using topic themes for multi-document summarization
Harabagiu, S. M., & Lăcătuşu, F. (2010) · ACM Transactions on Information Systems · doi:10.1145/1777432.1777436 · Cited by 40
Enriching news events with meta-knowledge information
Thompson, P., Nawaz, R., McNaught, J., & Ananiadou, S. (2016) · Language Resources and Evaluation · doi:10.1007/s10579-016-9344-9 · Cited by 40
Ontology population: Approaches and design aspects
Lubani, M., Noah, S. A. M., & Mahmud, R. B. (2018) · Journal of Information Science · doi:10.1177/0165551518801819 · Cited by 40
Named-Entity Recognition on Indonesian Tweets using Bidirectional LSTM-CRF
Wintaka, D. C., Bijaksana, M. A., & Asror, I. (2019) · Procedia Computer Science · doi:10.1016/j.procs.2019.08.161 · Cited by 40
Method and dataset entity mining in scientific literature: A CNN + BiLSTM model with self-attention
Hou, L., Zhang, J., Wu, O., Yu, T., Wang, Z., Li, Z., Gao, J., Ye, Y., & Yao, R. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107621 · Cited by 43
Vlachidis, A., & Tudhope, D. S. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23485 · Cited by 39
Metadata
- Authors
- Grishman, R., & Sundheim, B.
- Affiliations
- New York University (US)
- Venue
- Proceedings of the 16th conference on Computational linguistics -
- Published
- 1996
- Volume
- 1
- Pages
- 466
- ISSN
- 1525-2477 (ISSN-L)
- OpenAlex
- W2068882115
- Cited by
- 1,408
- References
- 0
- Citations by year
- 2026: 26 · 2025: 37 · 2024: 62 · 2023: 74 · 2022: 64 · 2021: 81 · 2020: 64 · 2019: 73 · 2018: 68 · 2017: 50 · 2016: 61 · 2015: 71 · 2014: 65 · 2013: 117 · 2012: 83
- Type
- conference abstract
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Conference on Computational Linguistics (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- MUC6, information extraction, benchmark tasks, natural language processing evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
