The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
GERBIL: General Entity Annotator Benchmarking Framework
Open accessAbstract
We present GERBIL, an evaluation framework for semantic entity annotation. The rationale behind our framework is to provide developers, end users and researchers with easy-to-use interfaces that allow for the agile, fine-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights pertaining to the extension, integration and use of annotation applications. In particular, GERBIL provides comparable results to tool developers so as to allow them to easily discover the strengths and weaknesses of their implementations with respect to the state of the art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in machine-processable format, allowing for the efficient querying and post-processing of evaluation results. Finally, the tool diagnostics provided by GERBIL allows deriving insights pertaining to the areas in which tools should be further refined, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. GERBIL aims to become a focal point for the state of the art, driving the research agenda of the community by presenting comparable objective evaluation results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition
Tjong Kim Sang, E. F., & De Meulder, F. (2003) · Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - · doi:10.3115/1119176.1119195 · Cited by 2,788
Reproducible research in computational science
Peng, R. D. (2011) · Science · doi:10.1126/science.1213847 · Cited by 1,449 · cited 1×
“Indeed, it is now well established that scientists spend between 60 and 80% of their time preparing data for experiments [14, 18, 31].”
Cited by
Survey on Aspect-Level Sentiment Analysis
Schouten, K., & Frasincar, F. (2016) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2015.2485209 · Cited by 832
Old is Gold: Linguistic Driven Approach for Entity and Relation Linking of Short Text
Sakor, A., Onando Mulang’, I., Singh, K., Shekarpour, S., Esther Vidal, M., Lehmann, J., & Auer, S. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1243 · Cited by 86
A Semantic Parsing and Reasoning-Based Approach to Knowledge Base Question Answering
Abdelaziz, I., Ravishankar, S., Kapanipathi, P., Roukos, S., & Gray, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i18.17988 · Cited by 26
“We also report Macro-F1-QALD (Usbeck et al. 2015), a recommended metric to use for evaluating QALD dataset.”
MethodAgent based dynamic thresholding for adaptive anomaly detection using reinforcement learning
Yang, X., Howley, E., & Schukat, M. (2025) · Neural Computing and Applications · doi:10.1007/s00521-024-10536-0 · Cited by 11
Metadata
- Authors
- Usbeck, R., Röder, M., Ngonga Ngomo, A.-C., Baron, C., Both, A., Brümmer, M., Ceccarelli, D., Cornolti, M., Cherix, D., Eickmann, B., Ferragina, P., Lemke, C., Moro, A., Navigli, R., Piccinno, F., Rizzo, G., Sack, H., Speck, R., Troncy, R., Waitelonis, J., & Wesemann, L.
- Venue
- Proceedings of the 24th International Conference on World Wide Web
- Published
- 2015-05-18
- Pages
- 1133-1143
- OpenAlex
- W2137489006
- Cited by
- 190
- References
- 2
- FWCI
- 22.54
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: GERBIL: General Entity Annotator Benchmarking Framework. https://app.cochat.ai/papers/10.1145/2736277.2741626
- The paper
- Usbeck, R., Röder, M., Ngonga Ngomo, A.-C., Baron, C., Both, A., Brümmer, M., Ceccarelli, D., Cornolti, M., Cherix, D., Eickmann, B., Ferragina, P., Lemke, C., Moro, A., Navigli, R., Piccinno, F., Rizzo, G., Sack, H., Speck, R., Troncy, R., . . . Wesemann, L. (2015). GERBIL: General Entity Annotator Benchmarking Framework. Proceedings of the 24th International Conference on World Wide Web. https://doi.org/10.1145/2736277.2741626
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.