PreprintCited by 1Topic ModelingNatural Language Processing TechniquesComputational and Text Analysis Methods
Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering
Open accessEllinger, Lukas, Fichtl, Alexander, Anschütz, Miriam, Groh, Georg·2026·arXiv
DOI 10.48550/arxiv.2605.26620arXiv 2605.26620
Abstract
Natural language conveys information at varying levels of granularity, from fine-grained references to broad descriptions. While granularity is fundamental to human communication, existing measures mostly capture surface detail or sentence specificity. We introduce Granuscore, a reference-free measure of granularity that leverages structural properties of a hierarchical embedding space. Granuscore reliably recovers hierarchical orderings on the Granola-EQ dataset and captures expected differences in granularity across discourse contexts. Across domains, we further show that Granuscore explains non-linear variation in sentence specificity beyond sentence length. Finally, we apply Granuscore to four question-answering benchmarks and analyze how granularity differs for questions, gold answers, and model outputs across response outcomes. The analysis reveals consistent differences in model behavior and provides a principled lens for characterizing the difficulty of QA datasets. Together, the results position Granuscore as a scalable, broadly applicable tool for analyzing granularity in text.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
DataCite · 10.48550/arxiv.2605.26620
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 39
- Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next- Generation Hyperparameter Optimization Frame- work. In The 25th ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining, pages 2623-2631.
- Miriam Anschütz, Thanh Mai Pham, Eslam Nasrallah, Maximilian Müller, Cristian-George Craciun, and Georg Groh. 2025. German4All -A Dataset and Model for Readability-Controlled Paraphrasing in German. In Proceedings of the 18th International Natural Language Generation Conference, pages 390-407, Hanoi, Vietnam. Association for Computa- tional Linguistics.
- Jiaoyan Chen, Yuan He, Ian Horrocks, and Zhangdie Yuan. 2024. Language Models as Hierarchy En- coders. In Advances in Neural Information Pro- cessing Systems 37, pages 14690-14711, Vancouver, BC, Canada. Neural Information Processing Systems Foundation, Inc. (NeurIPS). · DOI
- Aileen Cheng, Alon Jacovi, Amir Globerson, Ben Golan, Charles Kwong, Chris Alberti, Connie Tao, Eyal Ben-David, Gaurav Singh Tomar, Lukas Haas, Yonatan Bitton, Adam Bloniarz, Aijun Bai, An- drew Wang, Anfal Siddiqui, Arturo Bajuelos Castillo, Aviel Atias, Chang Liu, Corey Fry, and 46 others. 2025. The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality. arXiv preprint. ArXiv:2512.10791 [cs]. · DOI
- Jacob Cohen. 2013. Statistical Power Analysis for the Behavioral Sciences, 2 edition. Routledge, New York. · DOI · Paper page
- R.A. Day and B. Gastel. 2012. How to Write and Pub- lish a Scientific Paper. Cambridge University Press.
- DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingx- uan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. DeepSeek-V3 Technical Re- port. arXiv preprint. ArXiv:2412.19437 [cs. · DOI · Paper page
- Peter Dixon. 1987. The processing of organizational and component step information in written directions. Journal of Memory and Language, 26(1):24-35. · DOI
- Alex Djalali, David Clausen, Sven Lauer, Katrin Schultz, and Christopher Potts. 2011. Modeling Expert Effects and Common Ground Using Ques- tions under Discussion. In AAAI Fall Symposium: Building Representations of Common Ground with Intelligent Agents.
- Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2025. The Faiss library. arXiv preprint. ArXiv:2401.08281 [cs. · DOI
- Lukas Ellinger, Miriam Anschütz, and Georg Groh. 2025. Simplifications Are Absolutists: How Sim- plified Language Reduces Word Sense Awareness in LLM-Generated Definitions. In Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing -Natural Language Processing in the Generative AI era, pages 342-351, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.
- Yifan Gao, Yang Zhong, Daniel Preoţiuc-Pietro, and Junyi Jessy Li. 2019. Predicting and Analyzing Lan- guage Specificity in Social Media Posts. Proceed- ings of the AAAI Conference on Artificial Intelligence, 33(01):6415-6422. · DOI
- Jerry R. Hobbs. 1985. Granularity. In Proceedings of the 9th International Joint Conference on Artificial Intelligence -Volume 1, IJCAI'85, pages 432-435, San Francisco, CA, USA.
- Jie Huang, Kevin Chen-Chuan Chang, Jinjun Xiong, and Wen-mei Hwu. 2023. Can Language Models Be Specific? How? In Findings of the Association for Computational Linguistics: ACL 2023, pages 716- 727, Toronto, Canada. Association for Computational Linguistics. · DOI
- Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. 2025. Why Language Models Hallucinate. arXiv preprint. ArXiv:2509.04664 [cs. · DOI · Paper page
- Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A Highly Efficient Gradi- ent Boosting Decision Tree. In Advances in Neural Information Processing Systems, volume 30.
- Giyeong Kim. 2006. Relationship between index term specificity and relevance judgment. Information Pro- cessing & Management, 42(5):1218-1229. · DOI
- Wei-Jen Ko, Greg Durrett, and Junyi Jessy Li. 2019. Domain Agnostic Real-Valued Specificity Prediction. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):6610-6617. · DOI
- Junyi Jessy Li, Bridget O'Daniel, Yi Wu, Wenli Zhao, and Ani Nenkova. 2016. Improving the Annotation of Sentence Specificity. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3921-3927, Por- torož, Slovenia. European Language Resources As- sociation (ELRA).
- Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214-3252, Dublin, Ireland. Association for Computational Linguistics. · DOI · Paper page
- Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Daniel Weld. 2020. S2ORC: The Semantic Scholar Open Research Corpus. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pages 4969-4983, Online. As- sociation for Computational Linguistics. · DOI · Paper page
- George A. Miller. 1994. WordNet: A Lexical Database for English. In Human Language Technology: Pro- ceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
- Rutu Mulkar-Mehta, Jerry Hobbs, and Eduard Hovy. 2011. Granularity in Natural Language Discourse. In Proceedings of the Ninth International Conference on Computational Semantics (IWCS 2011).
- OECD. 2024. Do Adults Have the Skills They Need to Thrive in a Changing World?: Survey of Adult Skills 2023. OECD Skills Studies. · DOI
- Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groen- eveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, and 49 others. 2025. Olmo 3. arXiv preprint. ArXiv:2512.13961 [cs]. · DOI
- Hiroki Onozeki and Michimasa Inaba. 2025. Enhanc- ing Coherence and Interestingness in Knowledge- Grounded Dialogue Generation. In Proceedings of the 18th International Natural Language Generation Conference, pages 1-19, Hanoi, Vietnam. Associa- tion for Computational Linguistics.
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics. · DOI · Paper page
- Eleanor Rosch, Carolyn B Mervis, Wayne D Gray, David M Johnson, and Penny Boyes-Braem. 1976. Basic objects in natural categories. Cognitive Psy- chology, 8(3):382-439. · DOI · Paper page
- Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple Entity-Centric Ques- tions Challenge Dense Retrievers. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6138-6148, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page
- Marlene Stoll, Martin Kerwer, Klaus Lieb, and Anita Chasiotis. 2022. Plain language summaries: A sys- tematic review of theory, guidelines and empirical research. PLOS ONE, 17(6):e0268789. · DOI · Paper page
- J.M. Swales. 1990. Genre Analysis. Cambridge Ap- plied Linguistics. Cambridge University Press.
- Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, and 41 others. 2022. LaMDA: Language Models for Dia- log Applications. arXiv preprint. ArXiv:2201.08239 [cs. · DOI · Paper page
- Denny Vrandečić and Markus Krötzsch. 2014. Wiki- data: a free collaborative knowledgebase. Commun. ACM, 57(10):78-85. · DOI
- Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems, volume 33, pages 5776-5788.
- Miriam Wanner, Benjamin Van Durme, and Mark Dredze. 2024. DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation. arXiv preprint. ArXiv:2412.13175 · DOI
- Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Measuring short-form factuality in large language models. arXiv preprint. ArXiv:2411.04368 [cs]. · DOI · Paper page
- Tianyi Wu, Jingwei Ni, Bryan Hooi, Jiaheng Zhang, Elliott Ash, See-Kiong Ng, Mrinmaya Sachan, and Markus Leippold. 2025. Balancing Truthfulness and Informativeness with Uncertainty-Aware Instruction Fine-Tuning. arXiv preprint. ArXiv:2502.11962 · DOI
- An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayi- heng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 oth- ers. 2025. Qwen3 Technical Report. arXiv preprint. ArXiv:2505.09388 [cs. · DOI · Paper page
- Gal Yona, Roee Aharoni, and Mor Geva. 2024. Narrow- ing the Knowledge Evaluation Gap: Open-Domain Question Answering with Multi-Granularity An- swers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 6737-6751, Bangkok, Thailand. Association for Computational Linguistics. · DOI
Cited by
Sorted by citations
AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality
Fichtl, A. M., Ellinger, L., Kelber, J., Olík, K., & Groh, G. (2026) · arXiv · doi:10.48550/arxiv.2608.03581
“• An evaluation suite combining LLM-as-a-judge scoring, Granuscore, and an overlap metric based on concern decomposition and paper-grounded LLM matching against an in-corpus human baseline.”
Self-citation
- Authors
- Ellinger, L., Fichtl, A., Anschütz, M., & Groh, G.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- Pages
- 373-390
- arXiv
- 2605.26620
- OpenAlex
- W7162577376
- Cited by
- 0
- References
- 0
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Technische Universität München
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- hierarchical embedding, question answering, discourse structure, model output analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
