Supporting Accessible Data Visualization Through Audio Data Narratives
The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Abstract
Online data visualizations play an important role in informing public opinion but are often inaccessible to screen reader users. To address the need for accessible data representations on the web that provide direct, multimodal, and up-to-date access to the data, we investigate audio data narratives –which combine textual descriptions and sonification (the mapping of data to non-speech sounds). We conduct two co-design workshops with screen reader users to define design principles that guide the structure, content, and duration of a data narrative. Based on these principles and relevant auditory processing characteristics, we propose a dynamic programming approach to automatically generate an audio data narrative from a given dataset. We evaluate our approach with 16 screen reader users. Findings show with audio narratives, users gain significantly more insights from the data. Users describe data narratives help them better extract and comprehend the information in both the sonification and description.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Paas, F. G. W. C. (1992) · Journal of Educational Psychology · doi:10.1037/0022-0663.84.4.429 · Cited by 2,150 · cited 1×
“To assess cognitive load in understanding the information, we measured users’ self-perceived mental efort, on a 9-point Paas Likert scale [57].”
Method(2003) · Computer Graphics Forum · doi:10.1111/1467-8659.00652 · Cited by 1,436
Narrative Visualization: Telling Stories with Data
Segel, E., & Heer, J. (2010) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2010.179 · Cited by 1,334
Lo, S., & Andrews, S. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01171 · Cited by 973
The intelligibility of speech as a function of the context of the test materials.
Miller, G. A., Heise, G. A., & Lichten, W. (1951) · Journal of Experimental Psychology · doi:10.1037/h0062491 · Cited by 769
CONSIDERATIONS IN THE CHOICE OF INTEROBSERVER RELIABILITY ESTIMATES
Hartmann, D. P. (1977) · Journal of Applied Behavior Analysis · doi:10.1901/jaba.1977.10-103 · Cited by 762
Visualization of Time-Oriented Data
Aigner, W., Miksch, S., Schumann, H., & Tominski, C. (2011) · Human-Computer Interaction Series · doi:10.1007/978-0-85729-079-3 · Cited by 688
Empirical Studies in Information Visualization: Seven Scenarios
Lam, H., Bertini, E., Isenberg, P., Plaisant, C., & Carpendale, S. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.279 · Cited by 623
Exploratory Analysis of Spatial and Temporal Data
(2006) · doi:10.1007/3-540-31190-4 · Cited by 530
Auditory Scene Analysis: The Perceptual Organization of Sound
Smoliar, S., & Bregman, A. S. (1991) · Computer Music Journal · doi:10.2307/3680919 · Cited by 502
Toward measuring visualization insight
North, C. (2006) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2006.70 · Cited by 456 · cited 2×
“data and what insights they gained from the data [55].”
BackgroundPrins, N., & Kingdom, F. A. A. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01250 · Cited by 429
The Science of Visual Data Communication: What Works
Franconeri, S. L., Padilla, L. M., Shah, P., Zacks, J. M., & Hullman, J. (2021) · Psychological Science in the Public Interest · doi:10.1177/15291006211051956 · Cited by 360
An Insight-Based Methodology for Evaluating Bioinformatics Visualizations
Saraiya, P., North, C., & Duca, K. (2005) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2005.53 · Cited by 343
(2017) · doi:10.1145/2998181.2998364 · Cited by 258 · cited 1×
“These approaches have been investigated for general-purpose images (e.g. in social media, web search) [28, 75] as well as for describing specifc types of data visualizations and data features [17, 23, 38, 51].”
BackgroundThe processing of structured and unstructured tonal sequences
Deutsch, D. (1980) · Perception & Psychophysics · doi:10.3758/bf03204881 · Cited by 241
Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content
Lundgard, A., & Satyanarayan, A. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114770 · Cited by 155
(2013) · doi:10.1145/2470654.2481374 · Cited by 137 · cited 2×
“Prior works with visual narratives have investigated methods to automatically include relevant annotations that tie external context to a data visualization [36, 69].”
BackgroundMethodAccessible Visualization: Design Space, Opportunities, and Challenges
Kim, N. W., Joyner, S. C., Riegelhuth, A., & Kim, Y. (2021) · Computer Graphics Forum · doi:10.1111/cgf.14298 · Cited by 137
IEEE Symposium on Information Visualization (InfoVis 05)
(2005) · IEEE Symposium on Information Visualization · doi:10.1109/infvis.2005.1532121 · Cited by 125
Twitter A11y: A Browser Extension to Make Twitter Images Accessible
Gleason, C., Pavel, A., McCamey, E., Low, C., Carrington, P., Kitani, K. M., & Bigham, J. P. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376728 · Cited by 121
Data Sonification for Users with Visual Impairment: A Case Study with Georeferenced Data
Zhao, H., Plaisant, C., Shneiderman, B., & Lazar, J. (2008) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1352782.1352786 · Cited by 120
Representing financial time series based on data point importance
Fu, T.-C., Chung, F.-L., Luk, R. W. P., & Ng, C.-M. (2007) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2007.04.009 · Cited by 99
Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content
Lundgard, A., & Satyanarayan, A. (2022) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 97
Evaluating the Effect of Style in Information Visualization
Vande Moere, A., Tomitsch, M., Wimmer, C., Christoph, B., & Grechenig, T. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.221 · Cited by 92
Understanding Screen-Reader Users’ Experiences with Online Data Visualizations
Sharif, A., Chintalapati, S. S., Wobbrock, J. O., & Reinecke, K. (2021) · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3441852.3471202 · Cited by 85 · cited 6×
“While tabular representations provide direct access to the data, there are several limitations including: overloaded speech feedback [59] and working memory [37 ,68], and lack of an overall picture of the data structure [59, 65, 66].”
BackgroundMethodInfluentialWalker, B. N., & Mauney, L. M. (2010) · ACM Transactions on Accessible Computing · doi:10.1145/1714458.1714459 · Cited by 84
Comparing two haptic interfaces for multimodal graph rendering
Yu, W., & Brewster, S. (2003) · Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. HAPTICS 2002 · doi:10.1109/haptic.2002.998934 · Cited by 83
Inclusive data visualization for people with disabilities: a call to action
Marriott, K., Lee, B., Butler, M., Cutrell, E., Ellis, K., Goncu, C., Hearst, M., McCoy, K., & Szafir, D. A. (2021) · Interactions · doi:10.1145/3457875 · Cited by 83
Comparing Effectiveness and Engagement of Data Comics and Infographics
Wang, Z., Wang, S., Farinella, M., Murray-Rust, D., Henry Riche, N., & Bach, B. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300483 · Cited by 82
Stangl, A., Verma, N., Fleischmann, K. R., Morris, M. R., & Gurari, D. (2021) · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3441852.3471233 · Cited by 81 · cited 1×
“BVI users’ preferences on image descriptions can vary depending on the media source and information-seeking goals [67].”
BackgroundButler, M., Holloway, L. M., Reinders, S., Goncu, C., & Marriott, K. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445207 · Cited by 72
Guiding Novice Web Workers in Making Image Descriptions Using Templates
Morash, V. S., Siu, Y.-T., Miele, J. A., Hasty, L., & Landau, S. (2015) · ACM Transactions on Accessible Computing · doi:10.1145/2764916 · Cited by 69
Psychology of auditory perception
Lotto, A. J., & Holt, L. L. (2010) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.123 · Cited by 68
Stimulus complexity and information integration in the spontaneous interpretations of line graphs
Carswell, C. M., Emery, C., & Lonon, A. M. (1993) · Applied Cognitive Psychology · doi:10.1002/acp.2350070407 · Cited by 62
Sonification of Astronomical Data
Díaz-Merced, W., Candey, R. M., Brickhouse, N. S., Schneps, M. H., Mannone, J. C., Brewster, S. A., & Kolenberg, K. (2011) · Proceedings of the International Astronomical Union · doi:10.1017/s1743921312000440 · Cited by 61
Sawe, N., Chafe, C., & Treviño, J. (2020) · Frontiers in Communication · doi:10.3389/fcomm.2020.00046 · Cited by 61
Ferres, L., Parush, A., Roberts, S., & Lindgaard, G. (2006) · Lecture notes in computer science · doi:10.1007/11788713_163 · Cited by 52
Exploring the use of structured musical stimuli to communicate simple diagrams: the role of context
Alty, J., & Rigas, D. I. (2004) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2004.08.003 · Cited by 52
Effect of segmentation on financial time series pattern matching
Wan, Y., Gong, X., & Si, Y. (2015) · Applied Soft Computing · doi:10.1016/j.asoc.2015.10.012 · Cited by 45
Auditory and visual scene analysis: an overview
Kondo, H. M., van Loon, A. M., Kawahara, J., & Moore, B. C. J. (2017) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2016.0099 · Cited by 44
Siu, A. F., Fan, D., Kim, G. S.-H., Rao, H. V., Vazquez, X., O'Modhrain, S., & Follmer, S. (2021) · Proceedings of the 18th International Web for All Conference · doi:10.1145/3430263.3452432 · Cited by 37
Self-citationCharacterizing Automated Data Insights
Law, P.-M., Endert, A., & Stasko, J. (2020) · 2020 IEEE Visualization Conference (VIS) · doi:10.1109/vis47514.2020.00041 · Cited by 36
Non-visual access to graphical information on COVID-19
Holloway, L., Butler, M., Reinders, S., & Marriott, K. (2020) · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3373625.3418015 · Cited by 26 · cited 3×
“updated automatically, descriptions can become outdated, creating a mismatch with the information presented visually [35, 66].”
BackgroundNees, M. A., & Walker, B. N. (2008) · ACM Transactions on Applied Perception · doi:10.1145/1402236.1402237 · Cited by 24
Sonification of numerical data for education
Vines, K., Hughes, C. S., Alexander, L., Calvert, C., Colwell, C., Holmes, H., Kotecki, C., Parks, K., & Pearson, V. K. (2019) · Open Learning The Journal of Open Distance and e-Learning · doi:10.1080/02680513.2018.1553707 · Cited by 24
What are Data Insights to Professional Visualization Users?
Law, P.-M., Endert, A., & Stasko, J. (2020) · Visualization · doi:10.1109/vis47514.2020.00043 · Cited by 17
AUDiaL: A Natural Language Interface to Make Statistical Charts Accessible to Blind Persons
Murillo-Morales, T., & Miesenberger, K. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58796-3_44 · Cited by 17
Ohshiro, K., Hurst, A., & DuBois, L. (2021) · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3441852.3476533 · Cited by 17
The Technology of Binaural Understanding
Colburn, H. S. (2021) · The Journal of the Acoustical Society of America · doi:10.1121/10.0008979 · Cited by 8
Using sonification for mining time series data
Last, M., & Gorelik, A. (2008) · doi:10.1145/1509212.1509220 · Cited by 4
AudioFunctions.web Multimodal Exploration of Mathematical Function Graphs
Ahmetovic, D., Bernareggi, C., Joao, G., Mascetti, S., & Capietto, A. (2019) · ACM Proceedings · Cited by 3
Research works printed but not linked: 22
- Benjamin Bach , Moritz Stefaner , Jeremy Boy , Steven Drucker , Lyn Bartram , Jo Wood , Paolo Ciuccarelli , Yuri Engelhardt , Ulrike Koeppen , and Barbara Tversky . 2018. Narrative design patterns for data-driven storytelling. In Data-driven storytelling , pp. 107 - 133 . AK Peters/CRC Press . Benjamin Bach, Moritz Stefaner, Jeremy Boy, Steven Drucker, Lyn Bartram, Jo Wood, Paolo Ciuccarelli, Yuri Engelhardt, Ulrike Koeppen, and Barbara Tversky. 2018. Narrative design patterns for data-driven storytelling. In Data-driven storytelling, pp. 107-133. AK Peters/CRC Press.
- Matthew Brehmer , and Tamara Munzner . 2013. A multi-level typology of abstract visualization tasks. IEEE transactions on visualization and computer graphics, 19(12) , pp. 2376 - 2385 . Matthew Brehmer, and Tamara Munzner. 2013. A multi-level typology of abstract visualization tasks. IEEE transactions on visualization and computer graphics, 19(12), pp.2376-2385.
- Lorna M. Brown , Stephen Brewster , Ramesh Ramloll , Wai Yu , and Beate Riedel . 2002 . Browsing modes for exploring sonified line graphs . In Proceedings of the British Human-Computer Interface Conference. 6–9. Lorna M. Brown, Stephen Brewster, Ramesh Ramloll, Wai Yu, and Beate Riedel. 2002. Browsing modes for exploring sonified line graphs. In Proceedings of the British Human-Computer Interface Conference. 6–9.
- Lorna M. Brown , Stephen A. Brewster , S. A. Ramloll , R. Burton , and Beate Riedel . 2003 . Design guidelines for audio presentation of graphs and tables . International Conference on Auditory Display (ICAD). Lorna M. Brown, Stephen A. Brewster, S. A. Ramloll, R. Burton, and Beate Riedel. 2003. Design guidelines for audio presentation of graphs and tables. International Conference on Auditory Display (ICAD).
- Stuart T. Card , Jock D. Mackinlay , and Ben Scheiderman . 1999. Readings in Information Visualization , using vision to think . San Francisco : Morgan Kaufmann . Stuart T. Card, Jock D. Mackinlay, and Ben Scheiderman. 1999. Readings in Information Visualization, using vision to think. San Francisco: Morgan Kaufmann.
- Jinho Choi , Sanghun Jung , Deok Gun Park , Jaegul Choo, and Niklas Elmqvist. 2019 . Visualizing for the non-visual: Enabling the visually impaired to use visualization. In Computer Graphics Forum , volume 38 , pages 249– 260 . Wiley Online Library. Jinho Choi, Sanghun Jung, Deok Gun Park, Jaegul Choo, and Niklas Elmqvist. 2019. Visualizing for the non-visual: Enabling the visually impaired to use visualization. In Computer Graphics Forum, volume 38, pages 249–260. Wiley Online Library.
- Diana Deutsch . 1999. Grouping mechanisms in music . In The psychology of music , pages 299– 348 . Elsevier. Diana Deutsch. 1999. Grouping mechanisms in music. In The psychology of music, pages 299–348. Elsevier.
- Mounya Elhilali , Juanjuan Xiang , Shihab A. Shamma , and Jonathan Z . Simon . 2009 . Interaction between attention and bottom-up saliency mediates the representation of foreground and back- ground in an auditory scene. PLoS biology, 7(6):e1000129. Mounya Elhilali, Juanjuan Xiang, Shihab A. Shamma, and Jonathan Z. Simon. 2009. Interaction between attention and bottom-up saliency mediates the representation of foreground and back- ground in an auditory scene. PLoS biology, 7(6):e1000129.
- John H. Flowers , Laura E. Whitwer , Douglas C. Grafel , and Cheryl A . Kotan . 2001 . Sonification of daily weather records: Issues of perception, attention and memory in design choices. Faculty Publications, Department of Psychology , page 432 . John H. Flowers, Laura E. Whitwer, Douglas C. Grafel, and Cheryl A. Kotan. 2001. Sonification of daily weather records: Issues of perception, attention and memory in design choices. Faculty Publications, Department of Psychology, page 432.
- Nicholas A. Giudice . 2018. Navigating without vision: Principles of blind spatial cognition. In Handbook of behavioral and cognitive geography . Edward Elgar Publishing . Nicholas A. Giudice. 2018. Navigating without vision: Principles of blind spatial cognition. In Handbook of behavioral and cognitive geography. Edward Elgar Publishing.
- Jenna L. Gorlewicz , Jennifer L. Tennison , Hari P. Palani , and Nicholas A . Giudice . 2018 . The graphical access challenge for people with visual impairments: Positions and pathways forward. In Interactive Multimedia-Multimedia Production and Digital Storytelling. IntechOpen . Jenna L. Gorlewicz, Jennifer L. Tennison, Hari P. Palani, and Nicholas A. Giudice. 2018. The graphical access challenge for people with visual impairments: Positions and pathways forward. In Interactive Multimedia-Multimedia Production and Digital Storytelling. IntechOpen.
- Michael Held , and Richard M . Karp . 1962 . A dynamic programming approach to sequencing problems. Journal of the Society for Industrial and Applied mathematics 10, no. 1: 196-210. Michael Held, and Richard M. Karp. 1962. A dynamic programming approach to sequencing problems. Journal of the Society for Industrial and Applied mathematics 10, no. 1: 196-210.
- Thomas Hermann , Andy Hunt , and John G . Neuhoff . 2011 . The sonification handbook. Logos Verlag Berlin . Thomas Hermann, Andy Hunt, and John G. Neuhoff. 2011. The sonification handbook. Logos Verlag Berlin.
- Johan Kildal , and Stephen A. Brewster . 2006. Providing a size-independent overview of non-visual tables . In Proceedings of the International Conference on Auditory Display (ICAD). Johan Kildal, and Stephen A. Brewster. 2006. Providing a size-independent overview of non-visual tables. In Proceedings of the International Conference on Auditory Display (ICAD).
- Edward Kim , and Kathleen F . McCoy. 2018. Multimodal deep learning using images and text for information graphic classification . In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 143-148) . Edward Kim, and Kathleen F. McCoy. 2018. Multimodal deep learning using images and text for information graphic classification. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (pp. 143-148).
- Michael A. Nees and Bruce N. Walker. 2006. Relative intensity of auditory context for auditory graph design . In Proceedings of the International Conference on Auditory Display (ICAD). Michael A. Nees and Bruce N. Walker. 2006. Relative intensity of auditory context for auditory graph design. In Proceedings of the International Conference on Auditory Display (ICAD).
- Michael A. Nees and Bruce N. Walker. 2007. Listener, task, and auditory graph: Toward a conceptual model of auditory graph comprehension . In Proceedings of the International Conference on Auditory Display (ICAD). Michael A. Nees and Bruce N. Walker. 2007. Listener, task, and auditory graph: Toward a conceptual model of auditory graph comprehension. In Proceedings of the International Conference on Auditory Display (ICAD).
- Rameshsharma Ramloll , Stephen Brewster , Wai Yu , and Beate Riedel . 2001. Using non-speech sounds to improve access to 2D tabular numerical information for visually impaired users. In People and Computers XV—Interaction without Frontiers , pp. 515 - 529 . Springer , London . Rameshsharma Ramloll, Stephen Brewster, Wai Yu, and Beate Riedel. 2001. Using non-speech sounds to improve access to 2D tabular numerical information for visually impaired users. In People and Computers XV—Interaction without Frontiers, pp. 515-529. Springer, London.
- 10.5555/3235128 · doi:10.5555/3235128
- Tony Stockman , Christopher Frauenberger , and Greg Hind . 2005 . Interactive sonification of spreadsheets . In Proceedings of the International Conference on Auditory Display (ICAD). Tony Stockman, Christopher Frauenberger, and Greg Hind. 2005. Interactive sonification of spreadsheets. In Proceedings of the International Conference on Auditory Display (ICAD).
- Charles D. Stolper , Bongshin Lee , Nathalie Henry Riche, and John Stasko . 2016 . Emerging and recurring data-driven storytelling techniques: Analysis of a curated collection of recent stories. Charles D. Stolper, Bongshin Lee, Nathalie Henry Riche, and John Stasko. 2016. Emerging and recurring data-driven storytelling techniques: Analysis of a curated collection of recent stories.
- Qiyu Zhi , Alvitta Ottley , and Ronald Metoyer . 2019. Linking and layout: Exploring the integration of text and visualization in storytelling . In Computer Graphics Forum , volume 38(3), pages 675– 685 . Wiley Online Library . Qiyu Zhi, Alvitta Ottley, and Ronald Metoyer. 2019. Linking and layout: Exploring the integration of text and visualization in storytelling. In Computer Graphics Forum, volume 38(3), pages 675–685. Wiley Online Library.
Cited by
Understanding Visual Arts Experiences of Blind People
Li, F. M., Zhang, L., Bandukda, M., Stangl, A. J., Shinohara, K., Findlater, L., & Carrington, P. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580941 · Cited by 93
A survey of immersive visualization: Focus on perception and interaction
Zhang, Y., Wang, Z., Zhang, J., Shan, G., & Tian, D. (2023) · Visual Informatics · doi:10.1016/j.visinf.2023.10.003 · Cited by 78
Chart Reader: Accessible Visualization Experiences Designed with Screen Reader Users
Thompson, J. R., Martinez, J. J., Sarikaya, A., Cutrell, E., & Lee, B. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581186 · Cited by 43
Jiang, L., Jung, C., Phutane, M., Stangl, A. J., & Azenkot, S. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642238 · Cited by 59
Alam, M. Z. I., Islam, S., & Hoque, E. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581641.3584099 · Cited by 57
Metadata
- Authors
- Siu, A., S-H Kim, G., O'Modhrain, S., & Follmer, S.
- Affiliations
- Stanford University (US)
- Stanford University, Stanford University, United States
- University of Michigan, United States
- Venue
- CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2022-04-29
- Pages
- 1-19
- OpenAlex
- W4225163156
- Cited by
- 64
- References
- 51
- Citations by year
- 2026: 12 · 2025: 13 · 2024: 18 · 2023: 18 · 2022: 3
- FWCI
- 18.95
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Funders
- National science foundation (2016789, 2016363)
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Keywords
- accessible data visualization, sonification, screen readers, multimodal representation, dynamic programming, co-design workshops, auditory processing, web accessibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.