Understanding and Improving Drilled-Down Information Extraction from Online Data Visualizations for Screen-Reader Users
Open accessThe DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index; 9 of the 25 sources its claims cite were checked for support.
Abstract
Inaccessible online data visualizations can significantly disenfranchise screen-reader users from accessing critical online information. Current accessibility measures, such as adding alternative text to visualizations, only provide a high-level overview of data, limiting screen-reader users from exploring data visualizations in depth. In this work, we build on prior research to develop taxonomies of information sought by screen-reader users to interact with online data visualizations granularly through role-based and longitudinal studies with screen-reader users. Utilizing these taxonomies, we extended the functionality of VoxLens—an open-source multi-modal system that improves the accessibility of data visualizations—by supporting drilled-down information extraction. We assessed the performance of our VoxLens enhancements through task-based user studies with 10 screen-reader and 10 non-screen-reader users. Our enhancements “closed the gap” between the two groups by enabling screen-reader users to extract information with approximately the same accuracy as non-screen-reader users, reducing interaction time by 22% in the process.
Overview
CoChat Summary
TL;DR
This paper develops taxonomies of holistic and drilled-down information that screen-reader users (SRUs) seek from online data visualizations, then uses them to extend VoxLens for granular extraction from multi-series line graphs and geospatial maps. In a task-based study, SRUs using the enhanced VoxLens extracted information 5.6% more accurately on average than non-SRUs, closing a previously reported accuracy gap, though they still took 92.8% more interaction time.
Key findings
- Role-based (12 SRUs) and longitudinal (7 SRUs) studies produced taxonomies of holistic and drilled-down information needs. The most issued command was Value (28.6%); by visualization type, the most issued command was Factor for single-series bar graphs (26.7%), Average for multi-series line graphs (22.0%), and Value for geospatial maps (46.4%).
- In the task-based study with 10 SRUs using enhanced VoxLens and 10 non-SRUs using no tools, there was no significant effect of SRU on accuracy of extracted information (p ≈ .906); SRUs were 5.6% more accurate on average than non-SRUs, whereas prior work reported non-SRUs outperforming SRUs by 62%. Visualization type significantly affected accuracy (χ²(2, N=300)=9.35, p<.05, Cramer’s V=0.12); single-series bar graphs had the best performance (92% SRU, 98% non-SRU), followed by multi-series line graphs (88%, 84%) and geospatial maps (86%, 70%).
- SRU had a significant main effect on interaction time (F(1,18)=36.94, p<.001, η²p=0.69): SRUs averaged 42.4 seconds (SD=21.2) versus 22.0 seconds (SD=16.5) for non-SRUs, 92.8% higher. Visualization type also significantly affected interaction time (F(2,275.3)=6.14, p<.05, η²p=0.04), and age had a significant effect (F(1,16.9)=8.21, p<.05, η²p=0.33), with participants over 50 having 44.0% higher interaction times.
- NASA-TLX ratings showed no significant effect of SRU on any of the six workload dimensions, suggesting comparable perceived workload levels.
Methods
- The authors conducted a Wizard-of-Oz role-based study with 12 SRUs (M=50.3 years, SD=13.6) using explorer, teacher, and news reporter roles, and a 12-day longitudinal study with 7 SRUs (M=48.1 years, SD=8.7). Participants interacted with visualizations drawn from 30 curated examples (10 each of single-series bar graphs, multi-series line graphs, and geospatial maps); data included interaction logs and semi-structured interviews.
- Quantitative analysis used a mixed Poisson regression for command counts; qualitative analysis used semantic thematic analysis, combining 33 initial codes into 18 axial codes, with 90.1% inter-rater agreement.
- VoxLens was extended with chartType and dataModule parameters, factor-level categorization, regional categorization, expanded ranking vocabulary, factor-level and dependent-variable range queries, and visually hidden tables. Evaluation used a 2×3 mixed-factorial task-based study (10 SRUs using enhanced VoxLens vs. 10 non-SRUs without tools; 15 tasks per participant; 300 total trials), analyzed with mixed logistic regression for accuracy and a linear mixed model for interaction time, plus NASA-TLX analyzed with Aligned Rank Transform.
Limitations
- The paper states that exploration was limited to single-series bar graphs, multi-series line graphs, and geospatial maps; future work should cover other complex visualizations such as three-dimensional graphs and stacked bar charts, and physical interfaces such as tactile maps.
- The paper states that enhancements were limited by the keyword-matching algorithm and Web Speech API voice recognition, and that factor-level categorization struggled with prepositions or adverbs of time (e.g., “five years ago,” “between 2017 and 2019”).
- Evident from the text: sample sizes were small (12, 7, and 10+10 participants), and the authors note all participants were from the United States, which may limit generalization of regional categorization and findings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
9 Valid · 1 Not valid · 12 Not checked · 88 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Prior to the enhancements presented in this work, VoxLens was limited to only simple visualizations created using two-dimensional single-series data, such as single-series bar graphs.”
§ Understanding and Improving Drilled-Down Information Extraction from Online Data Visualizations for Screen-Reader Users > 3.2 Overview of VoxLens
What the sources say
- Supportedcites [70] 70. Ather Sharif, Olivia H. Wang, Alida T. Muongchan, Katharina Reinecke, and Jacob O. Wobbrock. 2022. VoxLens: Making Online Data Visualizations Accessible with an Interactive JavaScript Plug-In. In CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 478, 19 pages. https://doi.org/10.1145/ 3491102.3517431 source
The Design of VoxLens section states that VoxLens supports only two-dimensional single-series data. Section 3.1.4 identifies bar graphs among the chart types used, so a single-series bar graph is an apt example rather than an exclusive limit. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Out of these five, extremum, average and overall trend were already implemented in VoxLens.”
§ Understanding and Improving Drilled-Down Information Extraction from Online Data Visualizations for Screen-Reader Users > 4.1 Our Additions to VoxLens
What the sources say
- Supportedcites [70] 70. Ather Sharif, Olivia H. Wang, Alida T. Muongchan, Katharina Reinecke, and Jacob O. Wobbrock. 2022. VoxLens: Making Online Data Visualizations Accessible with an Interactive JavaScript Plug-In. In CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 478, 19 pages. https://doi.org/10.1145/ 3491102.3517431 source
Section 3.1.3 says VoxLens computes maximum and minimum data points and the average. Section 3.1.4 says its Sonification mode enables listeners to interpret the overall data trend; section 5.3.3 reports participants doing so. The paper separately notes that overall-trend information was not yet provided through Summary mode, but the claim does not specify that mode. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“(By contrast, using the original version of VoxLens, SRUs performed 15% less accurately than non-SRUs [70].)”
§ Understanding and Improving Drilled-Down Information Extraction from Online Data Visualizations for Screen-Reader Users > 1 INTRODUCTION
What the sources say
- Supportedcites [70] 70. Ather Sharif, Olivia H. Wang, Alida T. Muongchan, Katharina Reinecke, and Jacob O. Wobbrock. 2022. VoxLens: Making Online Data Visualizations Accessible with an Interactive JavaScript Plug-In. In CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 478, 19 pages. https://doi.org/10.1145/ 3491102.3517431 source
Section 6.1 reports that VoxLens reduced the information-extraction accuracy gap between screen-reader users and non-screen-reader users from 62% to 15%. Section 5.1 adds that the remaining difference between the groups was not statistically significant; the claim describes the reported gap, not a significant effect. (LLM, conf=0.92)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Several factors contribute to the difference in interaction times between SRUs and non-SRUs [7,70], including the duration of the auditory responses.”
§ Understanding and Improving Drilled-Down Information Extraction from Online Data Visualizations for Screen-Reader Users > 6.2 Continuous Reduction of Interaction Time
What the sources say
- Not checked — no full text of the sourcecites [7] 7. Jeffrey P. Bigham, Anna C. Cavender, Jeremy T. Brudvik, Jacob O. Wobbrock, and Richard E. Ladner. 2007. WebinSitu: A Comparative Analysis of Blind and Sighted Browsing Behavior. In Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility (Tempe, Arizona, USA) (Assets '07). Association for Computing Machinery, New York, NY, USA, 51-58. https://doi. org/10.1145/1296843.1296854 source
- Supportedcites [70] 70. Ather Sharif, Olivia H. Wang, Alida T. Muongchan, Katharina Reinecke, and Jacob O. Wobbrock. 2022. VoxLens: Making Online Data Visualizations Accessible with an Interactive JavaScript Plug-In. In CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 478, 19 pages. https://doi.org/10.1145/ 3491102.3517431 source
Section 6.1 directly discusses the interaction-time gap between screen-reader users and non-screen-reader users. It identifies time spent hearing sonified data, issuing a voice command, pressing a key combination, and the duration of the auditory response as possible contributors. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“The original VoxLens [70] reduced this gap, resulting in SRUs extracting information only 15% less accurately than non-SRUs.”
§ Understanding and Improving Drilled-Down Information Extraction from Online Data Visualizations for Screen-Reader Users > 6.1 Substantial Improvement in Accuracy
What the sources say
- Supportedcites [70] 70. Ather Sharif, Olivia H. Wang, Alida T. Muongchan, Katharina Reinecke, and Jacob O. Wobbrock. 2022. VoxLens: Making Online Data Visualizations Accessible with an Interactive JavaScript Plug-In. In CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Association for Computing Machinery, New York, NY, USA, Article 478, 19 pages. https://doi.org/10.1145/ 3491102.3517431 source
Section 6.1 states that VoxLens reduced the information-extraction accuracy gap between screen-reader and non-screen-reader users from 62% to 15%. The claim accurately paraphrases that reported result. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
50 more claims
Create a free account to see all 110 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 36
- It may state two separate claims as one. 21
- The quote only partly states this claim. 1
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- ConsistentF(1, 18) = 36.94reported p < .001 · recomputed p < .001
“F (1,18)=36.94,p < .001”
- ConsistentF(2, 275.3) = 6.14reported p < .05 · recomputed p = .002
“F (2,275.3)=6.14, p < .05”
- ConsistentF(1, 16.9) = 8.21reported p < .05 · recomputed p = .011
“F (1,16.9)=8.21,p < .05”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding and Improving Drilled-Down Information Extraction from Online Data Visualizations for Screen-Reader Users > 5.2.1 Accuracy of Extracted Information (AEI).
“There was a significant effect of Visualization Type (𝑉𝑇 ) on 𝐴𝐸𝐼 overall (𝜒 2 (2, 𝑁 =300)=9.35,𝑝<.05,Cramer's 𝑉 =0.12).”
§ Understanding and Improving Drilled-Down Information Extraction from Online Data Visualizations for Screen-Reader Users > 8 CONCLUSION
“| 𝑉𝑇 | 20 | 9.33 | < .05 | .12 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding and Improving Drilled-Down Information Extraction from Online Data Visualizations for Screen-Reader Users > 2.1 Need for Accessible Data Visualizations
“Their results show that due to the inaccessibility of online data visualizations, SRUs extract information 61% less accurately and spend 211% more time interacting than non-SRUs.”
§ Understanding and Improving Drilled-Down Information Extraction from Online Data Visualizations for Screen-Reader Users > 6.1 Substantial Improvement in Accuracy
“Sharif et al. [68] reported that SRUs are 62% less accurate than non-SRUs at extracting information from online data visualizations.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Understanding and Improving Drilled-Down Information Extraction from Online Data Visualizations for Screen-Reader Users > 5.2.2 Interaction Time (IT).
“Screen-Reader User (𝑆𝑅𝑈 ) had a significant main effect on Interaction Time (IT) (𝐹 (1,18)=36.94,𝑝<.001, 𝜂 2 𝑝 =0.69).”
§ Understanding and Improving Drilled-Down Information Extraction from Online Data Visualizations for Screen-Reader Users > 8 CONCLUSION
“| 𝑆𝑅𝑈 | 1 | 16.98 | 36.94 | < .001 | 0.69 |”
Cited works that were retracted
Works this paper cites that were later retracted, from the retraction notices publishers deposit. Major when the notice came out before this paper was published.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
Cited work 10.1007/s13369-021-05889-w · retracted 2021-07-13 · notice 10.1007/s13369-021-05889-w · this paper published 2023-04-30
Citations
Cites
1 retracted
RETRACTED ARTICLE: Interactive IoT Data Visualization for Decision Making in Business Intelligence
Zhang, L., Vinodhini, B., & Maragatham, T. (2021) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-021-05889-w · Cited by 12
RetractedUsing thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
Qualitative Evaluation and Research Methods
Welch, J. K., & Patton, M. Q. (1992) · The Modern Language Journal · doi:10.2307/330063 · Cited by 3,306 · cited 1×
“We used semantic thematic analysis [54, 64] to analyze the interview sessions from our role-playing and longitudinal studies.”
MethodBostock, M., Ogievetsky, V., & Heer, J. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.185 · Cited by 3,165 · cited 1×
“Therefore, we generated all visualizations using D3 [8].”
MethodThe aligned rank transform for nonparametric factorial analyses using only anova procedures
Wobbrock, J. O., Findlater, L., Gergle, D., & Higgins, J. J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978963 · Cited by 2,854 · cited 1×
“nonparametric Aligned Rank Transform procedure [23, 79] to statistically analyze the effects of SRU (levels: yes, no).”
MethodSelf-citationStatistical analysis of repeated measures data using SAS procedures.
Littell, R. C., Henry, P. R., & Ammerman, C. B. (1998) · Journal of Animal Science · doi:10.2527/1998.7641216x · Cited by 2,825 · cited 1×
“gression model [29] and a linear mixed model [27, 51] to analyze AEI and IT , respectively.”
MethodLog-normal distributions across the sciences: keys and clues
LIMPERT, E., STAHEL, W. A., & ABBT, M. (2001) · BioScience · doi:10.1641/0006-3568(2001)051[0341:lndats]2.0.co;2 · Cited by 2,621 · cited 2×
“Therefore, we applied a logarithmic transformation before conducting our analysis, following standard practice for time measures [5, 38, 50].”
BackgroundMethodAnderson, T. W., & Darling, D. A. (1954) · Journal of the American Statistical Association · doi:10.1080/01621459.1954.10501232 · Cited by 2,450 · cited 1×
“Anderson-Darling goodness-of-fit tests of normality [3] confirmed that log-transformed interaction times were conditionally normally distributed (p ≈ .”
BackgroundDoing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars.
Maguire, M., & Delahunt, B. (2017) · All Ireland Journal of Higher Education · doi:10.62707/aishej.v9i3.335 · Cited by 2,099
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
The SAGE Encyclopedia of Social Science Research Methods
Lewis-Beck, M., Bryman, A., & Futing Liao, T. (2004) · Sage eBooks · doi:10.4135/9781412950589 · Cited by 1,439
The Sage encyclopedia of social science research methods
(2004) · Choice Reviews Online · doi:10.5860/choice.42-0062 · Cited by 1,337
Eta-squared and partial eta-squared in fixed factor anova designs
Cohen, J. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300111 · Cited by 1,333 · cited 1×
“Partial etasquared (η2 p ) is a measure of effect size [18].”
BackgroundBuchenau, M., & Suri, J. F. (2000) · Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques · doi:10.1145/347642.347802 · Cited by 1,237 · cited 2×
“We conducted role-based and longitudinal user studies [14, 48] with 12 and seven SRUs, respectively, to understand their information needs from online data visualizations.”
MethodCONSIDERATIONS IN THE CHOICE OF INTEROBSERVER RELIABILITY ESTIMATES
Hartmann, D. P. (1977) · Journal of Applied Behavior Analysis · doi:10.1901/jaba.1977.10-103 · Cited by 762 · cited 2×
“1%, demonstrating a high level of agreement between raters [31, 35].”
BackgroundMethodA Multi-Level Typology of Abstract Visualization Tasks
Brehmer, M., & Munzner, T. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.124 · Cited by 747 · cited 1×
“We finalized the question categories for our task-based user study based on Brehmer and Munzner’s task topology [11] and prior work on the type of questions users ask of graphs [10, 42, 68].”
MethodGegenfurtner, A., Lehtinen, E., & Säljö, R. (2011) · Educational Psychology Review · doi:10.1007/s10648-011-9174-7 · Cited by 715 · cited 1×
“In contrast, non-screen-reader users can explore data visualizations by rapidly interpreting visual patterns [1, 28, 55].”
BackgroundWizard of Oz studies — why and how
Dahlbäck, N., Jönsson, A., & Ahrenberg, L. (1993) · Knowledge-Based Systems · doi:10.1016/0950-7051(93)90017-n · Cited by 706
An Aligned Rank Transform Procedure for Multifactor Contrast Tests
Elkin, L. A., Kay, M., Higgins, J. J., & Wobbrock, J. O. (2021) · The 34th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3472749.3474784 · Cited by 664 · cited 1×
“nonparametric Aligned Rank Transform procedure [23, 79] to statistically analyze the effects of SRU (levels: yes, no).”
MethodSelf-citationThe analysis of binomial data by a generalized linear mixed model
GILMOUR, A. R., ANDERSON, R. D., & RAE, A. L. (1985) · Biometrika · doi:10.1093/biomet/72.3.593 · Cited by 304 · cited 2×
“We employed a mixed Poisson regression model [29] to analyze 𝐶𝑁𝑇 , as is standard practice for count data.”
MethodAn effect size primer: A guide for clinicians and researchers.
Ferguson, C. J. (2015) · Methodological issues and strategies in clinical research (4th ed.). · doi:10.1037/14805-020 · Cited by 286
Patterns for How Users Overcome Obstacles in Voice User Interfaces
Myers, C., Furqan, A., Nebolsky, J., Caro, K., & Zhu, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173580 · Cited by 213 · cited 1×
“Our system does not process the user’s query when our algorithm does not find a keyword match—an unfortunate and known limitation of voice assistants [16, 59, 73].”
BackgroundAdomavičius, G., & Kwon, Y. (2011) · ACM Conference on Recommender Systems (RecSys) · Cited by 209
Svanaes, D., & Seland, G. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985753 · Cited by 198 · cited 1×
“For our role-based study, we employed a role-playing methodology [75], prompting our participants to explore data visualizations from different perspectives.”
MethodLogarithmic Transformations in ANOVA
Berry, D. A. (1987) · Biometrics · doi:10.2307/2531826 · Cited by 195 · cited 1×
“Therefore, we applied a logarithmic transformation before conducting our analysis, following standard practice for time measures [5, 38, 50].”
MethodRemote usability evaluations with disabled people.
Petrie, H., Hamilton, F., King, N., & Pavan, P. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124942 · Cited by 157 · cited 1×
“We were unsuccessful at finding SRUs who were actual teachers or news reporters, as is often the case with recruiting disabled participants, particularly those with specialties [65].”
BackgroundAccessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content
Lundgard, A., & Satyanarayan, A. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114770 · Cited by 155
WebinSitu: a comparative analysis of blind and sighted browsing behavior
Bigham, J. P., Cavender, A. C., Brudvik, J. T., Wobbrock, J. O., & Ladner, R. E. (2007) · Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1296843.1296854 · Cited by 153 · cited 1×
“Several factors contribute to the difference in interaction times between SRUs and non-SRUs [7, 70], including the duration of the auditory responses.”
BackgroundSelf-citationLearning non-visual graphical information using a touch-based vibro-audio interface
Giudice, N. A., Palani, H. P., Brenner, E., & Kramer, K. M. (2012) · Proceedings of the 14th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/2384916.2384935 · Cited by 133 · cited 1×
“1 Several prior works have attempted to improve the accessibility of online data visualizations [2, 30, 43, 44, 69, 70, 72], such as by”
BackgroundTransformations: An Introduction and a Bibliography
Hoyle, M. H. (1973) · International Statistical Review · doi:10.2307/1402836 · Cited by 127 · cited 1×
“Therefore, we applied a logarithmic transformation before conducting our analysis, following standard practice for time measures [5, 38, 50].”
MethodData Sonification for Users with Visual Impairment: A Case Study with Georeferenced Data
Zhao, H., Plaisant, C., Shneiderman, B., & Lazar, J. (2008) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1352782.1352786 · Cited by 120 · cited 1×
“Several researchers have developed solutions to enhance the experiences of SRUs in extracting information from data visualizations, including auto-generation of alt-text [44, 53, 58, 69], sonification [2, 4, 13, 26, 36, 57, 83], data summarization [43], tables [21], andmulti-modality [70].”
BackgroundAnswering Questions about Charts and Generating Visual Explanations
Kim, D. H., Hoque, E., & Agrawala, M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376467 · Cited by 120 · cited 1×
“We finalized the question categories for our task-based user study based on Brehmer and Munzner’s task topology [11] and prior work on the type of questions users ask of graphs [10, 42, 68].”
MethodOver the Shoulder Learning: Supporting Brief Informal Learning
Twidale, M. B. (2005) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-005-9007-7 · Cited by 116
Cross-Modal Equivalence of Visual and Auditory Scatterplots for Exploring Bivariate Data Samples
Flowers, J. H., Buhman, D. C., & Turnage, K. D. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778827151 · Cited by 107 · cited 1×
“Several researchers have developed solutions to enhance the experiences of SRUs in extracting information from data visualizations, including auto-generation of alt-text [44, 53, 58, 69], sonification [2, 4, 13, 26, 36, 57, 83], data summarization [43], tables [21], andmulti-modality [70].”
BackgroundVizTouch: automatically generated tactile visualizations of coordinate spaces
Brown, C., & Hurst, A. (2012) · Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction · doi:10.1145/2148131.2148160 · Cited by 106 · cited 1×
“Prior research has also developed solutions to make data visualizations accessible through other user interfaces, such as haptic graphs [78, 81] and 3-D printing [12, 40, 72].”
MethodRich Screen Reader Experiences for Accessible Data Visualization
Zong, J., Lee, C., Lundgard, A., Jang, J., Hajas, D., & Satyanarayan, A. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14519 · Cited by 89 · cited 1×
“Prior work has reported that alternative textual descriptions (“alt-text”) for visualizations are often missing [1, 3].”
BackgroundVoxLens: Making Online Data Visualizations Accessible with an Interactive JavaScript Plug-In
Sharif, A., Wang, O. H., Muongchan, A. T., Reinecke, K., & Wobbrock, J. O. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517431 · Cited by 86 · cited 10×
“1 Several prior works have attempted to improve the accessibility of online data visualizations [2, 30, 43, 44, 69, 70, 72], such as by”
BackgroundMethodResultInfluentialSelf-citationUnderstanding Screen-Reader Users’ Experiences with Online Data Visualizations
Sharif, A., Chintalapati, S. S., Wobbrock, J. O., & Reinecke, K. (2021) · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3441852.3471202 · Cited by 85 · cited 10×
“Prior research has identified the need for accessible online data visualizations, shedding light on the disenfranchisement caused by inaccessible visualizations for SRUs [45, 52, 55, 68].”
BackgroundMethodInfluentialSelf-citationInclusive data visualization for people with disabilities: a call to action
Marriott, K., Lee, B., Butler, M., Cutrell, E., Ellis, K., Goncu, C., Hearst, M., McCoy, K., & Szafir, D. A. (2021) · Interactions · doi:10.1145/3457875 · Cited by 83 · cited 4×
“In contrast, non-screen-reader users can explore data visualizations by rapidly interpreting visual patterns [1, 28, 55].”
BackgroundInfluentialExtraordinary computing: religion as a lens for reconsidering the home
Wyche, S. P., & Grinter, R. E. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518817 · Cited by 74
Tickers and Talker: An Accessible Labeling Toolkit for 3D Printed Models
Shi, L., Zelzer, I., Feng, C., & Azenkot, S. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858507 · Cited by 69 · cited 2×
“1 Several prior works have attempted to improve the accessibility of online data visualizations [2, 30, 43, 44, 69, 70, 72], such as by”
BackgroundMethodHow accessible is my visualization? Evaluating visualization accessibility with Chartability
Elavsky, F., Bennett, C., & Moritz, D. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14522 · Cited by 68 · cited 1×
“Researchers have recommended techniques to improve the accessibility of online data visualizations [17, 22, 52, 62, 68, 74].”
MethodSociotechnical Considerations for Accessible Visualization Design
Lundgard, A., Lee, C., & Satyanarayan, A. (2019) · 2019 IEEE Visualization Conference (VIS) · doi:10.1109/visual.2019.8933762 · Cited by 67 · cited 3×
“[52] presented a set of sociotechnical considerations for accessible visualization designs, identifying participatory design and the usage of Accessible Rich Internet Application (ARIA) attributes as crucial elements in creating online data visualizations.”
BackgroundMethodVisualizing Ranges over Time on Mobile Phones: A Task-Based Crowdsourced Evaluation
Brehmer, M., Lee, B., Isenberg, P., & Choe, E. K. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865234 · Cited by 57 · cited 1×
“We finalized the question categories for our task-based user study based on Brehmer and Munzner’s task topology [11] and prior work on the type of questions users ask of graphs [10, 42, 68].”
MethodDisenfranchising the Disabled: The Inaccessibility of Internet-Based Health Information
Davis, J. J. (2002) · Journal of Health Communication · doi:10.1080/10810730290001701 · Cited by 51 · cited 1×
“However, the essential visual nature of data visualizations inherently disenfranchises people who cannot see [20, 47, 55, 68].”
BackgroundevoGraphs — A jQuery plugin to create web accessible graphs
Sharif, A., & Forouraghi, B. (2018) · 2018 15th IEEE Annual Consumer Communications & Networking Conference (CCNC) · doi:10.1109/ccnc.2018.8319239 · Cited by 42 · cited 3×
“Several researchers have developed solutions to enhance the experiences of SRUs in extracting information from data visualizations, including auto-generation of alt-text [44, 53, 58, 69], sonification [2, 4, 13, 26, 36, 57, 83], data summarization [43], tables [21], andmulti-modality [70].”
BackgroundSelf-citationTuarob, S., Wettayakorn, P., Phetchai, P., Traivijitkhun, S., Lim, S., Noraset, T., & Thaipisutikul, T. (2021) · Financial Innovation · doi:10.1186/s40854-021-00269-7 · Cited by 42
Grosjean, S., Ciocca, J.-L., Gauthier-Beaupré, A., Poitras, E., Grimes, D., & Mestre, T. (2022) · Digital Health · doi:10.1177/20552076221081695 · Cited by 41 · cited 1×
“Recent work in politics [46, 84], health [32, 63, 85], finance [24, 76], and business analytics [6, 82] indicate the importance and wide adoption of online data visualizations.”
BackgroundSoundBar: exploiting multiple views in multimodal graph browsing
McGookin, D. K., & Brewster, S. A. (2006) · Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles · doi:10.1145/1182475.1182491 · Cited by 41 · cited 1×
“Several researchers have developed solutions to enhance the experiences of SRUs in extracting information from data visualizations, including auto-generation of alt-text [44, 53, 58, 69], sonification [2, 4, 13, 26, 36, 57, 83], data summarization [43], tables [21], andmulti-modality [70].”
BackgroundMultimodal Deep Learning using Images and Text for Information Graphic Classification
Kim, E., & McCoy, K. F. (2018) · Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3234695.3236357 · Cited by 39 · cited 3×
“Several researchers have developed solutions to enhance the experiences of SRUs in extracting information from data visualizations, including auto-generation of alt-text [44, 53, 58, 69], sonification [2, 4, 13, 26, 36, 57, 83], data summarization [43], tables [21], and multi-modality [70].”
BackgroundUsing Web Standards to Design Accessible Data Visualizations in Professional Communication
Strantz, A. (2021) · IEEE Transactions on Professional Communication · doi:10.1109/tpc.2021.3091784 · Cited by 37 · cited 1×
“Researchers have recommended techniques to improve the accessibility of online data visualizations [17, 22, 52, 62, 68, 74].”
MethodNative voice, self-concept and the moral case for personalized voice technology
Nathanson, E. (2016) · Disability and Rehabilitation · doi:10.3109/09638288.2016.1139193 · Cited by 32
Towards accessible graphs in HTML-based scientific articles
Mirri, S., Peroni, S., Salomoni, P., Vitali, F., & Rubano, V. (2017) · 2017 14th IEEE Annual Consumer Communications & Networking Conference (CCNC) · doi:10.1109/ccnc.2017.7983287 · Cited by 29 · cited 2×
“Several researchers have developed solutions to enhance the experiences of SRUs in extracting information from data visualizations, including auto-generation of alt-text [44, 53, 58, 69], sonification [2, 4, 13, 26, 36, 57, 83], data summarization [43], tables [21], andmulti-modality [70].”
BackgroundVitro: Designing a Voice Assistant for the Scientific Lab Workplace
Cambre, J., Liu, Y., Taylor, R. E., & Kulkarni, C. E. (2019) · Designing Interactive Systems Conference · doi:10.1145/3322276.3322298 · Cited by 29 · cited 1×
“Our system does not process the user’s query when our algorithm does not find a keyword match—an unfortunate and known limitation of voice assistants [16, 59, 73].”
BackgroundPost-WIMP Interaction for Information Visualization
Lee, B., Srinivasan, A., Isenberg, P., & Stasko, J. (2021) · Foundations and Trends® in Human–Computer Interaction · doi:10.1561/1100000081 · Cited by 29
"Play PRBLMS": Identifying and Correcting Less Accessible Content in Voice Interfaces
Springer, A., & Cramer, H. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173870 · Cited by 27
Experiences of using role playing andwiki in requirements engineering course projects
Liang, P., & de Graaf, O. (2010) · doi:10.1109/reet.2010.5633115 · Cited by 26 · cited 1×
“Building on and following recommendations from prior work [39, 49, 56], we identified three roles that provided in-depth perspectives: (1) explorer ; (2) teacher; and (3) news reporter.”
BackgroundTowards More Accessible Visualizations for Color-Vision-Deficient Individuals
Oliveira, M. M. (2013) · Computing in Science & Engineering · doi:10.1109/mcse.2013.113 · Cited by 26 · cited 1×
“Researchers have recommended techniques to improve the accessibility of online data visualizations [17, 22, 52, 62, 68, 74].”
MethodBusiness Intelligence for IT Governance of a Technology Company
Biagi, V., Patriarca, R., & Gravio, G. D. (2021) · Data · doi:10.3390/data7010002 · Cited by 20
Sharif, A., Zhang, A. M., Shih, A., Wobbrock, J. O., & Reinecke, K. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3550363 · Cited by 20
Self-citationUser Modelling for Personalized Question Answering
Quarteroni, S., & Manandhar, S. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74782-6_34 · Cited by 18 · cited 1×
“Therefore, we recommend using personalized designs [60, 66] by identifying usage patterns and individualized preferences of users to improve overall performance.”
MethodDesigning Accessible Visualizations: The Case of Designing a Weather Map for Blind Users
Carroll, D., Chakraborty, S., & Lazar, J. K. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39188-0_47 · Cited by 17
Multimodal Exploration of Mathematical Function Graphs with AudioFunctions.web
Ahmetovic, D., Cantù, N., Bernareggi, C., Guerreiro, J., Mascetti, S., & Capietto, A. (2019) · Proceedings of the 16th International Web for All Conference · doi:10.1145/3315002.3332438 · Cited by 16 · cited 2×
“Several researchers have developed solutions to enhance the experiences of SRUs in extracting information from data visualizations, including auto-generation of alt-text [44, 53, 58, 69], sonification [2, 4, 13, 26, 36, 57, 83], data summarization [43], tables [21], andmulti-modality [70].”
BackgroundInformation Graphic Summarization using a Collection of Multimodal Deep Neural Networks
Kim, E., Onweller, C., & McCoy, K. F. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412146 · Cited by 14
Zhao, L., & Ye, W. (2022) · Convergence The International Journal of Research into New Media Technologies · doi:10.1177/13548565211069872 · Cited by 13 · cited 1×
“Recent work in politics [46, 84], health [32, 63, 85], finance [24, 76], and business analytics [6, 82] indicate the importance and wide adoption of online data visualizations.”
BackgroundAccessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content
Lundgard, A., & Satyanarayan, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04406 · Cited by 11 · cited 1×
“Several researchers have developed solutions to enhance the experiences of SRUs in extracting information from data visualizations, including auto-generation of alt-text [44, 53, 58, 69], sonification [2, 4, 13, 26, 36, 57, 83], data summarization [43], tables [21], andmulti-modality [70].”
BackgroundZheng, S., Edwards, J. R., Dudeck, M. A., Patel, P. R., Wattenmaker, L., Mirza, M., Tejedor, S. C., Lemoine, K., Benin, A. L., & Pollock, D. A. (2021) · JMIR Public Health and Surveillance · doi:10.2196/23528 · Cited by 10
Hurst, A. K. (2018) · International Web for All Conference (W4A) · doi:10.1145/3192714.3192715 · Cited by 8 · cited 1×
“Prior research has also developed solutions to make data visualizations accessible through other user interfaces, such as haptic graphs [78, 81] and 3-D printing [12, 40, 72].”
MethodRaising the Issue of Ethics through Use of Scenarios
Hunt, T. (1982) · Journalism & Mass Communication Educator · doi:10.1177/107769588203700116 · Cited by 7 · cited 1×
“Building on and following recommendations from prior work [39, 49, 56], we identified three roles that provided in-depth perspectives: (1) explorer ; (2) teacher ; and (3) news reporter .”
BackgroundEr, X., & Sun, Y. (2021) · International Conference on Machine Learning and Intelligent Systems Engineering (MLISE) · doi:10.1109/mlise54096.2021.00057 · Cited by 7 · cited 1×
“Recent work in politics [46, 84], health [32, 63, 85], finance [24, 76], and business analytics [6, 82] indicate the importance and wide adoption of online data visualizations.”
BackgroundBibliometric Visualisation of Computer Science and COVID-19: A Review and Proposed Method
Patrick, R. R., & Junaini, S. N. (2021) · IEEE Student Conference on Research and Development (SCOReD) · doi:10.1109/scored53546.2021.9652741 · Cited by 2
Review of Qualitative Evaluation and Research Methods (2nd ed.).
indicated, N. A. (1991) · Contemporary Psychology: A Journal of Reviews · doi:10.1037/031327
Hart (1991) · doi:10.1002/1097-4679(199107)47:4
The Analysis of Binomial Data by a Generalized Linear Mixed Model
Gilmour, A. R., Anderson, R. D., & Rae, A. L. (1985) · Biometrika · doi:10.2307/2336731
Accessible data visualization in higher education
Konecki, M., LaPierre, C., & Jervis, K. (2018) · 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO) · doi:10.23919/mipro.2018.8400136
Research works printed but not linked: 7
- Lorna M. Brown , Stephen A. Brewster , Ramesh Ramloll , Mike Burton , and Beate Riedel . 2003 . Design guidelines for audio presentation of graphs and tables . In Proceedings of the 9th International Conference on Auditory Display. Citeseer , Boston University, USA, 284–287. Lorna M. Brown, Stephen A. Brewster, Ramesh Ramloll, Mike Burton, and Beate Riedel. 2003. Design guidelines for audio presentation of graphs and tables. In Proceedings of the 9th International Conference on Auditory Display. Citeseer, Boston University, USA, 284–287.
- Brigitte N. Frederick . 1999. Fixed- , random-, and mixed-effects ANOVA models: A user-friendly guide for increasing the generalizability of ANOVA results . In Advances in Social Science Methodology, B. Thompson (Ed.) . JAI Press , Stamford, Connecticut , 111–122. Brigitte N. Frederick. 1999. Fixed-, random-, and mixed-effects ANOVA models: A user-friendly guide for increasing the generalizability of ANOVA results. In Advances in Social Science Methodology, B. Thompson (Ed.). JAI Press, Stamford, Connecticut, 111–122.
- Melita Hajdinjak and France Mihelic . 2004. Conducting the Wizard-of-Oz Experiment.Informatica (Slovenia) 28, 4 ( 2004 ), 425–429. Melita Hajdinjak and France Mihelic. 2004. Conducting the Wizard-of-Oz Experiment.Informatica (Slovenia) 28, 4 (2004), 425–429.
- Jake Holland . 2017. New York Times ’ Upshot editor discusses data visualization, storytelling. https://dailynorthwestern.com/ 2017 /05/03/campus/new-york-times-upshot-editor-discusses-data-visualization-storytelling/. ( Accessed on 03/05/2022). Jake Holland. 2017. New York Times’ Upshot editor discusses data visualization, storytelling. https://dailynorthwestern.com/2017/05/03/campus/new-york-times-upshot-editor-discusses-data-visualization-storytelling/. (Accessed on 03/05/2022).
- Michal Kubovics and Pavel Bielik . 2021 . Visualization of Data And Keywords in Online Journalism . Marketing Identity 9 , 1 (2021), 121 – 133 . Michal Kubovics and Pavel Bielik. 2021. Visualization of Data And Keywords in Online Journalism. Marketing Identity 9, 1 (2021), 121–133.
- Ather Sharif , Andrew M Zhang , Anna Shih , Jacob O Wobbrock , and Katharina Reinecke . 2022 . Understanding and Improving Information Extraction From Online Geospatial Data Visualizations for Screen-Reader Users. In The 24th International ACM SIGACCESS Conference on Computers and Accessibility ( Athens, Greece) (ASSETS ’22). Association for Computing Machinery, New York, NY, USA, To appear pages. Ather Sharif, Andrew M Zhang, Anna Shih, Jacob O Wobbrock, and Katharina Reinecke. 2022. Understanding and Improving Information Extraction From Online Geospatial Data Visualizations for Screen-Reader Users. In The 24th International ACM SIGACCESS Conference on Computers and Accessibility (Athens, Greece) (ASSETS ’22). Association for Computing Machinery, New York, NY, USA, To appear pages.
- 10.5555/645443.652687 · doi:10.5555/645443.652687
Cited by
GenAssist: Making Image Generation Accessible
Huh, M., Peng, Y.-H., & Pavel, A. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606735 · Cited by 67
“However, both text-to-image generation and GenAssist do not yet support images that are information-rich or densely structured such as information visualizations [64, 65] or diagrams [3, 66].”
BackgroundSeo, J., Kamath, S. S., Zeidieh, A., Venkatesh, S., & McCurry, S. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675660 · Cited by 25
“Future iterations were made to VoxLens to improve the drill-down information extraction from data visualizations by screen reader users [36], however this implementation does not extend to visualizations like box plots, scatter plots, and heatmaps.”
ChartA11y: Designing Accessible Touch Experiences of Visualizations with Blind Smartphone Users
Zhang, Z., Thompson, J. R., Shah, A., Agrawal, M., Sarıkaya, A., Wobbrock, J. O., Cutrell, E. B., & Lee, B. (2024) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3675611 · Cited by 37
Self-citationMatplotAlt: A Python Library for Adding Alt Text to Matplotlib Figures in Computational Notebooks
Nylund, K., Mankoff, J. C., & Potluri, V. (2025) · Computer Graphics Forum · doi:10.1111/cgf.70119 · Cited by 17
“Chart summaries provide an overall picture of Matplotlib figures, but consumers of these artifacts may also want to drill down to the underlying data [SCWR21,SZRW23].”
Conveying Uncertainty in Data Visualizations to Screen-Reader Users Through Non-Visual Means
Sharif, A., Zhong, R., & Wang, Y. (2023) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3597638.3614502 · Cited by 13
Self-citation
Metadata
- Authors
- Sharif, A., Zhang, A. M., Reinecke, K., & Wobbrock, J. O.
- Affiliations
- University of Washington (US)
- Venue
- 20th International Web for All Conference
- Publisher
- ACM
- Published
- 2023-04-30
- Pages
- 18-31
- OpenAlex
- W4366762618
- Semantic Scholar
- bc044f00f72063821d24ccb069b2114c04c07fcb
- Cited by
- 21
- References
- 53
- Citations by year
- 2026: 5 · 2025: 8 · 2024: 6 · 2023: 2
- FWCI
- 8.79
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Web for All Conference (W4A) (publishedVersion, cc-by)
- Funders
- University of Washington
- Topics
- Social Sciences › Human Factors and Ergonomics › Digital Accessibility for Disabilities
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Tactile and Sensory Interactions
- Keywords
- multimodal accessibility, alternative text, web accessibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.