Towards More Accessible Scientific PDFs for People with Visual Impairments: Step-by-Step PDF Remediation to Improve Tag Accuracy

Probably not a substantial claim: may not pass S3 (load-bearing).

All participants would recommend PAVE 2.0 for novice users.

“Overall, fifteen participants stated that they would prefer to use PAVE 2.0 in the future, and all participants would recommend it for novice users.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors overlooked a problematic red-green color combination in Step 2, which a participant highlighted later.

“However, we overlooked a problematic redgreen color combination in Step 2 which a participant highlighted later.”

May combine two claims.

Experienced and novice users could improve the auto-tagging results by 23.2 pp. and 18.3 pp. respectively through manual correction using PAVE 2.0.

“Additionally, our results showed that experienced and novice users could improve the auto-tagging results by 23.2 pp. and 18.3 pp.respectively through manual correction using PAVE 2.0.”

May combine two claims.

Using Adobe Acrobat Pro, the first page had an incorrect reading order in 17 out of 19 PDFs, representing 25% of experienced users' errors and 80% of novice users' errors.

“In contrast, using Adobe Acrobat Pro, the first page had an incorrect reading order in 17 out of 19 PDFs, representing 25% of experienced users' errors and 80% of novice users' errors.”

May combine two claims.

The AI model reaches an Edit score of 97.2% on the im2latexv2 test dataset, indicating an average error rate of 3 symbols per 100 symbols in an equation.

“The model reaches an Edit score of 97.2% on the im2latexv2 test dataset [34], indicating an average error rate of 3 symbols per 100 symbols in an equation.”

• cites [34]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

PAVE 2.0's auto-tagging reaches an average score of 56.9%, 19.4 pp. higher than Adobe Acrobat Pro.

“The auto-tagging results show that PAVE 2.0's auto-tagging reaches an average score of 56.9%, 19.4 pp.higher than Adobe Acrobat Pro.”

Classified as probably not a substantial claim.

Our tool is designed for scientific documents.

“Furthermore, as our tool is designed for scientific documents, we specifically targeted individuals with a scientific and academic background.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

PAVE 2.0's mathematical formula recognition works reliably for single-line mathematical expressions but shows weaknesses for single mathematical symbols or multi-line mathematical expressions.

“The mathematical formula recognition works reliably for single-line mathematical expressions but shows weaknesses for single mathematical symbols or multi-line mathematical expressions.”

May combine two claims.

PAVE 2.0's auto-detecting cannot detect captions, footnotes, and other infrequently used content types, and typically tags them as paragraphs.

“Due to the limitation of classes in the training dataset, it cannot detect captions, footnotes, and other infrequently used content types, and typically tags them as paragraphs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We recruited 19 participants for our study.

“We recruited 19 participants for our study.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Adobe Acrobat Pro's auto-tagging function often fails to detect inline headings and the correct heading level.

“Adobe Acrobat Pro's auto-tagging function often fails to detect inline headings and the correct heading level.”

May combine two claims.

The scores for participants who reached Step 7 were 87.5% for novice users and 100% for experienced users.

“The scores for participants who reached Step 7 were 87.5% for novice users and 100% for experienced users.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Participant location distribution: eleven were in Switzerland, five in Austria, one in Germany, and two in Spain.

“Eleven participants were located in Switzerland, five were in Austria, one was in Germany, and two were in Spain.”

May combine two claims.

The first page was incorrect in 15 of 19 PDFs remediated with PAVE 2.0, accounting for 78% of errors among experienced users and 38% of errors among novice users.

“Hence, the first page was incorrect in 15 of 19 PDFs remediated with PAVE 2.0, accounting for 78% of errors among experienced users and 38% of errors among novice users.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Adobe Acrobat Pro's auto-tagging usually detects the list structure very well, but the list tag sometimes does not contain the complete list.

“While the list structure is usually detected very well, the list tag sometimes does not contain the complete list.”

May combine two claims.

The difference between experienced and novice users is relatively small for both tools, averaging a 4.9 pp. difference with PAVE 2.0 and an average difference of 2.8 pp. with Adobe Acrobat Pro.

“However, the difference between experienced and novice users is relatively small for both tools, averaging a 4.9 pp.difference with PAVE 2.0 and an average difference of 2.8 pp. with Adobe Acrobat Pro.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participant gender distribution: thirteen identified as female, four as male, one as non-binary, and one did not disclose their gender.

“Thirteen participants identified as female, four as male, one as non-binary, and one did not disclose their gender.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Participant age distribution: one was aged 18-30, nine were 31-40, six were 41-50, and three were 51-60.

“One of the participants was aged between 18 and 30, nine were aged between 31 and 40, six were aged between 41 and 50, and three were aged between 51 and 60.”

Probably not a substantial claim: may not pass S3 (load-bearing).

CHI papers also contained most headings per page with 3.16 (ASSETS: 3.11, ICCHP: 2.08).

“CHI papers also contained most headings per page with 3.16 (ASSETS: 3.11, ICCHP: 2.08).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

PAVE 2.0's auto-detecting of regions works reliably for scientific documents such as conference papers, but shows weaknesses with non-scientific documents.

“We observed that the auto-detecting of the regions in PAVE 2.0 works reliably for scientific documents, such as conference papers, but shows weaknesses with non-scientific documents.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

If tables are recognized correctly by Adobe Acrobat Pro, the table structure is recognized often very well.

“Similarly, if tables are recognized correctly, the table structure is recognized often very well.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two participants did not like that the grey boxes overlapped leaving them unsure about whether they had selected the correct text.

“Two participants did not like that the grey boxes overlapped leaving them unsure about whether they had selected the correct text.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

On average, Menzies et al. observed 2.0 tables per paper (1.7 in our sample), 9.2 figures per paper (5.0 in our sample), and 0.3 formulas per paper (0.1 in our sample).

“On average, they observed 2.0 tables per paper (1.7 in our sample), 9.2 figures per paper (5.0 in our sample), and 0.3 formulas per paper (0.1 in our sample).”

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The component counts in our sample differ slightly from those reported by Menzies et al. for 330 ASSETS papers between 2011 and 2020.

“The component counts in our sample differ slightly from those reported by Menzies et al. [21] for 330 ASSETS papers between 2011 and 2020.”

• cites [21]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

In some papers, the caption tag is nested within the figure tag, and in others, it appears after the figure tag.

“We also observed that in some papers, the caption tag is nested within the figure tag, and in others, it appears after the figure tag.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two participants were disappointed that lists could not be selected.

“However, two participants were disappointed that lists could not be selected.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The other 2 participants would consider using the tool, but wanted to test it further before making a final decision.

“The other 2 participants would consider using the tool, but wanted to test it further before making a final decision.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Three participants reported that they liked the reading order tool.

“Three participants reported that they liked the reading order tool.”

May combine two claims.

Experienced users spent 15 minutes and 3 seconds for the first four steps, which is around 2 minutes longer than with the Ally tool.

“However, experienced users spent 15 minutes and 3 seconds for the first four steps, which is around 2 minutes longer than with the Ally tool.”

May combine two claims.

The average score indicates that the conference papers are slightly more accessible compared to the results with Adobe Acrobat Pro, but clearly lower than with PAVE 2.0.

“The average score indicates that the conference papers are slightly more accessible compared to the results with Adobe Acrobat Pro, but clearly lower than with PAVE 2.0.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two participants stated they would prefer to use Adobe Acrobat Pro in the future.

“The last two participants stated they would prefer to use Adobe Acrobat Pro in the future.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Adobe Acrobat Pro's user interface lacks intuitiveness.

“Similar to previous studies [5,16,32], we found that the weaknesses of Adobe Acrobat Pro are the unclear workflow and the lack of intuitiveness of the user interface.”

• cites [5]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [16]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [32]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Only one participant did not add an alternative text for the tagged image.

“However, only one of the participants did not add an alternative text for the tagged image.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Formulas are the least popular component (CHI: 0.11 formulas per page, ICCHP: 0.04, and ASSETS: 0.01).

“Formulas instead are the least popular component (CHI: 0.11 formulas per page, ICCHP: 0.04, and ASSETS: 0.01).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Nine participants reported that they liked the report and it gave them some guidance.

“Nine participants reported that they liked the report and it gave them some guidance.”

May combine two claims.

Novice users spent 13 minutes and 21 seconds for the first four steps, which is similar to previous findings regarding the four subtasks of the Ally prototype.

“Novice users spent 13 minutes and 21 seconds for the first four steps, which is similar to previous findings regarding the four subtasks of the Ally prototype (the Ally prototype only has these four subtasks in its remediation process).”

May combine two claims.

All five participants who finished before the 20-minute time limit with Adobe Acrobat Pro mentioned that the PDF was probably not completely accessible, but they did not know how to fix the remaining issues.

“However, all five participants mentioned that the PDF was probably not completely accessible, but they did not know how to fix the remaining issues.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ASSETS papers contained more lists per page (0.21) than CHI papers (0.17) and ICCHP papers (0.10), while for the other component types CHI papers contained the most components per page.

“Interestingly, ASSETS papers contained more lists per page (0.21) than CHI papers (0.17) and ICCHP papers (0.10), while for the other component types CHI papers contained the most components per page.”

May combine two claims.

Eighteen participants found PAVE 2.0 simpler, while one participant preferred the usability of Adobe Acrobat Pro.

“Regarding ease of use, eighteen participants found PAVE 2.0 simpler, while one participant preferred the usability of Adobe Acrobat Pro.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One of the four lists was incorrectly tagged.

“Additionally, ten participants did not recognize that one of the four lists was incorrectly tagged and that the table was tagged as a paragraph by the automatic tagging.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Figures are the most popular component of the four components (tables, lists, figures, and formulas) for all conferences (CHI: 0.61 figures per page, ICCHP: 0.52, and ASSETS: 0.4).

“Figures are the most popular component of the four components (tables, lists, figures, and formulas) for all conferences (CHI: 0.61 figures per page, ICCHP: 0.52, and ASSETS: 0.4).”

Probably not a substantial claim: may not pass S3 (load-bearing).

One participant had to give up because Adobe Acrobat Pro stopped working multiple times.

“One participant had to give up because Adobe Acrobat Pro stopped working multiple times.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Three participants stated they would not use the tool again.

“Nevertheless, sixteen participants reported they would use the tool again, while three stated they would not.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Fourteen participants used the Adobe accessibility report as a guide after we had introduced it to them.

“Fourteen participants used the Adobe accessibility report as a guide after we had introduced it to them.”

May combine two claims.

The table, captions, some headings, parts of lists, and mathematical formulas were often tagged as paragraphs.

“As a result, the table, captions, some headings, parts of lists, and mathematical formulas were often tagged as paragraphs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two participants stated their preference would depend on the specific PDF, alternating between Adobe Acrobat Pro and PAVE 2.0.

“Two participants stated their preference would depend on the specific PDF, alternating between Adobe Acrobat Pro and PAVE 2.0.”

May combine two claims.

Novice users were quicker with PAVE 2.0, while experienced users spent more time with PAVE 2.0 than with Adobe Acrobat Pro.

“Novice users were quicker with PAVE 2.0, while experienced users spent more time with PAVE 2.0 than with Adobe Acrobat Pro.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The table was tagged as a paragraph by the automatic tagging.

“Additionally, ten participants did not recognize that one of the four lists was incorrectly tagged and that the table was tagged as a paragraph by the automatic tagging.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The evaluation shows the great value ASSETS and CHI put on tagging figures with alt text, while the ICCHP papers never included alternative text.

“Furthermore, the evaluation shows the great value ASSETS and CHI put on tagging figures with alt text, while the ICCHP papers never included alternative text.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Sixteen participants responded that they believed PAVE 2.0 could enhance their current or future work with accessible PDFs.

“Sixteen participants responded that they believed PAVE 2.0 could enhance their current or future work with accessible PDFs.”

May combine two claims.

The tag accuracy and tagging strategy vary greatly from paper to paper for all conferences.

“We observed that the tag accuracy and tagging strategy vary greatly from paper to paper for all conferences.”

May combine two claims.

Novice users were faster and needed fewer instructions than experienced users with PAVE 2.0.

“Interestingly, novice users were faster and needed fewer instructions than experienced users with PAVE 2.0.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Sixteen participants reported they would use the tool again.

“Nevertheless, sixteen participants reported they would use the tool again, while three stated they would not.”

May combine two claims.

Novice users spent on average 5 minutes and 48 seconds per page, while experienced users spent 6 minutes and 35 seconds in our study with PAVE 2.0.

“Novice users spent on average 5 minutes and 48 seconds per page, while experienced users spent 6 minutes and 35 seconds in our study with PAVE 2.0.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The math editor allows straightforward creation and modification of a mathematical formula's alternative text, even for novice users.

“Furthermore, we developed a math editor, allowing straightforward creation and modification of a mathematical formula's alternative text, even for novice users.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The user study participants all had a scientific or academic background and were interested in accessible PDFs.

“The user study participants all had a scientific or academic background and were interested in accessible PDFs, which limits the generalization of the answers.”

Classified as probably not a substantial claim. May combine two claims.

We presented thirteen criteria and a score to manually evaluate tag accuracy in a PDF, addressing aspects that are not fully covered by existing accessibility checkers.

“Additionally, we presented thirteen criteria and a score to manually evaluate the tag accuracy in a PDF, addressing aspects that are not fully covered by existing accessibility checkers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Two participants reported finding the PDF remediation process fun and enjoyable.

“Two participants even reported finding our PDF remediation process fun and enjoyable.”

Classified as probably not a substantial claim.

We developed a PDF remediation process with eight steps.

“Inspired by Pradhan et al. [32], we developed a PDF remediation process with eight steps.”

• cites [32]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

PDF remediation is valuable for making new PDFs accessible at time of their creation, but also for potentially making the vast quantity of inaccessible PDFs already in existence accessible to individuals who use screen readers as well.

“This process is therefore valuable for making new PDFs accessible at time of their creation, but also for potentially making the vast quantity of inaccessible PDFs already in existence accessible to individuals who use screen readers as well.”

• AssumptionRests on an assumption: PDF remediation is applicable regardless of the process used to create the PDF originally.

Probably not a substantial claim: may not pass S3 (load-bearing).

Nganji et al. [27] found that only 15.5% of 200 articles from four journals between 2014 and 2018 contained tags.

“Nganji et al. [27] manually analyzed 200 articles from four journals between 2014 and 2018.They found that only 15.5% of the documents contained tags.”

• cites [27]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable).

There is a need for methods and tools that allow people to remediate PDFs with a high degree of accessibility.

“Hence, there is a need for methods and tools that allow people to remediate PDFs with a high degree of accessibility [5,16,32].”

• cites [5]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [16]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [32]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

A big part of the user frustration was that participants did not know what was auto-tagged by PAVE and how to override it.

“They found that a big part of the user frustration was that the participants did not know what was auto-tagged by PAVE and how to override it.”

• cites [16]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

An accessible PDF can be created in two ways.

“An accessible PDF can be created in two ways.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The major improvement of Ally was that the user no longer interacted directly with the PDF structure tree.

“The major improvement to existing tools was that the user no longer interacted directly with the PDF structure tree which represents the logical structure of the content of a PDF.”

• cites [32]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Ally splits the process into multiple logical subtasks to give the user more guidance.

“Instead, Ally splits the process into multiple logical subtasks to give the user more guidance.”

• cites [32]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The prototype is not fully accessible, particularly the drawing functionalities.

“While the prototype is not fully accessible (particularly the drawing functionalities) to streamline development”

Probably not a substantial claim: may not pass S3 (load-bearing).

Wang et al. [45] developed SciA11y, a method to transform a PDF into accessible HTML.

“For instance, Wang et al. [45] developed SciA11y, a method to transform a PDF into an accessible HTML.”

• cites [45]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Conversion errors occur and are not detected in PDF-to-HTML conversion methods like SciA11y.

“However, conversion errors do occur and are not detected which is problematic.”

• cites [45]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Many other tools besides SciA11y can convert PDFs into HTML, as shown by Pathirana et al. [28].

“Besides SciA11y, there are many other tools available that can convert PDFs into HTML, as shown in the evaluation by Pathirana et al. [28].”

• cites [28]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The user interface was designed with accessibility as a priority, e.g., using a high-contrast color palette.

“It was designed with accessibility as a priority, e.g., using a highcontrast color palette.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Drye et al. [10] found that half of participants did not know how to create an accessible document.

“Additionally, half of the participants did not know how to create an accessible document.”

• cites [10]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Current processes for making PDFs accessible are time-consuming and error-prone.

“However, the current processes for making PDFs accessible are still time-consuming and error-prone, and require expert knowledge [16], challenges that many authors have likely experienced.”

• cites [16]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Pradhan et al. developed the Ally prototype to improve the user experience of remediation tools using best practices from HCI research.

“Pradhan et al. [32] developed the Ally prototype to improve the user experience of remediation tools using best practices from HCI research.”

• cites [32]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

Four of these steps (Region, Reading Order, Heading Structure, and Meta Information) are necessary for every PDF, regardless of its content.

“Four of these steps (Region, Reading Order, Heading Structure, and Meta Information) are necessary for every PDF, regardless of its content.”

May combine two claims.

This work incorporates the findings of Pradhan et al. and extends it to a complete AI-supported PDF remediation process.

“This work incorporates the findings of Pradhan et al. and extends it to a complete AI-supported PDF remediation process.”

• cites [32]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

2.4% of the US population is not able to easily read most scientific PDFs.

“Consequently, 2.4% of the US population is not able to easily read most scientific PDFs [25].”

• cites [25]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Bigham et al. [5] found that current tools for making conference PDFs accessible are too complicated and not usable for inexperienced users.

“They highlighted that current tools for this task are too complicated and not usable for inexperienced users.”

• cites [5]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

PAVE was developed by Doblies et al. in 2014 and is the basis for PAVE 2.0.

“Doblies et al. [9] developed the web application PAVE in 2014 upon which our prototype (PAVE 2.0) is based.”

• cites [9]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The author has to follow specific guidelines when creating the document to ensure that it is exported correctly as an accessible PDF.

“However, not all document editors have this feature, and the author has to follow specific guidelines when creating the document to ensure that it is exported correctly as an accessible PDF.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A comparison study with Adobe Acrobat Pro found that a semi-automatic tool like PAVE is necessary to foster PDF accessibility.

“A comparison study with Adobe Acrobat Pro [16] revealed that a semi-automatic tool like PAVE is necessary to foster PDF accessibility”

• cites [16]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

PDF is the dominant format in STEM fields.

“This means that people with visual impairments cannot properly read most PDFs with screen readers, posing a significant problem -especially in STEM fields, where PDF is the dominant format [5].”

• cites [5]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Rajkumar et al. [16] found that most researchers and practitioners in STEM fields are unhappy with existing tools.

“Similarly, Rajkumar et al. [16] found that most researchers and practitioners in STEM fields are unhappy with the existing tools.”

• cites [16]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Drye et al. [10] found that one-third of participants could not define what accessibility of documents means.

“The study revealed that one-third of the participants could not define what accessibility of documents means.”

• cites [10]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The vast majority of research literature is still not accessible to everyone, despite recent improvements in the accessibility of scientific literature.

“Even though the accessibility of scientific literature has improved in recent years, the vast majority of research literature is still not accessible to everyone.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The step-by-step approach ensures a logical workflow and prevents the user from creating invalid tags or invalid nesting of tags in the PDF structure tree.

“This step-by-step approach ensures a logical workflow and prevents the user from creating invalid tags or invalid nesting of tags in the PDF structure tree.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The remaining four steps (Tables, Lists, Figures, and Mathematical Formulas) are content-dependent.

“The remaining four steps (Tables, Lists, Figures, and Mathematical Formulas) are content-dependent, reflecting their common presence in scientific PDFs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Darvishy et al. [8] found that only 11.5% of 2,500 papers from repositories of five German-speaking universities in Switzerland (2018-2022) contained tags.

“Darvishy et al. [8] investigated 2,500 papers in repositories of five German-speaking universities in Switzerland from 2018 to 2022 in a semi-automatic way and found that only 11.5% of the papers contained tags.”

• cites [8]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The first method of creating an accessible PDF involves exporting it directly from document editing software, provided that the software has an accessible PDF export option.

“The first method involves exporting an accessible PDF directly from the document editing software, provided that the software has an accessible PDF export option.”

Classified as probably not a substantial claim. May combine two claims.

Ally's prototype presented user interfaces for four subtasks (regions, headers, reading order, and tables), but not for the complete PDF remediation process.

“Their prototype presented user interfaces for four subtasks (regions, headers, reading order, and tables), but not for the complete PDF remediation process.”

• cites [32]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

PAVE at that time did not meet user expectations regarding intuitiveness or user experience.

“but PAVE at that time did not meet user expectations regarding intuitiveness or user experience.”

• cites [16]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

PAC allows users to analyze and reveal accessibility issues in a PDF, but these issues cannot be corrected directly in the tool.

“PAC allows users to analyze and reveal accessibility issues in a PDF but these cannot be corrected directly in the tool.”

• cites [42]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Hiding different aspects of the visualization reduces visual clutter and cognitive load.

“Depending on the step, the user can hide different aspects of the visualization to reduce visual clutter and cognitive load [41].”

• cites [41]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The idea of PAVE was to provide a simple, semi-automatic process for non-experts to remediate PDFs.

“The idea of PAVE was to provide a simple, semi-automatic process for non-experts to remediate PDFs.”

• cites [9]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Not all document editors have an accessible PDF export option.

“However, not all document editors have this feature, and the author has to follow specific guidelines when creating the document to ensure that it is exported correctly as an accessible PDF.”

May combine two claims.

The PDF accessibility analyses presented in Section 2 evaluate the accessibility of a PDF based mainly on metadata and machine-checkable criteria, but these methods do not reveal other concerns such as whether all headings are correctly tagged.

“The PDF accessibility analyses presented in Section 2 evaluate the accessibility of a PDF based mainly on metadata and machinecheckable criteria.However, these methods do not reveal other concerns, for example whether all headings are correctly tagged [18].”

• cites [18]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Thirteen manually checkable criteria were defined to evaluate the tag accuracy of a tagged PDF using the Matterhorn Protocol, the best-practice guide of the PDF Association, and the synthetic errors defined by Pradhan et al.

“Therefore, we defined thirteen manually checkable criteria to evaluate the tag accuracy of a tagged PDF using the Matterhorn Protocol [29], the best-practice guide of the PDF Association [30], and the synthetic errors defined by Pradhan et al.”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [30]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Pradhan et al.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The automatic region detection method ensures that all PDF operators are marked.

“This ensures that all PDF operators are marked.”

Classified as probably not a substantial claim. May combine two claims.

Automatic table recognition has become popular in recent years, but errors still occur.

“In recent years, automatic table recognition has become popular, but errors still occur [17].”

• cites [17]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

We compared the efficiency and effectiveness of our novel process and tool with the de facto standard PDF remediation tool Adobe Acrobat Pro.

“We compared the efficiency and effectiveness of our novel process and tool with the de facto standard PDF remediation tool Adobe Acrobat Pro.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Recommendations for shorter alt texts exist.

“To encourage brevity a word counter is integrated into the alternative text editor, adhering to recommendations for shorter alt texts [46].”

• cites [46]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

One of the most promising table annotating methods is the PDF Table Annotation tool.

“One of the most promising is the PDF Table Annotation tool [15].”

• cites [15]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In PAVE 2.0, the user can only create a valid list structure.

“As in the previous steps, the user can only create a valid list structure.”

Classified as probably not a substantial claim.

An AI model based on the work of Schmitt-Koopmann et al. predicts the corresponding LaTeX code.

“After opening the math editor, an AI model based on the work of Schmitt-Koopmann et al. [36] predicts the corresponding La-TeX code.”

• cites [36]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

This kind of annotation is widely used in annotation tools such as PAWLS or PDFAnno.

“This kind of annotation is widely used in annotation tools such as PAWLS [26] or PDFAnno [38].”

• cites [26]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [38]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In PAVE 2.0, all other meta information required for the UA standard is set automatically.

“All other meta information required for the UA standard is set automatically.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A researcher who was not the interviewer assigned codes to relevant content during the first cycle of analysis.

“Following Saldaña's method [33], a researcher who was not the interviewer assigned codes to relevant content during the first cycle of analysis.”

• cites [33]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The interviewer and the researcher who had assigned the initial codes executed the second cycle together.

“The interviewer and the researcher who had assigned the initial codes executed the second cycle together.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The reasons are that experienced users tagged the PDF manually or deleted larger parts of the automatically created tags.

“The reasons are that experienced users tagged the PDF manually or deleted larger parts of the automatically created tags.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The interviews were transcribed automatically with MS Teams/OneDrive and manually corrected to ensure a high-quality transcript.

“The interviews were transcribed automatically with MS Teams/OneDrive and manually corrected to ensure a high-quality transcript.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the sake of ecological validity, we allowed the participants to use any additional support they wanted, such as Google.

“For the sake of ecological validity, we wanted the task to reflect real world use of PDF remediation tools, so we allowed the participants to use any additional support they wanted, such as Google.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The interviews were analyzed using deductive and inductive coding.

“The interviews were analyzed using deductive and inductive coding.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Headings are the most used option to navigate websites.

“Research has shown that headings are the most used option to navigate websites [47].”

• cites [47]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The color of the outline boxes indicates the type of region for an easy visual overview.

“The color of the outline boxes indicates the type of region for an easy visual overview [49].”

• cites [49]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Research by Williams et al. indicates that longer alt texts are often of higher quality.

“Nonetheless, research by Williams et al. [48] indicates that longer alt texts are often of higher quality.”

• cites [48]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

To mitigate bias and learning effects, we counterbalanced the order of the tools.

“To mitigate bias and learning effects, we counterbalanced the order of the tools.”

May combine two claims.

Alternative texts for mathematical formulas must adhere to specific rules to prevent ambiguity, which makes it challenging to write valid alternative text, even for experts.

“Alternative texts for mathematical formulas must adhere to specific rules to prevent ambiguity, which makes it challenging to write valid alternative text, even for experts [12].”

• cites [12]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We created two slightly different versions of the PDFs and counterbalanced them to ensure that the order of the PDF versions did not affect the results.

“We created two slightly different versions of the PDFs and counterbalanced them to ensure that the order of the PDF versions did not affect the results.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The analysis of the tag accuracy was done by the interviewer for all PDFs.

“The analysis of the tag accuracy was done by the interviewer for all PDFs.”

• AssumptionRests on an assumption: The interviewer who analyzed tag accuracy had sufficient expertise to apply the criteria reliably.

Probably not a substantial claim: may not pass S3 (load-bearing).

An initial coding scheme was developed based on the research questions and the interview guide.

“Prior to the analysis, an initial coding scheme was developed based on our research questions and the interview guide.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The auto-tagging scores depend heavily on the characteristics of the PDF itself.

“We want to emphasize that these auto-tagging scores depend heavily on the characteristics of the PDF itself.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In PAVE 2.0, existing invalid list structures are updated to be valid structures.

“Furthermore, existing invalid list structures are updated to be valid structures.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Automating the other steps can influence the PDF remediation speed similarly depending on the used documents.

“Nevertheless, automating the other steps can influence the PDF remediation speed similarly depending on the used documents.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

User interfaces for currently non-AI-supported steps may need further adaptation to maintain effectiveness and efficiency.

“Nevertheless, user interfaces for currently non-AI-supported steps may need further adaptation to maintain effectiveness and efficiency.”

Classified as probably not a substantial claim. May combine two claims.

The difficulty arises because these functions were unexpected for most users and require some training.

“This difficulty arises because these functions were unexpected for most users and require some training.”

Probably not a substantial claim: may not pass S2 (checkable).

A highly automated system is necessitated.

“The quantitative feedback revealed that around 20% of participants would be willing to spend only up to 1 minute per page making PDFs accessible, necessitating a highly automated system.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A reason for the high reading order score of the ICCHP PDFs could be the simpler one-column layout, compared to the two-column layout of ASSETS and CHI.

“A reason for the high reading order score of the ICCHP PDFs could be the simpler one-column layout, compared to the two-column layout of ASSETS and CHI.”

Probably not a substantial claim: may not pass S4 (based).

Reliance on automation comes with its own challenges, particularly when users do not verify the accuracy of automated outputs.

“However, this reliance on automation comes with its own challenges, particularly when users do not verify the accuracy of automated outputs.”

Probably not a substantial claim: may not pass S2 (checkable).

There is a tension in the integration of AI into accessibility tools.

“This situation reveals a tension in the integration of AI into accessibility tools.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

The tension hinges on the balance between user trust in automated solutions and the critical need for such solutions to encourage, or perhaps even require, user verification.

“This tension hinges on the balance between user trust in automated solutions and the critical need for such solutions to encourage, or perhaps even require, user verification.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The inconsistent tagging strategy indicates that the papers were tagged with different interpretations or no guidance.

“This inconsistent tagging strategy indicates that the papers were tagged with different (interpretations) or no guidance.”

Probably not a substantial claim: may not pass S2 (checkable).

Multiple options permitted by current accessibility standards contribute to confusion and unnecessary complexity.

“We believe that these multiple options permitted by current accessibility standards contribute to confusion and unnecessary complexity.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

An improved figure step with specialized AI models for scientific graphics and images could reduce the burden of the PDF remediation process of CHI papers significantly.

“Hence, we believe an improved figure step with specialized AI models for scientific graphics and images could reduce the burden of the PDF remediation process of CHI papers significantly.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One reason was because they did not find sufficient information about the automatic correction on the help page of Adobe Acrobat Pro.

“One reason was because they did not find sufficient information about the automatic correction on the help page of Adobe Acrobat Pro.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The wish for automation is also due to authors being aware of accessibility issues, but not knowing how to fix them.

“We believe that this wish for automation is also due to authors being aware of accessibility issues, but not knowing how to fix them.”

May combine two claims.

Accessibility checks are helpful, but they can also provide the user with a false impression of the accessibility of a PDF.

“This indicates, similar to the findings of Kumar et al. [18], that accessibility checks are helpful, but they can also provide the user with a false impression of the accessibility of a PDF.”

• cites [18]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The short duration during which participants used the tools means that people's experience and performance in the real world might differ.

“Particularly the short duration during which participants used the tools means that people's experience and performance in the real world might differ.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The interview included speculative questions.

“Additionally, the interview included speculative questions.”

May combine two claims.

The novel PDF remediation process reduces the knowledge necessary to remediate PDFs and makes the method suitable for novice and experienced users.

“Our work builds upon this and introduces a novel PDF remediation process which reduces the knowledge necessary to remediate PDFs and, as a result, makes the method suitable for novice and experienced users.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

Authors feel it is the publisher's responsibility to make PDFs accessible.

“Research [16] showed that authors feel it is the publisher's responsibility to make PDFs accessible.”

• cites [16]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The step-by-step process requires minimal expert knowledge and time from authors.

“Our study demonstrated that our step-by-step process enables both novice and experienced users to remediate a PDF with around 90% higher tag accuracy compared to Adobe Acrobat Pro, requiring minimal expert knowledge and time from authors.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The responses to speculative questions might not reflect actual user behavior or preferences in real-world use.

“These responses might not reflect actual user behavior or preferences in real-world use.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The user currently does not have the option to tag footnotes or links.

“Additionally, the user currently does not have the option to tag footnotes or links.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The PDF parsing library utilized cannot reliably parse all PDFs, leading to potential parsing errors.

“Additionally, the PDF parsing library utilized cannot reliably parse all PDFs, leading to potential parsing errors.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The prototype currently only supports documents that are not overly complex, e.g., not containing tables that span multiple pages, irregular tables or multi-column lists.

“Our prototype currently only supports documents that are not overly complex, e.g., not containing tables that span multiple pages, irregular tables or multi-column lists.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Making a PDF accessible requires knowledge about the content, which would allow authors to make PDFs accessible more efficiently compared to publishers.

“However, making a PDF accessible requires knowledge about the content, which would allow authors to make PDFs accessible more efficiently compared to publishers.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Integrating the step-by-step process into a conference publishing workflow would require authors to spend approximately an additional hour when submitting a 10-page paper.

“We estimate it would require authors to spend approximately an additional hour when submitting a 10page paper.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The use of one PDF (with two versions) limits the generalization of the results, especially the auto-tagging results.

“Furthermore, the use of one PDF (with two versions) limits the generalization of the results, especially the auto-tagging results.”