The DOI matches its Crossref record; 3 of the 29 references it cites were found in a registry or an index, and 26 are not checked yet; none of the 10 sources its claims cite is checked for support yet.
ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Models
Open accessAbstract
Benjamin Newman, Yoonjoo Lee, Aakanksha Naik, Pao Siangliulue, Raymond Fok, Juho Kim, Daniel S Weld, Joseph Chee Chang, Kyle Lo. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.
Checked 2026-11-10 · 26 references not checked yet
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces ARXIVDIGESTABLES, a dataset of 2,228 literature review tables from ArXiv, and DECON-TEXTEVAL, an automatic evaluation method for comparing generated and human-authored tables despite surface-form differences. It finds that language models reconstruct reference schemas better with more grounding context, and that generated aspects not matching gold can still be useful.
Key findings
- ARXIVDIGESTABLES contains 2,228 high-quality tables, 7,542 unique papers, 7,634 columns, and 43,905 values, curated from about 2.5 million extracted tables; 1,985 tables come from computer science.
- DECON-TEXTEVAL, using decontext featurization and sentence-transformer scoring at threshold 0.7, achieves 70–85% precision and is nearly twice as precise as prompting Llama 3 (70B), which had 37–55% precision on predicted matches.
- Schema reconstruction improves with additional context: low-context prompts perform worst, while in-text references and in-context examples perform best, though even the best systems are far from perfect; newer tables have recalls on average 1–3 percentage points lower.
- Human evaluation found no significant advantage for matched over unmatched generated aspects in usefulness, specificity, or insightfulness; value evaluation showed adding context significantly improved partial matches, with complete/partial/none rates of 21.13%/22.54%/56.34% for column names, 18.84%/31.30%/49.86% with captions, and 22.65%/31.77%/45.59% with in-text references.
Methods
- The task is decomposed into schema generation and value generation. Schema generation uses input paper titles and abstracts; value generation prompts models with an aspect-paper pair and the paper’s full text, then rewrites values for table display.
- Experiments use Mixtral 8x22 and GPT-3.5-Turbo, with additional context conditions: generated caption, gold caption, gold caption plus in-text references, and few-shot examples retrieved by caption embedding similarity.
- Evaluation includes DECON-TEXTEVAL for schema alignment, using featurizers (name, values, decontext) and scorers (exact match, Jaccard, sentence transformers, Llama 3), plus human evaluation of generated aspect quality and value matches.
Limitations
- The paper states that it only studies ArXiv papers, overrepresenting fields like computer science and underrepresenting medicine, humanities, and social science, and all high-quality tables are in English.
- The paper states that reconstructing tables remains difficult, many generated columns do not match reference columns, and human evaluation of novel-column utility is costly; future work should develop automatic metrics that correlate with human utility.
- Evident from the text: value evaluation uses reference schemas as input to bypass schema alignment, so it is not end-to-end; the paper also notes many value matches lack lexical overlap or require inference, indicating automated value metrics may be insufficiently sensitive.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 10 cited sources checked · 0 supported · 0 flagged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Comparing outputs from the baseline to the Caption+In-text References condition, we find that the latter tends to output more specific aspects.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 6.1 Schema Evaluation Results
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Interestingly, unlike schema reconstruction, we observe that incorporating additional context does not seem to improve value generation accuracy; we dig deeper into this during human evaluation.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 6.2 Value Evaluation Results
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, like schema alignment, models are far from perfect in value generation.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 6.2 Value Evaluation Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, we produce a set of 2,228 highquality tables.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Final Filter and Manual Verification
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The full pipeline filters from 2.5 million starting tables published in 800,000 papers to 2,228 tables published in 1,723 papers.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 2 Creating ARXIVDIGESTABLES
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For (1), we find that language models have higher recall by conditioning on more context that specifies the purpose of the table (e.g., captions, in-line references, other example tables).”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 1 Introduction
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our approach overcomes the difficulty in matching semantically-similar but lexicallydifferent column names by using a language model to expand column names into descriptions grounded in documents.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 1 Introduction
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 4 presents results from this assessment, showing that adding additional context leads to a significant improvement in partial matches.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 6.2 Value Evaluation Results
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesThe paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While DE-CONTEXTEVAL is effective at matching generated and reference table columns, and we test providing different additional context to steer the table generation models, many generated table columns do not match with the reference columns.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Limitations
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To start, we consider approximately 800,000 papers that have LaTeX source available.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 2 Creating ARXIVDIGESTABLES
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- It may state two separate claims as one. 26
- Probably not a substantial claim. 22
- The quote only partly states this claim. 2
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(1) We see that low context prompts (e.g., a baseline with no additional context, caption-only) perform the worst while high context prompts (e.g., in-text references, in-context examples) perform best.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 6.1 Schema Evaluation Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“We also experimented using GPT-4 (Achiam et al., 2023) for these difficult cases, but still found errors due to insufficient layout information being maintained in the conversion from LaTeX source to XML.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Additional information
What the sources say
- Not matchedcites Achiam et al., 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Llama3 scorers have high recall, but low precision.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Ethical Considerations and Broader Impact
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Reference table schemas are the gold standard for evaluating generated schemas.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Unfortunately, sometimes XML-formatted tables have column headers that span multiple rows, rows can have insufficient numbers of columns, cells may span multiple columns rows, etc.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Additional information
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“That said, the rows in the tables in our benchmark do include citations, so readers can trace values back to their sources.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Ethical Considerations and Broader Impact
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, aspects from Caption+In-text Reference are shown to be more useful and specific than the Baseline's, but were less insightful.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 6.1 Schema Evaluation Results
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More context improves recall, but does not lead to completely reproducing reference table schemas.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Ethical Considerations and Broader Impact
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Reference table schemas are the gold standard for evaluating generated schemas.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The generated table has also proposed two novel aspects that are still relevant and useful, like”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 1 Introduction
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach is fast, but initial experiments found it more prone to hallucinations and generic column names (e.g.,”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Generation
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As a first filtering pass, we remove tables that are likely to be misparsed or unusable, filtering those with fewer than 400 or more than 15,000 characters.”
- Also stated in: § 2 Creating ARXIVDIGESTABLES · plain
“This leaves approximately 211,000 tables.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 2 Creating ARXIVDIGESTABLES
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § 2 Creating ARXIVDIGESTABLES · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The generated table has reconstructed two gold aspects: the pink and blue aspects are the same, despite surface form differences (e.g.,”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 1 Introduction
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To obtain column topics, we manually annotate columns in ∼50 tables-∼38% are about datasets, ∼20% are about methods, and the rest are on other”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Ethical Considerations and Broader Impact
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We filter out any tables that have fewer than two matched citations, leaving us with 44,617 tables.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 2 Creating ARXIVDIGESTABLES
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These tables compare and contrast a total of 7,542 unique papers using a total of 7,634 columns and 43,905 values.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 1 Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“(1) Exact match has very low recall, as expected, serving as our conservative bound.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Calibrating Schema Alignment
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Reconstructing tables is difficult.While DE-CONTEXTEVAL is effective at matching generated and reference table columns, and we test providing different additional context to steer the table generation models, many generated table columns do not match with the reference columns.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Limitations
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After this step in the process we remove any tables where the algorithm failed and any tables that now have fewer than two rows, leaving 47,876 tables.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 2 Creating ARXIVDIGESTABLES
Strength:plain· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can see that our system generates a different (and redundant) set of aspects compared to the reference.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Ethical Considerations and Broader Impact
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Reference table schemas are the gold standard for evaluating generated schemas.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Combining with a small textual similarity model results in a matcher that is nearly twice more precise than prompting Llama 3 (70B), which often hallucinates matches.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 1 Introduction
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These tables are not manually checked, are filtered less stringently, and do not have linked full-texts.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Additional information
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: The high-quality dataset meets the desiderata.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(2) Interestingly, though adding context improves reconstruction, it does not make the task trivial -even the best performing systems are far from perfect.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 6.1 Schema Evaluation Results
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see that scorers continue to follow the same trend observed during schema alignment, with the sentence transformer scorer being fairly permissive while an exact match is overly strict.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 6.2 Value Evaluation Results
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To enable future work to improve on this pipeline, we also release a set of 22, 283 medium-quality tables (seeFigure 3) with less strict filtering alongside which filters we ran to produce it along with quality metadata (See §A.3).”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Ethical Considerations and Broader Impact
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
- The authors state this as an assumption: The medium-quality tables will enable future work to improve on this pipeline.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find a majority (1,985) of the tables come from computer science publications, with others coming from Physics, Quantitative Biology, Statistics, Math, and other fields (See Appendix A.4).”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Dataset Statistics
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find ∼40% of the columns are categorical or boolean, which are more suitable for supporting inter-paper comparisons, while the other ∼60% are more descriptive.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Ethical Considerations and Broader Impact
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reference table separately splits out the various data types while the generated one has a single”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Ethical Considerations and Broader Impact
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: Reference table schemas are the gold standard for evaluating generated schemas.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests an interesting tradeoff between our reconstruction objective, and possibly a different objective like creativity.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 6.1 Schema Evaluation Results
Strength:hedged· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“First, reference tables tend to present information concisely, making column headers and values hard to interpret without additional context (e.g., a column might be named”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Challenges
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Tables with multi-row or hierarchical headers are allowed.These can sometimes lead to misformatted tables.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Additional information
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, all of the tables in our high quality set are in English, even though literature review tables may also be used in other languages.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Limitations
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This potentially makes value generation difficult because all of the values have to come from the title and abstract.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Additional information
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This potentially makes schema generation difficult, as any additional context has to come from the caption (if present).”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Additional information
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In contrast, schema generation from a set of documents remains relatively under-explored, even though it is a crucial and effortful part of the manual literature review process.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 1 Introduction
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✗ Based: failsProbably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, many matches have no lexical overlap (e.g.,”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 6.2 Value Evaluation Results
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one. The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The key challenge in assessing how well a generated table reconstructs a reference table lies in determining schema alignmentsidentifying which columns convey the same information despite different phrasing.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Challenges
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This approach is slower but allows us to overcome context window limits and leverage prior work in aspect-based question answering to perform value generation.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Generation
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This suggests that novel generated aspects are of comparable quality (usefulness, specificity, insightfulness) to gold aspects or even have a higher quality (usefulness of aspects from Caption+In-text References).”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 6.1 Schema Evaluation Results
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“At the core of this * Equal contributions.sensemaking process is identifying a schema, a set of important aspects that are useful for comparing and contrasting prior literature (Russell et al., 1993).”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 1 Introduction
What the sources say
- Not matchedcites Russell et al., 1993
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Prior work like Zhang and Balog (2018) infers new schemas from pre-existing ones, while recent work like Wang et al. (2024) assumes users can clearly articulate a schema in a short natural language query to infer aspects directly.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 1 Introduction
What the sources say
- Not matchedcites Zhang and Balog (2018)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Wang et al. (2024)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
None of the sources cited for this claim could be matched to a paper in Crossref, so there is nothing to check it against.
In the paper
“Conducting literature reviews by reading and synthesizing information across a large set of documents is vital for scientists to stay abreast of their fields yet is increasingly laborious as the number of scientific publications grows exponentially (Jinha, 2010;Bornmann et al., 2021).”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > 1 Introduction
What the sources say
- Not matchedcites Jinha, 2010
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Bornmann et al., 2021
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“due to stochasticity in the generation process, models occasionally recover.”
§ ARXIVDIGESTABLES: Synthesizing Scientific Literature into Tables using Language Models > Additional information
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✗ Based: failsProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
Scientists create literature review tables with rows as publications and columns as a schema.
“When conducting literature reviews, scientists often create literature review tables”
The authors state this as an assumption.
Rows without citations in a table refer to the source paper containing the table.
“Rows without citations are assumed to refer to the source paper containing the table.”
The paper relies on this without support.
Titles and abstracts usually have enough information to form useful schemas.
“We represent input papers using their titles and abstracts, which usually have enough information to form useful schemas and are easier to fit in the context window of models.”
The paper relies on this without support.
Citation information is usually contained within a cell in a table.
“At this stage, the citation information is usually contained within a cell in a table.”
The paper relies on this without support.
The high-quality dataset meets the desiderata.
“In addition to our high-quality dataset that is likely to meet our desiderata”
The paper relies on this without support.
Reference table schemas are the gold standard for evaluating generated schemas.
“Mean (SD) ratings from human assessments of generated aspects that match the gold schema (M) with those that do not (NM).”
The paper relies on this without support.
The human evaluation criteria (Usefulness, Specificity, Insightfulness) adequately capture column quality.
“When performing human evaluation for the novel schema, we assessed each column based on the following criteria:”
The authors state this as an assumption.
The medium-quality tables will enable future work to improve on this pipeline.
“To enable future work to improve on this pipeline, we also release a set of 22, 283 medium-quality tables”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Minimum row count conflicts with the final filtering rule
MinorTable 1 reports a minimum of one paper (row) per table, but the appendix says the final dataset excludes tables with fewer than two rows.
“Min Max Median Mean Total Papers 1 35 3.0 4.944 11016 Aspects 2 13 3.0 3.426 7634”
“After these steps, we remove any tables that have fewer than two citations, rows or columns, leaving us with our final set.”
Two different totals are given for table rows
MinorThe Table 1 caption prints the total as 11,0016 rows, while the table itself reports 11016.
“Of the 11,0016 total rows there are 7,542 unique papers.”
“Min Max Median Mean Total Papers 1 35 3.0 4.944 11016”
Citations
Cites
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
The cost structure of sensemaking
Russell, D. M., Stefik, M., Pirolli, P. L. T., & Card, S. K. (1993) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/169059.169209 · Cited by 793
Luan, Y., He, L., Ostendorf, M., & Hajishirzi, H. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1360 · Cited by 422
Article 50 million: an estimate of the number of scholarly articles in existence
JINHA, A. E. (2010) · Learned Publishing · doi:10.1087/20100308 · Cited by 404
The Pyramid Method: Incorporating human content selection variation in summarization evaluation
Nenkova, A., Passonneau, R., & McKeown, K. (2007) · ACM Transactions on Speech and Language Processing · doi:10.1145/1233912.1233913 · Cited by 307
Is chatGPT a good nlg evaluator? A preliminary study
Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J. (2023) · Proceedings of the 4th New Frontiers in Summarization Workshop · doi:10.18653/v1/2023.newsum-1.1 · Cited by 235
QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization
Zhong, M., Yin, D., Yu, T., Zaidi, A., Mutuma, M., Jha, R., Awadallah, A. H., Celikyilmaz, A., Liu, Y., Qiu, X., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.472 · Cited by 178
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., & Gardner, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.365 · Cited by 138
Self-citationMSˆ2: Multi-Document Summarization of Medical Studies
DeYoung, J., Beltagy, I., van Zuylen, M., Kuehl, B., & Wang, L. L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.594 · Cited by 120
Lu, Y., Dong, Y., & Charlin, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.648 · Cited by 113
Flagged first, then by citations
Research works printed but not linked: 14
- Fan Bai, Junmo Kang, Gabriel Stanovsky, Dayne Fre- itag, and Alan Ritter. 2023. Schema-driven infor- mation extraction from heterogeneous tables. ArXiv, abs/2305.14336. · arXiv
- Lutz Bornmann, Robin Haunschild, and Rüdiger Mutz. 2021. Growth rates of modern science: a latent piecewise growth curve approach to model publi- cation numbers from established and new literature databases. Humanities and Social Sciences Commu- nications, 8(1):1-15.
- Kyle Yingkai Gao and Jamie Callan. 2017. Scien- tific table search using keyword queries. ArXiv, abs/1707.03423. · arXiv
- Hayato Hashimoto, Kazutoshi Shinoda, Hikaru Yokono, and Akiko Aizawa. 2017. Automatic generation of review matrices as multi-document summarization of scientific papers. In BIRNDL@SIGIR.
- Keti Korini, Ralph Peeters, and Christian Bizer. 2022. Sotab: the wdc schema. org table annotation bench- mark. In CEUR Workshop Proceedings, volume 3320, pages 14-19. RWTH Aachen.
- Yoonjoo Lee, Kyungjae Lee, Sunghyun Park, Dasol Hwang, Jaehyeon Kim, Hong-in Lee, and Moontae Lee. 2023. QASA: advanced question answering on scientific articles. In Proceedings of the 40th Interna- tional Conference on Machine Learning, ICML'23. JMLR.org.
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. 2024. Llmscore: Unveiling the power of large language models in text-to-image syn- thesis evaluation. Advances in Neural Information Processing Systems, 36. · DOI
- Nafise Sadat Moosavi, Andreas Rücklé, Dan Roth, and Iryna Gurevych. 2021. SciGen: a dataset for reasoning-aware text generation from scientific ta- bles. In NeurIPS Datasets and Benchmarks.
- Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
- Darsh J. Shah, L. Yu, Tao Lei, and Regina Barzi- lay. 2021. Nutri-bullets hybrid: Consensual multi- document summarization. In North American Chap- ter of the Association for Computational Linguistics.
- Shuo Zhang and Krisztian Balog. 2018. On-the-fly table generation. The 41st International ACM SIGIR Con- ference on Research & Development in Information Retrieval.
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA.
Cited by
Sorted by citations
From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery
Zheng, T., Deng, Z., Tsang, H. T., Wang, W., Bai, J., Wang, Z., & Song, Y. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.895 · Cited by 104
AutoLabs: cognitive multi-agent systems with self-correction for autonomous chemical experimentation
Panapitiya, G., Saldanha, E., Job, H., & Hess, O. (2026) · Scientific Reports · doi:10.1038/s41598-026-45593-z · Cited by 34
Table-R1: Inference-Time Scaling for Table ReasoningPreprint
Yang, Z., Chen, L., Cohan, A., & Zhao, Y. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.23621 · Cited by 26
A Survey of AI ScientistsPreprint
Tie, G., Zhou, P., & Sun, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.23045 · Cited by 23
Fok, R., Chang, J. C., Radensky, M., Siangliulue, P., Bragg, J., Zhang, A. X., & Weld, D. S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2504.18496 · Cited by 12
Self-citation
Metadata
- Authors
- Newman, B., Lee, Y., Naik, A., Siangliulue, P., Fok, R., Kim, J., Weld, D. S., Chang, J. C., & Lo, K.
- Affiliations
- University of Washington (US)
- ORCID
- Venue
- Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
- Published
- 2024
- Pages
- 9612-9631
- OpenAlex
- W4404783021
- Semantic Scholar
- c7face35e84f2cb04fb1600d54298799aa0ed189
- Cited by
- 19
- References
- 3
- Citations by year
- 2026: 9 · 2025: 10
- FWCI
- 3.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- language models, information extraction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
