The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.
Is Your LLM Outdated? A Deep Look at Temporal Generalization
Open accessAbstract
Chenghao Zhu, Nuo Chen, Yufei Gao, Yunyi Zhang, Prayag Tiwari, Benyou Wang. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper introduces temporal generalization for LLMs and FreshBench, evaluating compression intelligence via BPC and future event prediction via Good Judgment Open questions. It finds prevalent Nostalgia Bias, temporal degeneration, and that stronger/newer models can decline faster while open-source models show better long-term adaptability.
Key findings
- Temporal bias is common: Nostalgia Bias is the most frequent, Neophilia Bias the least; models favor older knowledge, with peak accuracy before 2020 and more pronounced Nostalgia for more distant past periods. In sampled content, Wiki was classified as 97.83% historical vs. BBC 91.86% recent.
- Compression intelligence shows stable BPC on Wikipedia but increasing BPC on BBC news; larger models deteriorate faster on Wiki (correlation coefficient 0.504). arXiv BPC dropped sharply around March 2024, coinciding with ChatGPT’s reported peak of over 1.8 billion visits that month.
- Future event prediction declines over time: LLaMA-7B’s accuracy dropped from 0.11 in March 2023 to 0.08 by March 2024. Newer models perform better initially but can decline faster; GPT-4-231106 had the largest drop at 39.39%, while Command R+, Mixtral-8x22B-Instruct-v0.1, DeepSeek-V2-Chat, and LLaMA-3.1-405B-Instruct showed slower declines.
- Miniature models underperform in temporal tasks: Gemini.pro fell from 0.37 in March 2024 to 0.17 by July 2024; GPT-4 mini had 0.21 accuracy in April 2024, about 0.1 lower than full-sized GPT-4 in the same period. Closed-source models slightly outperform open-source initially, but open-source models demonstrate better stability.
Methods
- The paper defines cutoff time as the model release date, splitting evaluation into past, present, and future; it defines Temporal Bias (Nostalgia/Neophilia) and Temporal Degeneration, and computes a Temporal Bias Index (TBI) by fitting BPC over time.
- Data include crawled Yahoo, BBC, Reddit, Wikipedia, arXiv, GitHub, Quora, and Wattpad, plus external BBC News and Wikitext sampled every three months with 50 entries per dataset. Future prediction questions come from Good Judgment Open: 2,769 questions up to 2024-10-01, earliest proposed 2015-09-01, latest closed 2024-10-01.
- Analysis uses BPC normalized by UTF-8 character length, prediction accuracy as correct/total, and hypothesis tests for Neophilia Bias, Nostalgia Bias, and Temporal Degeneration using a normal CDF p-value. Experiments consumed about 300 GPU hours on NVIDIA A800 for compression and about 50 GPU hours for prediction.
Limitations
- The paper states that collected text may not fully reflect language diversity and complexity, that the English-centric questions and answers limit applicability to non-English regions and global events, and that text-loss evaluation requires model logits, which is challenging for closed-source or API-accessed models.
- The text makes evident that future event prediction uses a retrospective approach, treating data available after a question was posed and before market closure as “future” data rather than actual future data.
- The text also notes inherent randomness in the question datasets, which the hypothesis testing is designed to address but does not eliminate.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 9 cited sources checked · 0 supported · 0 flagged
- Findingreasoning
The Claude-3-opus-20240229 model shows the least decline among the Claude and GPT model iterations.
“Notably, the Claude-3-opus-20240229 model shows the least decline, indicating a potentially better tuning for broader applications or general datasets.”
Quote in the paperSubstantial claim - Findingown data
GPT-4-231106 experienced the largest drop (39.39%) among the compared models.
“Similarly, the progression from GPT-3.5-turbo-230613 to GPT-4-231106 shows an increasing trend in both pre-release accuracy and the magnitude of decline post-release, with the GPT-4-231106 model experiencing the largest drop (39.39%).”
Quote in the paperNot a substantial claim - Findingown data
A sharp decrease in BPC occurred around March 2024 in the arXiv dataset.
“a sharp decrease in BPC occurred around March 2024”
Quote in the paperSubstantial claim - Findingnone given
There is a positive correlation between model attributes such as model size and release date and performance decay.
“There is a positive correlation between model attributes (such as model size and release date) and performance decay.”
Quote in the paperNot a substantial claim - Findingreasoning
Performance on language likelihood does not always align with specific abilities measured by existing benchmarks.
“it does not always align with specific abilities measured by existing benchmarks”
Quote in the paperSubstantial claim
- Findingown data
The further back in time from release, the more pronounced and significant the Nostalgia Bias becomes.
“we observe that the further back in time we go, the more pronounced the Nostalgia Bias becomes, and it is also more significant.”
Quote in the paperSubstantial claim - Findingown data
Calculations in Appendix I reveal variations in how well each model can adapt specific text over time.
“However, our calculations in Appendix I reveal variations in how well each model can adapt specific text over time.”
Quote in the paperSubstantial claim - Findingreasoning
Models show stronger Nostalgia Bias when evaluated with BBC text than Wiki text.
“Models show stronger Nostalgia Bias when evaluated with BBC text than Wiki text.”
Quote in the paperNot a substantial claim - Findingown data
Using validation loss of new text to evaluate models is mostly effective.
“Our analysis indicates that using validation loss of new text to evaluate models is mostly effective.”
Quote in the paperSubstantial claim - Findingnone given
ArXiv BPC has a huge decline after GPT is widely used.
“ArXiv BPC has a huge decline after GPT is widely used, as mentioned in Key Findings.”
Quote in the paperSubstantial claim - Findingown data
Models tend to be stronger in earlier periods before release, with performance gradually declining over time.
“Models tend to be stronger in earlier periods before release, with performance gradually declining over time.”
Quote in the paperSubstantial claim - Findingown data
In a sample of approximately 500 data pieces from Wiki and BBC, gpt-4o classified Wiki content as 97.83% Historical and 2.17% Others.
“Wiki: Historical: 97.83%, Othersfoot_1 :2.17%.”
Quote in the paperSubstantial claim- Rests on an assumption: GPT-4o's classification of Wiki and BBC content is accurate.
- Rests on an assumption: The approximately 500 sampled data pieces are representative of the Wiki and BBC datasets.
- Findingown data
Models such as Command R+, Mixtral-8x22B-Instruct-v0.1, DeepSeek-V2-Chat, and LLaMA-3.1-405B-Instruct demonstrate slower declines in accuracy.
“Notably, models such as Command R+, Mixtral-8x22B-Instruct-v0.1, DeepSeek-V2-Chat, and LLaMA-3.1-405B-Instructdemonstrate slower declines in accuracy”
Quote in the paperSubstantial claim - Findingown data
In both pre- and post-release scenarios, proprietary models such as GPT-4 and Claude-3.5 slightly outperform their open-source counterparts.
“in both pre-and post-release scenarios, proprietary models such as GPT-4 and Claude-3.5 slightly outperform their open-source counterparts.”
Quote in the paperNot a substantial claim - Findingown data
When predicting outcomes for events known only post-release, all models exhibited substantial declines in performance.
“However, when predicting outcomes for events known only post-release, all models exhibited substantial declines in performance.”
Quote in the paperSubstantial claim - Findingreasoning
Claude-3-sonnet-20240229 and Claude-3.5-Sonnet-20240620 show more significant declines than Claude-3-opus-20240229.
“This contrasts with the more significant declines seen in Claude-3-sonnet-20240229 and Claude-3.5-Sonnet-20240620,suggesting that these iterations may have undergone specific architectural tweaks, utilized different training data scopes, or implemented optimization strategies that did not generalize as effectively.”
Quote in the paperSubstantial claim - Findingcitation
The timing of the sharp decrease in arXiv BPC coincides with the peak in ChatGPT usage in March 2024.
“This timing coincides with the peak in ChatGPT usage, as reported by Exploding Topicsfoot_0 , which saw over 1.8 billion visits in that month.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Exploding Topics
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingnone given
In more dynamic environments such as the BBC dataset, the impact of model size on performance decay is smaller.
“In more dynamic environments (such as the BBC dataset), the impact of model size on performance decay is smaller.”
Quote in the paperSubstantial claim - Findingnone given
Newer models generally exhibit stronger performance.
“Newer models generally exhibit stronger performance.”
Quote in the paperNot a substantial claim - Findingown data
Newer models exhibit much stronger future prediction capabilities.
“In contrast, newer models exhibit much stronger future prediction capabilities.”
Quote in the paperSubstantial claim - Findingnone given
Miniature models significantly underperform compared to their full-sized counterparts in temporal tasks.
“Miniature models signif-icantly underperform compared to their full -sized counterparts in temporal tasks.”
Quote in the paperNot a substantial claim - Findingown data
In a sample of approximately 500 data pieces from Wiki and BBC, gpt-4o classified BBC content as 91.86% Recent and 8.14% Historical.
“BBC: Recent: 91.86%, Historical: 8.14%.”
Quote in the paperSubstantial claim- Rests on an assumption: GPT-4o's classification of Wiki and BBC content is accurate.
- Rests on an assumption: The approximately 500 sampled data pieces are representative of the Wiki and BBC datasets.
- Findingown data
Phi-1.5 and Baichuan2-13B series show Nostalgia Bias in all three periods before release.
“Phi-1.5 and Baichuan2-13B series show Nostalgia Bias in all three periods before release.”
Quote in the paperNot a substantial claim - Findingown data
Prompts v5 (Future Recall) and v4 (Precise Thinking) demonstrated relatively better effectiveness for Llama3.1-8B-Instruct, Qwen2.5-7B, and Baichuan2-7B-Chat.
“Analyzing the results from Llama3.1-8B-Instruct,Qwen2.5-7B and Baichuan2-7B-Chat, we found that prompts v5(Future Recall) and v4(Precise Thinking) demonstrated relatively better effectiveness.”
Quote in the paperSubstantial claim - Findingreasoning
The correlation between BPC and scores for chat models strengthens with longer text lengths.
“For chat models in this task, the correlation be-tween BPC and scores is logical-higher probabilities of generating the text correlate with higher scores, and this correlation strengthens with longer text lengths.”
Quote in the paperNot a substantial claim - Findingown data
LLMs often struggle with temporal biases, affecting reliability in dynamic scenarios requiring accurate forecasting.
“Our experiments reveal that LLMs often struggle with temporal biases, affecting reliability in dynamic scenarios requiring accurate forecasting.”
Quote in the paperSubstantial claim- Rests on an assumption: BPC (bits per character) is a valid measure of a model's understanding and prediction of world events.
- Rests on an assumption: Declines in benchmark accuracy over time reflect degradation in temporal generalization rather than other factors.
- Findingown data
Base models exhibit a stronger correlation with established benchmarks.
“Our findings suggest that: 1) Base models exhibit a stronger correlation with established benchmarks, implying that these models, in their fundamental form, possess a robust capacity for modeling language.”
Quote in the paperSubstantial claim- Rests on an assumption: BPC (bits per character) is a valid measure of a model's understanding and prediction of world events.
- Findingown data
The scale of models observed does not significantly alter outcomes.
“From Fig. 12 we can conclude that the scale of models being observed does not significantly alter outcomes, suggesting that performance correlations across benchmarks are generally consistent regardless of model size.”
Quote in the paperSubstantial claim- Rests on an assumption: BPC (bits per character) is a valid measure of a model's understanding and prediction of world events.
- Findingown data
Models tend to better comprehend earlier events.
“Specifically, we urge the community to take note of this phenomenon: models tend to better comprehend earlier events.”
Quote in the paperSubstantial claim - Findingown data
Different models respond variably to the designed prompts.
“Specifically, different models respond variably to the designed prompts.”
Quote in the paperSubstantial claim - Findingown data
For base models, a familiar text results in lower scores on related comprehension tasks.
“However, an inverse trend is observed with base models, where a familiar text paradoxically results in lower scores on related comprehension tasks.”
Quote in the paperSubstantial claim- Rests on an assumption: BPC (bits per character) is a valid measure of a model's understanding and prediction of world events.
- Findingown data
Miniature versions of models underperform compared to their full-sized counterparts in temporal tasks.
“The data reveals that miniature versions of models underperform compared to their full-sized counterparts in temporal tasks.”
Quote in the paperSubstantial claim - Findingown data
The mini version of GPT-4 had an accuracy of 0.21 in April 2024, around 0.1 lower than the full-sized GPT-4 in the same period.
“Similarly, the mini version of GPT-4 showed considerably lower performance across multiple time periods, particularly in April 2024, where its accuracy was just 0.21, around 0.1 lower than the full-sized GPT-4 in the same period.”
Quote in the paperSubstantial claim - Findingown data
For Qwen2.5-7B, all modified prompts generally outperformed the base prompt in most scenarios.
“In contrast, for the Qwen2.5-7Bmodel, all modified prompts generally outperformed the base prompt in most scenarios.”
Quote in the paperNot a substantial claim - Findingown data
All models reached their peak accuracy before 2020.
“all models reached their peak accuracy before 2020.”
Quote in the paperSubstantial claim - Findingown data
For chat models, higher probabilities of generating the text correlate with higher scores.
“For chat models in this task, the correlation be-tween BPC and scores is logical-higher probabilities of generating the text correlate with higher scores, and this correlation strengthens with longer text lengths.”
Quote in the paperSubstantial claim- Rests on an assumption: BPC (bits per character) is a valid measure of a model's understanding and prediction of world events.
- Findingown data
For Yi-6B, there were scarcely any discernible differences in performance across the several prompts employed.
“For Yi-6B, there were scarcely any discernible differences in performance across the several prompts we employed.”
Quote in the paperNot a substantial claim - Findingown data
Miniature versions of models show significant declines in temporal performance.
“Miniature versions of models show significant declines in temporal performance.”
Quote in the paperSubstantial claim - Interpretationown data
In dynamic environments, different models have significantly varying adaptation capabilities.
“These findings indicate that in dynamic environments, different models have significantly varying adaptation capabilities.”
Quote in the paperSubstantial claim - Premisecitation
LLMs can perform future event prediction.
“LLMs could perform future event prediction, as seen in Bonde (2022) which employed GPT-3 in a few-shot setting to tackle binary questions from the Metaculus platform.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Bonde (2022)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Interpretationreasoning
New Wiki texts with recent temporal markers do not necessarily imply that the content is new in a temporal context.
“New Wiki texts with Figure 4: BPC on BBC news text Figure 5: BPC on arXiv text recent temporal markers do not necessarily imply that the content is new in a temporal context.”
Quote in the paperNot a substantial claim - Interpretationown data
It is reasonable that Wiki text has a flat curve in BPC.
“So, it is reasonable that Wiki text has a flat curve in BPC.”
Quote in the paperSubstantial claim - Premisecitation
Knowledge-based assessments such as MMLU and C-Eval are prone to data manipulation.
“Knowledge-based assessments, exemplified by MMLU Hendrycks et al. (2021) and C-Eval Huang et al. (2023), are prone to data manipulation”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Hendrycks et al. (2021)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Huang et al. (2023)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Study design and conductreasoning
Temporal generalization evaluation reduces data contamination by limiting access to future content.
“This approach significantly reduces the issue of data contamination by limiting access to future content.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Models gradually degrade during the pre-release period, rather than experiencing a rapid decline around the cutoff period.
“This suggests that the models gradually degrade during the pre-release period, rather than experiencing a rapid decline around the cutoff period.”
Quote in the paperSubstantial claim - Interpretationreasoning
Relying solely on language likelihood as an indicator of overall performance might overlook crucial aspects of cognitive and interpretative abilities.
“Thus, relying solely on language likelihood as an indicator of overall per- formance might overlook crucial aspects of cognitive and interpretative abilities that are essential for more complex applications.”
Quote in the paperSubstantial claim- Rests on an assumption: BPC (bits per character) is a valid measure of a model's understanding and prediction of world events.
- Interpretationreasoning
Models' training data may disproportionately reflect earlier times.
“This tendency highlights the models' alignment with the informational characteristics prevalent prior to the latter half of the observed period, suggesting their training data may disproportionately reflect earlier times.”
Quote in the paperSubstantial claim - Interpretationown data
Performance correlations across benchmarks are generally consistent regardless of model size.
“From Fig. 12 we can conclude that the scale of models being observed does not significantly alter outcomes, suggesting that performance correlations across benchmarks are generally consistent regardless of model size.”
Quote in the paperSubstantial claim - Interpretationreasoning
Fine-tuning may alter the base models' understanding of the texts, shifting their processing in ways that do not favor traditional comprehension metrics.
“This indicates that fine-tuning may alter the base models' understanding of the texts, perhaps shifting their processing in ways that do not favor traditional comprehension metrics.”
Quote in the paperSubstantial claim- Rests on an assumption: BPC (bits per character) is a valid measure of a model's understanding and prediction of world events.
- Interpretationown data
Existing benchmarks have shortcomings related to temporal shifts.
“The findings highlight shortcomings in existing benchmarks related to temporal shifts and underscore the urgent need for improvements in LLM training and updating processes to enhance adaptability and mitigate biases.”
Quote in the paperSubstantial claim - Interpretationown data
There is a notable trend toward Nostalgia, where models demonstrate optimal performance on data from the earlier part of the timeline analyzed.
“This observation points to a notable trend toward”
Quote in the paperSubstantial claim - Interpretationreasoning
Base models, in their fundamental form, possess a robust capacity for modeling language.
“Our findings suggest that: 1) Base models exhibit a stronger correlation with established benchmarks, implying that these models, in their fundamental form, possess a robust capacity for modeling language.”
Quote in the paperSubstantial claim
- It may state two separate claims as one. 29
- Probably not a substantial claim. 26
- The quote only partly states this claim. 2
It may state two separate claims as one.
Findingnone givenWithin model series such as Claude, Gemini, and GPT, newer versions are becoming more powerful but also experience more rapid performance decline.
“Within model series such as Claude,Gemini, and GPT, newer versions are becoming more powerful but also experience more rapid performance decline.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataThe progression from GPT-3.5-turbo-230613 to GPT-4-231106 shows an increasing trend in both pre-release accuracy and the magnitude of decline post-release.
“Similarly, the progression from GPT-3.5-turbo-230613 to GPT-4-231106 shows an increasing trend in both pre-release accuracy and the magnitude of decline post-release, with the GPT-4-231106 model experiencing the largest drop (39.39%).”
Quote in the paperSubstantial claimIt may state two separate claims as one.
FindingreasoningModels perform stably on Wikipedia but show more variation on BBC.
“Models perform stably on Wikipedia but show more variation on BBC.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataLLaMA-7B's accuracy dropped from 0.11 in March 2023 to 0.08 by March 2024, with no improvement in later evaluation periods.
“For example, LLaMA-7B's accuracy dropped from 0.11 in March 2023 to 0.08 by March 2024, with no improvement in later evaluation periods.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingnone givenModels prefer older knowledge, with nonuniform knowledge distribution skewed towards the past.
“Models prefer older knowledge, with nonuniform knowledge distribution skewed towards the past.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
FindingreasoningThe overall BPC for BBC texts shows an increasing trend, indicating gradual performance degeneration.
“The overall BPC for BBC texts shows an increasing trend, indicating gradual performance degeneration.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataNostalgia Bias is the most common bias, followed by periods where neither bias is significant, with Neophilia Bias being the least frequent.
“As shown in Tab. 2, among the biases, Nostalgia is the most common, followed by periods where neither bias is significant (shown as -), with Neophilia Bias being the least frequent.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataNewer models show higher correlations between model attributes and performance decay, meaning that despite advanced capabilities, they may deteriorate faster.
“Additionally, newer models show higher correlations, meaning that despite their advanced capabilities, they may deteriorate faster.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one. The quote only partly states this claim.
Findingnone givenClosed-source models perform well initially but struggle with temporal generalization, while open-source models show better stability.
“Close-source models perform well initially but struggle with temporal generalization, while open-source models show better stability.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Findingnone givenPerformance on language likelihood is a strong indicator of a model's linguistic capabilities.
“while performance on language likelihood is a strong indicator of a model's linguistic capabilities”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataIndividual models show considerable differences in performance on new data; Yi and Baichuan2 series adapt relatively well to new texts, while Phi-1.5 and OPT series exhibit poorer generalization.
“Individual models show considerable differences in performance on new data; for instance, Yi, and Baichuan2 series adapt relatively well to new texts, while Phi-1.5 and OPT series exhibit poorer generalization (see Appendix I).”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingnone givenThe distribution of knowledge is not uniform, nor does it tend to be closer to the present.
“The distribution of knowledge is not uniform, nor does it tend to be closer to the present.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataEarlier models perform worse, while newer advanced models perform better in future predictions.
“Earlier models perform worse, while newer advanced models perform better in future predictions.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingnone givenModels generally favor older knowledge and prefer information from more distant past periods.
“This observation reveals that models generally favor older knowledge and prefer information from more distant past periods.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataMistral-7B-v0.1 shows Neophilia Bias in 20 to 40 months before release but more significant Nostalgia in another two periods, which the authors attribute to random fluctuations rather than a consistent trend.
“Mistral-7B-v0.1 show Neophilia Bias in 20 to 40 months before release but show more significant Nostalgia in another two periods, which we attribute to random fluctuations rather than a consistent trend.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataIn the Wiki dataset, larger models deteriorate faster, with a correlation coefficient of 0.504.
“In the Wiki dataset, larger models deteriorate faster (correlation coefficient of 0.504), suggesting a trade-off between capability and flexibility.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingnone givenClosed-source models tend to perform slightly better at the beginning, yet open-source models demonstrate superior generalization capabilities.
“Closed-source models tend to perform slightly better at the beginning, yet opensource models demonstrate superior generalization capabilities.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Findingnone givenModels like Claude-3.5-Sonnet-20240620, GPT-4-231106, and GPT-4o show rapid declines in performance post-release.
“post-release.Conversely, models like Claude-3.5-Sonnet-20240620,GPT-4-231106, and GPT-4o show rapid declines in performance,”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataWhile the ranking of models varied slightly between datasets, the performance of individual models remained relatively stable over time within each dataset type.
“In general, while the ranking of models varied slightly between datasets, the performance of individual models remained relatively stable over time within each dataset type”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataGemini Pro had an accuracy of 0.37 in March 2024, which dropped to 0.17 by July 2024.
“For instance, Gemini.prohad an accuracy of 0.37 in March 2024, but this dropped sharply to 0.17 by July 2024, indicating a significant loss in precision for temporal tasks.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
Findingown dataThe model achieved lowest accuracy on the Finance (58.1%), Business (55.4%), and Economic Indicators (62.3%) categories.
“In addition, our exploratory analysis revealed significant performance variations across question categories, with the model achieving lowest accuracy on”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Findingown dataThe understanding of world events by models lingers around 2015, with a trend towards even earlier years.
“However, our findings indicate that the understanding of world events (even in cases where models can remember) lingers around 2015, with a trend towards even earlier years (beyond the time range of our test data).”
Quote in the paperSubstantial claim- Rests on an assumption: The Good Judgment Open questions used are representative of real-world events for evaluating temporal generalization.
- Rests on an assumption: BPC (bits per character) is a valid measure of a model's understanding and prediction of world events.
It may state two separate claims as one.
Findingown dataThe model achieved stronger performance on the Foreign Policy (81.2%), Security and Conflict (79.6%), and Leader Entry/Exit (83.7%) categories.
“compared to stronger performance on”
Quote in the paperNot a substantial claimProbably not a substantial claim.
InterpretationreasoningThe preference for older knowledge poses a risk as users expect up-to-date knowledge close to the model's release date.
“This poses a risk as users expect up-to-date knowledge close to the model's release date.”
Quote in the paperSubstantial claimIt may state two separate claims as one.
PremisecitationOpen-dialogue evaluations such as MT-Bench and Alpaca-Eval rely on subjective human or model-based judgments, making their outcomes susceptible to dialogue structure and potentially compromising assessment validity.
“Opendialogue evaluations like MT-Bench Zheng et al. (2023) and Alpaca-Eval Li et al. (2023a), rely on subjective human or model-based judgments, making their outcomes susceptible to the influence of dialogue structure, and thereby, potentially compromising assessment validity.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Zheng et al. (2023) source
- cites Li et al. (2023a)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. The paper may give no reason why it holds.
Study design and conductreasoningDefining the cutoff time as the model's release date ensures no data leakage for the model.
“Although the data after the cutoff time is not strictly future data, this approach ensures no data leakage for the model.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Study design and conductreasoningTemporal generalization evaluation eliminates the possibility of data leakage through temporal boundaries.
“eliminating the possibility of data leakage through temporal boundaries”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Premisenone givenCompression Intelligence is limited by its inapplicability to closed-source models, where key indicators like logits or losses are inaccessible.
“Compression Intelligence is limited by its inapplicability to closedsource models, where key indicators like logits or losses are inaccessible.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Interpretationnone givenAfter the cutoff time, a model's understanding of future trends declines, leading to temporal degeneration.
“However, after the cutoff time, the model's understanding of future trends declines, leading to temporal degeneration.”
Quote in the paperNot a substantial claimProbably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
Study design and conductnone givenTemporal generalization evaluation enables objective evaluation results without relying on subjective interpretations.
“enabling objective evaluation results without relying on subjective interpretations”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
InterpretationreasoningIt appears that humans may be adapting their writing to more closely resemble LLM outputs, rather than LLMs significantly enhancing their ability to model human language.
“In essence, it appears that humans may be adapting their writing to more closely resemble LLM outputs, rather than LLMs significantly enhancing their ability to model human language.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
Premisenone givenBBC texts are news articles and are highly time-sensitive.
“Conversely, BBC texts are news articles and are highly timesensitive.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Study design and conductnone givenThe Wikipedia dataset is continuously updated, so it is not easily subject to direct contamination.
“Since the Wikipedia dataset is continuously updated, it is not easily subject to direct contamination.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it.
PremisecitationIntegrating news content can significantly improve the accuracy of predictions in real-time scenarios.
“Halawi et al. (2024) showcased how integrating news content can significantly improve the accuracy of predictions in real-time scenarios.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Halawi et al. (2024)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Interpretationnone givenIf a model shows a decline in performance on Wikipedia, it is likely that the model is overfitting.
“If a model shows a decline in performance on Wikipedia, it is likely that the model is overfitting.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Premisenone givenLanguage likelihood can be biased by models finely tuned to specific formats or domains, skewing overall evaluation.
“Additionally, language likelihood can be biased by models finely tuned to specific formats or domains, skewing overall evaluation and presenting a distorted view of a model's general performance.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
PremisecitationKnowledge-based assessments raise concerns about real-world applicability and the risk of data leakage.
“applicability and the risk of data leakage Wei et al. (2023); Golchin and Surdeanu (2024).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Wei et al. (2023)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Golchin and Surdeanu (2024)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim.
InterpretationreasoningThe nonuniform, past-skewed knowledge distribution poses a potential risk in using these models because there is an expectation that knowledge should be as up-to-date as possible, ideally close to the model's release date.
“This poses a potential risk in our utilization of these models because there is an expectation that the knowledge utilized should be as up-to-date as possible, ideally close to the model's release date.”
Quote in the paperSubstantial claimProbably not a substantial claim.
Study design and conductreasoningThe ongoing online evaluation allows continuous evaluation with the latest data, ensuring the assessment reflects the model's current capabilities and the evolving information landscape.
“This approach also allows continuous evaluation with the latest data, ensuring the assessment reflects the model's current capabilities and the evolving information landscape.”
Quote in the paperSubstantial claimProbably not a substantial claim.
Study design and conductreasoningThe adapted BPC metric allows more equitable assessment of model performance across different languages and tokenization schemes by normalizing log likelihood based on character count.
“our adapted Bits Per Character (BPC) metric offers a comparison between models by normalizing the log likelihood based on character count, irrespective of the tokenization method, allowing for a more equitable assessment of model performance across different languages and tokenization schemes”
Quote in the paperSubstantial claimThe quote only partly states this claim.
Interpretationown dataMini models exhibit poor generalization over time, with performance rapidly declining in long-term prediction tasks.
“they exhibit poor generalization over time”
Quote in the paperSubstantial claimProbably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Study design and conductnone givenThe proportion and diversity of collected texts may fall short of fully reflecting the diversity and complexity of language.
“While we strive to collect textual data from a variety of sources such as news articles, scientific papers, and forum discussions, the proportion and diversity of the texts may still fall short of fully reflecting the diversity and complexity of language.”
Quote in the paperNot a substantial claimProbably not a substantial claim.
Study design and conductnone givenThe peak accuracy periods were analyzed based on the average accuracy of all questions within three-month intervals, to provide a stable and robust assessment of each model's performance.
“The peak accuracy periods analyzed are based on the average accuracy of all questions within three-month intervals, to provide a stable and robust assessment of each model's performance.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
Interpretationnone givenPrior to the cutoff time, an LLM may exhibit a Nostalgia bias skewed toward past dates or a Neophilia bias skewed toward future dates.
“In general, prior to the cutoff time, an LLM might exhibit a Nostalgia bias that is skewed toward past dates (i.e., aligning better with historical data) or a Neophilia bias that is skewed toward future dates ( favoring more recent information).”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper may give no reason why it holds.
Interpretationnone givenAfter the cutoff time, an LLM's understanding of future trends declines, leading to temporal degeneration.
“However, after the cutoff time, the model's understanding of future trends declines, leading to temporal degeneration.”
Quote in the paperNot a substantial claimIt may state two separate claims as one.
Interpretationown dataThe specific impacts of prompts vary from model to model and are closely related to the model's instruction-following capabilities.
“The specific impacts vary from model to model, which is closely related to the model's instruction-following capabilities.”
Quote in the paperNot a substantial claimProbably not a substantial claim.
Interpretationnone givenThe evaluation framework and datasets introduced allow for an in-depth assessment of model performance over time.
“The evaluation framework and datasets we introduce allow for an in-depth assessment of model performance over time.”
Quote in the paperSubstantial claimProbably not a substantial claim.
InterpretationreasoningThe evaluation method allows examining the model's ability to process and predict real-world events accurately and efficiently.
“This approach allows us to examine the model's ability to process and predict real-world events accurately and efficiently.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
Study design and conductreasoningThe evaluation method is resistant to data contamination, objective, and free from biases in question construction and judgment.
“This ensures that our evaluation is resistant to data contamination, objective, and free from biases in question construction and judgment.”
Quote in the paperNot a substantial claim- Rests on an assumption: BPC (bits per character) is a valid measure of a model's understanding and prediction of world events.
It may state two separate claims as one.
Interpretationown dataThe observation that understanding lingers around 2015 is contrary to the initial hypothesis that models may overfit current events.
“This observation is contrary to our initial hypothesis that models may overfit current events, which is significantly more concerning.”
Quote in the paperSubstantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Study design and conductnone givenThe English-centric nature of the questions and answers limits the benchmark's applicability to non-English speaking regions and global events.
“Also, the English-centric nature of the questions and answers limits the benchmark's applicability to non-English speaking regions and global events.”
Quote in the paperNot a substantial claimProbably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
Study design and conductnone givenEvaluations on text loss require access to the model's logits, which is challenging for closed-source models or models accessed via APIs.
“Moreover, such evaluations on text loss require access to the model's logits, which is challenging for closed-source models or models accessed via APIs.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
There is a gap in understanding how well LLMs can generalize across temporal contexts.
“there remains a gap in understanding how well these models can generalize across temporal contexts”
The paper relies on this without support.
GPT-4o's classification of Wiki and BBC content is accurate.
“asked gpt-4o to classify the content.”
The paper relies on this without support.
Data available after a question was posed and before the prediction market's closure can serve as valid future data for evaluating model prediction.
“We use data available after the event's question was posed and before the prediction market's closure, treating this as “future” data for model evaluation.”
The paper relies on this without support.
Formatting answers as ranges within a prediction market scenario ensures objectivity.
“To ensure objectivity in our predictions, answers are typically formatted as ranges within a prediction market scenario.”
The paper relies on this without support.
The approximately 500 sampled data pieces are representative of the Wiki and BBC datasets.
“we uniformly sampled approximately 500 data pieces from both the Wiki and BBC datasets”
The paper relies on this without support.
The Good Judgment Open questions used are representative of real-world events for evaluating temporal generalization.
“For this experiment, we sourced all questions from Good Judgment Open using web scraping.”
The paper relies on this without support.
BPC (bits per character) is a valid measure of a model's understanding and prediction of world events.
“Our method of evaluation is based on facts and computation loss.”
The paper relies on this without support.
Declines in benchmark accuracy over time reflect degradation in temporal generalization rather than other factors.
“We rigorously define and quantify temporal generalization and bias within LLMs, uncovering significant challenges in predicting and adapting to future contexts.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
A model improved despite the claim that all models declined
MinorThe text says all models had substantial post-release declines, but Table 4 reports Baichuan2-7B-Chat improving from 0.37 to 0.39, with a negative percentage decline.
“when predicting outcomes for events known only post-release, all models exhibited substantial declines in performance.”
“| Baichuan2-7B-Chat | 0.37 | 0.39 | -0.02 | -5.41% |”
Baichuan-13B-Chat appears with two release dates and TBI sets
MinorTable 8 lists the same model name twice, once as released in June 2023 and once in July 2023, with different TBI values.
“| Baichuan-13B-Chat | Jun 2023 | -11.7 | 685.7 | 4.8 |”
“| Baichuan-13B-Chat | Jul 2023 | -7.0 | 717.2 | 12.1 |”
The sign of the Temporal Bias Index is reversed in Table 8
MinorThe definition makes TBI the slope of BPC over time, and the main text says a positive slope means increasing BPC. Table 8 instead says a negative TBI indicates increasing BPC.
“Negative TBI values indicate a trend of increasing BPC, which refers to decreasing performance on MMLU.”
“A positive BPC slope indicates an increase in BPC over time, reflecting performance degeneration.”
Citations
Cites
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
Humans or LLMs as the Judge? A Study on Judgement Bias
Chen, G. H., Chen, S., Liu, Z., Jiang, F., & Wang, B. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.474 · Cited by 76
Self-citationExploring the Impact of Temporal Bias in Point-of-Interest Recommendation
Rahmani, H. A., Naghiaei, M., Tourani, A., & Deldjoo, Y. (2022) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3523227.3551481 · Cited by 14
Humans or LLMs as the Judge? A Study on Judgement Biases
Chen, G. H., Chen, S., Liu, Z., Jiang, F., & Wang, B. (2024) · arXiv · doi:10.48550/arxiv.2402.10669 · Cited by 9
Self-citation
Flagged first, then by citations
Research works printed but not linked: 9
- Dale Dorsey. 2021. The Possibility of Temporal Bias. In A Theory of Prudence. Oxford University Press. · DOI
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Infor- mation Processing Systems.
- Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junx- ian He. 2024. Compression represents intelligence linearly. CoRR, abs/2404.09937. · DOI · Paper page
- Yoan Russac, Christina Katsimerou, Dennis Bohle, Olivier Cappé, Aurélien Garivier, and Wouter M Koolen. 2021. A/b/n testing with control in the pres- ence of subpopulations. In Advances in Neural In- formation Processing Systems, volume 34, pages 25100-25110.
- Suproteem Sarkar and Keyon Vafa. 2024. Lookahead bias in pretrained language models. SSRN Electronic Journal.
- Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, Rafael Valdece Sousa Bastos, and Philip E. Tetlock. 2024. Wisdom of the silicon crowd: Llm ensemble prediction capabilities rival human crowd accuracy. ArXiv.
- Michael Zhang and Eunsol Choi. 2021. SituatedQA: In- corporating extra-linguistic contexts into QA. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 7371- 7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
- Hao Zhou, Vamsi K Ithapu, Sathya Narayanan Ravi, Vikas Singh, Grace Wahba, and Sterling C Johnson. 2016. Hypothesis testing in unsupervised domain adaptation with applications in alzheimer's disease. In Advances in Neural Information Processing Sys- tems, volume 29.
Cited by
Sorted by citations
Six Fallacies in Substituting Large Language Models for Human Participants
Lin, Z. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251357566 · Cited by 30
On the Fundamental Limits of LLMs at ScalePreprint
Mohsin, M. A., Umer, M., Bilal, A., Memon, Z., Qadir, M. I., Bhattacharya, S., Rizwan, H., Gorle, A. R., Kazmi, M. Z., Amir, N., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.12869 · Cited by 29
Content moderation by LLM: from accuracy to legitimacy
Huang, T. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11328-1 · Cited by 24
Safety challenges of AI in medicine in the era of large language modelsPreprint
Wang, X., Zhang, N. X., He, H., Nguyen, T. T. H., Yu, K., Deng, H., Brandt, C., Bitterman, D. S., Pan, L., Cheng, C., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.18968 · Cited by 23
An agentic AI framework for ingestion and standardization of single-cell RNA-seq data analysis
Nouri, N., Artzi, R., & Savova, V. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-025-00064-0 · Cited by 22
ProveRAG: Provenance-Driven Vulnerability Analysis With Automated Retrieval-Augmented LLMs
Fayyazi, R., Trueba, S. H., Zuzak, M., & Yang, S. J. (2025) · IEEE Access · doi:10.1109/access.2025.3638251 · Cited by 18
Challenges in Guardrailing Large Language Models for SciencePreprint
Pantha, N., Ramasubramanian, M., Gurung, I., Maskey, M., & Ramachandran, R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.08181 · Cited by 16
Security and privacy in LLMs: A comprehensive survey of threats and mitigation strategies
Berini, A. D. E., Jamil, N., Benrazek, A., Lakas, A., Ismail, L., Ferrag, M. A., & Lam, K. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104241 · Cited by 13
Wang, S., Zhao, F., Bu, D., Lu, Y., Gong, M., Liu, H., Yang, Z., Zeng, X., Yuan, Z., Wan, B., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64142-2 · Cited by 13
MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher AssistantsPreprint
Ding, D., Wang, T., Zhu, C., Tao, M., Jiang, Y. E., & Zhou, W. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.01887 · Cited by 11
Self-citationLarge-Scale, Longitudinal Study of Large Language Models During the 2024 US Election SeasonPreprint
Cen, S. H., Ilyas, A., Driss, H., Park, C., Hopkins, A., Podimata, C., & Mądry, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.18446 · Cited by 9
LLMLagBench: Identifying Temporal Training Boundaries in Large Language ModelsPreprint
Pęzik, P., Kaczyński, K., Szymańska, M., Żarnecki, F., Deckert, Z., Kwiatkowski, J., & Janowski, W. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.12116 · Cited by 7
A Survey of RWKVPreprint
Li, Z., Xia, T., Chang, Y., & Yuan, W. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.14847 · Cited by 6
Sun, D., Yang, D., Shen, Y., Jiao, Y., Tan, Z., Feng, J., Zhong, L., Wang, J., Wei, P., & Gu, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.09713 · Cited by 6
Li, Z., Xia, T., Chang, Y., & Wu, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130711 · Cited by 5
The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language ModelsPreprint
Gienapp, L., Schröder, C., Schweter, S., Akiki, C., Schlatt, F., Zimmermann, A., Genêt, P., & Potthast, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.13996 · Cited by 5
A Survey of Hallucination in Large Language Models
何, 金. (2026) · Artificial Intelligence and Robotics Research · doi:10.12677/airr.2026.151016 · Cited by 4
A Generative AI–Based Technical Data Extraction Tool for IoT Application Systems
Kong, D., Funabiki, N., Kyaw, H. H. S., Kotama, I. N. D., Zhu, Z., & Rahmadani, A. A. (2026) · Sensors · doi:10.3390/s26041081 · Cited by 4
Evaluating large language models for pharmacotherapy simulations: a mixed-methods study
Farrag, A. N., El-Zeiny, A., & Ali, A. M. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02626-1 · Cited by 2
Kumar, N. D., Ramavath, S. K., & Venkatesh, V. (2025) · International Generative AI and Computational Language Modelling Conference (GACLM) · doi:10.1109/gaclm67198.2025.11232293 · Cited by 2
TiMoE: Time-Aware Mixture of Language ExpertsPreprint
Faro, R., Fan, D., Alphaidze, T., & Jaggi, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.08827 · Cited by 2
Prior, M., Schultz, A., & Grabmair, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.23497 · Cited by 2
Raza, F. M., Yang, S. L., Srinivasan, S., & DeFranco, J. F. (2026) · Computer · doi:10.1109/mc.2026.3664470 · Cited by 1
Towards Knowledge Alignment in Code LLMs: Contrastive Unlearning for Evolving APIs
Tran, H. Q., Vu, D. H., Dinh, T. N., Nguyen, A. H. D., Vu, A. N. H., Bui, A. M. T., & Than, N. P. (2026) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme67971.2026.00068 · Cited by 1
Impacts of Racial Bias in Historical Training Data for News AIPreprint
Bhargava, R., Jespersen, M. H., Ndulue, E. B., & Dsouza, V. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.16901 · Cited by 1
NeuralDB: Scaling Knowledge Editing in LLMs to 100,000 Facts with Neural KV DatabasePreprint
Fei, W., Shi, H., Xu, J. M., Peng, J., Li, J., Zhang, J., Bai, B. B., Han, W., Chen, Z., & Niu, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.18028
DRBO: Mitigating Short Board Effect via Dynamic Reward Balancing in Multi-reward LLM Optimization
Chen, N., Gao, Y., Jin, Y., Hu, Y., Gao, A., Yan, L., & Wang, B. (2025) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2025.findings-emnlp.468
Self-citationScalable Multi-resolution Graph Representation Learning: Towards Knowledge-Enhanced Language Models
Namazi, R. (2024) · MacSphere (McMaster University) · doi:10.71548/14930
A Conceptual Framework for Simulated Self-Assessment and Meta-Evaluation of Generative AI Models
Yotov, K., Hadzhikoleva, S., Hadzhikolev, E. N., Milev, M., & Rachovski, T. (2026) · AI · doi:10.3390/ai7040134
Egbueri, J. C., Agbasi, J. C., Usman, A. G., Uwajingba, H. C., & Abba, S. I. (2026) · Springer hydrogeology · doi:10.1007/978-3-032-18853-3_6
LLMs for Venture Capital Investment: Approaches and Open Problems
Galang, J. A. C., Mosca, E., Malberg, S., & Groh, G. (2026) · IFIP advances in information and communication technology · doi:10.1007/978-3-032-30809-2_12
Deep Research in the Era of Agentic AI: Requirements and Limitations for Scholarly Research
Jaradeh, M. Y., & Auer, S. (2025) · RENATE · doi:10.34657/39960
Language Models as Forecasters
Yuan, M. (2026) · Apollo (University of Cambridge) · doi:10.17863/cam.136270
Metadata
- Authors
- Zhu, C., Chen, N., Gao, Y., Zhang, Y., Tiwari, P., & Wang, B.
- Affiliations
- Shenzhen Research Institute of Big Data (CN)
- Chinese University of Hong Kong, Shenzhen (CN)
- Halmstad University (SE)
- Venue
- Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
- Published
- 2025
- Pages
- 7433-7457
- OpenAlex
- W4411119852
- Cited by
- 33
- References
- 3
- Citations by year
- 2026: 12 · 2025: 17
- FWCI
- 10.67
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Natural Science Foundation of China (12326608)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- temporal generalization, large language models, temporal grounding, temporal reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
