CoChat
Sign Up
PreprintCited by 2scientometrics and bibliometrics researchLibrary Science and Information SystemsWeb visibility and informetrics

The DOI matches its Crossref record; 12 of the 14 references it cites were found in a registry or an index; 3 of the 5 sources its claims cite were checked for support.

Analysis of the Deletions of DOIs: What Factors Undermine Their Persistence and to What Extent?

Open access
Kikkawa, Jiro, Takaku, Masao, Yoshikane, Fuyuki·2022·Lecture Notes in Computer Science
DOI 10.1007/978-3-031-16802-4_13arXiv 2207.12018
Open this paper in CoChat — free accountOpen PDF

Abstract

Digital Object Identifiers (DOIs) are regarded as persistent; however, they are sometimes deleted. Deleted DOIs are an important issue not only for persistent access to scholarly content but also for bibliometrics, because they may cause problems in correctly identifying scholarly articles. However, little is known about how much of deleted DOIs and what causes them. We identified deleted DOIs by comparing the datasets of all Crossref DOIs on two different dates, investigated the number of deleted DOIs in the scholarly content along with the corresponding document types, and analyzed the factors that cause deleted DOIs. Using the proposed method, 708,282 deleted DOIs were identified. The majority corresponded to individual scholarly articles such as journal articles, proceedings articles, and book chapters. There were cases of many DOIs assigned to the same content, e.g., retracted journal articles and abstracts of international conferences. We show the publishers and academic societies which are the most common in deleted DOIs. In addition, the top cases of single scholarly content with a large number of deleted DOIs were revealed. The findings of this study are useful for citation analysis and altmetrics, as well as for avoiding deleted DOIs.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

The paper identifies deleted Crossref DOIs by comparing all Crossref DOI datasets from March 2017 and January 2021, finding 708,282 deleted DOIs. Most were alias DOIs for individual scholarly articles, often caused by multiple DOI assignments, retractions, conference abstracts, and re-registration by publishers or societies.

Key findings

  • 708,282 deleted DOIs were identified, equal to 0.59% of all DOIs in 2021; 94.29% (667,869) were Alias DOIs, indicating most deletions came from multiple DOI assignments to the same content.
  • The top document types were journal articles (78.35%), proceedings articles (8.48%), and book chapters (6.77%), so most deleted DOIs corresponded to individual scholarly articles rather than whole books or journals.
  • The top prefixes included Wiley (94,471 deleted DOIs; P1 13.34%, P2 2.00%), Elsevier BV (85,643; 12.09%, 0.58%), JSTOR (66,232; 9.35%, 2.82%), American Psychological Association (51,846; 7.32%, 6.74%), and Society of Petroleum Engineers (35,499; 5.01%, 87.62%).
  • For DOI name changes among Alias/Primary pairs, 69.74% changed only the suffix, 23.46% changed both prefix and suffix, and 6.80% changed only the prefix; the most frequent suffix change was deleting a slash once (10.56%).

Methods

  • The authors compared two datasets of all Crossref DOIs: March 2017 (87,542,203 unique DOIs) and January 2021 (120,684,617 unique DOIs), yielding a difference set of 711,198 candidate DOIs.
  • Candidates were classified using the “Which RA?” API, DOI-link HTTP headers, and the Crossref REST API; 708,282 were treated as deleted DOIs, including Alias DOIs, defunct DOIs, DOIs without redirects, and DOIs with deleted metadata descriptions.
  • Analyses covered basic statistics, Crossref metadata document types and Alias-DOI counts per Primary DOI, prefix-level deletion ratios P1 and P2, and suffix-change patterns using Levenshtein similarity and diff-lcs character diffs.

Limitations

  • The paper states it could not identify DOIs deleted before March 2017 or after March 2021, could not cover other unreachable contents, and defined deleted DOIs only as deletions of identifiers and metadata.
  • The paper states it could not classify all causes of deleted DOIs because it focused on the most frequent cases, and it is unclear whether the observed factors represent exceptional cases.
  • The paper states that quantitative analysis of deleted DOIs’ effects on citation analysis and altmetrics remains important, and that DOIs for preprint servers such as arXiv were beyond scope; the method also depends on two snapshot dates and API/HTTP responses at the time of checking.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/978-3-031-16802-4_13

Cites

12 of 14 references verified0 citation errors · 2 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

3 of 5 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    The study identified 708,282 deleted DOIs by comparing Crossref DOI datasets from March 2017 and January 2021.

    “Using the proposed method, 708,282 deleted DOIs were identified.”
    • Also stated in: § 4.1 Basic Statistics of Deleted DOIs · plain
      “We identified 708,282 deleted DOIs by applying the method described in Section 3.1.”
    • Also stated in: § 5 Conclusion · plain
      “we identified 708,282 deleted DOIs that existed in March 2017 and did not exist in January 2021 using the proposed method.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [9] Himmelstein, D., Wheeler, K., Greene, C.: Metadata for all DOIs in Crossref: JSON MongoDB exports of all works from the Crossref API. figshare (2017). https://doi.org/10.6084/m9.figshare.4816720.v1 source
    • cites [2] Crossref: January 2021 Public Data File from Crossref. Academic Torrents. https://doi.org/10.13003/gu3dqmjvg4 source
    • cites [10] Kemp, J.: New public data file: 120+ million metadata records (2021). https://www.crossref.org/blog/new-public-data-file-120-million-metadata-records/ source

      The announcement describes a Crossref metadata file containing records registered through January 7, 2021 (section labeled “2026 September 17”; no page number). Our copy does not report a comparison with a March 2017 dataset or identify 708,282 deleted DOIs. Searches for that number and for the proposed dataset comparison found no supporting passage. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Findingnone given

    In case #2, all articles published in Volume 11 of Energy Procedia were withdrawn.

    “In case #2, all articles published in Volume 11 of Energy Procedia were withdrawn.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The registrants with the largest numbers of deleted DOIs were major international commercial publishers, such as Wiley, Elsevier BV, and Springer Science and Business Media LLC (#1, #2, #9, and #10).

    “As indicated by Table 5, the registrants with the largest numbers of deleted DOIs were major international commercial publishers, such as Wiley, Elsevier BV, and Springer Science and Business Media LLC (#1, #2, #9, and #10).”
    Quote in the paperSubstantial claim
  • Findingown data

    There were cases where a large number of DOIs were assigned to a single content item.

    “On the other hand, focusing on the maximum value, there were cases where a large number of DOIs were assigned to a single content item.”
    Quote in the paperSubstantial claim
  • Findingown data

    The percentage of deleted DOIs relative to all DOIs in 2021 was 0.59%.

    “The percentage of deleted DOIs relative to all DOIs in 2021 was 0.59% (= 708, 282/120, 684, 617 * 100).”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim. 5
  • It may state two separate claims as one. 15

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data

    These results match those of Tkaczyk [18].

    “These results match those of Tkaczyk [18]”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [18] Tkaczyk, D.: Double trouble with DOIs - Crossref (2020). https://www.crossref.org/blog/double-trouble-with-dois/ source

      The Introduction’s “Overall results” section reports that Tkaczyk flagged 4,240 of 524,496 DOIs (0.8%) as duplicates. I searched for the cited work’s results, but the citing sentence does not identify which of its own results supposedly match them, so the comparison cannot be checked from our copy. (LLM, conf=0.74)

      Experimental — accuracy not yet measured

  • It may state two separate claims as one.

    Interpretationown data

    A large amount of scholarly content provided by JSTOR was transferred to other registrants, and their DOIs were re-registered with the prefixes of new registrants.

    “indicate that a large amount of scholarly content provided by JSTOR was transferred to other registrants, and their DOIs were re-registered with the prefixes of new registrants”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [18] Tkaczyk, D.: Double trouble with DOIs - Crossref (2020). https://www.crossref.org/blog/double-trouble-with-dois/ source

      The Introduction reports that JSTOR’s sampled DOIs had duplicates deposited by eight other members, and that JSTOR deposited certain shared DOIs first. This supports the existence of new DOIs for matching content under other registrants, but the paper does not establish that JSTOR’s content was transferred to them or quantify such a transfer as large. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • It may state two separate claims as one.

    Findingown data

    IGI Global and the American Psychological Association (#7 and #4) had relatively high P2 values of 17.51% and 6.74%, respectively.

    “IGI Global and the American Psychological Association (#7 and #4) had relatively high P 2 values of 17.51% and 6.74%, respectively.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    The minimum, maximum, median, and standard deviation of the number of alias DOIs corresponding to each primary DOI were 1, 1,474, 1, and 2.49, respectively.

    “The minimum value, maximum value, median, and standard deviation of the number of alias DOIs corresponding to each primary DOI were 1, 1,474, 1, and 2.49, respectively.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it.

    Findingown data

    Case #10 involved an abstract of an international conference published in another journal.

    “Case #10 involved an abstract of an international conference published in another journal.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    These publishers registered numerous DOIs, but few of them were deleted, as the P2 values for these publishers were low.

    “These publishers registered numerous DOIs, but few of them were deleted, as the P 2 values for these publishers were low.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Findingown data

    The Society of Petroleum Engineers (#5) had a remarkably high P2 value of 87.62%; thus, most of the DOIs registered by this registrant with this prefix were deleted.

    “In contrast, the”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    Cases #4-#9 were similar to #3 and all involved the Journal of the American College of Cardiology.

    “Cases #4-#9 were similar to #3 and all involved the Journal of the American College of Cardiology.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    Of the 45,430 cases of 'only the prefix changed,' 35,176 involved the prefix '10.2523' of Alias DOIs and the prefix '10.2118' of Primary DOIs, and the corresponding registrant was the Society of Petroleum Engineers.

    “Of the 45,430 cases of”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    The group 'only the suffix changed' had a large percentage of cases with Sim values of > 0.7, indicating that most of the cases had a high degree of similarity among suffixes.

    “The group”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Findingown data

    The prefix '10.2307' was changed to '10.1090 (American Mathematical Society)' and '10.1080 (Informa UK Limited)' in 22,198 and 10,010 cases, respectively.

    “The prefix”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given

    This problem would be caused by the registration agency improperly managing their content.

    “This problem would be caused by the registration agency improperly managing their content.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationreasoning

    This content would not be the target of citation analysis and altmetrics and did not have a critical impact on them.

    “This content would not be the target of citation analysis and altmetrics and did not have a critical impact on them.”
    Quote in the paperSubstantial claim

  • It may state two separate claims as one.

    Interpretationown data

    The majority of cases were those in which two DOIs were registered for the same content, and one of them was deleted because the median value was 1.

    “The majority of cases were those in which two DOIs were registered for the same content, and one of them was deleted because the median value was 1.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationown data

    Case #3 involved abstracts of an international conference, where DOIs were registered to the abstracts of the presentations, but the publisher appeared to change the policy for registering DOIs to a set of abstracts.

    “Case #3 involved abstracts of an international conference, where DOIs were registered to the abstracts of the presentations, but the publisher appeared to change the policy for registering DOIs to a set of abstracts.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationreasoning

    The Primary DOIs with large numbers of Alias DOIs were caused by the activities of publishers and academic societies rather than by the deletion of multiple DOIs mistakenly registered to the same scholarly articles.

    “The Primary DOIs with large numbers of Alias DOIs were caused by the activities of publishers and academic societies rather than by the deletion of multiple DOIs mistakenly registered to the same scholarly articles.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: In the DOI system, a single and unique DOI is registered for each content item.

  • Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.

    Interpretationnone given

    Regarding JSTOR, the deletions of DOI seem to be caused by the re-registration of DOIs to the same article published by content holders other than JSTOR.

    “Regarding JSTOR, the deletions of DOI seem to be caused by the re-registration of DOIs to the same article published by content holders other than JSTOR.”
    Quote in the paperNot a substantial claim

  • It may state two separate claims as one.

    Interpretationown data

    The prefix '10.2523' is listed as #5 in Table 5, and these cases were due to DOIs being re-registered by the same publisher.

    “This is why the prefix”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.

    Interpretationnone given

    'both the prefix and the suffix changed' and 'only the prefix changed' were cases of deletions in which the DOIs were re-registered with the content holder's prefix when the content holder changed.

    “In contrast,”
    Quote in the paperNot a substantial claim
19 of 19

  • The paper relies on this without support.

    DOIs are regarded as persistent and are not deleted once registered.

    “Once registered, the DOIs are regarded as persistent and are not deleted.”

  • The paper relies on this without support.

    In the DOI system, a single and unique DOI is registered for each content item.

    “In the DOI system, a single and unique DOI is registered for each content item.”

  • The paper relies on this without support.

    Deleted DOIs cause problems for persistent access to scholarly content and for bibliometric analysis, including citation analysis and altmetrics.

    “These deletions cause problems not only for persistent access to scholarly content but also for bibliometric analysis, including citation analysis and altmetrics.”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • A January 2021 dataset is described as a March 2021 dataset

    Minor

    The methods identify January 2021 as the later dataset, but the explanation of Non-Crossref DOIs says that dataset was from March 2021.

1 of 1

Cited works that were retracted

Works this paper cites that were later retracted, from the retraction notices publishers deposit. Major when the notice came out before this paper was published.

  • Cites a retracted work: 10.1016/s1876-6102(14)00453-6

    Major

    The cited work was retracted on 2011-01-01 (notice 10.1016/s1876-6102(14)00453-6), before this paper was published (2022). Its authors could have known.

    Cited work 10.1016/s1876-6102(14)00453-6 · retracted 2011-01-01 · notice 10.1016/s1876-6102(14)00453-6 · this paper published 2022

  • Cites a retracted work: 10.1016/s1876-6102(14)00454-8

    Major

    The cited work was retracted on 2011-01-01 (notice 10.1016/s1876-6102(14)00454-8), before this paper was published (2022). Its authors could have known.

    Cited work 10.1016/s1876-6102(14)00454-8 · retracted 2011-01-01 · notice 10.1016/s1876-6102(14)00454-8 · this paper published 2022

2 of 2

Citations

Cites

2 retracted

Flagged first, then by citations

Research works printed but not linked: 3

  1. Cornell University: New arXiv articles are now automatically assigned DOIs $$|$$arXiv.org blog (2022). https://blog.arxiv.org/2022/02/17/new-arxiv-articles-are-now-automatically-assigned-dois/
  2. Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. Sov. Phys.-Dokl. 10(8), 707–710 (1966)
  3. The International DOI Foundation: Factsheet DOI Resolution Documentation - 4. Which RA? (2020). https://www.doi.org/factsheets/DOIProxy.html#whichra

Cited by

Sorted by citations

Related

Metadata

Authors
Kikkawa, J., Takaku, M., & Yoshikane, F.
Affiliations
University of Tsukuba (JP)
Corresponding author
Kikkawa, Jiro
ORCID
Venue
Lecture Notes in Computer Science
Publisher
Springer International Publishing
Published
2022
Pages
161-174
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
DOI
10.1007/978-3-031-16802-4_13
arXiv
2207.12018
OpenAlex
W4295940503
Cited by
2
References
9
Citations by year
2026: 1 · 2022: 1
FWCI
1.21
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Japan Society for the Promotion of Science (22K18147, 21K21303)
Topics
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Physical Sciences › Computer Science › Information Systems › Library Science and Information Systems
  • Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
Keywords
digital object identifiers, persistent identifiers, Crossref, retracted article, altmetrics, citation analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-08, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-08 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review