Cited by 1,023ERP Systems Implementation and ImpactInformation Systems Theories and ImplementationInformation Technology Governance and Strategy

Learning to Implement Enterprise Systems: An Exploratory Study of the Dialectics of Change

Open access
Robey, Daniel, Ross, Jeanne W., Boudreau, Marie-Claude·2002·Journal of Management Information Systems

Abstract

This paper reports on a comparative case study of 13 industrial firms that implemented an enterprise resource planning (ERP) system. It compares firms based on their dialectic learning process. All firms had to overcome knowledge barriers of two types: those associated with the configuration of the ERP package, and those associated with the assimilation of new work processes. We found that both strong core teams and carefully managed consulting relationships addressed configuration knowledge barriers. User training that included both technical and business processes, along with a phased implementation approach, helped firms to overcome assimilation knowledge barriers. However, all firms in this study experienced ongoing concerns with assimilation knowledge barriers, and we observed two different approaches to address them. In a piecemeal approach, firms concentrated on the technology first and deferred consideration of process changes. In a concerted approach, both the technology and process changes were undertaken together. Although most respondents clearly stated a preference for either piecemeal or concerted change, all firms engaged in practices that reflected a combination of these approaches.

AI snapshot

Written from the paper’s full text.

TL;DR

This comparative case study examines 13 North American manufacturing firms that implemented enterprise resource planning systems, using a dialectic learning lens to analyze how they overcame knowledge barriers. It distinguishes configuration and assimilation barriers, and contrasts piecemeal with concerted implementation approaches.

Key findings

  • Firms faced two main knowledge barriers: configuring the ERP package and assimilating new work processes. More successful firms managed consultants carefully and transferred external knowledge; less successful firms either ignored consultant advice or became overdependent, with one firm paying 60% of project cost to consultants.
  • Training budgets ranged from 1% to 20% of total project budget. More successful firms supplemented system training with education on new processes and change management, while incremental implementation helped users assimilate new work processes.
  • Eight firms used a piecemeal approach, implementing ERP before or separately from process reengineering, while five used a concerted approach, redesigning processes alongside ERP. Both groups ultimately had to change systems and processes together, and the distinction was clearer in stated philosophy than in practice; no evidence showed concerted approaches led to greater success.
  • Outcomes were mixed: positive impacts included supply chain efficiency, improved financial accounting, greater data visibility, and process-centered thinking; negative impacts included data inaccuracy, lost reporting capabilities, user resistance, strained customer relationships, and loss of skilled people. Most respondents said it was too early to tell ultimate benefits.

Methods

  • Comparative case study of 13 North American manufacturing firms or divisions that had completed a major ERP implementation from SAP, Oracle, Baan, or PeopleSoft. The original sample had 16 sites, but 3 had insufficient data; firms initially had to exceed $500 million in revenue, though this criterion was later relaxed.
  • One author interviewed 16 sites as part of a larger ERP study. The analysis used a dialectic process perspective, focusing on obstacles encountered during implementation and the mechanisms used to overcome them.
  • Firms were categorized by relative success in overcoming configuration and assimilation knowledge barriers, and by whether they followed piecemeal or concerted change processes.

Limitations

  • The paper states that many firms were still in post-implementation adjustment, most said it was too early to tell ultimate benefits, and the authors could not gauge ultimate success. It also notes that interviews with senior executives gave no direct access to situated learning, and that the study does not address whether ERP is an effective match for organizations or verify concerns about lost functionality.
  • Evident from the text: the sample is limited to 13 North American manufacturing firms or divisions, partly selected through sponsor and consulting contacts, which constrains generalizability. The data are retrospective interview accounts, and outcomes were unstable at the time of study, making causal or long-term conclusions difficult.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1080/07421222.2002.11045713

Cites

58 of 58 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 3 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Only two cases (PlastiCo and ContainerCo) had mixed results, successful in configuring ERP but less successful in assimilating new work processes.

    “Only two cases (PlastiCo and ContainerCo) had mixed results, in that they were judged successfiil in configunng ERP but less successfiil in assimilating new work processes.”
    Quote in the paperNot a substantial claim
  • Findingown data

    All firms had to overcome two types of knowledge barriers: configuration of the ERP package and assimilation of new work processes.

    “All firms had to overcome knowledge bamers of two types: those associated with the configuration of the ERP package, and those associated with the assimilation of new work processes.”
    Quote in the paperSubstantial claim
  • Findingown data

    User knowledge barriers were overcome more effectively when change was introduced incrementally.

    “Although ERP is often associated with radical change, our results suggest that user knowledge barriers were overcome more effectively when change was introduced incrementallv”
    Quote in the paperSubstantial claim
  • Findingown data

    Eight companies adopted a piecemeal approach.

    “The eight companies that adopted a piecemeal approach (TeleCo, HealthCo, MetalCo, WearCo, PlastiCo, CommCo AutoCo, and EquipCo) implemented ERP either prior to or apart from a business process reengineering initiative.”
    Quote in the paperSubstantial claim
  • Findingown data

    The remaining five firms pursued a concerted change approach.

    “The remaining five firms (ComputerCo, TextileCo, IndusCo, ContainerCo, and PharmaCo) pursued a concerted change approach, focusing on business process redesign along with ERP implementation.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 30
  • Classified as probably not a substantial claim. 12
  • The quote only partly states this claim. 2
  • May combine two claims.

    Findingown data

    Observed ERP implementation approaches were piecemeal and concerted.

    “We also observed different ERP implementation approaches: piecemeal and concerted.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Respondents described a mixture of positive and negative impacts, but most said it was too early to tell ultimate benefits.

    “Respondents described a mixture of positive and negative impacts resulting from their ERP implementation, but most said that it was “too early to tell” what ultimate benefits might result.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    In less successful cases, companies either depended too little on consulting advice or became over dependent.

    “In the less successfiil cases, companies either depended too little on consulting advice or became over dependent.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Negative outcomes included data inaccuracy, loss of reporting capabilities, resistance by users, strained customer relationships, and loss of skilled people.

    “Negative outcomes included problems of data inaccuracy, loss of reporting capabilities, resistance by users, strained relationships with customers, and loss of skilled people.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    Performance improvements included greater efficiency within supply chains, improved financial accounting, greater data visibility and analysis capability, and more process-centered thinking.

    “Performance improvements included greater efficiency within supply chains, improved financial accounting, greater data visibility and analysis capability, and more process-centered thinking.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    EquipCo attempted to circumvent process change by customizing and changing program code to make the system fit its existing processes and experienced extraordinary technological problems as a result.

    “The one firm (EquipCo) that attempted to circumvent process change by customizing and changing program code to make the system fit its existing processes experienced extraordinary technological problems as a result of its modifications.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingnone given

    Firms that were less successful in the management of their consultant relationships made one of two mistakes: they were either too dependent on consultants' advice or they distrusted them.

    “Firms that were less successful in the management of their consultant relationships made one of two mistakes: they were either too dependent on consultants' advice or they distrusted them.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    At PharmaCo, a respondent estimated that it could take about three years to get full knowledge of the software.

    “one respondent from PharmaCo estimated that it could take about tliree years to get ftill knowledge of the software”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingnone given

    Firms differ in their implementation approach, some choosing a piecemeal approach and others a concerted approach.

    “Moreover, firms differ in their implementation approach, some choosing a piecemeal approach and others a concerted approach.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    Although standard business processes were part of the corporate model for the first ERP implementation, fears were that divisions would modify their individual processes later.

    “Although standard business processes were part of the corporate model for the first ERP implementation, fears were that divisions would modify their individual processes later.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    The processes designed into EquipCo's system failed to leverage the capabilities of the ERP and severely limited the firm's ability to use the data effectively.

    “In addition, it was clear that the processes that had been designed into the system failed to leverage the capabilities of the ERP and severely limited the firm's ability to use the data effectively.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    ContainerCo actually implemented its ERP modules into traditional functional departments, foregoing the advantages potentially associated with process integration.

    “However, ContainerCo actually implemented its ERP modules into traditional fiinctional departments, foregoing the advantages potentially associated with process integration.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingown data

    The most successful firms in the sample limited their dependence on external consultants and took measures to ensure the transfer of external knowledge to the organization.

    “However, the most successful firms in our sample limited their dependence on external consultants and took measures to ensure the transfer of external knowledge to the organization.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    ERP was regarded as dismantling not only existing business processes but also the understanding of how work ought to be structured.

    “ERP was regarded as dismantling not only existing business processes but also the understanding of how work ought to be structured.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Findingreasoning

    Although the piecemeal approach was adopted to separate the two types of learning requirements, vanilla versions required some business processes to change immediately.

    “Although this approach was adopted to separate the two types of leaming requirements, vanilla versions required some business processes to change immediately.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingcitation

    Firms adopting either approach also faced a future with ERP upgrades and new software releases, and few firms regarded their implementation as complete simply because the installed version was working.

    “Moreover, firms adopting either approach also faced a future with ERP upgrades and new software releases (Glass, 1998), and few firms regarded their implementation as “complete” simply because the installed version was working.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Glass, 1998 source
  • May combine two claims.

    Findingcitation

    Although the concerted approach increased the learning challenge, these firms were more likely to report full transformations rather than wait for the future.

    “Although this increased the learning challenge, these tlrms were more likely to report fiill transformations rather than wait for the lliture.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. The quote only partly states this claim.

    Findingown data

    Many respondents were candidly doubtful that the business changes would indeed follow the ERP implementation, citing the resistant organizational cultures out in the operating divisions.

    “Many respondents were candidly doubtful that the business changes would indeed follow the BRP implementation, citing the resistant organizational cultures out in the operating divisions.”
    Quote in the paperSubstantial claim
  • The quote only partly states this claim.

    Findingown data

    According to respondents, ERP implementation challenged established knowledge in two ways.

    “According to respondents, BRP implementation challenged established knowledge in two ways.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingreasoning

    Core teams became a key repository of new knowledge and helped to distribute knowledge throughout organizations as they came into contact with user areas.

    “Not only did core teams become a key repository of new knowledge, but they also helped to distribute knowledge throughout organizations as they carne into contact with user areas.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingnone given

    Whereas most firms had once built information systems around the requirements of their existing operational processes, most were concluding that big packages sometimes demanded that business processes adapt to assumptions built into the software.

    “Whereas most firms had once built information systems around the requirements of their existing operational processes, most were concluding that big packages sometimes demanded that business processes adapt to assumptions built into the software.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingown data

    Respondents described organizational forces that promoted learning about new systems and processes as well as forces that sought to maintain the status quo.

    “In summary, respondents descnbed organizational forces that promoted leammg about new systems and processes as well as forces that sought to maintain the status quo.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Findingnone given

    The effectiveness of core teams depended on their size, representation, and their ability to avoid turnover of key members.

    “The effectiveness of core teams depended on their size, representation, and their ability to avoid turnover of key members.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim.

    Premisecitation

    Past research has revealed that changes induced by information technology are often resisted, unanticipated, or contradictory.

    “Because past research has revealed that changes induced by mformation technology are often resisted, unanticipated, or contradictory, a theory embedding a motor that acknowledges the opposing forces contnbuting to change promises useful analytical power (Robey and Boudreau, 1999).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Robey and Boudreau, 1999 source
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premiseown data

    ERP systems required working more cross-functionally than before because of their integrated design.

    “For example, ERP systems required working more cross-fiinctionally than before simply because of their integrated design.”
    Quote in the paperNot a substantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    These statements indicate the presence of barriers that require substantial efforts to overcome.

    “Clearly, these statements indicate the presence of barriers that require substantial efforts to overcome.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationreasoning

    Concerted change is practically forced upon most firms by virtue of ERP's integrated nature; old business processes simply cannot be performed on new software.

    “Concerted change is practically forced upon most firms by virtue of ERP's integrated nature; old business processes simply cannot be performed on new software.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Because ERP technology is so new and complex, it is difficult for a company to embark on such an endeavor without external knowledge.

    “Because HRP technology is so new and complex, it is difficult for a company to embark on such an endeavor without external knowledge.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: ERP technology is so new and complex that it is difficult for a company to embark on an ERP implementation without external knowledge.
  • May combine two claims.

    Interpretationreasoning

    ERP tends to be associated with integrated, process-centered models of organization, and moving successfully toward such newer business and organizational models requires substantial learning.

    “ERP tends to be associated with integrated, process-centered models of organization, and moving successtlilly toward such newer business and organizational models requires substantial learning.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ERP tends to be associated with integrated, process-centered models of organization.
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Researchers agree that organizational factors are most critical to successful ERP implementation.

    “Despite the significant technical challenges posed by ERP, researchers agree that organizational factors are most critical to successful ERP implementation (Constantinos, 1999).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Constantinos, 1999

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationreasoning

    The concerted group of firms was more realistic in acknowledging the tight coupling between technology and organizational change, while many firms in the piecemeal group appeared naive in their expectation that they could easily defer attention to business process changes.

    “We concluded that the concerted group of firms was more realistic in acknowledging the tight coupling between technology and organizational change, while many firms in the piecemeal group appeared naive in their expectation that they could easily defer attention to business process changes.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    ERP solutions removed some functionality associated with legacy systems, but that may have been desirable for firms seeking greater integration.

    “Certainly, ERP solutions removed some fiinctionality associated with legacy systems, but that may have been desirable for firms seeking greater integration.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationreasoning

    Firms in the piecemeal ERP group had not simply deferred part of their task to the future; in some cases they may have increased the total learning challenge as a result of this approach.

    “Thus, firms in the piecemeal ERP group had not simply deferred part of their task to the ftiture; in some cases they may have increased the total learning challenge as a result of this approach.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationnone given

    Assimilation challenges demand intensive employee education and appear to be more effective with incremental implementation plans.

    “The assimilation challenges demand intensive employee education and appear to be more effective with incremental implementation plans.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Configuration challenges demand a core team that is carefully selected, motivated with incentives, and empowered to act, as well as an effectively managed consulting relationship.

    “The configuration challenges demand a core team that is carefully selected, motivated with incentives, and empowered to act, as well as an effectively managed consulting relationship.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationreasoning

    These findings imply that firms implementing ERP may not have the luxury to proceed in piecemeal fashion, even though many of them tried.

    “These findings imply that firms implementing ERP may not have the luxury to proceed m piecemeal fashion, even though many of them tned.”
    Quote in the paperSubstantial claim
  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The paper's use of learning constructs was drawn directly from the interview notes, which contained numerous references to learning, learning curves, and knowledge.

    “Rather, our use of learning constructs was drawn directly from the interview notes, which contained numerous references to learning, learning curves, and knowledge.”
    Quote in the paperNot a substantial claim
  • May combine two claims.

    Interpretationreasoning

    Learning was not concentrated at the individual level; the structures and processes of entire divisions needed to change, and references to cultural change captured the scope of the learning required at the organizational level.

    “Rather, the structures and processes of entire divisions needed to change, and references to cultural change captured the scope of the learning required at the organizational level.”
    Quote in the paperSubstantial claim
  • May combine two claims.

    Interpretationreasoning

    Caches of organizational memory formerly guarded by division managers become transparent in the world of ERP, and enduring assumptions about responsibility, accountability, and shared understandings about what it takes to succeed are all challenged.

    “Caches of organizational memory formerly guarded by division managers become transparent in the world of ERP, and enduring assumptions about responsibility, accountability, and the shared understandings about what it takes to succeed are all challenged.”
    Quote in the paperSubstantial claim
39 of 39
  • The paper relies on this without support.

    ERP software packages are prepackaged and allow customization only through configuration tables that reflect business rules.

    “First, the software was prepackaged, allowmg for customizahon only through tables that the firm could configure m order to reflect its business rules.”
  • The paper relies on this without support.

    Replacing legacy systems with ERP disrupts processes built on those systems and replaces them with more standardized, crossfunctional processes.

    “Second, as firms replaced existing legacy systems with an ERP, they disrupted the processes that were built on those systems and replaced them with more standardized, crossftanctional processes.”
  • The paper relies on this without support.

    ERP tends to be associated with integrated, process-centered models of organization.

    “ERP tends to be associated with integrated, process-centered models of organization”
  • The paper relies on this without support.

    ERP systems typically require organizations to replace large portions of what they know, not only about technical infrastructures but also about business processes.

    “ERP systems typically require orgamzations to replace large portions of what they know, not only about technical mfrastructures but also about business processes.”
  • The paper relies on this without support.

    ERP technology is so new and complex that it is difficult for a company to embark on an ERP implementation without external knowledge.

    “Because HRP technology is so new and complex, it is difficult for a company to embark on such an endeavor without external knowledge.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • ES implementations comprise an extensive redesign of business processes and the deployment of new software to support the new business processes.

    “ES implementations comprise two main parts—an extensive redesign of business processes and the deployment of new software to support the new business processes (Morris and Venkatesh 2010; Robey et al. 2002).”

    From Enterprise System Implementation and Employee Job Performance: Understanding the Role of Advice Networks1 (2014) · cites this paper as Robey et al. 2002

    Section 4.3.2 describes firms that redesigned processes while implementing ERP software, supporting the two-part description for a concerted approach. But it also reports eight firms that implemented ERP before or apart from process reengineering and intended to defer redesign, so extensive redesign was not part of every implementation. (LLM, conf=0.88)

    Experimental — accuracy not yet measured
  • Despite a large body of ERP research literature from a number of different perspectives, there is not an adequate understanding and explanation about how and why these varying outcomes occur.

    “Despite a large body of ERP research literature from a number of different perspectives, there is not an adequate understanding and explanation about how and why these varying outcomes occur.”

    From An Explanatory Framework for Achieving Business Benefits from ERP Systems (2012) · cites this paper as Robey, Ross, & Boudreau's (2002)

    Section 2.1 says ERP impact studies report mixed outcomes and offer little theoretical explanation of how and why those outcomes occur. The prior-research discussion also says studies have taken both variance and process approaches. However, the Introduction says academic ERP research had only begun to appear, rather than describing a large body of research literature. (LLM, conf=0.86)

    Experimental — accuracy not yet measured
  • A new ES introduces uncertainty in the work environment in the transition period and often results in realignments of business processes and work unit interdependencies.

    “A new ES introduces uncertainty in the work environment in the transition period (Kolodny et al. 1996) and often results in realignments of business processes and work unit interdependencies (Robey et al. 2002).”
  • An ES implementation comes with changes to business processes and the introduction of new software applications.

    “As noted earlier, an ES implementation comes with changes to business processes and the introduction of new software applications (Davenport 2000; Robey et al. 2002; Ross and Vitale 2000).”
  • Most studies to date have concentrated on description rather than explanation.

    “Robey, Ross, & Boudreau's (2002) contention that most studies to date have concentrated on description rather than explanation still applies.”

    From An Explanatory Framework for Achieving Business Benefits from ERP Systems (2012) · cites this paper as Robey, Ross, & Boudreau's (2002)

Cites

Flagged first, then by citations

6 of the 64 references couldn’t be matched to a known paper.

Research works printed but not linked: 4

  1. [20]20. Koh, C, Soh, C, and Markus, M.L. "A Process Theory Approach to Analyzing ERP Implementation and Impacts: The Case of Revel Asia," Journal of Information Technology Cases and Applications, (2:1), 2000, 4-23.
  2. [24]24. Markus, M.L., Tanis, C, and van Fenema, "MultiSite ERP Implementations," Communications of the ACM, (43:4) 2000, 42^6.
  3. [26]26. Miles, M.B. and Huberman, A.M., Qualitative Data Analysis: An Expended Source Book, Second Edition, Sage, Thousand Oaks, CA, 1994.
  4. [50]50. Willcocks, L.P. and Sykes, R., "The Role of the CIO and IT Function in HRP," Communications of the ACM, (43:4), 2000, 32

Cited by

Sorted by citations

Authors
Robey, D., Ross, J. W., & Boudreau, M.-C.
Venue
Journal of Management Information Systems
Published
2002-07
Volume
19
Issue
1
Pages
17-46
ISSN
  • 0742-1222 (ISSN-L)
  • 1557-928X
OpenAlex
W2098925541
Cited by
1,023
References
64
Citations by year
2026: 5 · 2025: 6 · 2024: 11 · 2023: 16 · 2022: 18 · 2021: 15 · 2020: 20 · 2019: 36 · 2018: 25 · 2017: 37 · 2016: 45 · 2015: 65 · 2014: 62 · 2013: 80 · 2012: 59
FWCI
83.70
Type
journal article
Language
en
Open access
Locations
  • Journal of Management Information Systems (publishedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
Topics
  • Social Sciences › Business, Management and Accounting › Management Information Systems › ERP Systems Implementation and Impact
  • Social Sciences › Sociology and Political Science › Information Systems Theories and Implementation
  • Social Sciences › Business, Management and Accounting › Management Information Systems › Information Technology Governance and Strategy
Keywords
enterprise resource planning, ERP implementation, organizational learning, knowledge barriers, ERP configuration, workflow changes, consulting relationships, user training, phased implementation, industrial enterprises

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review