How to build models for government: criteria driving model acceptance in policymaking
Open accessThe DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
Models are used to inform policymaking and underpin large amounts of government expenditure. Several authors have observed a discrepancy between the actual and potential use of models in government. While there have been several studies investigating model acceptance in government, it remains unclear under what conditions models are accepted. In this paper, we address the question “What criteria affect model acceptance in policymaking?”, the answer to which will contribute to the wider understanding of model use in government. We employ a thematic coding approach to identify the acceptance criteria for the eight models in our sample. Subsequently, we compare our findings with existing literature and use qualitative comparative analysis to explore what configurations of the criteria are observed in instances of model acceptance. We conclude that model acceptance is affected by a combination of the model’s characteristics, the supporting infrastructure and organizational factors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Uptake of models in government is as much a social process as it is a technical one [50].”
From Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making (2018) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(1990) · Cited by 24,217
Technology Acceptance Model 3 and a Research Agenda on Interventions
Venkatesh, V., & Bala, H. (2008) · Decision Sciences · doi:10.1111/j.1540-5915.2008.00192.x · Cited by 8,159 · cited 2×
“This idea is supported by findings on the adoption of information technology in general (Venkatesh and Bala 2008).”
ResultRules, Games, and Common-Pool Resources
Ostrom, E., Gardner, R., & Walker, J. (1994) · University of Michigan Press eBooks · doi:10.3998/mpub.9739 · Cited by 3,732 · cited 1×
“Such work could employ frameworks more suited for analysis of dynamic social processes, such as the institutional analysis and development framework (Ostrom et al. 1994), or the repositories of actor-network theory (Mol 2013).”
MethodSet Relations in Social Research: Evaluating Their Consistency and Coverage
Ragin, C. C. (2006) · Political Analysis · doi:10.1093/pan/mpj019 · Cited by 1,784 · cited 3×
“notion of significance and assesses the degree to which a solution or set of solutions agrees in showing the outcome in question (Ragin 2006).”
BackgroundInfluentialStandards of Good Practice in Qualitative Comparative Analysis (QCA) and Fuzzy-Sets
Schneider, C. Q., & Wagemann, C. (2010) · Comparative Sociology · doi:10.1163/156913210x12493538729793 · Cited by 1,473 · cited 1×
“Using the procedure put forward by Schneider and Wagemann (2010), we conducted a crisp set qualitative comparative analysis (QCA) on both the eleven criteria and the grouped criteria.”
MethodActor-Network Theory: sensitive terms and enduring tensions
MOL, A. P. J. (2010) · UvA-DARE (University of Amsterdam) · Cited by 859
Techniques and Procedures for Developing Grounded Theory
Straus, A. (1998) · Medical Entomology and Zoology · Cited by 841
The rules of the game called psychological science
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612459060 · Cited by 800
Reconsidering evidence-based policy: Key issues and challenges
Head, B. W. (2010) · Policy and Society · doi:10.1016/j.polsoc.2010.03.001 · Cited by 527
Rihoux, B. (2006) · International Sociology · doi:10.1177/0268580906067836 · Cited by 461
Integrated assessment and modelling: features, principles and examples for catchment management
JAKEMAN, A., & LETCHER, R. (2003) · Environmental Modelling & Software · doi:10.1016/s1364-8152(03)00024-0 · Cited by 424
Environmental Science & Policy
Murdiyarso, D. (2007) · Cited by 235
Boulanger, P.-M., & Bréchet, T. (2005) · Ecological Economics · doi:10.1016/j.ecolecon.2005.07.033 · Cited by 227 · cited 1×
“Moreover, Boulanger and Bréchet (2005) demonstrate that different modelling techniques have different strengths and weaknesses for use in policy.”
BackgroundMinsky, M. (1965) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 210
Nilsson, M., Jordan, A., Turnpenny, J., Hertin, J., Nykvist, B., & Russel, D. (2008) · Policy Sciences · doi:10.1007/s11077-008-9071-1 · Cited by 204 · cited 6×
“& Daniel Antony Kolkman d.kolkman@surrey.ac.uk 1 Centre for Research in Social Simulation, University of Surrey, Guildford GU2 7XH, Surrey, UK Nilsson et al. (2008) argue that during the twentieth century, the issues facing policymakers have become increasingly complex.”
BackgroundMethodInfluentialA methodology for the design and development of integrated models for policy support
van Delden, H., Seppelt, R., White, R. W., & Jakeman, A. J. (2010) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2010.03.021 · Cited by 182 · cited 9×
“…et al. 2006; van Delden et al. 2011), interpretability (Baesens et al. 2009; van Delden et al. 2011), model performance (Baesens et al. 2009; van Delden et al. 2011), the user interface (McIntosh et al. 2011; van Delden et al. 2011) and the model development process (McIntosh et al. 2005;…”
BackgroundMethodResultInfluentialThe design features of forecasting support systems and their effectiveness
Fildes, R. A., Goodwin, P. B., & Lawrence, M. J. (2005) · Decision Support Systems · doi:10.1016/j.dss.2005.01.003 · Cited by 155 · cited 1×
“Previous research has suggested a variety of such criteria that can drive or constrain model acceptance, for instance, userfriendliness (Fildes et al. 2006; van Delden et al. 2011), interpretability (Baesens et al. 2009; van Delden et al. 2011), model performance (Baesens et al. 2009; van Delden et…”
BackgroundTHE POLICY AND POLITICS OF POLICY APPRAISAL: EMERGING TRENDS AND NEW DIRECTIONS
Turnpenny, J. R., Radaelli, C. M., Jordan, A., & Jacob, K. H. (2009) · Journal of European Public Policy · doi:10.1080/13501760902872783 · Cited by 152 · cited 3×
“The use of models in policy has been said to fall short of its potential (Mcintosh et al. 2008; Turnpenny et al. 2009).”
BackgroundTurtles all the way down: bounded rationality in an evidence-based age
Botterill, L. C., & Hindmoor, A. (2012) · Policy Studies · doi:10.1080/01442872.2011.626315 · Cited by 137 · cited 2×
“Policymakers are also constrained by resource limitations in terms of time and financial inputs (Botterill and Hindmoor 2012).”
BackgroundMcIntosh, B. S., Seaton, R. A. F., & Jeffrey, P. J. (2006) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2005.12.015 · Cited by 116 · cited 5×
“McIntosh et al. (2007) assert that many support-tool technologies contain models.”
BackgroundInfluentialGeyer, R. (2011) · Political Studies · doi:10.1111/j.1467-9248.2011.00903.x · Cited by 115 · cited 1×
“For instance, complexity clashes with the culture of auditing and target setting, as it obscures the very causality, reductionism, predictability and determinism on which the latter is based (Geyer 2012).”
BackgroundA SWOT Analysis of Planning Support Systems
Vonk, G., Geertman, S. C. M., & Schot, P. P. (2007) · Environment and Planning A Economy and Space · doi:10.1068/a38262 · Cited by 112 · cited 1×
“In addition, the failure of previous technologies to live up to expectations has made users reluctant to use models (Vonk et al. 2007).”
BackgroundImproving the Adoption and Use of Planning Support Systems in Practice
Vonk, G., & Geertman, S. C. M. (2008) · Applied Spatial Analysis and Policy · doi:10.1007/s12061-008-9011-7 · Cited by 89 · cited 1×
“Drivers and constraints to model acceptance in the literature Vonk and Geertman (2008) suggested that users prefer simple models over advanced ones, user-friendliness is of importance, organizational support for implementing models is often limited, managers consider the implementation of models to…”
BackgroundThe roles of computer models in the environmental policy life cycle
van Daalen, C., Dresen, L., & Janssen, M. A. (2002) · Environmental Science & Policy · doi:10.1016/s1462-9011(02)00040-0 · Cited by 85 · cited 3×
“Despite these advantages and the increase of model use in policy (van Daalen et al. 2002), model use has been argued to fall short of its potential (Mcintosh et al. 2008; Turnpenny et al. 2009).”
BackgroundDeliberative Policy Analysis as Practical Reason: Integrating Empirical and Normative Arguments
(2017) · Routledge eBooks · doi:10.4324/9781315093192-27 · Cited by 82 · cited 1×
“The effects of the move towards evidence-based policymaking have been criticized (e.g. Fischer 2007).”
Background50 years of data mining and OR: upcoming trends and challenges
Baesens, B., Mues, C., Martens, D., & Vanthienen, J. (2009) · Journal of the Operational Research Society · doi:10.1057/jors.2008.171 · Cited by 72 · cited 4×
“…previous studies on model use: Participation in the development process (McIntosh et al. 2005; Stalpers et al. 2009; McIntosh et al. 2011), tractability (Baesens et al. 2009; van Delden et al. 2011), efficiency, (Baesens et al. 2009; van Delden et al. 2011), the presence (van Delden et al. 2011).”
BackgroundInfluentialOn the Design of Computer-Based Models for Integrated Environmental Science
McIntosh, B. S., Jeffrey, P. J., Lemon, M., & Winder, N. (2005) · Environmental Management · doi:10.1007/s00267-004-0172-0 · Cited by 66 · cited 4×
“Of the eleven criteria identified, six have been associated with model acceptance in previous studies on model use: Participation in the development process (McIntosh et al. 2005; Stalpers et al. 2009; McIntosh et al. 2011), tractability (Baesens et al.”
BackgroundMethodResultInfluentialMcIntosh, B. S., Giupponi, C., Voinov, A. A., Smith, C. S., Matthews, K. B., Monticino, M. G., Kolkman, M. J., Crossman, N. D., van Ittersum, M. K., Haase, D., et al. (2008) · Developments in Integrated Environmental Assessment · doi:10.1016/s1574-101x(08)00603-0 · Cited by 55
The process of using a forecasting support system
Goodwin, P. B., Fildes, R. A., Lawrence, M. J., & Nikolopoulos, K. I. (2007) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2007.05.016 · Cited by 43 · cited 1×
“Goodwin et al. (2007) proposed that a model should be acceptable to users, be easy to use, offer a flexible range of methods, be viable for commercial exploitation, and facilitate the appropriate mix of judgment and statistical methods.”
BackgroundDiez, E., & McIntosh, B. S. (2010) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2010.04.003 · Cited by 40 · cited 1×
“Diez and Mcintosh (2011) put forward this idea and suggest that model use can differ across geographical contexts.”
BackgroundScholten, H. (2008) · doi:10.18174/122078 · Cited by 16 · cited 1×
“In effect, models underpin large amounts of government expenditure and motivate decisions that affect people’s lives (Scholten 2008; Treasury 2013).”
BackgroundReconciling model results with user needs to improve climate policy
Stalpers, S. I. P., Ierland, E. C. V., & Kroeze, C. (2009) · Environmental Science & Policy · doi:10.1016/j.envsci.2009.08.004 · Cited by 16 · cited 4×
“Of the eleven criteria identified, six have been associated with model acceptance in previous studies on model use: Participation in the development process (McIntosh et al. 2005; Stalpers et al. 2009; McIntosh et al. 2011), tractability (Baesens et al.”
BackgroundMethodInfluentialPreface: Thematic issue on the assessment and evaluation of environmental models and software
McIntosh, B. S., Alexandrov, G. A., Matthews, K. B., Myšiak, J., & van Ittersum, M. K. (2010) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2010.08.008 · Cited by 16 · cited 6×
“McIntosh et al.(2011) distinguish four levels of acceptance: (1) model development has been completed and presented to its intended users, (2) the users have been trained in the use of a model, but there is limited evidence of actual use, (3) the model has been used on a one-off basis, and (4) the…”
BackgroundMethodResultInfluentialComputerized models: tools for assessing the future of complex systems?
van Ittersum, M. K., & Sterk, B. (2015) · Edward Elgar Publishing eBooks · doi:10.4337/9781783477043.00016 · Cited by 12 · cited 2×
“Yet, only some of those studies present evidence of the actual use of the model in question (van Ittersum and Sterk 2015).”
BackgroundGuidelines for policy modellers – 30 years on: New tricks or old dogs?
Syme, G. J., BENNETT, D., Macpherson, D. K., & Thomas, J. F. (2011) · International Congress on Modelling and Simulation (MODSIM) · doi:10.36334/modsim.2011.g6.syme · Cited by 5 · cited 3×
“Moreover, research on the broader subject of model use is argued to be in its infancy (McIntosh et al. 2007) and some contend that this research field has failed to progress over the last few decades (Syme et al. 2011).”
BackgroundHappe, K., & Balmann, A. (2008) · RePEc: Research Papers in Economics · doi:10.22004/ag.econ.6588 · Cited by 4
Roosenschoon, O. R., Reis, S., Turnpenny, J. R., Adele, C., Jacob, K. H., Wascher, D. M., Weiland, S., Helming, K., Podhora, A., & Wien, J. J. F. (2012) · ScholarsArchive (Brigham Young University) · Cited by 3
Barbero (2014) · doi:10.1016/j · Cited by 1
Research works printed but not linked: 1
- 30. Treasury, H. M. (2013). Review of quality assurance of Government analytical models: Final Report.
Cited by
Veale, M., Van Kleek, M., & Binns, R. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174014 · Cited by 490
“Uptake of models in government is as much a social process as it is a technical one [50].”
BackgroundModel-based policymaking or policy-based modelling? How energy models and energy policy interact
Süsser, D., Ceglarz, A., Gaschnig, H., Stavrakas, V., Flamos, A., Giannakidis, G., & Lilliestam, J. (2021) · Energy Research & Social Science · doi:10.1016/j.erss.2021.101984 · Cited by 207
“These problems include the inability of models to answer specific questions that users need answered [22], low transparency of models [25], lack of trust in models by policymakers, inability of models to deliver timely support for decision-making, missing capacities in institutions to make use of complex modelling, th…”
BackgroundComputational Models That Matter During a Global Pandemic Outbreak: A Call to Action
Squazzoni, F., Polhill, G. J., Edmonds, B., Ahrweiler, P., Antosz, P., Scholz, G., Chappin, E. J. L., Borit, M., Verhagen, H., Giardini, F., et al. (2020) · Journal of Artificial Societies and Social Simulation · doi:10.18564/jasss.4298 · Cited by 174
“A danger is that a dominant coalition with an accepted model can use it to support their policy for long periods, simply updating the model over time, while leaving its basic assumptions unchanged (Kolkman et al. 2016).”
BackgroundSelf-citationA framework for characterising and evaluating the effectiveness of environmental modelling
Hamilton, S. H., Fu, B., Guillaume, J. H. A., Badham, J., Elsawah, S., Gober, P. A., Hunt, R. J., Iwanaga, T., Jakeman, A. J., Ames, D. P., et al. (2019) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2019.04.008 · Cited by 112
“Others have also found that the complexity of models may contribute to model rejection (Kolkman et al. 2016).”
BackgroundSüsser, D., Gaschnig, H., Ceglarz, A., Stavrakas, V., Flamos, A., & Lilliestam, J. (2021) · Energy · doi:10.1016/j.energy.2021.121909 · Cited by 85
Metadata
- Authors
- Kolkman, D. A., Campo, P., Balke-Visser, T., & Gilbert, N.
- Affiliations
- University of Surrey (GB)
- Venue
- Policy Sciences
- Published
- 2016-05-03
- Volume
- 49
- Issue
- 4
- Pages
- 489-504
- ISSN
- 0032-2687 (ISSN-L)
- 1573-0891
- OpenAlex
- W2346893305
- Cited by
- 39
- References
- 37
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 1 · 2023: 6 · 2022: 7 · 2021: 6 · 2020: 4 · 2019: 4 · 2018: 2 · 2016: 1
- FWCI
- 6.69
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Policy Sciences (publishedVersion, cc-by)
- Surrey Research Insight Open Access (The University of Surrey) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- University of Surrey (GB)
- Engineering and Physical Sciences Research Council (GB)
- Topics
- Social Sciences › Public Administration › Public Policy and Administration Research
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Social Sciences › Decision Sciences › Management Science and Operations Research › Complex Systems and Decision Making
- Keywords
- model acceptance, policy making, government model, decision support, model characteristics, organizational factors, supporting infrastructure
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.