The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Have We Learned Anything New About the Use of Evaluation?
Abstract
“Use of evaluation” used to mean the use of results for making program decisions. Now we are aware of its larger dimensions. Many aspects of evaluation can be used-ideas and insights, signals that are sent by the very fact of evaluating, the processes of learning through cooperation with evaluation, the choice of outcome indicators that herald success, even the study design. “Use” encompasses a broad array of effects by multiple classes of users. The paper highlights the importance of considering further potential collective users-the program organization, client groups, and civil society.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The results of the data analysis have undermined certain myths, which Weiss (1998) refers to as the enlightenment function.”
From Data teams for school improvement (2016) · cites this paper as Weiss (1998)
Citations
Cites
Agendas, Alternatives, and Public Policies
Perry, J. L., & Kingdon, J. W. (1985) · Journal of Policy Analysis and Management · doi:10.2307/3323801 · Cited by 11,238
(1987) · Cited by 6,910
Policy Change and Learning: An Advocacy Coalition Approach
Roberts, A. S., Sabatier, P., & Jenkins‐Smith, H. C. (1994) · Canadian Public Policy · doi:10.2307/3551961 · Cited by 3,599
Organizational Learning II: Theory, Method, and Practice.
Greenwood, D. J., Argyris, C., & Schön, D. A. (1997) · Industrial and Labor Relations Review · doi:10.2307/2525281 · Cited by 3,455
Information in Organizations as Signal and Symbol
Feldman, M. S., & March, J. G. (1981) · Administrative Science Quarterly · doi:10.2307/2392467 · Cited by 1,724
Utilization-focused evaluation : the new century text
Patton, M. Q. (1997) · Cited by 1,469
Knowledge Creep and Decision Accretion
Weiss, C. H. (1980) · Knowledge · doi:10.1177/107554708000100303 · Cited by 669
Self-citationSystematic Thinking for Social Action
Tullock, G., & Rivlin, A. M. (1972) · Journal of money credit and banking · doi:10.2307/1991056 · Cited by 367
Professional evaluation: social impact and political consequences
(1993) · Choice Reviews Online · doi:10.5860/choice.31-1583 · Cited by 289
The New American Political System
Williams, P. M. (1979) · International Affairs · doi:10.2307/2616385 · Cited by 275
Research works printed but not linked: 7
- Alkin, M. C., 1990, Debates on evaluation
- Breslau, D., Theory and Society
- Huberman, M., 1989, Knowledge in Society: The International Journal of Knowledge Transfer, 2, 6
- Senge, P. M., The fifth discipline: The art and practice of the learning organizations
- Shadish, W. R., 1997, EVALTALK message
- Shulha, L. M., American Evaluation Association meeting
- Weiss, C. H., 1967, House of Representatives committee on government operations, The use of social research in federal domestic programs, 426
Cited by
Sorted by citations
Wiltsey Stirman, S., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012) · Implementation Science · doi:10.1186/1748-5908-7-17 · Cited by 1,355
Diffusion Theory and Knowledge Dissemination, Utilization, and Integration in Public Health
Green, L. W., Ottoson, J. M., García, C., & Hiatt, R. A. (2009) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.031308.100049 · Cited by 953
Nutley, S., Walter, I., & Davies, H. T. O. (2003) · Evaluation · doi:10.1177/1356389003009002002 · Cited by 386
Data-based decision-making for school improvement: Research insights and gaps
Schildkamp, K. (2019) · Educational Research · doi:10.1080/00131881.2019.1625716 · Cited by 347
Inclusive Evaluation: Implications of Transformative Theory for Evaluation
Mertens, D. M. (1999) · American Journal of Evaluation · doi:10.1016/s1098-2140(99)80105-2 · Cited by 282
A Framework for Evaluating Environmental Policy Instruments: Context and Key Concepts
Mickwitz, P. (2003) · Evaluation · doi:10.1177/135638900300900404 · Cited by 249
Working with evaluation stakeholders: A rationale, step-wise approach and toolkit
Bryson, J. M., Patton, M. Q., & Bowman, R. A. (2010) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2010.07.001 · Cited by 228
Parkhurst, J. (2016) · London School of Economics and Political Science Research Online (London School of Economics and Political Science) · doi:10.4324/9781315675008 · Cited by 756
The Renaissance of Field Experimentation in Evaluating Interventions
Shadish, W. R., & Cook, T. D. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.60.110707.163544 · Cited by 208
Defining and Teaching Evaluative Thinking
Buckley, J., Archibald, T., Hargraves, M., & Trochim, W. M. K. (2015) · American Journal of Evaluation · doi:10.1177/1098214015581706 · Cited by 181
Data-based decision making in education: challenges and opportunities
Schildkamp, K., Lai, M., & Earl, L. (2013) · Springer eBooks · doi:10.1007/978-94-007-4816-3 · Cited by 211
Co-production of social research: strategies for engaged scholarship
Martin, S. J. (2010) · Public Money & Management · doi:10.1080/09540962.2010.492180 · Cited by 176
Assessing the Societal Impact of Research: The Relational Engagement Approach
Ozanne, J. L., Davis, B., Murray, J. B., Grier, S. A., Benmecheddal, A., Downey, H., Ekpo, A. E., Garnier, M., Hietanen, J., Gall‐Ely, M. L., et al. (2016) · Journal of Public Policy & Marketing · doi:10.1509/jppm.14.121 · Cited by 174
Data teams for school improvement
Schildkamp, K., Poortman, C. L., & Handelzalts, A. (2016) · School Effectiveness and School Improvement · doi:10.1080/09243453.2015.1056192 · Cited by 229
Data-based Decision Making: An Overview
Lai, M. K., & Schildkamp, K. (2013) · Data-based Decision Making in Education · doi:10.1007/978-94-007-4816-3_2 · Cited by 167
Askim, J. (2007) · International Review of Administrative Sciences · doi:10.1177/0020852307081152 · Cited by 167
Estabrooks, C. A., Derksen, L., Winther, C., Lavis, J. N., Scott, S. D., Wallin, L., & Profetto‐McGrath, J. (2008) · Implementation Science · doi:10.1186/1748-5908-3-49 · Cited by 166
Cashmore, M., Richardson, T., Hilding-Ryedvik, T., & Emmelin, L. (2010) · Environmental Impact Assessment Review · doi:10.1016/j.eiar.2010.01.004 · Cited by 162
Receptivity to Research in Policing
Lum, C. M., Telep, C. W., Koper, C. S., & Grieco, J. (2012) · Justice Research and Policy · doi:10.3818/jrp.14.1.2012.61 · Cited by 162
Xiao, H., & Smith, S. L. J. (2007) · Annals of Tourism Research · doi:10.1016/j.annals.2006.09.001 · Cited by 150
Evidence and Policy and Naming and Shaming
Pawson, R. (2002) · Policy Studies · doi:10.1080/0144287022000045993 · Cited by 135
What works to promote research-policy engagement?
Oliver, K. A., Hopkins, A., Boaz, A., Guillot-Wright, S. P., & Cairney, P. (2022) · Evidence & Policy · doi:10.1332/174426421x16420918447616 · Cited by 122
Solving the disjuncture between research and practice: Telehealth trends in the 21st century
Miller, E. A. (2007) · Health Policy · doi:10.1016/j.healthpol.2006.09.011 · Cited by 120
An analytical framework to discuss the usability of (environmental) indicators for policy
Bauler, T. (2011) · Ecological Indicators · doi:10.1016/j.ecolind.2011.05.013 · Cited by 128
Fleischer, D. N., & Christie, C. A. (2009) · American Journal of Evaluation · doi:10.1177/1098214008331009 · Cited by 111
Triggs, P., & John, P. (2004) · Journal of Computer Assisted Learning · doi:10.1111/j.1365-2729.2004.00101.x · Cited by 103
A política da avaliação de políticas públicas
de Faria, C. A. P. (2005) · Revista Brasileira de Ciências Sociais · doi:10.1590/s0102-69092005000300007 · Cited by 103
Bozeman, B. L., & Youtie, J. (2017) · Research Policy · doi:10.1016/j.respol.2017.06.003 · Cited by 101
Exploring the Necessary Conditions for Evaluation Use in Program Change
Ledermann, S. (2011) · American Journal of Evaluation · doi:10.1177/1098214011411573 · Cited by 106
Theory-based evaluation and the social impact of the arts
Galloway, S. (2009) · Cultural Trends · doi:10.1080/09548960902826143 · Cited by 94
Suárez-Herrera, J. C., Springett, J., & Kagan, C. M. (2009) · Evaluation · doi:10.1177/1356389009105884 · Cited by 84
Symposium on Implementing Public Policy: Learning from Theory and Practice
Schofield, J., & Sausman, C. (2004) · Public Administration · doi:10.1111/j.0033-3298.2004.00392.x · Cited by 83
The effects of the use of a school self-evaluation instrument
Schildkamp, K., Visscher, A. J., & Luyten, H. (2009) · School Effectiveness and School Improvement · doi:10.1080/09243450802605506 · Cited by 82
Reforms, Organizational Change and Performance in Higher Education
Pinheiro, R. M., Aarrevaara, T., Berg, L. N., Fumasoli, T., Geschwind, L., Hansen, H. F., Hernes, H., Kivistö, J., Lind, J. K., Lyytinen, A., et al. (2019) · Springer eBooks · doi:10.1007/978-3-030-11738-2 · Cited by 80
Milley, P., Szijarto, B., Svensson, K., & Cousins, J. B. (2018) · Evaluation · doi:10.1177/1356389018763242 · Cited by 76
School performance feedback systems in the USA and in The Netherlands: a comparison
Schildkamp, K., & Teddlie, C. (2008) · Educational Research and Evaluation · doi:10.1080/13803610802048874 · Cited by 76
Ettelt, S., Mays, N., & Allen, P. W. (2014) · Journal of Social Policy · doi:10.1017/s0047279414000865 · Cited by 82
Toward an Evidence-Based Approach to Preventing Crime
Welsh, B. C., & Farrington, D. P. (2001) · The Annals of the American Academy of Political and Social Science · doi:10.1177/000271620157800110 · Cited by 73
Evaluation use in the organizational context – changing focus to improve theory
Højlund, S. (2014) · Evaluation · doi:10.1177/1356389013516053 · Cited by 71
Ehlers, L. H., Vestergaard, M., Kidholm, K., Bonnevie, B., Pedersen, P. H., Jørgensen, T., Jensen, M. F., Kristensen, F. B., & Kjølby, M. (2006) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462306051178 · Cited by 76
The politics of performance measurement: ‘Evaluation use as mediator for politics’
Bjørnholt, B., & Larsen, F. (2014) · Evaluation · doi:10.1177/1356389014551485 · Cited by 68
Principles and purposes of European Union Cohesion policy evaluation
Batterbury, S. C. E. (2006) · Regional Studies · doi:10.1080/00343400600600504 · Cited by 153
Advancing the evaluation of integrated knowledge translation
Kreindler, S. A. (2018) · Health Research Policy and Systems · doi:10.1186/s12961-018-0383-0 · Cited by 65
Spatial Planning, Territorial Development, and Territorial Impact Assessment
Medeiros, E. (2019) · Journal of Planning Literature · doi:10.1177/0885412219831375 · Cited by 62
Hellrung, K., & Hartig, J. (2013) · Educational Research Review · doi:10.1016/j.edurev.2012.09.001 · Cited by 63
Ottoson, J. M. (2009) · New Directions for Evaluation · doi:10.1002/ev.310 · Cited by 60
Escorza, T. E. (2014) · RELIEVE - Revista Electrónica de Investigación y Evaluación Educativa · doi:10.7203/relieve.9.1.4348 · Cited by 62
Design Principles for Data Visualization in Evaluation
Evergreen, S. D. H., & Metzner, C. (2013) · New Directions for Evaluation · doi:10.1002/ev.20071 · Cited by 57
Social Innovation, Societal Change, and the Role of Policies
Lukesch, R., Ludvig, A., Slee, B., Weiss, G., & Živojinović, I. (2020) · Sustainability · doi:10.3390/su12187407 · Cited by 66
Data-based decision making : conclusions and a data use framework
Schildkamp, K., & Lai, M. (2013) · Springer eBooks · doi:10.1007/978-94-007-4816-3_10 · Cited by 51
Siron, S., Dagenais, C., & Ridde, V. (2015) · International Journal of Public Health · doi:10.1007/s00038-015-0716-5 · Cited by 56
OECD Environmental Performance Review Programme
Lehtonen, M. (2005) · Evaluation · doi:10.1177/1356389005055536 · Cited by 54
Owen, R. (2021) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2021.3094992 · Cited by 53
Ottoson, J. M., Green, L. W., Beery, W. L., Senter, S., Cahill, C., Pearson, D. C., Greenwald, H. P., Hamre, R., & Leviton, L. C. (2009) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2008.10.010 · Cited by 52
What Makes Research Useful for Public School Educators?
Neal, J. W., Neal, Z. P., Lawlor, J., Mills, K. J., & McAlindon, K. (2017) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-017-0834-x · Cited by 54
Oliver, K. A., Lorenc, T., & Tinkler, J. (2019) · Evaluation · doi:10.1177/1356389019850847 · Cited by 53
Bridging Education Research and Education Policymaking
Kirst, M. W. (2000) · Oxford Review of Education · doi:10.1080/713688533 · Cited by 51
BERA at 30. Have we come of age?†
Furlong, J. J. (2004) · British Educational Research Journal · doi:10.1080/01411920410001689670 · Cited by 50
The use and usability of evaluation outputs: A social practice approach
Saunders, M. (2012) · Evaluation · doi:10.1177/1356389012459113 · Cited by 48
Evaluation use in evaluation systems – the case of the European Commission
Højlund, S. (2014) · Evaluation · doi:10.1177/1356389014550562 · Cited by 48
Brewster, L., Aveling, E., Martin, G. P., Tarrant, C., & Dixon‐Woods, M. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2014-003732 · Cited by 48
Tamaki, E. M., Tanaka, O. Y., Felisberto, E., Alves, C. K. D. A., Junior, M. D., Bezerra, L. C. D. A., Calvo, M. C. M., & de Miranda, A. S. (2012) · Ciência & Saúde Coletiva · doi:10.1590/s1413-81232012000400007 · Cited by 47
Mickwitz, P., & Melanen, M. (2009) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2008.12.003 · Cited by 46
The Complexity of Evidence: Reflections on Research Utilisation in a Heavily Politicised Policy Area
Monaghan, M. (2009) · Social Policy and Society · doi:10.1017/s1474746409990157 · Cited by 45
Evaluation units as knowledge brokers: Testing and calibrating an innovative framework
Olejniczak, K., Raimondo, E., & Kupiec, T. (2016) · Evaluation · doi:10.1177/1356389016638752 · Cited by 44
The importance of evaluating programs of school, family and community partnerships
Epstein, J. L., & Sheldon, S. B. (2019) · Aula Abierta · doi:10.17811/rifie.48.1.2019.31-42 · Cited by 45
The influence of competences and support on school performance feedback use
Vanhoof, J., Verhaeghe, G., Verhaeghe, J. P., Valcke, M., & Petegem, P. V. (2010) · Educational Studies · doi:10.1080/03055698.2010.482771 · Cited by 44
Explaining the political use of evaluation in international organizations
Eckhard, S., & Jankauskas, V. (2020) · Policy Sciences · doi:10.1007/s11077-020-09402-2 · Cited by 46
When Data Teams Struggle: Learning from Less Successful Data Use Efforts
Schildkamp, K., & Datnow, A. (2020) · Leadership and Policy in Schools · doi:10.1080/15700763.2020.1734630 · Cited by 63
Smith, A. W., & Leech, R. (2010) · International Journal of Sport Policy and Politics · doi:10.1080/19406940.2010.519341 · Cited by 42
Hickey, G. M., Forest, P., Sandall, J., Lalor, B. M., & Keenan, R. J. (2013) · Science and Public Policy · doi:10.1093/scipol/sct004 · Cited by 42
Bunn, F., Trivedi, D. P., Alderson, P., Hamilton, L. K., Martin, A. E., Pinkney, E., & Iliffe, S. (2015) · Health Technology Assessment · doi:10.3310/hta19280 · Cited by 42
Shifting focus: The impacts of sustainable seafood certification
van Putten, I. E., Longo, C., Arton, A., Watson, M., Anderson, C. M., Himes‐Cornell, A., Obregón, C., Robinson, L. M., & van Steveninck, T. J. (2020) · PLoS ONE · doi:10.1371/journal.pone.0233237 · Cited by 49
Changing lives? Critical evaluation of a school-based athlete role model intervention
Armour, K., & Duncombe, R. (2011) · Sport Education and Society · doi:10.1080/13573322.2011.608941 · Cited by 44
Evaluation of Programs: Reading Carol H. Weiss
Msila, V., & Setlhako, M. A. (2013) · Universal Journal of Educational Research · doi:10.13189/ujer.2013.010408 · Cited by 38
Knowledge utilization: Implications for evaluation
Blake, S., & Ottoson, J. M. (2009) · New Directions for Evaluation · doi:10.1002/ev.311 · Cited by 37
Knowledge brokerage context factors – What matters in knowledge exchange in impact assessment?
Saarela, S., Söderman, T., & Lyytimäki, J. M. (2014) · Environmental Science & Policy · doi:10.1016/j.envsci.2014.09.001 · Cited by 37
The utilization of training program evaluation in corporate universities
Bober, C. F., & Bartlett, K. R. (2004) · Human Resource Development Quarterly · doi:10.1002/hrdq.1111 · Cited by 37
Meadow, A. M., & Owen, G. (2021) · doi:10.2458/10150.658313 · Cited by 38
The utilisation of a school self‐evaluation instrument
Schildkamp, K., & Visscher, A. (2010) · Educational Studies · doi:10.1080/03055690903424741 · Cited by 36
Szijarto, B., Milley, P., Svensson, K., & Cousins, J. B. (2017) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2017.08.010 · Cited by 36
What is the proposed role of research evidence in England’s ‘self-improving’ school system?
Godfrey, D. (2017) · Oxford Review of Education · doi:10.1080/03054985.2017.1329718 · Cited by 36
Welsh, B. C., Yohros, A., & Zane, S. N. (2020) · Aggression and Violent Behavior · doi:10.1016/j.avb.2020.101511 · Cited by 36
Analysing Models as a Knowledge Technology in Transport Planning
Gudmundsson, H. (2011) · Transport Reviews · doi:10.1080/01441647.2010.532884 · Cited by 35
Toth, C. (2018) · The WAC Clearinghouse; University Press of Colorado eBooks · doi:10.37514/per-b.2018.0155.2.04 · Cited by 36
Owen, R., Vedanthachari, L. N., & Hussain, J. (2023) · Venture Capital · doi:10.1080/13691066.2023.2205606 · Cited by 34
Centering Indigenous knowledge in suicide prevention: a critical scoping review
Sjoblom, E., Ghidei, W., Leslie, M., James, A., Bartel, R., Campbell, S., & Montesanti, S. R. (2022) · BMC Public Health · doi:10.1186/s12889-022-14580-0 · Cited by 37
The use of school self‐evaluation results in the Netherlands and Flanders
Schildkamp, K., Vanhoof, J., Petegem, P. V., & Visscher, A. J. (2011) · British Educational Research Journal · doi:10.1080/01411926.2010.528556 · Cited by 34
Tangney, P., & Howes, M. J. (2015) · Environment and Planning C Government and Policy · doi:10.1177/0263774x15602023 · Cited by 34
Sandford, R. A., Beckett, A. E., & Giulianotti, R. (2021) · Sport Education and Society · doi:10.1080/13573322.2021.1902299 · Cited by 33
Evaluators' descriptions of process use: An exploratory study
Harnar, M. A., & Preskill, H. S. (2007) · New Directions for Evaluation · doi:10.1002/ev.241 · Cited by 33
Training evaluation based on cases of Taiwanese benchmarked high‐tech companies
Lien, B. Y., Hung, R. Y. Y., & McLean, G. N. (2007) · International Journal of Training and Development · doi:10.1111/j.1468-2419.2007.00268.x · Cited by 33
Martin, S. J., Downe, J., Grace, C., & Nutley, S. (2010) · Evaluation · doi:10.1177/1356389009350119 · Cited by 34
Lyytimäki, J. M., Gudmundsson, H., & Sørensen, C. H. (2011) · Sustainable Development · doi:10.1002/sd.530 · Cited by 32
Promoting Evaluation through Collaboration
O’Sullivan, R. G., & D’Agostino, A. (2002) · Evaluation · doi:10.1177/135638902401462466 · Cited by 32
Revising Road Safety Policy: The Role of Systematic Evidence in Switzerland
Frey, K. (2010) · Governance · doi:10.1111/j.1468-0491.2010.01503.x · Cited by 31
Daigneault, P. (2014) · Evaluation and Program Planning · doi:10.1016/j.evalprogplan.2014.04.003 · Cited by 31
van der Kleij, F. M., & Eggen, T. J. H. M. (2013) · Studies in Educational Evaluation · doi:10.1016/j.stueduc.2013.04.002 · Cited by 31
Revisiting Foresight Rationales: What Lessons from the Social Sciences and Humanities?
Barrè, R., & Keenan, M. (2008) · Springer eBooks · doi:10.1007/978-3-540-68811-2_4 · Cited by 30
The non-use and influence of UK Energy Sector Indicators
Lehtonen, M. (2012) · Ecological Indicators · doi:10.1016/j.ecolind.2012.10.026 · Cited by 30
Metadata
- Authors
- Weiss, C. H.
- Affiliations
- Harvard University (US)
- Corresponding author
- Weiss, Carol H.
- Venue
- American Journal of Evaluation
- Publisher
- SAGE Publications
- Published
- 1998-03
- Volume
- 19
- Issue
- 1
- Pages
- 21-33
- ISSN
- 1098-2140 (ISSN-L)
- 1557-0878
- OpenAlex
- W4245760083
- Cited by
- 384
- References
- 20
- Citations by year
- 2026: 8 · 2025: 12 · 2024: 14 · 2023: 18 · 2022: 20 · 2021: 18 · 2020: 31 · 2019: 17 · 2018: 17 · 2017: 25 · 2016: 21 · 2015: 17 · 2014: 18 · 2013: 12 · 2012: 20
- FWCI
- 7.98
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Journal of Evaluation (publishedVersion)
- Topics
- Social Sciences › Public Administration › Evaluation and Performance Assessment
- Social Sciences › Public Administration › Public Policy and Administration Research
- Social Sciences › Economics, Econometrics and Finance › Finance › Community Development and Social Impact
- Keywords
- program evaluation, program decision-making, outcome indicators, research design, learning from evaluation, civil society, program organization, client groups
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review