The DOI matches its Crossref record.
Fourth-Generation Evaluation
Abstract
Naturalistic program evaluation strategies are gaining acceptance within the evaluation profession. They are especially useful in the examination of the transaction, or process, components of programs. One of their disadvantages is that they demand extensive quantities of evaluator time and effort which are difficult to arrange. Guba refers to evaluation conducted within the naturalistic paradigm (as distinct from the use of naturalistic techniques) as fourth-generation evaluation. This article describes an evaluation study in the health field that attempted to stay within the naturalistic paradigm while mitigating some of the drawbacks of the methodological process of naturalistic evaluation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
S., N. L. (2017) · doi:10.1177/1609406917733847 · Cited by 14,051
Mixed Methods Research: A Research Paradigm Whose Time Has Come
Johnson, R. B., & Onwuegbuzie, A. J. (2004) · Educational Researcher · doi:10.3102/0013189x033007014 · Cited by 11,170
Qualitative Quality: Eight “Big-Tent” Criteria for Excellent Qualitative Research
Tracy, S. J. (2010) · Qualitative Inquiry · doi:10.1177/1077800410383121 · Cited by 7,536
Toward a definition of mixed methods research
Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007) · Journal of Mixed Methods Research · doi:10.1177/1558689806298224 · Cited by 6,464
REVIEW OF COMMUNITY-BASED RESEARCH: Assessing Partnership Approaches to Improve Public Health
Israel, B. A., Schulz, A. J., Parker, E. A., & Becker, A. B. (1998) · Annual Review of Public Health · doi:10.1146/annurev.publhealth.19.1.173 · Cited by 5,893
Verification strategies for establishing reliability and validity in qualitative research
Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002) · International Journal of Qualitative Methods · doi:10.1177/160940690200100202 · Cited by 4,978
Member Checking: A Tool to Enhance Trustworthiness or Merely a Nod to Validation?
Birt, L., Scott, S., Cavers, D., Campbell, C., & Walter, F. (2016) · Qualitative Health Research · doi:10.1177/1049732316654870 · Cited by 3,970
Quality and trustworthiness in qualitative research in counseling psychology.
Morrow, S. L. (2005) · Journal of Counseling Psychology · doi:10.1037/0022-0167.52.2.250 · Cited by 3,199
Smith, B., & McGannon, K. R. (2018) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2017.1317357 · Cited by 3,074
Critical analysis of strategies for determining rigor in qualitative inquiry
Morse, J. M. (2015) · Qualitative Health Research · doi:10.1177/1049732315588501 · Cited by 2,809
Two Decades of Developments in Qualitative Inquiry: A Personal, Experiential Perspective
Patton, M. Q. (2002) · Qualitative Social Work · doi:10.1177/1473325002001003636 · Cited by 2,564
Understanding and Evaluating Qualitative Research
Fossey, E., Harvey, C., Mcdermott, F., & Davidson, L. (2002) · Australian & New Zealand Journal of Psychiatry · doi:10.1046/j.1440-1614.2002.01100.x · Cited by 2,311
Validity in qualitative research
Whittemore, R., Chase, S. K., & Mandle, C. L. (2001) · Qualitative Health Research · doi:10.1177/104973201129119299 · Cited by 2,144
Participatory learning for sustainable agriculture
Pretty, J. N. (1995) · World Development · doi:10.1016/0305-750x(95)00046-f · Cited by 1,753
LOORBACH, D. (2010) · Governance · doi:10.1111/j.1468-0491.2009.01471.x · Cited by 1,748
Quality in Qualitative Research
Seale, C. (1999) · Qualitative Inquiry · doi:10.1177/107780049900500402 · Cited by 1,743
The Development of Constructivist Grounded Theory
Mills, J., Bonner, A., & Francis, K. (2006) · International Journal of Qualitative Methods · doi:10.1177/160940690600500103 · Cited by 1,678
Emerging Criteria for Quality in Qualitative and Interpretive Research
Lincoln, Y. S. (1995) · Qualitative Inquiry · doi:10.1177/107780049500100301 · Cited by 1,568
Qualitative research methods in health technology assessment: a review of the literature.
Murphy, Dingwall, Greatbatch, Parker, & Watson (1998) · Health Technology Assessment · doi:10.3310/hta2160 · Cited by 1,107
Validity in qualitative research revisited
Cho, J., & Trent, A. (2006) · Qualitative Research · doi:10.1177/1468794106065006 · Cited by 1,037
Fertile Obsession: Validity After Poststructuralism
Lather, P. (1993) · The Sociological Quarterly · doi:10.1111/j.1533-8525.1993.tb00112.x · Cited by 937
Yilmaz, K. (2013) · European Journal of Education · doi:10.1111/ejed.12014 · Cited by 934
Huitema, D., Mostert, E., Egas, W., Moellenkamp, S., Pahl-Wostl, C., & Yalcin, R. (2009) · Ecology and Society · doi:10.5751/es-02827-140126 · Cited by 912
Validity and Qualitative Research: An Oxymoron?
Onwuegbuzie, A. J., & Leech, N. L. (2007) · Quality & Quantity · doi:10.1007/s11135-006-9000-3 · Cited by 832
Quality control in qualitative research
Stiles, W. B. (1993) · Clinical Psychology Review · doi:10.1016/0272-7358(93)90048-q · Cited by 832
Schwandt, T. A., Lincoln, Y. S., & Guba, E. G. (2007) · New Directions for Evaluation · doi:10.1002/ev.223 · Cited by 829
Sampling in Qualitative Research: Insights from an Overview of the Methods Literature
Gentles, S., Charles, C., Ploeg, J., & McKibbon, K. A. (2015) · The Qualitative Report · doi:10.46743/2160-3715/2015.2373 · Cited by 771
Program evaluation models and related theories: AMEE Guide No. 67
Frye, A. W., & Hemmer, P. A. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.668637 · Cited by 640
Maher, C., Hadfield, M., Hutchings, M., & de Eyto, A. (2018) · International Journal of Qualitative Methods · doi:10.1177/1609406918786362 · Cited by 632
Data Analysis Strategies for Mixed-Method Evaluation Designs
Caracelli, V. J., & Greene, J. C. (1993) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737015002195 · Cited by 627
Metatriangulation: Building Theory from Multiple Paradigms
Lewis, M. W., & Grimes, A. J. (1999) · The Academy of Management Review · doi:10.2307/259348 · Cited by 623
Judging the quality of qualitative inquiry: Criteriology and relativism in action
Sparkes, A. C., & Smith, B. (2009) · Psychology of Sport and Exercise · doi:10.1016/j.psychsport.2009.02.006 · Cited by 596
Adopting a constructivist approach to grounded theory: Implications for research design
Mills, J., Bonner, A., & Francis, K. (2006) · International Journal of Nursing Practice · doi:10.1111/j.1440-172x.2006.00543.x · Cited by 593
A Comparative Discussion of the Notion of 'Validity' in Qualitative and Quantitative Research
Winter, G. (2000) · The Qualitative Report · doi:10.46743/2160-3715/2000.2078 · Cited by 592
Braun, V., & Clarke, V. (2023) · Health Psychology Review · doi:10.1080/17437199.2022.2161594 · Cited by 582
Enhancing the Trustworthiness of Qualitative Research in International Business
Sinkovics, R. R., Penz, E., & Ghauri, P. N. (2008) · Management International Review · doi:10.1007/s11575-008-0103-z · Cited by 573
Quantitative and Qualitative Research: Beyond the Debate
Gelo, O., Braakmann, D., & Benetka, G. (2008) · Integrative Psychological and Behavioral Science · doi:10.1007/s12124-008-9078-3 · Cited by 537
Rigour and grounded theory research
Chiovitti, R. F., & Piran, N. (2003) · Journal of Advanced Nursing · doi:10.1046/j.0309-2402.2003.02822.x · Cited by 523
Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N., French, D. P., McIntosh, E., et al. (2021) · Health Technology Assessment · doi:10.3310/hta25570 · Cited by 522
Latulippe, K., Hamel, C., & Giroux, D. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6731 · Cited by 458
The ‘‘what’’and ‘‘how’’of case study rigor: three strategies based on published work
Gibbert, M., & Ruigrok, W. (2010) · Organizational Research Methods · doi:10.1177/1094428109351319 · Cited by 506
Peer Debriefing in Qualitative Research: Emerging Operational Models
Spall, S. (1998) · Qualitative Inquiry · doi:10.1177/107780049800400208 · Cited by 490
FABRICATION AS ETHICAL PRACTICE: Qualitative inquiry in ambiguous Internet contexts
Markham, A. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2011.641993 · Cited by 484
Implementation of a hermeneutic inquiry in nursing: philosophy, rigour and representation
Koch, T. (1996) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.1996.17224.x · Cited by 454
The Goodness of Qualitative Research
Peshkin, A. (1993) · Educational Researcher · doi:10.2307/1176170 · Cited by 447
The elephant in the living room: or extending the conversation about the politics of evidence
Denzin, N. K. (2009) · Qualitative Research · doi:10.1177/1468794108098034 · Cited by 445
Using Qualitative Methods for Causal Explanation
Maxwell, J. A. (2004) · Field Methods · doi:10.1177/1525822x04266831 · Cited by 441
Residents’ attitudes to tourism: a longitudinal study of 140 articles from 1984 to 2010
Nunkoo, R., Smith, S. L. J., & Ramkissoon, H. (2013) · Journal of Sustainable Tourism · doi:10.1080/09669582.2012.673621 · Cited by 440
Analyzing student interactions and meaning construction in computer bulletin board discussions
Pena-Shaff, J. B., & Nicholls, C. (2004) · Computers & Education · doi:10.1016/j.compedu.2003.08.003 · Cited by 437
Shifting Conceptions of Validity in Educational Measurement: Implications for Performance Assessment
Moss, P. A. (1992) · Review of Educational Research · doi:10.2307/1170738 · Cited by 434
Achieving trustworthiness in qualitative research: A pan-paradigmatic perspective
Williams, E. N., & Morrow, S. L. (2009) · Psychotherapy Research · doi:10.1080/10503300802702113 · Cited by 411
Generalizability in qualitative research: a tale of two traditions
Carminati, L. (2018) · Qualitative Health Research · doi:10.1177/1049732318788379 · Cited by 401
van der Giesen, C., Cucurachi, S., Guinée, J., Kramer, G. J., & Tukker, A. (2020) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2020.120904 · Cited by 386
How Do You Modernize a Health Service? A Realist Evaluation of Whole‐Scale Transformation in London
GREENHALGH, T., HUMPHREY, C., HUGHES, J., MACFARLANE, F., BUTLER, C., & PAWSON, R. (2009) · The Milbank Quarterly · doi:10.1111/j.1468-0009.2009.00562.x · Cited by 380
Broadening the range of designs and methods for impact evaluations
Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B. (2012) · doi:10.22163/fteval.2012.100 · Cited by 369
Reconceptualizing Bias in Teaching Qualitative Research Methods
Roulston, K., & Shelton, S. A. (2015) · Qualitative Inquiry · doi:10.1177/1077800414563803 · Cited by 360
Post Qualitative Inquiry in an Ontology of Immanence
St. Pierre, E. A. (2019) · Qualitative Inquiry · doi:10.1177/1077800418772634 · Cited by 359
Hoppe, R. (2005) · Poiesis & Praxis · doi:10.1007/s10202-005-0074-0 · Cited by 353
Framing Experience: Concept Maps, Mind Maps, and Data Collection in Qualitative Research
Wheeldon, J., & Faubert, J. (2009) · International Journal of Qualitative Methods · doi:10.1177/160940690900800307 · Cited by 347
Researching public health: Behind the qualitative-quantitative methodological debate
Baum, F. (1995) · Social Science & Medicine · doi:10.1016/0277-9536(94)e0103-y · Cited by 335
Hermeneutic phenomenology: philosophical perspectives and current use in nursing research
Annells, M. (1996) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.1996.tb00041.x · Cited by 334
Cultural Consultation: A Model of Mental Health Service for Multicultural Societies
Kirmayer, L. J., Groleau, D., Guzder, J., Blake, C., & Jarvis, E. (2003) · The Canadian Journal of Psychiatry · doi:10.1177/070674370304800302 · Cited by 334
Big data in the policy cycle: Policy decision making in the digital era
Höchtl, J., Parycek, P., & Schöllhammer, R. (2016) · Journal of Organizational Computing and Electronic Commerce · doi:10.1080/10919392.2015.1125187 · Cited by 331
Ploeg, J., Davies, B., Edwards, N., Gifford, W., & Miller, P. E. (2007) · Worldviews on Evidence-Based Nursing · doi:10.1111/j.1741-6787.2007.00106.x · Cited by 321
Wallerstein, N. (1999) · Social Science & Medicine · doi:10.1016/s0277-9536(99)00073-8 · Cited by 292
Eakin, J. M., & Mykhalovskiy, E. (2003) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2003.00392.x · Cited by 287
Greaves, C. J., & Farbus, L. (2006) · Journal of the Royal Society for the Promotion of Health · doi:10.1177/1466424006064303 · Cited by 284
Emotional labor and nursing: an under‐appreciated aspect of caring work
Henderson, A. (2001) · Nursing Inquiry · doi:10.1046/j.1440-1800.2001.00097.x · Cited by 283
Connelly, F. M., & Clandinin, D. J. (1999) · Issues in Educational Research · doi:10.1016/b978-008043349-3/50013-x · Cited by 283
Novice to Expert: An Exploration of How Professionals Learn
Daley, B. J. (1999) · Adult Education Quarterly · doi:10.1177/074171369904900401 · Cited by 275
Physics students' epistemologies and views about knowing and learning
Roth, W., & Roychoudhury, A. (1994) · Journal of Research in Science Teaching · doi:10.1002/tea.3660310104 · Cited by 266
Sipe, L. R. (2000) · Reading Research Quarterly · doi:10.1598/rrq.35.2.4 · Cited by 262
Wu, Y.-H., Wrobel, J., Cornuet, M., Kerhervé, H., Damnée, S., & Rigaud, A.-S. (2014) · Clinical Interventions in Aging · doi:10.2147/cia.s56435 · Cited by 261
Luft, J. A., Kurdziel, J. P., Roehrig, G. H., & Turner, J. (2004) · Journal of Research in Science Teaching · doi:10.1002/tea.20004 · Cited by 257
Know-how to lead digital transformation: The case of local governments
Pittaway, J. J., & Montazemi, A. R. (2020) · Government Information Quarterly · doi:10.1016/j.giq.2020.101474 · Cited by 289
A closer look at the role of mentor teachers in shaping preservice teachers' professional identity
Izadinia, M. (2015) · Teaching and Teacher Education · doi:10.1016/j.tate.2015.08.003 · Cited by 242
Dimensions of everyday eating and drinking episodes
Bisogni, C. A., Falk, L. W., Madore, E., Blake, C. E., Jastran, M., Sobal, J., & Devine, C. M. (2007) · Appetite · doi:10.1016/j.appet.2006.09.004 · Cited by 201
What a Long, Strange Trip It's Been…: Twenty-Five Years of Qualitative and New Paradigm Research
Lincoln, Y. S. (2010) · Qualitative Inquiry · doi:10.1177/1077800409349754 · Cited by 185
Why adolescents don't disclose incidents of bullying and harassment
deLara, E. W. (2012) · Journal of School Violence · doi:10.1080/15388220.2012.705931 · Cited by 189
Lincoln, Y. S. (2007) · The Blackwell Encyclopedia of Sociology · doi:10.1002/9781405165518.wbeosn006 · Cited by 158
Li, J., & De Luca, R. (2014) · Studies in Higher Education · doi:10.1080/03075079.2012.709494 · Cited by 154
Tolman, D. L., & Szalacha, L. A. (1999) · Psychology of Women Quarterly · doi:10.1111/j.1471-6402.1999.tb00338.x · Cited by 151
Qualitative and quantitative inquiries are not incompatible: A call for a new partnership
Reichardt, C. S., & Rallis, S. F. (1994) · New Directions for Program Evaluation · doi:10.1002/ev.1670 · Cited by 151
Emotions and Positional Identity in Becoming a Social Justice Science Teacher: Nicole's Story
Rivera Maulucci, M. S. (2013) · Journal of Research in Science Teaching · doi:10.1002/tea.21081 · Cited by 147
Calder, M. J., Richter, M. S., Mao, Y., Kovacs Burns, K., Mogale, R. S., & Danko, M. (2016) · Canadian Journal of Higher Education · doi:10.47678/cjhe.v46i2.184585 · Cited by 141
Critical Realism as emancipatory action: the case for realistic evaluation in practice development
Wilson, V., & McCormack, B. (2006) · Nursing Philosophy · doi:10.1111/j.1466-769x.2006.00248.x · Cited by 119
Weaving an interpretivist stance throughout mixed methods research
McChesney, K., & Aldridge, J. (2019) · International Journal of Research & Method in Education · doi:10.1080/1743727x.2019.1590811 · Cited by 118
Abendroth, M., Lutz, B. J., & Young, M. E. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2011.10.003 · Cited by 116
Finding Voice and Passion: Critical Race Theory Methodology in Science Teacher Education
Mensah, F. M. (2019) · American Educational Research Journal · doi:10.3102/0002831218818093 · Cited by 109
Creating a culture of data use for continuous improvement: A case study of an Edison Project school
Sutherland, S. (2004) · American Journal of Evaluation · doi:10.1177/109821400402500302 · Cited by 106
Rycroft-Malone, J., Wilkinson, J., Burton, C. R., Harvey, G., McCormack, B., Graham, I., & Staniszewska, S. (2013) · Journal of Health Services Research & Policy · doi:10.1177/1355819613498859 · Cited by 98
When academics integrate research skill development in the curriculum
Willison, J. W. (2012) · Higher Education Research & Development · doi:10.1080/07294360.2012.658760 · Cited by 92
Goberna-Tricas, J., Banús-Giménez, M. R., Palacio-Tauste, A., & Linares-Sancho, S. (2011) · Midwifery · doi:10.1016/j.midw.2010.10.004 · Cited by 88
Transforming Beef By-products into Valuable Ingredients: Which Spell/Recipe to Use?
Henchion, M., McCarthy, M., & O’Callaghan, J. (2016) · Frontiers in Nutrition · doi:10.3389/fnut.2016.00053 · Cited by 83
Riazi, A. M., Rezvani, R., & Ghanbar, H. (2023) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2023.100065 · Cited by 79
Denzin, N. K. (2015) · The Blackwell Encyclopedia of Sociology · doi:10.1002/9781405165518.wbeost050.pub2 · Cited by 75
Hall, V., Conboy-Hill, S., & Taylor, D. (2011) · Journal of Medical Internet Research · doi:10.2196/jmir.1917 · Cited by 71
An exploration of the hydration care of older people: A qualitative study
Godfrey, H., Cloete, J., Dymond, E., & Long, A. (2012) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2012.04.009 · Cited by 67
University of Melbourne, Brunzell, T., Stokes, H., University of Melbourne, Waters, L., & University of Melbourne (2018) · Australian Journal of Teacher Education · doi:10.14221/ajte.2018v43n2.7 · Cited by 66
Canton-Thompson, J., Gebert, K. M., Thompson, B., Jones, G., Calkin, D., & Donovan, G. (2008) · Journal of Forestry · doi:10.1093/jof/106.8.416 · Cited by 65
Metadata
- Authors
- Neufeld, A., & Ryan, A. G.
- Venue
- Canadian Journal of Program Evaluation
- Publisher
- University of Victoria Libraries
- Published
- 1988-09
- Volume
- 3
- Issue
- 2
- Pages
- 77-91
- ISSN
- 0834-1516 (ISSN-L)
- 1496-7308
- OpenAlex
- W4362505439
- Cited by
- 5,913
- References
- 0
- FWCI
- 22.54
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.