The DOI matches its Crossref record; 93 of the 93 references it cites were found in a registry or an index.
Best Practice Recommendations for Designing and Implementing Experimental Vignette Methodology Studies
Abstract
We describe experimental vignette methodology (EVM) as a way to address the dilemma of conducting experimental research that results in high levels of confidence regarding internal validity but is challenged by threats to external validity versus conducting nonexperimental research that usually maximizes external validity but whose conclusions are ambiguous regarding causal relationships. EVM studies consist of presenting participants with carefully constructed and realistic scenarios to assess dependent variables including intentions, attitudes, and behaviors, thereby enhancing experimental realism and also allowing researchers to manipulate and control independent variables. We describe two major types of EVM aimed at assessing explicit (i.e., paper people studies) and implicit (i.e., policy capturing and conjoint analysis) processes and outcomes. We offer best practice recommendations regarding the design and implementation of EVM studies based on a multidisciplinary literature review, discuss substantive domains and topics that can benefit from implementing EVM, address knowledge gaps regarding EVM such as the need to increase realism and the number and diversity of participants, and address ways to overcome some of the negative perceptions about EVM by pointing to exemplary articles that have used EVM successfully.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2002) · Cited by 13,211
Ireland, R. D., Elfring, T., & Hmieleski, K. M. (2011) · In Search of Research Excellence · doi:10.4337/9781849807630.00017 · Cited by 4,553
What We Know and Don’t Know About Corporate Social Responsibility: A Review and Research Agenda
Aguinis, H., & Glavas, A. (2012) · Journal of Management · doi:10.1177/0149206311436079 · Cited by 3,558
Self-citationThe international personality item pool and the future of public-domain personality measures
Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. G. (2006) · Journal of Research in Personality · doi:10.1016/j.jrp.2005.08.007 · Cited by 3,185
Benefits of Training and Development for Individuals and Teams, Organizations, and Society
Aguinis, H., & Kraiger, K. (2009) · Annual Review of Psychology · doi:10.1146/annurev.psych.60.110707.163505 · Cited by 1,453
Self-citationRESEARCH METHODOLOGY IN MANAGEMENT: CURRENT PRACTICES, TRENDS, AND IMPLICATIONS FOR FUTURE RESEARCH.
Scandura, T. A., & Williams, E. A. (2000) · Academy of Management Journal · doi:10.2307/1556348 · Cited by 1,186
The Use of Vignettes in Survey Research
Alexander, C. S., & Becker, H. J. (1978) · Public Opinion Quarterly · doi:10.1086/268432 · Cited by 1,169
The Vignette Technique in Survey Research
Finch, J. (1987) · Sociology · doi:10.1177/0038038587021001008 · Cited by 1,126
Aguinis, H., Gottfredson, R. K., & Culpepper, S. A. (2013) · Journal of Management · doi:10.1177/0149206313478188 · Cited by 1,090
Self-citationExperimental Vignette Studies in Survey Research
Atzmüller, C., & Steiner, P. M. (2010) · Methodology · doi:10.1027/1614-2241/a000014 · Cited by 997
Research in Corporate Social Performance and Policy
Walters, K. D., & Preston, L. E. (1982) · The Academy of Management Review · doi:10.2307/257239 · Cited by 969
Barney, J., & Felin, T. (2013) · Academy of Management Perspectives · doi:10.5465/amp.2012.0107 · Cited by 839
Recommendations for Increasing Replicability in Psychology
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., et al. (2013) · European Journal of Personality · doi:10.1002/per.1919 · Cited by 823
The application of vignettes in social and nursing research
Hughes, R., & Huby, M. (2002) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2002.02100.x · Cited by 701
Research in Corporate Social Performance and Policy.
Preston, L. E., & Walters, K. D. (1982) · Academy of Management Review · doi:10.5465/amr.1982.4285292 · Cited by 615
The "Science of the Sophomore" Revisited: From Conjecture to Empiricism
Gordon, M. E., Slade, L. A., & Schmitt, N. (1986) · The Academy of Management Review · doi:10.2307/258340 · Cited by 574
Analyzing Decision Making: Metric Conjoint Analysis
Wittink, D. R., & Louviere, J. J. (1989) · Journal of Marketing Research · doi:10.2307/3172612 · Cited by 568
A review of research methods in IO/OB work-family research.
Casper, W. J., Eby, L. T., Bordeaux, C., Lockwood, A., & Lambert, D. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.1.28 · Cited by 558
Podsakoff, P. M., MacKenzie, S. B., Podsakoff, N. P., & Bachrach, D. G. (2008) · Journal of Management · doi:10.1177/0149206308319533 · Cited by 523
The yin and yang of progress in social psychology: Seven koan.
McGuire, W. J. (1973) · Journal of Personality and Social Psychology · doi:10.1037/h0034345 · Cited by 490
Unpacking the uncertainty construct: Implications for entrepreneurial action
McKelvie, A., Haynie, J. M., & Gustavsson, V. (2011) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2009.10.004 · Cited by 475
The Time Has Come: Bayesian Methods for Data Analysis in the Organizational Sciences
Kruschke, J. K., Aguinis, H., & Joo, H. (2012) · Organizational Research Methods · doi:10.1177/1094428112457829 · Cited by 457
Self-citationFactorial Survey Methods for Studying Beliefs and Judgments
Jasso, G. (2006) · Sociological Methods & Research · doi:10.1177/0049124105283121 · Cited by 439
Podsakoff, N. P., Podsakoff, P. M., MacKenzie, S. B., Maynes, T. D., & Spoelma, T. M. (2014) · Journal of Organizational Behavior · doi:10.1002/job.1911 · Cited by 425
Louviere, J. J. (1988) · Sage eBooks · doi:10.4135/9781412983365 · Cited by 409
KEEN TO HELP? MANAGERS' IMPLICIT PERSON THEORIES AND THEIR SUBSEQUENT EMPLOYEE COACHING
HESLIN, P. A., VANDEWALLE, D., & LATHAM, G. P. (2006) · Personnel Psychology · doi:10.1111/j.1744-6570.2006.00057.x · Cited by 408
Thirty Years of Conjoint Analysis: Reflections and Prospects
Green, P. E., Krieger, A. M., & Wind, Y. (2001) · Interfaces · doi:10.1287/inte.31.3s.56.9676 · Cited by 388
WEIGHT‐BASED DISCRIMINATION IN EMPLOYMENT: PSYCHOLOGICAL AND LEGAL ASPECTS
ROEHLING, M. V. (1999) · Personnel Psychology · doi:10.1111/j.1744-6570.1999.tb00186.x · Cited by 381
Designing experiments that generalize
Highhouse, S. (2009) · Organizational Research Methods · doi:10.1177/1094428107300396 · Cited by 353 · cited 2×
“Specifically, it takes more time and effort to conduct an experiment, involving the creation of experimental materials, recruiting of participants, administering the experimental treatments, and addressing ethical challenges in administering the treatments compared to using archival data or an online survey (Aguinis &…”
MethodFactorial Surveys: Using Vignettes to Study Professional Judgement
Taylor, B. J. (2005) · British Journal of Social Work · doi:10.1093/bjsw/bch345 · Cited by 342
Hybrid Models for Conjoint Analysis: An Expository Review
Green, P. E. (1984) · Journal of Marketing Research · doi:10.1177/002224378402100204 · Cited by 324
An Ounce of Prevention Is Worth a Pound of Cure: Improving Research Quality Before Data Collection
Aguinis, H., & Vandenberg, R. J. (2014) · Annual Review of Organizational Psychology and Organizational Behavior · doi:10.1146/annurev-orgpsych-031413-091231 · Cited by 304
Self-citationMcNemar, Q. (1946) · Psychological Bulletin · doi:10.1037/h0060985 · Cited by 302
Scenarios in Business Ethics Research: Review, Critical Assessment, and Recommendations
Weber, J. (1992) · Business Ethics Quarterly · doi:10.2307/3857568 · Cited by 299
“I Care about Nature, but …”: Disengaging Values in Assessing Opportunities that Cause Harm
Shepherd, D. A., Patzelt, H., & Baron, R. A. (2013) · Academy of Management Journal · doi:10.5465/amj.2011.0776 · Cited by 289
Aiman-Smith, L., Scullen, S. E., & Barr, S. H. (2002) · Organizational Research Methods · doi:10.1177/109442802237117 · Cited by 289
Designing Vignette Studies in Marketing
Wason, K. D., Polonsky, M. J., & Hyman, M. R. (2002) · Australasian Marketing Journal · doi:10.1016/s1441-3582(02)70157-2 · Cited by 284
Grant, A. M., & Wall, T. D. (2009) · Organizational Research Methods · doi:10.1177/1094428108320737 · Cited by 284
The role of affect and affective congruence in perceptions of leaders: an experimental study
Newcombe, M. J., & Ashkanasy, N. M. (2002) · The Leadership Quarterly · doi:10.1016/s1048-9843(02)00146-7 · Cited by 280
Employment Discrimination in Organizations: Antecedents and Consequences
Goldman, B. M., Gutek, B. A., Stein, J. H., & Lewis, K. (2006) · Journal of Management · doi:10.1177/0149206306293544 · Cited by 277
Karren, R. J., & Barringer, M. W. (2002) · Organizational Research Methods · doi:10.1177/109442802237115 · Cited by 274
Methodological Wishes for the Next Decade and How to Make Wishes Come True
Aguinis, H., & Edwards, J. R. (2014) · Journal of Management Studies · doi:10.1111/joms.12058 · Cited by 267
Self-citationThe Use of Vignettes in Qualitative Research into Social Work Values
Wilks, T. (2004) · Qualitative Social Work · doi:10.1177/1473325004041133 · Cited by 264
Effects of message, source, and context on evaluations of employee voice behavior.
Whiting, S. W., Maynes, T. D., Podsakoff, N. P., & Podsakoff, P. M. (2012) · Journal of Applied Psychology · doi:10.1037/a0024871 · Cited by 261
Using Experience Sampling Methodology to Advance Entrepreneurship Theory and Research
Uy, M. A., Foo, M.-D., & Aguinis, H. (2010) · Organizational Research Methods · doi:10.1177/1094428109334977 · Cited by 261
Self-citationInnovations in Experimental Design in Attitude Surveys
Sniderman, P. M., & Grob, D. B. (1996) · Annual Review of Sociology · doi:10.1146/annurev.soc.22.1.377 · Cited by 260
Shepherd, D. A., & Zacharakis, A. (1999) · Venture Capital · doi:10.1080/136910699295866 · Cited by 254
Testing management theories: critical realist philosophy and research methods
Miller, K. D., & Tsang, E. W. K. (2011) · Strategic Management Journal · doi:10.1002/smj.868 · Cited by 248
Jasny, B. R., Chin, G., Chong, L., & Vignieri, S. (2011) · Science · doi:10.1126/science.334.6060.1225 · Cited by 245
EASTMAN, K. K. (1994) · Academy of Management Journal · doi:10.2307/256678 · Cited by 245
Leader self-sacrifice and leadership effectiveness: The moderating role of leader self-confidence
Cremer, D. D., & Knippenberg, D. V. (2004) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2004.04.002 · Cited by 243
The Analysis of Factorial Surveys
HOX, J. J., KREFT, I. G. G., & HERMKENS, P. L. J. (1991) · Sociological Methods & Research · doi:10.1177/0049124191019004003 · Cited by 240
Contingent Leadership and Effectiveness of Trauma Resuscitation Teams.
Yun, S., Faraj, S., & Sims, H. P. (2005) · Journal of Applied Psychology · doi:10.1037/0021-9010.90.6.1288 · Cited by 230
Decision Research: A Field Guide
Mayne, A. J. (1992) · Journal of the Operational Research Society · doi:10.1057/jors.1992.8 · Cited by 226
Management Science on the Credibility Bubble: Cardinal Sins and Various Misdemeanors.
BEDEIAN, A. G., TAYLOR, S. G., & MILLER, A. N. (2010) · Academy of Management Learning & Education · doi:10.5465/amle.2010.56659889 · Cited by 223
Multilevel Entrepreneurship Research: Opportunities for Studying Entrepreneurial Decision Making
Shepherd, D. A. (2011) · Journal of Management · doi:10.1177/0149206310369940 · Cited by 218
I Need Time! Exploring Pathways to Compliance under Institutional Complexity
Raaijmakers, A. G. M., Vermeulen, P. A. M., Meeus, M. T. H., & Zietsma, C. (2015) · Academy of Management Journal · doi:10.5465/amj.2011.0276 · Cited by 194
De Cremer, D., Mayer, D. M., van Dijke, M., Schouten, B. C., & Bardes, M. (2009) · Journal of Applied Psychology · doi:10.1037/a0014782 · Cited by 183
Dangers in applying results from experimental social psychology.
Argyris, C. (1975) · American Psychologist · doi:10.1037/h0076834 · Cited by 174
Conjoint Analysis in Entrepreneurship Research: A Review and Research Agenda
Lohrke, F. T., Holloway, B. B., & Woolley, T. W. (2010) · Organizational Research Methods · doi:10.1177/1094428109341992 · Cited by 172
Leadership style and regulatory mode: Value from fit?
Benjamin, L., & Flynn, F. J. (2006) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2006.01.008 · Cited by 172
Just the two of us: Misalignment of theory and methods in examining dyadic phenomena.
Krasikova, D. V., & LeBreton, J. M. (2012) · Journal of Applied Psychology · doi:10.1037/a0027962 · Cited by 164
Research Methodology in Organizational Studies
Podsakoff, P. M., & Dalton, D. R. (1987) · Journal of Management · doi:10.1177/014920638701300213 · Cited by 161
Multiple Segment Factorial Vignette Designs
Ganong, L. H., & Coleman, M. (2006) · Journal of Marriage and Family · doi:10.1111/j.1741-3737.2006.00264.x · Cited by 160
eLancing: A review and research agenda for bridging the science–practice gap
Aguinis, H., & Lawal, S. O. (2013) · Human Resource Management Review · doi:10.1016/j.hrmr.2012.06.003 · Cited by 158
Self-citationTaking the reins: The effects of new leader status and leadership style on team performance.
Sauer, S. J. (2011) · Journal of Applied Psychology · doi:10.1037/a0022741 · Cited by 158
Effects of organizational citizenship behaviors on selection decisions in employment interviews.
Podsakoff, N. P., Whiting, S. W., Podsakoff, P. M., & Mishra, P. (2010) · Journal of Applied Psychology · doi:10.1037/a0020948 · Cited by 151
Bucar, B., Glas, M., & Hisrich, R. D. (2003) · Journal of Business Venturing · doi:10.1016/s0883-9026(01)00083-0 · Cited by 149
Evaluating the performance of paper people.
Murphy, K. R., Herr, B. M., Lockhart, M. C., & Maguire, E. (1986) · Journal of Applied Psychology · doi:10.1037/0021-9010.71.4.654 · Cited by 142
The social role theory of unethical leadership
Hoyt, C. L., Price, T. L., & Poatsy, L. (2013) · The Leadership Quarterly · doi:10.1016/j.leaqua.2013.07.001 · Cited by 129
Analytical mindsets in turnover research
Allen, D. G., Hancock, J. I., Vardaman, J. M., & McKee, D. N. (2013) · Journal of Organizational Behavior · doi:10.1002/job.1912 · Cited by 128
Spector, P. E. (1981) · Sage eBooks · doi:10.4135/9781412985673 · Cited by 128
Priem, R. L., & Harrison, D. A. (1994) · Strategic Management Journal · doi:10.1002/smj.4250150405 · Cited by 124
Conducting field experiments using eLancing's natural environment
Aguinis, H., & Lawal, S. O. (2012) · Journal of Business Venturing · doi:10.1016/j.jbusvent.2012.01.002 · Cited by 117
Self-citationHobson, C. J., & Hobson, F. W. (1983) · Academy of Management Review · doi:10.5465/amr.1983.4284668 · Cited by 108
Burstin, K., Doughtie, E. B., & Raphaeli, A. (1980) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.1980.tb00699.x · Cited by 106
Matsukura, H., Yoneda, T., & Ishida, H. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.40 · Cited by 105
Melone, N. P. (1994) · Organization Science · doi:10.1287/orsc.5.3.438 · Cited by 104
The Factorial Survey: An Experimental Method to Replicate Real World Problems
Ludwick, R. E., & Zeller, R. A. (2001) · Nursing Research · doi:10.1097/00006199-200103000-00009 · Cited by 103
Pelletier, K. L. (2011) · The Leadership Quarterly · doi:10.1016/j.leaqua.2011.09.011 · Cited by 103
The Impact of Poor Performers on Team Outcomes: An Empirical Examination of Attribution Theory*
Taggar, S., & Neubert, M. J. (2004) · Personnel Psychology · doi:10.1111/j.1744-6570.2004.00011.x · Cited by 95
Pierce, C. A., Aguinis, H., & Adams, S. K. R. (2000) · Academy of Management Journal · doi:10.2307/1556415 · Cited by 87
Self-citationConsideration of the Incomplete Block Design for Policy-Capturing Research
Graham, M., & Cable, D. M. (2001) · Organizational Research Methods · doi:10.1177/109442810141002 · Cited by 86
Aguinis, H., & Adams, S. K. R. (1998) · Group & Organization Management · doi:10.1177/1059601198234005 · Cited by 74
Self-citationData analytic trends and training in strategic management
Shook, C. L., Ketchen, D. J., Cycyota, C. S., et al. (2003) · Strategic Management Journal · doi:10.1002/smj.352 · Cited by 68
Hobson, C. J., & Gibson, F. W. (1983) · Academy of Management Review · doi:10.2307/258265 · Cited by 63
Using virtual reality technology in organizational behavior research
Pierce, C. A., & Aguinis, H. (1997) · Journal of Organizational Behavior · Cited by 53
Self-citationPriem, R. L., Walters, B. A., & Li, S. (2010) · Journal of Management · doi:10.1177/0149206310372258 · Cited by 47
Paper people versus direct observation: An empirical examination of laboratory methodologies
Woehr, D. J., & Lance, C. E. (1991) · Journal of Organizational Behavior · doi:10.1002/job.4030120504 · Cited by 44
Unfairness begets unfairness: Victim derogation bias in employee ratings
Skarlicki, D. P., & Turner, R. A. (2014) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2013.11.004 · Cited by 43
Comparing the Efficacy of Policy-Capturing Weights and Direct Estimates for Predicting Job Choice
Slaughter, J. E., Richard, E. M., & Martin, J. H. (2006) · Organizational Research Methods · doi:10.1177/1094428105279936 · Cited by 41
“EXTRACTING” POLICY FROM JUDICIAL OPINIONS: THE DANGERS OF POLICY CAPTURING IN A FIELD SETTING
Roehling, M. V. (1993) · Personnel Psychology · doi:10.1111/j.1744-6570.1993.tb00881.x · Cited by 40
The “top-of-the-line” influence on the buyer–seller relationship
Donoho, C. L. (2003) · Journal of Business Research · doi:10.1016/s0148-2963(02)00442-3 · Cited by 8
Cited by
A Critical Review and Best‐Practice Recommendations for Control Variable Usage
Bernerth, J. B., & Aguinis, H. (2016) · Personnel Psychology · doi:10.1111/peps.12103 · Cited by 1,731
Self-citationResearch Methods in Business Studies
Ghauri, P., Grønhaug, K., & Strange, R. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108762427 · Cited by 1,198
Employee Green Behavior: A Theoretical Framework, Multilevel Review, and Future Research Agenda
Norton, T. A., Parker, S. L., Zacher, H., & Ashkanasy, N. M. (2015) · Organization & Environment · doi:10.1177/1086026615575773 · Cited by 981
On doing better science: From thrill of discovery to policy implications
Antonakis, J. (2017) · The Leadership Quarterly · doi:10.1016/j.leaqua.2017.01.006 · Cited by 475
On doing relevant and rigorous experiments: Review and recommendations
Lonati, S., Quiroga, B. F., Zehnder, C., & Antonakis, J. (2018) · Journal of Operations Management · doi:10.1016/j.jom.2018.10.003 · Cited by 354
Metadata
- Authors
- Aguinis, H., & Bradley, K. J.
- Venue
- Organizational Research Methods
- Publisher
- SAGE Publications
- Published
- 2014-08-25
- Volume
- 17
- Issue
- 4
- Pages
- 351-371
- ISSN
- 1094-4281 (ISSN-L)
- 1552-7425
- OpenAlex
- W1977793234
- Cited by
- 2,021
- References
- 93
- FWCI
- 118.03
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.