Statistical Methods for the Validation of Questionnaires: Discrepancy between Theory and Practice
Abstract
Summary: Objectives: Questionnaires used in epidemiological studies should be validated. However, unclarity exists about the appropriate statistical methods and interpretation of validation studies. Thus, we investigated the theory and practice of statistical evaluation approaches. Methods: Using three platforms, a literature review, own simulations, and a validation study performed by ourselves, we worked out relevant limitations, advantages, and new important aspects of evaluation methods. Results: Our systematic literature review, based on physical activity questionnaires, revealed that correlation coefficients are still the common approach in validation studies, found in 41 of 46 reviewed publications (89.1%). This practice has been criticized in the theoretically oriented literature for more than 20 years. Appropriate evaluation methods as recommended by Bland and Altman were found in only ten publications (21.7 %). We showed that serious bias in questionnaires can be revealed by Bland-Altman plots but may remain undetected by correlation coefficients. With our simulations we refuted the argument that correlation coefficients properly investigate whether a questionnaire ranks the subjects sufficiently well. Further, with Bland-Altman analyses we could evaluate differential errors with respect to case-control status in our validation study. Yet, this was not possible with correlation coefficients, because they generally do not identify systematic bias. In addition, we show a potential pitfall in the interpretation of Bland-Altman plots that might occur in specific rare instances. Conclusions: The commonly used correlation approach can yield misleading conclusions in validation studies. A more frequent and proper use of the Bland-Altman methods would be desirable to improve epidemiological data quality.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Correlation coefficients may be misleading as a measure of validity.
“because correlation coefficients may be misleading [18].”
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Sorted by citations
Warren, J. M., Ekelund, U., Besson, H., Mezzani, A., Geladas, N., & Vanhees, L. (2010) · European Journal of Cardiovascular Prevention & Rehabilitation · doi:10.1097/hjr.0b013e32832ed875 · Cited by 587 · see citing works
Accelerometry and physical activity questionnaires - a systematic review
Skender, S., Ose, J., Chang-Claude, J., Paskow, M., Brühmann, B., Siegel, E. M., Steindorf, K., & Ulrich, C. M. (2016) · BMC Public Health · doi:10.1186/s12889-016-3172-0 · Cited by 449 · see citing works
“Prior research has shown that bias in questionnaires can be revealed by Bland–Altman plots, while it may remain undetected by the use of correlation coefficients [87].”
Self-citationPredicting outcomes in radiation oncology—multifactorial decision support systems
Lambin, P., van Stiphout, R. G. P. M., Starmans, M. H. W., Velazquez, E. R., Nalbantov, G. I., Aerts, H. J. W. L., Roelofs, E., van Elmpt, W., Boutros, P. C., Granone, P. M., et al. (2012) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2012.196 · Cited by 396 · see citing works
Validity of activity monitors in health and chronic disease: a systematic review
Remoortel, H. V., Giavedoni, S., Raste, Y., Burtin, C., Louvaris, Z., Gimeno‐Santos, E., Langer, D., Glendenning, G. A., Hopkinson, N. S., Vogiatzis, I., et al. (2012) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/1479-5868-9-84 · Cited by 277 · see citing works
“A systematic review of the statistical methods used to validate physical activity questionnaires revealed similar findings, with the majority of the studies using correlation analysis compared to Bland and Altman analysis [153].”
MethodEstimating activity energy expenditure: how valid are physical activity questionnaires?
Neilson, H. K., Robson, P. J., Friedenreich, C. M., & Csizmadi, I. (2008) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/87.2.279 · Cited by 211 · see citing works
“Even with sufficient sample sizes, the limitations of correlation coefficients are well documented (47, 49, 50, 94), with the literature emphasizing 2 potential pitfalls.”
BackgroundMethodResultInfluentialDecision support systems for personalized and participative radiation oncology
Lambin, P., Zindler, J. D., Vanneste, B. G. L., Voorde, L. V. D., Eekers, D. B. P., Compter, I., Panth, K. M., Peerlings, J., Larue, R. T. H. M., Deist, T. M., et al. (2016) · Advanced Drug Delivery Reviews · doi:10.1016/j.addr.2016.01.006 · Cited by 154 · see citing works
“should be validated as is the case for laboratory measurements of organ function or parameters measured from blood [74,75].”
BackgroundPhillips, S. M., Summerbell, C., Hobbs, M., Hesketh, K. R., Saxena, S. K., Muir, C., & Hillier-Brown, F. C. (2021) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-021-01132-9 · Cited by 127 · see citing works
Reliability and agreement studies: a guide for clinical investigators
Hernáez, R. (2015) · Gut · doi:10.1136/gutjnl-2014-308619 · Cited by 98 · see citing works
Relative validity of a food frequency questionnaire used in the Inter99 study
Toft, U., Kristoffersen, L., Ladelund, S., Bysted, A., Jakobsen, J., Lau, C. J., Jørgensen, T., Borch‐Johnsen, K., & Ovesen, L. (2007) · European Journal of Clinical Nutrition · doi:10.1038/sj.ejcn.1602815 · Cited by 90 · see citing works
“However, all these four methods have limitations and cannot determine validity alone (Bellach, 1993; Carstensen, 2004; Schmidt and Steindorf, 2006), and better alternatives to correlation coefficients and other methods commonly used in validation studies are needed.”
BackgroundEvenson, K. R., Chasan‐Taber, L., Downs, D. S., & Pearce, E. (2012) · Paediatric and Perinatal Epidemiology · doi:10.1111/j.1365-3016.2012.01311.x · Cited by 90 · see citing works
Ball, T., Joy, E. A., Goh, T. L., Hannon, J. C., Gren, L. H., & Shaw, J. M. (2014) · Primary Health Care Research & Development · doi:10.1017/s1463423613000479 · Cited by 55 · see citing works
“Evidence that determines the usefulness of a primary care PA questionnaire should include statistical agreement with a criterion measure of PA, as well as sensitivity to change, or repeatability (Bland and Altman, 1986; Luiz and Szklo, 2005; Schmidt and Steindorf, 2006; Mâsse, 2010).”
MethodCargill, J., Lucas, R. M., Gies, P. H., King, K., Swaminathan, A. D., Allen, M., & Banks, E. (2012) · Photochemistry and Photobiology · doi:10.1111/j.1751-1097.2012.01221.x · Cited by 52 · see citing works
“Agreement was further assessed using the Bland–Altman (BA) method (30), by plotting the difference between the questionnaire and diary measurements of time outdoors for each participant against the mean of these measurements, which allows both random and systematic errors to be examined visually (31).”
MethodCellular Telephones Measure Activity and Lifespace in Community‐Dwelling Adults: Proof of Principle
Schenk, A. K., Witbrodt, B. C., Hoarty, C. A., Carlson, R. H., Goulding, E. H., POTTER, J. F., & Bonasera, S. J. (2011) · Journal of the American Geriatrics Society · doi:10.1111/j.1532-5415.2010.03267.x · Cited by 52 · see citing works
McCaw, J. M., Forbes, K. M., Nathan, P. M., Pattison, P. E., Robins, G. L., Nolan, T. M., & McVernon, J. (2010) · BMC Infectious Diseases · doi:10.1186/1471-2334-10-166 · Cited by 45 · see citing works
Validation of a management competency assessment tool for health service managers
Howard, P. F., Liang, Z., Leggat, S. G., & Karimi, L. (2018) · Journal of Health Organization and Management · doi:10.1108/jhom-08-2017-0223 · Cited by 44 · see citing works
Position statement: Testing physical condition in a population – how good are the methods?
Jørgensen, T., Andersen, L. B., Froberg, K., Maeder, U., Smith, L. V. H., & Aadahl, M. (2009) · European Journal of Sport Science · doi:10.1080/17461390902862664 · Cited by 42 · see citing works
Dietary intake and biological measurement of folate: A qualitative review of validation studies
Park, J. Y., Vollset, S. E., Melse‐Boonstra, A., Chajès, V., Ueland, P. M., & Slimani, N. (2012) · Molecular Nutrition & Food Research · doi:10.1002/mnfr.201200105 · Cited by 32 · see citing works
“The use of Bland–Altman plot (graphs of the differences between the test and reference measurements against the mean of the two [47] with limits of agreement calculated as the mean difference plus and minus two standard deviations [48]) has been recommended when investigating validity as it assesses the agreement grap…”
MethodA review of how to conduct a surgical survey using a questionnaire
Hing, C. B., Smith, T. O., Hooper, L., et al. (2010) · The Knee · doi:10.1016/j.knee.2010.10.003 · Cited by 30 · see citing works
“population, the design of the questionnaire must be valid, reliable, unbiased and discriminatory [2,3].”
BackgroundHirsch, O., Keller, H., Albohn-Kühne, C., Krones, T., & Donner‐Banzhoff, N. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-71 · Cited by 27 · see citing works
“Schmidt and Steindorf [50] showed that the Bland-Altman method is adequate and robust for questionnaire data.”
BackgroundMethodInfluentialOutcome Analysis of Surgery for Blount Disease
Jones, J. K., Gill, L., John, M., Goddard, M. S., & Hambleton, I. R. (2009) · Journal of Pediatric Orthopaedics · doi:10.1097/bpo.0b013e3181b768d9 · Cited by 26 · see citing works
“-40 -20 0 20 40 femoral-tibial angle scores mda (0) mda (20) mda (-20)”
BackgroundInfluentialEvaluation of a short retrospective questionnaire for physical activity in women
Schmidt, M. E., Slanger, T. E., Chang-Claude, J. C., Wahrendorf, J., & Steindorf, K. (2006) · European Journal of Epidemiology · doi:10.1007/s10654-006-9042-9 · Cited by 26 · see citing works
“which might have resulted in misleading conclusions [11, 12].”
BackgroundSelf-citationLines, R. L. J., Ntoumanis, N., Thøgersen‐Ntoumani, C., McVeigh, J. A., Ducker, K. J., Fletcher, D., & Gucciardi, D. F. (2020) · Journal of science and medicine in sport · doi:10.1016/j.jsams.2020.03.004 · Cited by 23 · see citing works
Comparison of energy estimates in chronic kidney disease using doubly‐labelled water
Sridharan, S., Wong, J., Vilar, E., & Farrington, K. (2015) · Journal of Human Nutrition and Dietetics · doi:10.1111/jhn.12326 · Cited by 22 · see citing works
“The validity of activity questionnaires should not be assessed solely on correlations alone ((17,21,22)).”
BackgroundNitel Araştırmaların Raporlanmasında Standartlar: SRQR Kontrol Listesinin Türkçe Uyarlaması
Tekindal, M. T., Özden, S. A., Gedik, T. E., Ahmet, E. G. E., ERİM, F., & Tekindal, M. A. (2021) · Opus uluslararası toplum araştırmaları dergisi · doi:10.26466/opus.882177 · Cited by 23 · see citing works
Webster, S., Khan, A., & Nitz, J. C. (2011) · Journal of Clinical Gerontology and Geriatrics · doi:10.1016/j.jcgg.2011.06.003 · Cited by 17 · see citing works
Quality Food Products as a Tourist Attraction in the Province of Córdoba (Spain)
Millán, M. G. D., & de la Torre, M. G. M. V. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph191912754 · Cited by 17 · see citing works
Reliability and validity of a physical activity questionnaire in children
Platat, C., & Jarrar, A. H. (2011) · International Journal of Food Sciences and Nutrition · doi:10.3109/09637486.2011.644766 · Cited by 17 · see citing works
“Besides, as correlational analysis alone may not reveal the potential bias, Bland–Altman method was used in which a fair agreement was first obtained as defined by an ICC with value $0.40 (Schmidt and Steindorf 2006).”
MethodInfluentialMonyeki, M. A., Moss, S. J., Kemper, H. C. G., & Twisk, J. W. R. (2018) · Children · doi:10.3390/children5060071 · Cited by 17 · see citing works
Validation of two short questionnaires assessing physical activity in colorectal cancer patients
Henriksen, H. B., Berntsen, S., Paur, I., Zucknick, M., Skjetne, A. J., Bøhn, S. K., Henriksen, C., Smeland, S., Carlsen, M. H., & Blomhoff, R. (2018) · BMC Sports Science Medicine and Rehabilitation · doi:10.1186/s13102-018-0096-2 · Cited by 16 · see citing works
“questionnaire minus SWA) plotted against the average of the two measurements, for each individual subject, as well as to identify outliers [52, 53].”
MethodAssessing the level of agreement between the self‐ and interview‐administered Child‐OIDP
Rosel, E., Tsakos, G., Bernabé, E., Sheiham, A., & Bravo, M. (2010) · Community Dentistry And Oral Epidemiology · doi:10.1111/j.1600-0528.2010.00533.x · Cited by 16 · see citing works
“method provides consistently higher or lower scores than the other (17, 18).”
BackgroundTest-retest reliability of physical activity questionnaires in Parkinson’s disease
Ånfors, S., Kammerlind, A.-S. C., & Nilsson, M. H. (2021) · BMC Neurology · doi:10.1186/s12883-021-02426-y · Cited by 15 · see citing works
“Moreover, limits of agreement (LoA) were visualized with Bland-Altman plots [54] for the differences and means of the paired data in PASE and IPAQ-SF, a method recommended when evaluating questionnaires [55].”
MethodEke, H., Sjöblom, L., Lagerros, Y. T., et al. (2023) · Nutrition · doi:10.1016/j.nut.2023.112332 · Cited by 11 · see citing works
Olaya-Cuartero, J., Fernández‐Sáez, J., Østerlie, O., et al. (2022) · Biology · doi:10.3390/biology11060902 · Cited by 11 · see citing works
“However, correlation coefficient analyses are mostly used to determine which segment of the triathlon is most associated with the overall performance in the competition, but these would not be the most appropriate since calculation depends on variability between subjects [23] and is sensitive to the range of values un…”
MethodEnergy expenditure estimates in chronic kidney disease using a novel physical activity questionnaire
Sridharan, S., Vilar, E., Ramanarayanan, S., Davenport, A., & Farrington, K. (2020) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfaa377 · Cited by 9 · see citing works
Haddad, M., & Ebada, M. E. (2017) · Journal of Public Health Research · doi:10.4081/jphr.2017.818 · Cited by 8 · see citing works
Técnicas para estimar la estabilidad de una escala de medición en salud.
Simancas‐Pallares, M., Herazo, E., & Campo‐Arias, A. (2016) · Revista Ciencias Biomédicas · doi:10.32997/rcb-2016-2936 · Cited by 6 · see citing works
“Gráficos de Bland y Altman para las puntuaciones en la CES-D en adolescentes de Bucaramanga-Colombia (55). rias técnicas estadísticas para estimar la estabilidad de escalas.”
BackgroundPatient satisfaction with clinicians in colorectal 2-week wait clinics
Cock, K., & Kent, B. (2017) · British Journal of Nursing · doi:10.12968/bjon.2017.26.6.319 · Cited by 6 · see citing works
Assessing Levels of Agreement between Two Commonly Used Oral Health‐Related Quality of Life Measures
Bernabé, E., de Oliveira, C. M., Sheiham, A., & Tsakos, G. (2009) · Journal of Public Health Dentistry · doi:10.1111/j.1752-7325.2008.00112.x · Cited by 6 · see citing works
Differential Item Functioning of the Psychological Domain of the Menopause Rating Scale
Monterrosa-Castro, Á. D. J., Portela-Buelvas, K., Oviedo, H. C., et al. (2016) · BioMed Research International · doi:10.1155/2016/8790691 · Cited by 5 · see citing works
Inconclusiveness of psychometric testing of medication adherence questionnaires
Arnet, I., Eickhoff, C., Sahm, L. J., Caloz, B. S., Mittag, M., Schulz, M., & Allemann, S. S. (2024) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-024-03684-8 · Cited by 4 · see citing works
Knudsen, M. D., Carlsen, M. H., Hjartåker, A., Blomhoff, R., & Henriksen, H. B. (2024) · Food & Nutrition Research · doi:10.29219/fnr.v68.10366 · Cited by 3 · see citing works
“Bland-Altman plots were used to further explore the differences between methods and reproducibility of the same method (18, 19), such as under-or over -reporting (mean differences), limits of agreement, and presence of outliers in the data.”
MethodErfassung körperlicher Aktivität - Ein Überblick über Anspruch und Realität einer validen Messung
Eckert, K. G., Lange, M., & Wagner, P. (2013) · Springer eBooks · doi:10.1007/978-3-531-19063-1_5 · Cited by 2 · see citing works
Assessing the feasibility of introducing a Work Based Learning (WBL) module in SMEs
Hogeforster, M. (2025) · Economics & Sociology · doi:10.14254/2071-789x.2025/18-1/12 · Cited by 1 · see citing works
“Fuzzy logic as a methodology allows for this uncertainty to be taken into account (Schmidt & Steindorf, 2006).”
Heo, S., & Jin, I.-K. (2019) · Audiology and Speech Research · doi:10.21848/asr.2019.15.4.273 · Cited by 1 · see citing works
“일반적으로 설문지 검증 과정에서 사용되는 척도는 실행 가능성(feasibility), 신뢰도(reliability), 타당도(validity) 등이 있 다(de Yébenes et al., 2009; Schmidt & Steindorf, 2006).”
von Stein‐Lausnitz, M., Reissmann, D. R., Roggendorf, M. J., Sterzenbach, G., & Naumann, M. (2018) · Acta Odontologica Scandinavica · doi:10.1080/00016357.2018.1497804
Klinik çalışmalarda yapay zekâ uygulamalarının raporlanmasında CONSORT-AI protokolünün kullanımı
Tekindal, M. A., Tekindal, M. T., Demirsöz, M., et al. (2022) · Eurasian journal of veterinary sciences · doi:10.15312/eurasianjvetsci.2022.366
Huang, Q., and, M., & Wang, N. (2025) · Journal of Gerontological Nursing · doi:10.3928/00989134-20250812-02
Phillips, S. M., Summerbell, C., Hobbs, M., et al. (2021) · Figshare · doi:10.6084/m9.figshare.16928400
Phillips, S. M., Summerbell, C., Hobbs, M., et al. (2021) · Figshare · doi:10.6084/m9.figshare.16928412
Phillips, S. M., Summerbell, C., Hobbs, M., et al. (2021) · Figshare · doi:10.6084/m9.figshare.16928415
Phillips, S. M., Summerbell, C., Hobbs, M., et al. (2021) · Figshare · doi:10.6084/m9.figshare.16928397
Phillips, S. M., Summerbell, C., Hobbs, M., et al. (2021) · Figshare · doi:10.6084/m9.figshare.16928406
Phillips, S. M., Summerbell, C., Hobbs, M., et al. (2021) · Figshare · doi:10.6084/m9.figshare.16928391
Phillips, S. M., Summerbell, C., Hobbs, M., et al. (2021) · Figshare · doi:10.6084/m9.figshare.16928409
Phillips, S. M., Summerbell, C., Hobbs, M., et al. (2021) · Figshare · doi:10.6084/m9.figshare.16928394
Phillips, S. M., Summerbell, C., Hobbs, M., et al. (2021) · Figshare · doi:10.6084/m9.figshare.16928403
Phillips, S. M., Summerbell, C., Hobbs, M., et al. (2021) · Figshare · doi:10.6084/m9.figshare.16928382
- Authors
- Schmidt, M. E., & Steindorf, K.
- Affiliations
- German Cancer Research Center (DE)
- Corresponding author
- Steindorf, K.
- Venue
- Methods of Information in Medicine
- Publisher
- Georg Thieme Verlag KG
- Published
- 2006
- Volume
- 45
- Issue
- 04
- Pages
- 409-413
- ISSN
- 0026-1270 (ISSN-L)
- 2511-705X
- OpenAlex
- W4323284567
- Cited by
- 57
- References
- 0
- Citations by year
- 2025: 2 · 2024: 2 · 2023: 1 · 2022: 3 · 2021: 13 · 2020: 2 · 2019: 1 · 2018: 4 · 2017: 2 · 2016: 4 · 2015: 2 · 2014: 1 · 2013: 1 · 2012: 5
- FWCI
- 2.28
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Methods of Information in Medicine (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Social Sciences › Education › Health Education and Validation
- Keywords
- questionnaire validation, physical activity questionnaire, Bland-Altman analysis, correlation coefficient, measurement error, systematic bias, simulation study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
