Effect of an E-mental Health Approach to Workers' Health Surveillance versus Control Group on Work Functioning of Hospital Employees: A Cluster-RCT
Open accessThe DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: To evaluate an e-mental health (EMH) approach to workers' health surveillance (WHS) targeting work functioning (WF) and mental health (MH) of healthcare professionals in a randomised controlled trial. METHODS: Nurses and allied health professionals (N = 1140) were cluster-randomised at ward level to the intervention (IG) or control group (CG). The intervention consisted of two parts: (a) online screening and personalised feedback on impaired WF and MH, followed by (b) a tailored offer of self-help EMH interventions. CG received none of these parts. Primary outcome was impaired WF (Nurses Work Functioning Questionnaire), assessed at baseline and after three and six months. Analyses were performed in the positively screened subgroup (i) and in all participants (ii). RESULTS: Participation rate at baseline was 32% (NIG = 178; NCG = 188). Eighty-two percent screened positive for at least mild impairments in WF and/or MH (NIG = 139; NCG = 161). All IG-participants (N = 178) received part (a) of the intervention, nine participants (all positively screened, 6%) followed an EMH intervention to at least some extent. Regarding the subgroup of positively screened participants (i), both IG and CG improved over time regarding WF (non-significant between-group difference). After six months, 36% of positively screened IG-participants (18/50) had a relevant WF improvement compared to baseline, versus 28% (32/115) of positively screened CG-participants (non-significant difference). In the complete sample (ii), IG and CG improved over time but IG further improved between three and six months while CG did not (significant interaction effect). CONCLUSIONS: In our study with a full compliance rate of 6% and substantial drop-out leading to a small and underpowered sample, we could not demonstrate that an EMH-approach to WHS is more effective to improve WF and MH than a control group. The effect found in the complete sample of participants is not easily interpreted. Reported results may be useful for future meta-analytic work. TRIAL REGISTRATION: Dutch Trial Register NTR2786.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Most interventions (18/28, 64%) were focused on reducing stress [21-35] and depressive symptoms [23,31,36-39] in employees.”
From Acceptability of Web-Based Mental Health Interventions in the Workplace: Systematic Review (2022) · cites this paper as [27]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There were also no differences between these two groups at follow-up on scores of mental distress, impaired work functioning or impaired mental health”
From Web-based interventions for the management of stress in the workplace: Focus, form, and efficacy (2017) · cites this paper as [37]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Both the intervention and control groups improved in work functioning although there was no significant difference between the two”
From Web-based interventions for the management of stress in the workplace: Focus, form, and efficacy (2017) · cites this paper as [37]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371
Spitzer, R. L. (1999) · JAMA · doi:10.1001/jama.282.18.1737 · Cited by 9,802
Impact of Event Scale: A Measure of Subjective Stress
Horowitz, M., Wilner, N., & Alvarez, W. (1979) · Psychosomatic Medicine · doi:10.1097/00006842-197905000-00004 · Cited by 7,813 · cited 4×
“42 [ 52 ] Panic disorder Patient Health Questionnaire [ 53 ] , only assessed for participants identified as having anxiety complaints 2 answers affirmative on the first 4 items plus 4 symptoms affirmative on the following 11 items [ 54 ] Posttraumatic stress Dutch translation of the Impact of Event Scale [ 37 , 38 ] T…”
MethodInfluentialBush, K. (1998) · Archives of Internal Medicine · doi:10.1001/archinte.158.16.1789 · Cited by 6,196
Consort 2010 statement: extension to cluster randomised trials
Campbell, M. K., Piaggio, G., Elbourne, D. R., Altman, D. G., & for the CONSORT Group (2012) · BMJ · doi:10.1136/bmj.e5661 · Cited by 3,510
CONSORT statement: extension to cluster randomised trials
Campbell, M. K., Elbourne, D. R., & Altman, D. G. (2004) · BMJ · doi:10.1136/bmj.328.7441.702 · Cited by 2,557 · cited 1×
“The design, conduct and reporting of this study adhere to the Consolidated Standards of Reporting Trials guidelines [21,22].”
MethodA behavior model for persuasive design
Fogg, B. (2009) · Proceedings of the 4th International Conference on Persuasive Technology · doi:10.1145/1541948.1541999 · Cited by 2,328 · cited 1×
“Moreover, it could be useful to apply elements of persuasive design to encourage employees to follow an EMH intervention [47,48].”
BackgroundCONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials
Schulz, K. F., Altman, D. G., & Moher, D. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72352 · Cited by 2,165
Persuasive System Design Does Matter: a Systematic Review of Adherence to Web-based Interventions
Kelders, S. M., Kok, R. N., Ossebaard, H. C., & Van Gemert-Pijnen, J. E. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2104 · Cited by 1,308 · cited 1×
“Moreover, it could be useful to apply elements of persuasive design to encourage employees to follow an EMH intervention [47,48].”
Background
Research works printed but not linked: 3
- Terluin B (1998) De Vierdimensionele Klachtenlijst (4DKL) in de huisartspraktijk [The Four Dimensional Symptom Questionnaire (4DSQ)]. De Psycholoog 33: 18– 24.
- Brom D, Kleber RJ (1985) De Schok Verwerkings Lijst [The Impact of Event Scale]. Nederlands Tijdschrift voor de Psychologie 40: 164– 168.
- de Beurs E, Zitman FG (2006) De Brief Symptom Inventory (BSI): De betrouwbaarheid en validiteit van een handzaam alternatief voor de SCL-90. [The Brief Symptom Inventory (BSI): The reliability and validity of a brief alternative of the SCL-90]. Maandblad Geestelijke Volksgezondheid 61: 120– 141.
Cited by
Preventing occupational stress in healthcare workers
Ruotsalainen, J. H., Verbeek, J. H., Mariné, A., & Serra, C. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002892.pub5 · Cited by 563
Preventing occupational stress in healthcare workers
Marine, A., Ruotsalainen, J. H., Serra, C., & Verbeek, J. H. (2006) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002892.pub2 · Cited by 499
Preventing occupational stress in healthcare workers
Ruotsalainen, J. H., Verbeek, J. H., Mariné, A., & Serra, C. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002892.pub3 · Cited by 219
Kawakami, N., & Tsutsumi, A. (2016) · Journal of Occupational Health · doi:10.1539/joh.15-0001-er · Cited by 147
Phillips, E. A., Gordeev, V. S., & Schreyögg, J. (2019) · Scandinavian Journal of Work, Environment & Health · doi:10.5271/sjweh.3839 · Cited by 148
“Aikens et al, 2014 (82) + - + + Imamura et al, 2016 (65) + - + - + Allexandre et al, 2016 (84) + - - + Jonas et al, 2017 (83) + - + - + Billings et al, 2008 (61) ? - + Ketelaar et al, 2013 (85) + - - - ?”
Individual-level interventions for reducing occupational stress in healthcare workers
Tamminga, S. J., Emal, L. M., Boschman, J. S., Levasseur, A., Thota, A. B., Ruotsalainen, J. H., Schelvis, R. M. C., Nieuwenhuijsen, K., & van der Molen, H. F. (2023) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002892.pub6 · Cited by 193
Self-citationBolier, L., Ketelaar, S. M., Nieuwenhuijsen, K., Smeets, O., Gärtner, F. R., & Sluiter, J. K. (2014) · Internet Interventions · doi:10.1016/j.invent.2014.10.002 · Cited by 98
Self-citationInterventions on reducing burnout in physicians and nurses: A systematic review
Aryankhesal, A., Mohammadibakhsh, R., Hamidi, Y., Alidoost, S., Behzadifar, M., Sohrabi, R., & Farhadi, Z. (2019) · Medical Journal of The Islamic Republic of Iran · doi:10.47176/mjiri.33.77 · Cited by 93
Preventing occupational stress in healthcare workers
Ruotsalainen, J. H., Verbeek, J. H., Mariné, A., & Serra, C. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd002892.pub4 · Cited by 88
Interventions on reducing burnout in physicians and nurses: A systematic review.
(2019) · Europe PMC (PubMed Central) · doi:10.34171/mjiri.33.77 · Cited by 88
Sin, J., Galeazzi, G., McGregor, E., Collom, J., Taylor, A., Barrett, B., Lawrence, V., & Henderson, C. (2020) · Journal of Medical Internet Research · doi:10.2196/20581 · Cited by 79
“Outcomes with validated measures Comparisons (n), gender distributionc (%F/M/other), and age Intervention approach (n)b, gender distri- butionc (%F/M/other), and age Targeted CMDaReference; country AHSQe, PHQ-9f, GHSQg, AQoL-4Dh No tailored feedback, just generic advice (n=1431, US) Web-based assessment with tailored…”
BackgroundMethodInfluentialAnger, W. K., Dimoff, J. K., & Alley, L. (2024) · American Journal of Public Health · doi:10.2105/ajph.2023.307556 · Cited by 67
Stratton, E., Lampit, A., Choi, I., Gavelin, H. M., Aji, M., Taylor, J., Calvo, R. A., Harvey, S. B., & Glozier, N. (2022) · Journal of Medical Internet Research · doi:10.2196/37776 · Cited by 71
Moe-Byrne, T., Shepherd, J., Merecz-Kot, D., Sinokki, M., Naumanen, P., Hakkaart-van Roijen, L., & Van Der Feltz-Cornelis, C. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000123 · Cited by 55
“In contrast, a study [48,49] which included a subgroup analysis of participants with high stress levels, measured both posttraumatic stress and distress did not show significant improvement between the digital intervention group and the control group at six months follow up.”
BackgroundResultInfluentialPerry, L., Nicholls, R., Duffield, C. M., & Gallagher, R. D. (2017) · Journal of Advanced Nursing · doi:10.1111/jan.13345 · Cited by 70
Nikunlaakso, R., Selander, K., Oksanen, T., & Laitinen, J. H. (2022) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2022.06.004 · Cited by 59
Web-based interventions for the management of stress in the workplace: Focus, form, and efficacy
Ryan, C., Bergin, M., Chalder, T., & Wells, J. S. G. (2017) · Journal of Occupational Health · doi:10.1539/joh.16-0227-ra · Cited by 42
Connected Mental Health: Systematic Mapping Study
(2020) · doi:10.2196/19950 · Cited by 33
Acceptability of Web-Based Mental Health Interventions in the Workplace: Systematic Review
Scheutzow, J., Attoe, C., & Harwood, J. (2022) · JMIR Mental Health · doi:10.2196/34655 · Cited by 30
“7% [36], which is equivalent to a moderate individual-defined level of acceptability [23-25,27,36,39,44].”
BackgroundMethodInfluentialImamura, K., Asai, Y., Watanabe, K., Tsutsumi, A., Shimazu, A., Inoue, A., Hiro, H., Odagiri, Y., Yoshikawa, T., Yoshikawa, E., et al. (2018) · Journal of Occupational Health · doi:10.1539/joh.2017-0314-oa · Cited by 28
Ketelaar, S. M., Nieuwenhuijsen, K., Bolier, L., Smeets, O., & Sluiter, J. K. (2014) · Safety and Health at Work · doi:10.1016/j.shaw.2014.08.002 · Cited by 27
“From our previously performed RCT, we know that the control group (who had not yet received the intervention at that time) improved over time [9].”
BackgroundMethodResultInfluentialSelf-citationOptimizing Workplace Digital Mental Health Interventions: Systematic Review and Meta-Analysis
Stratton, E., Morris, R. W., Milton, A. C., Deady, M., Choi, I., & Glozier, N. (2025) · Journal of Medical Internet Research · doi:10.2196/71253 · Cited by 18
Lee, S., Lim, J., Lee, S., Heo, Y., & Jung, D. (2022) · BMC Family Practice · doi:10.1186/s12875-021-01622-6 · Cited by 18
“The most widespread method of online screening is person-tailored feedback, which provides the result individually by generating algorithms to set cutoff points and distinguish severity accordingly [17].”
MethodTsutsumi, A., Sasaki, N., Komase, Y., Watanabe, K., Inoue, A., Imamura, K., & Kawakami, N. (2020) · International Journal of Workplace Health Management · doi:10.1108/ijwhm-04-2020-0060 · Cited by 17
Aye, L. M., Tan, M., Schaefer, A., Thurairajasingam, S., Geldsetzer, P., Soon, L.-K., Reininghaus, U., Bärnighausen, T., & Su, T. T. (2024) · Digital Health · doi:10.1177/20552076241278313 · Cited by 26
“Ketelaar et al. 40 ; Low Utilization: Low program utilization diluting observed differences between groups.”
BackgroundInfluentialBoß, L., Ross, J., Reis, D., Pischel, S., Mallwitz, T., Brückner, H., Tanner, G., Nissen, H., Kalon, L., Schümann, M., et al. (2025) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2025.105076 · Cited by 13
Blekemolen, J., Hulshof, C. T. J., & Sluiter, J. K. (2016) · Occupational Medicine & Health Affairs · doi:10.4172/2329-6879.1000256 · Cited by 10
“[12] studied in a randomized controlled trial the effects of an Emental health intervention in a WHS program on mental health of hospital nurses and allied health professionals of a large hospital.”
BackgroundSelf-citationLópez-del-Hoyo, Y., Fernández-Martínez, S., Pérez-Aranda, A., Barceló-Soler, A., Guzman-Parra, J., Varela-Moreno, E., Campos, D., Monreal-Bartolomé, A., Beltrán-Ruiz, M., Moreno-Küstner, B., et al. (2022) · BMC Nursing · doi:10.1186/s12912-022-01089-5 · Cited by 9
“, depression, anxiety, stress) [25, 26, 40, 41] and, using the feedback provided by different healthcare workers (i.”
MethodRisk communication about work-related stress disorders in healthcare workers: a scoping review
Emal, L. M., Tamminga, S. J., Daams, J. G., Kežić, S., Timmermans, D. R. M., Schaafsma, F. G., & van der Molen, H. F. (2022) · International Archives of Occupational and Environmental Health · doi:10.1007/s00420-022-01851-x · Cited by 12
What is the purpose of the Stress Check Program?
Sugawara, N., Saito, M., & Nakamura, K. (2016) · Journal of Occupational Health · doi:10.1539/joh.16-0170-op · Cited by 7
Predictive value and construct validity of the work functioning screener‐healthcare (WFS‐H)
Boezeman, E. J., Nieuwenhuijsen, K., & Sluiter, J. K. (2016) · Journal of Occupational Health · doi:10.1539/joh.15-0056-oa · Cited by 7
Self-citationImpact of resilience and wellbeing program in nursing: an integrative review
Aljohani, N. H. Z., Alotibi, A. M., Aldhabi, A. A., & Alotibi, N. M. H. (2025) · BMC Nursing · doi:10.1186/s12912-025-04262-8 · Cited by 5
“The training programs targeting raising the nurses’ resilience showed promising results, but some study was not [20].”
Emal, L. M., Tamminga, S. J., Beumer, A., Kežić, S., Timmermans, D. R. M., Schaafsma, F. G., & van der Molen, H. F. (2025) · BMC Psychology · doi:10.1186/s40359-025-03542-x · Cited by 1
“Studies have shown that the DVM can improve mental well-being and increase the intention to change behavior, making it a valuable addition to preventive interventions [2, 24, 25, 27, 28].”
Caring for healthcare professionals: improving prevention in occupational healthcare
Ketelaar, S. M. (2014) · UvA-DARE (University of Amsterdam) · Cited by 3
Self-citationSiqueira, C. C. A. (2017) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.47.2017.tde-20032017-122017
Sakuraya, A., Tran, T. T. T., Sripo, N., Watanabe, K., Imamura, K., Boonyamalik, P., Sasaki, N., Tienthong, T., Asaoka, H., Iida, M., et al. (2025) · Journal of Occupational Health · doi:10.1093/joccuh/uiaf061
“The sample size in this study was calculated for the primary outcome (ie, depressive symptoms assessed by the Depression, Anxiety, and Stress Scale–21-item [DASS-21]), assuming that the effect size was set to 0.15.”
InfluentialKremers, S. H. M., Beulens, J. W. J., Middel, C. N. H., Elders, P. J. M., van der Beek, A. J., & Rutters, F. (2026) · JMIR Formative Research · doi:10.2196/85135
“…among health care workers using online mental health modules found that those who dropped out had reduced work-related vitality (work functioning and engagement), while a study investigating a 6-week app-based resilience training found no differences between completers and noncompleters [60,64,65].”
Metadata
- Authors
- Ketelaar, S. M., Nieuwenhuijsen, K., Gärtner, F. R., Bolier, L., Smeets, O., & Sluiter, J. K.
- Affiliations
- Amsterdam UMC Location University of Amsterdam (NL)
- Trimbos Institute (NL)
- Corresponding author
- Ketelaar, Sarah M.
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-09-12
- Volume
- 8
- Issue
- 9
- Pages
- e72546
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC3772000
- OpenAlex
- W2058569876
- Cited by
- 38
- References
- 48
- Citations by year
- 2026: 1 · 2025: 5 · 2024: 2 · 2023: 1 · 2022: 7 · 2020: 3 · 2019: 3 · 2018: 1 · 2017: 4 · 2016: 3 · 2015: 2 · 2014: 5
- FWCI
- 5.09
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Pure Amsterdam UMC (publishedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- ZonMw (208010001)
- Trimbos-instituut
- Instituut Gak
- Biotechnology and Biological Sciences Research Council
- Topics
- Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Workplace Health and Well-being
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Occupational Health and Burnout
- Keywords
- e-mental health, occupational health surveillance, work functioning, mental health, healthcare workers, nurses, allied health professionals, online screening, personalized feedback, self-help intervention, occupational health, hospital staff
- MeSH terms
- Adult, Female, Humans, Male, Mental Health, Middle Aged, Nurses, Patient Compliance, Population Surveillance, Surveys and Questionnaires, Treatment Outcome, Control Groups
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
