The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Statistical assessment of the learning curves of health technologies
Open accessAbstract
OBJECTIVES: (1) To describe systematically studies that directly assessed the learning curve effect of health technologies. (2) Systematically to identify 'novel' statistical techniques applied to learning curve data in other fields, such as psychology and manufacturing. (3) To test these statistical techniques in data sets from studies of varying designs to assess health technologies in which learning curve effects are known to exist. METHODS - STUDY SELECTION (HEALTH TECHNOLOGY ASSESSMENT LITERATURE REVIEW): For a study to be included, it had to include a formal analysis of the learning curve of a health technology using a graphical, tabular or statistical technique. METHODS - STUDY SELECTION (NON-HEALTH TECHNOLOGY ASSESSMENT LITERATURE SEARCH): For a study to be included, it had to include a formal assessment of a learning curve using a statistical technique that had not been identified in the previous search. METHODS - DATA SOURCES: Six clinical and 16 non-clinical biomedical databases were searched. A limited amount of handsearching and scanning of reference lists was also undertaken. METHODS - DATA EXTRACTION (HEALTH TECHNOLOGY ASSESSMENT LITERATURE REVIEW): A number of study characteristics were abstracted from the papers such as study design, study size, number of operators and the statistical method used. METHODS - DATA EXTRACTION (NON-HEALTH TECHNOLOGY ASSESSMENT LITERATURE SEARCH): The new statistical techniques identified were categorised into four subgroups of increasing complexity: exploratory data analysis; simple series data analysis; complex data structure analysis, generic techniques. METHODS - TESTING OF STATISTICAL METHODS: Some of the statistical methods identified in the systematic searches for single (simple) operator series data and for multiple (complex) operator series data were illustrated and explored using three data sets. The first was a case series of 190 consecutive laparoscopic fundoplication procedures performed by a single surgeon; the second was a case series of consecutive laparoscopic cholecystectomy procedures performed by ten surgeons; the third was randomised trial data derived from the laparoscopic procedure arm of a multicentre trial of groin hernia repair, supplemented by data from non-randomised operations performed during the trial. RESULTS - HEALTH TECHNOLOGY ASSESSMENT LITERATURE REVIEW: Of 4571 abstracts identified, 272 (6%) were later included in the study after review of the full paper. Some 51% of studies assessed a surgical minimal access technique and 95% were case series. The statistical method used most often (60%) was splitting the data into consecutive parts (such as halves or thirds), with only 14% attempting a more formal statistical analysis. The reporting of the studies was poor, with 31% giving no details of data collection methods. RESULTS - NON-HEALTH TECHNOLOGY ASSESSMENT LITERATURE SEARCH: Of 9431 abstracts assessed, 115 (1%) were deemed appropriate for further investigation and, of these, 18 were included in the study. All of the methods for complex data sets were identified in the non-clinical literature. These were discriminant analysis, two-stage estimation of learning rates, generalised estimating equations, multilevel models, latent curve models, time series models and stochastic parameter models. In addition, eight new shapes of learning curves were identified. RESULTS - TESTING OF STATISTICAL METHODS: No one particular shape of learning curve performed significantly better than another. The performance of 'operation time' as a proxy for learning differed between the three procedures. Multilevel modelling using the laparoscopic cholecystectomy data demonstrated and measured surgeon-specific and confounding effects. The inclusion of non-randomised cases, despite the possible limitations of the method, enhanced the interpretation of learning effects. CONCLUSIONS - HEALTH TECHNOLOGY ASSESSMENT LITERATURE REVIEW: The statistical methods used for assessing learning effects in health technology assessment have been crude and the reporting of studies poor. CONCLUSIONS - NON-HEALTH TECHNOLOGY ASSESSMENT LITERATURE SEARCH: A number of statistical methods for assessing learning effects were identified that had not hitherto been used in health technology assessment. There was a hierarchy of methods for the identification and measurement of learning, and the more sophisticated methods for both have had little if any use in health technology assessment. This demonstrated the value of considering fields outside clinical research when addressing methodological issues in health technology assessment. CONCLUSIONS - TESTING OF STATISTICAL METHODS: It has been demonstrated that the portfolio of techniques identified can enhance investigations of learning curve effects. (ABSTRACT TRUNCATED)
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
Information Theory and an Extension of the Maximum Likelihood Principle
Akaike, H. (1998) · Springer Series in Statistics · doi:10.1007/978-1-4612-1694-0_15 · Cited by 17,781
(1979) · Medical Entomology and Zoology · Cited by 11,252
Introduction to Statistical Quality Control
Sauter, R. M., & Montgomery, D. C. (1992) · Technometrics · doi:10.2307/1269251 · Cited by 7,842
SCHEFFE, H. (1960) · Soil Science · doi:10.1097/00010694-196006000-00016 · Cited by 5,516
(2006) · Technometrics · doi:10.1198/tech.2006.s417 · Cited by 4,950
Acquisition of cognitive skill
Anderson, J. R. (1982) · Psychological Review · doi:10.1037/0033-295x.89.4.369 · Cited by 3,663
Factors Affecting the Cost of Airplanes
WRIGHT, T. P. (1936) · Journal of the Aeronautical Sciences · doi:10.2514/8.155 · Cited by 2,958
Cognitive Skills and Their Acquisition
(2013) · Psychology Press eBooks · doi:10.4324/9780203728178 · Cited by 2,912
Mechanisms of skill acquisition and the law of practice
Newell, A., & Rosenbloom, P. S. (2018) · Figshare · doi:10.1184/r1/6607196 · Cited by 1,489
Accuracy of Pedicular Screw Placement In Vivo
GERTZBEIN, S. D., & ROBBINS, S. E. (1990) · Spine · doi:10.1097/00007632-199001000-00004 · Cited by 1,381
Ciric, I., Ragin, A., Baumgartner, C., & Pierce, D. (1997) · Neurosurgery · doi:10.1097/00006123-199702000-00001 · Cited by 1,029
Laparoscopic Nissen fundoplication: preliminary report.
Dallemagne, B., Weerts, J., Jehaes, C., Markiewicz, S., & Lombard, R. (1991) · PubMed · Cited by 677
Konrad, C., Schupfer, G., Wietlisbach, M., & Gerber, H. (1998) · Anesthesia & Analgesia · doi:10.1097/00000539-199803000-00037 · Cited by 655
The problem of inference from curves based on group data.
Estes, W. K. (1956) · Psychological Bulletin · doi:10.1037/h0045156 · Cited by 654
Time Series—A Biostatistical Introduction
Tsokos, C. P. (1992) · Technometrics · doi:10.1080/00401706.1992.10484918 · Cited by 648
Burton, P., Gurrin, L., & Sly, P. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980615)17:11<1261::aid-sim846>3.0.co;2-z · Cited by 635
Comparison of Conventional Anterior Surgery and Laparoscopic Surgery for Inguinal-Hernia Repair
Liem, M. S. L., van der Graaf, Y., van Steensel, C. J., Boelhouwer, R. U., Clevers, G.-J., Meijer, W. S., Stassen, L. P. S., Vente, J. P., Weidema, W. F., Schrijvers, A. J. P., et al. (1997) · New England Journal of Medicine · doi:10.1056/nejm199705293362201 · Cited by 541
The learning curve for laparoscopic cholecystectomy
Club, T. S. S., Moore, M. J., & Bennett, C. L. (1995) · The American Journal of Surgery · doi:10.1016/s0002-9610(99)80252-9 · Cited by 474
A User's Guide to MLwiN version 2.0
Rasbash, J., Steele, F., Browne, W. J., & Goldstein, H. (2004) · Bristol Research (University of Bristol) · Cited by 457
Darzi, A., Smith, S., & Taffinder, N. (1999) · BMJ · doi:10.1136/bmj.318.7188.887 · Cited by 347
Logan, G. D. (1992) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037//0278-7393.18.5.883 · Cited by 341
A Learning Curve for Laparoscopic Fundoplication: Definable, Avoidable, or a Waste of Time?
Watson, D. I., Baigrie, R. J., & Jamieson, G. G. (1996) · Annals of Surgery · doi:10.1097/00000658-199608000-00013 · Cited by 304
Bennett, C. L. (1997) · Archives of Surgery · doi:10.1001/archsurg.1997.01430250043009 · Cited by 303
Jowell, P. S., Baillie, J., Branch, M. S., Affronti, J., Browning, C. L., & Bute, B. P. (1996) · Annals of Internal Medicine · doi:10.7326/0003-4819-125-12-199612150-00009 · Cited by 300
Sample Size Calculations for Cluster Randomised Trials
Campbell, M., Grimshaw, J., Steen, N., & Changing Professional Practice in Europe Group (EU BIOMED II Concerted Action) (2000) · Journal of Health Services Research & Policy · doi:10.1177/135581960000500105 · Cited by 298
Surgical and Neurological Complications in a Series of 708 Epilepsy Surgery Procedures
Behrens, E., Schramm, J., Zentner, J., & König, R. (1997) · Neurosurgery · doi:10.1097/00006123-199707000-00004 · Cited by 294
Multilevel time series models with applications to repeated measures data
Goldstein, H., Healy, M. J. R., & Rasbash, J. (1994) · Statistics in Medicine · doi:10.1002/sim.4780131605 · Cited by 285
Monitoring the results of cardiac surgery by variable life-adjusted display
Lovegrove, J., Valencia, O., Treasure, T., Sherlaw-Johnson, C., & Gallivan, S. (1997) · The Lancet · doi:10.1016/s0140-6736(97)06507-0 · Cited by 285
Multilevel Models: Applications to Health Data
Rice, N., & Leyland, A. (1996) · Journal of Health Services Research & Policy · doi:10.1177/135581969600100307 · Cited by 272
THE EFFECTS OF INCREASING SKILL ON CYCLE TIME AND ITS CONSEQUENCES FOR TIME STANDARDS
DE JONG, J. R. (1957) · Ergonomics · doi:10.1080/00140135708964571 · Cited by 260
Regan, J. J., Yuan, H., & McAfee, P. C. (1999) · Spine · doi:10.1097/00007632-199902150-00023 · Cited by 245
Stone, G. W., Rutherford, B. D., McConahay, D. R., Johnson, W. L., Giorgi, L. V., Ligon, R. W., & Hartzler, G. O. (1990) · Journal of the American College of Cardiology · doi:10.1016/0735-1097(90)90285-w · Cited by 232
Laparoscopic colorectal surgery: Do we get faster?
Agachan, F., Joo, J. S., Sher, M., Weiss, E. G., Nogueras, J. J., & Wexner, S. D. (1997) · Surgical Endoscop · doi:10.1007/s004649900357 · Cited by 230
The Strategy-Specific Nature of Improvement: The Power Law Applies by Strategy Within Task
Delaney, P. F., Reder, L. M., Staszewski, J. J., & Ritter, F. E. (1998) · Psychological Science · doi:10.1111/1467-9280.00001 · Cited by 214
The Learning curve for totally extraperitoneal laparoscopic inguinal hernia repair
Liem, M. S. L., van Steensel, C. J., Boelhouwer, R. U., Weidema, W. F., Clevers, G.-J., Meijer, W. S., Vente, J. P., de Vries, L. S., & van Vroonhoven, T. J. M. V. (1996) · The American Journal of Surgery · doi:10.1016/s0002-9610(97)89569-4 · Cited by 204
Equipoise and the ethics of randomization.
Lilford, R. J., & Jackson, J. (1995) · Europe PMC (PubMed Central) · Cited by 201
Randomised controlled trial of laparoscopic versus open repair of inguinal hernia: early results
Lawrence, K., McWhinnie, D., Goodwin, A., Doll, H., Gordon, A., Gray, A., Britton, J., & Collin, J. (1995) · BMJ · doi:10.1136/bmj.311.7011.981 · Cited by 190
A note on functional relations obtained from group data.
Sidman, M. (1952) · Psychological Bulletin · doi:10.1037/h0063643 · Cited by 184
LAPAROSCOPIC ADRENALECTOMY: THE PREFERRED CHOICE? A COMPARISON TO OPEN ADRENALECTOMY
WINFIELD, H. N., HAMILTON, B. D., BRAVO, E. L., & NOVICK, A. C. (1998) · Journal of Urology · doi:10.1016/s0022-5347(01)62884-2 · Cited by 176
The Generalised Estimating Equations: An Annotated Bibliography
Ziegler, A., Kästner, C., & Blettner, M. (1998) · Biometrical Journal · doi:10.1002/(sici)1521-4036(199806)40:2<115::aid-bimj115>3.0.co;2-6 · Cited by 170
Analysis of 1500 laryngeal mask uses by one anaesthetist in adults undergoing routine anaesthesia
Brimacombe, J. (1996) · Anaesthesia · doi:10.1111/j.1365-2044.1996.tb07660.x · Cited by 166
Browne, M. W., & Du Toit, S. H. C. (1991) · Best methods for the analysis of change: Recent advances, unanswered questions, future directions. · doi:10.1037/10099-004 · Cited by 159
Complications and failures of video-assisted thoracic surgery: Experience from two centers in Asia
Yim, A. P. C., & Liu, H.-P. (1996) · The Annals of Thoracic Surgery · doi:10.1016/0003-4975(95)01097-1 · Cited by 153
Ployhart, R. E., & Hakel, M. D. (1998) · Personnel Psychology · doi:10.1111/j.1744-6570.1998.tb00744.x · Cited by 132
Complications of phonosurgery: Results of a national survey
Rosen, C. A. (1998) · The Laryngoscope · doi:10.1097/00005537-199811000-00020 · Cited by 130
Error in Medicine: What Have We Learned?
Bates, D. W., & Gawande, A. A. (2000) · Annals of Internal Medicine · doi:10.7326/0003-4819-132-9-200005020-00025 · Cited by 124
The backward curve: a method for the study of learning.
Hayes, K. J. (1953) · Psychological Review · doi:10.1037/h0056308 · Cited by 118
Transperitoneal Laparoscopic Adrenalectomy: Experience in 100 Patients
Terachi, T., Matsuda, T., Terai, A., Ogawa, O., Kakehi, Y., Kawakita, M., Shichiri, Y., MIKAMI, O. K., Takeuchi, H., Okada, Y., et al. (1997) · Journal of Endourology · doi:10.1089/end.1997.11.361 · Cited by 114
Learning curve for oesophageal cancer surgery
Sutton, D. N., Wayman, J., & Griffin, S. M. (1998) · British journal of surgery · doi:10.1046/j.1365-2168.1998.00962.x · Cited by 113
Trials and fast changing technologies: the case for tracker studies
Lilford, R. J., Braunholtz, D. A., Greenhalgh, R., & Edwards, S. J. (2000) · BMJ · doi:10.1136/bmj.320.7226.43 · Cited by 113
A method of objectively evaluating improvements in laparoscopic skills
Chung, J., & Sackier, J. (1998) · Surgical Endoscopy · doi:10.1007/s004649900795 · Cited by 113
Observer variability in the interpretation of contrast enhanced MRI of the breast
MUSSURAKIS, S., Buckley, D. L., Coady, A., Turnbull, L. W., & Horsman, A. (1996) · British Journal of Radiology · doi:10.1259/0007-1285-69-827-1009 · Cited by 85
LAPAROSCOPIC BILIARY INJURY: MORE THAN A LEARNING CURVE PROBLEM
Windsor, J. A., & Pong, J. (1998) · Australian and New Zealand Journal of Surgery · doi:10.1111/j.1445-2197.1998.tb04742.x · Cited by 84
Training in laparoscopic cholecystectomy
Hunter, J. G., Sackier, J. M., & Berci, G. (1994) · Surgical Endoscopy · doi:10.1007/bf02909489 · Cited by 83
Long, J. P., Fallick, M. L., LaRock, D., & Rand, W. (1998) · The Journal of Urology · doi:10.1016/s0022-5347(01)63953-3 · Cited by 83
Multilevel Analysis of Repeated Measures Data
van der Leeden, R. (1998) · Quality & Quantity · doi:10.1023/a:1004233225855 · Cited by 80
Considerations in the use of echocardiography in epidemiology. The Framingham Study.
Savage, D. D., Garrison, R. J., Kannel, W. B., Anderson, S. J., Feinleib, M., & Castelli, W. P. (1987) · Hypertension · doi:10.1161/01.hyp.9.2_pt_2.ii40 · Cited by 78
A ventilation‐exchange bougie for fibreoptic intubations with the laryngeal mask airway
Atherton, D. P. L., O’Sullivan, E., LOWE, D., & Charters, P. (1996) · Anaesthesia · doi:10.1111/j.1365-2044.1996.tb15047.x · Cited by 69
A cross-level units-of-analysis approach to individual differences in skill acquisition.
Eyring, J. D., Johnson, D. S., & Francis, D. J. (1993) · Journal of Applied Psychology · doi:10.1037/0021-9010.78.5.805 · Cited by 63
Learning Curve Characteristics of Pacing Lead Extraction with a Laser Sheath
Bracke, F. A. L. E., Meijer, A., & van Gelder, B. (1998) · Pacing and Clinical Electrophysiology · doi:10.1111/j.1540-8159.1998.tb01172.x · Cited by 59
Developing a standardized test for the assessment of suturing skill in novice microsurgeons
Starkes, J. L., Payk, I., & Hodges, N. J. (1998) · Microsurgery · doi:10.1002/(sici)1098-2752(1998)18:1<19::aid-micr5>3.0.co;2-p · Cited by 52
Laparoscopic splenectomy: lessons from the learning curve.
Poulin, E. C., & Mamazza, J. (1998) · Europe PMC (PubMed Central) · Cited by 50
The learing curve for intraoperative echocardiography during congenital heart surgery
Ungerleider, R. M., Greeley, W. J., Kanter, R. J., & KISSLO, J. A. (1992) · The Annals of Thoracic Surgery · doi:10.1016/0003-4975(92)91013-y · Cited by 48
Making cost assessments based on RCTs more useful to decision-makers
Baltussen, R. M. P. M., Ament, A. J. H. A., & Leidl, R. (1996) · Health Policy · doi:10.1016/s0168-8510(96)90023-8 · Cited by 42
Lang, G., Buckle, H. C., Hadley, H. R., Lui, P. D., & Stewart, S. (1994) · Urology · doi:10.1016/s0090-4295(94)80135-5 · Cited by 33
Current assessment of fetal losses as a direct consequence of chorionic villus sampling
Goldberg, J. D., Porter, A. E., & Golbus, M. S. (1990) · American Journal of Medical Genetics · doi:10.1002/ajmg.1320350206 · Cited by 31
PASTIS: A program for curve and distribution analyses
Cousineau, D., & Larochelle, S. (1997) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03210606 · Cited by 30
Transabdominal preperitoneal laparoscopic inguinal herniorrhaphy: results of 509 repairs.
Quilici, P. J., Greaney, E. M., Quilici, J., & Anderson, S. R. (1996) · PubMed · Cited by 21
INSTRUCTIONS AND FEEDBACK EFFECTS ON SPEED AND ACCURACY WITH DIFFERENT LEARNING CURVE MODELS
Buck, J. R., Wendy, S., & CHENG, J. (1993) · IIE Transactions · doi:10.1080/07408179308964326 · Cited by 20
Sinclair, B. G., Sandor, G. G. S., & Farquharson, D. F. (1997) · PubMed · Cited by 15
Sathe, S., Vohra, J. K., Chan, W., Wong, J. H., Gerloff, J., Riters, A., Hall, R. M., & Hunt, D. (1993) · Australian and New Zealand Journal of Medicine · doi:10.1111/j.1445-5994.1993.tb01748.x · Cited by 11
Analyzing Errors in Practice: A Vehicle for Assessing and Enhancing the Quality of Care
Deusinger, S. S. (1992) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300007923 · Cited by 11
Champault, G., Rizk, N. P., Catheline, J.-M., Riskalla, H., & Boutelier, P. (1996) · PubMed · Cited by 9
Successful coronary stenting in a 4-year-old child
Moore, J. W., & Buchbinder, M. (1998) · Catheterization and Cardiovascular Diagnosis · doi:10.1002/(sici)1097-0304(199806)44:2<147::aid-ccd5>3.0.co;2-6 · Cited by 8
Decrease in Costs for Management of Lower Airway Disease in the Pediatric Intensive Care Unit
Irazuzta, J. E., ZHANG, J., & Pandit, S. (1998) · Southern Medical Journal · doi:10.1097/00007611-199807000-00009 · Cited by 4
Cited by
International guidelines for groin hernia management
The HerniaSurge Group (2018) · Hernia · doi:10.1007/s10029-017-1668-x · Cited by 2,296
European Hernia Society guidelines on the treatment of inguinal hernia in adult patients
Simons, M. P., Aufenacker, T., Bay-Nielsen, M., Bouillot, J. L., Campanelli, G., Conze, J., de Lange, D., Fortelny, R., Heikkinen, T., Kingsnorth, A., et al. (2009) · Hernia · doi:10.1007/s10029-009-0529-7 · Cited by 1,617
Randomised trials in surgery: problems and possible solutions
McCulloch, P., Taylor, I., Sasako, M., Lovett, B., & Griffin, D. (2002) · BMJ · doi:10.1136/bmj.324.7351.1448 · Cited by 762
Tekkis, P. P., Senagore, A. J., Delaney, C. P., & Fazio, V. W. (2005) · Annals of Surgery · doi:10.1097/01.sla.0000167857.14690.68 · Cited by 745
Challenges in evaluating surgical innovation
Ergina, P. L., Cook, J. A., Blazeby, J. M., Boutron, I., Clavien, P.-A., Reeves, B. C., & Seiler, C. M. (2009) · The Lancet · doi:10.1016/s0140-6736(09)61086-2 · Cited by 653
Metadata
- Authors
- Ramsay, C. R., Grant, A. M., Wallace, S. A., Garthwaite, P. H., Monk, A. F., & Russell, I. T.
- Venue
- Health Technology Assessment
- Published
- 2001
- Volume
- 5
- Issue
- 12
- Pages
- 1-79
- ISSN
- 1366-5278 (ISSN-L)
- 2046-4924
- OpenAlex
- W2125647201
- Cited by
- 353
- References
- 76
- FWCI
- 16.11
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.