Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 47

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Understanding Expert Disagreement in Medical Data Analysis through Structured Adjudication

Schaekermann, Mike, Beaton, Graeme, Habib, Minahz, Lim, Andrew, Larson, Kate, Law, Edith·2019·Proceedings of the ACM on Human-Computer Interaction
DOI 10.1145/3359178
Open this paper in CoChat — free account

Abstract

Expert disagreement is pervasive in clinical decision making and collective adjudication is a useful approach for resolving divergent assessments. Prior work shows that expert disagreement can arise due to diverse factors including expert background, the quality and presentation of data, and guideline clarity. In this work, we study how these factors predict initial discrepancies in the context of medical time series analysis, examining why certain disagreements persist after adjudication, and how adjudication impacts clinical decisions. Results from a case study with 36 experts and 4,543 adjudicated cases in a sleep stage classification task show that these factors contribute to both initial disagreement and resolvability, each in their own unique way. We provide evidence suggesting that structured adjudication can lead to significant revisions in treatment-relevant clinical parameters. Our work demonstrates how structured adjudication can support consensus and facilitate a deep understanding of expert disagreement in medical data analysis.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3359178

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 47

  1. 10.1145/2531602.2531653 · DOI
  2. 10.15346/hc.v1i1.3 · DOI · Paper page
  3. Elham Bagheri , Justin Dauwels , Brian C. Dean , Chad G. Waters , M. Brandon Westover , and Jonathan J . Halford . 2017 . Interictal epileptiform discharge characteristics underlying expert interrater agreement. Clinical Neurophysiology , Vol. 128 , 10 (10 2017), 1994--2005. https://doi.org/10.1016/j.clinph.2017.06.252 Elham Bagheri, Justin Dauwels, Brian C. Dean, Chad G. Waters, M. Brandon Westover, and Jonathan J. Halford. 2017. Interictal epileptiform discharge characteristics underlying expert interrater agreement. Clinical Neurophysiology, Vol. 128, 10 (10 2017), 1994--2005. https://doi.org/10.1016/j.clinph.2017.06.252 · DOI
  4. A. Baker , K. Young , J. Potter , and I. Madan . 2010. A review of grading systems for evidence-based guidelines produced by medical specialties . Clinical Medicine , Vol. 10 , 4 (8 2010 ), 358--363. https://doi.org/10.7861/clinmedicine.10--4--358 A. Baker, K. Young, J. Potter, and I. Madan. 2010. A review of grading systems for evidence-based guidelines produced by medical specialties. Clinical Medicine, Vol. 10, 4 (8 2010), 358--363. https://doi.org/10.7861/clinmedicine.10--4--358 · doi:10.7861/clinmedicine.10--4--358
  5. Erin P. Balogh Bryan T. Miller and John R. Ball (Eds.). 2015. Improving Diagnosis in Health Care. National Academies Press Washington D.C. https://doi.org/10.17226/21794 Erin P. Balogh Bryan T. Miller and John R. Ball (Eds.). 2015. Improving Diagnosis in Health Care. National Academies Press Washington D.C. https://doi.org/10.17226/21794 · DOI · Paper page
  6. 10.1155/2011/406391 · DOI
  7. 10.1023/B:ARTI.0000046007.11806.9a · doi:10.1023/b:arti.0000046007.11806.9a
  8. Katarzyna Budzynska , Mathilde Janier , Juyeon Kang , Chris Reed , Patrick Saint-Dizier , Manfred Stede , and Olena Yaskorska . 2014 . Towards Argument Mining from Dialogue . In Computational Models of Argument - Proceedings of COMMA 2014 , Atholl Palace Hotel, Scottish Highlands, UK, September 9--12 , 2014. 185--196. https://doi.org/10.3233/978--1--61499--436--7--185 Katarzyna Budzynska, Mathilde Janier, Juyeon Kang, Chris Reed, Patrick Saint-Dizier, Manfred Stede, and Olena Yaskorska. 2014. Towards Argument Mining from Dialogue. In Computational Models of Argument - Proceedings of COMMA 2014, Atholl Palace Hotel, Scottish Highlands, UK, September 9--12, 2014. 185--196. https://doi.org/10.3233/978--1--61499--436--7--185 · doi:10.3233/978--1--61499--436--7--185
  9. 10.1145/3025453.3026044 · DOI
  10. 10.3115/v1/W15-1601 · DOI
  11. Carlos Ches n evar, Jarred McGinnis , Sanjay Modgil , Iyad Rahwan , Chris Reed , Guillermo Simari , Matthew South , Gerard Vreeswijk , and Steven Willmott . 2006 . Towards an argument interchange format . The Knowledge Engineering Review , Vol. 21 , 04 (12 2006), 293. https://doi.org/10.1017/S0269888906001044 Carlos Ches n evar, Jarred McGinnis, Sanjay Modgil, Iyad Rahwan, Chris Reed, Guillermo Simari, Matthew South, Gerard Vreeswijk, and Steven Willmott. 2006. Towards an argument interchange format. The Knowledge Engineering Review, Vol. 21, 04 (12 2006), 293. https://doi.org/10.1017/S0269888906001044 · DOI · Paper page
  12. Robin Cohen . 1987 . Analyzing the Structure of Argumentative Discourse. Comput. Linguist. , Vol. 13 , 1 -- 2 (1 1987), 11--24. http://dl.acm.org/citation.cfm?id=26386.26388 Robin Cohen. 1987. Analyzing the Structure of Argumentative Discourse. Comput. Linguist., Vol. 13, 1--2 (1 1987), 11--24. http://dl.acm.org/citation.cfm?id=26386.26388
  13. 10.5555/3306127.3331890 · doi:10.5555/3306127.3331890
  14. Norman Dalkey and Olaf Helmer . 1963 . An Experimental Application of the DELPHI Method to the Use of Experts . Management Science , Vol. 9 , 3 (4 1963), 458--467. https://doi.org/10.1287/mnsc.9.3.458 Norman Dalkey and Olaf Helmer. 1963. An Experimental Application of the DELPHI Method to the Use of Experts. Management Science, Vol. 9, 3 (4 1963), 458--467. https://doi.org/10.1287/mnsc.9.3.458 · DOI · Paper page
  15. Todd Davies and Reid Chandler . 2012. Online deliberation design. Democracy in motion: Evaluation the practice and impact of deliberative civic engagement ( 2012 ), 103--131. Todd Davies and Reid Chandler. 2012. Online deliberation design. Democracy in motion: Evaluation the practice and impact of deliberative civic engagement (2012), 103--131.
  16. Ryan Drapeau , Lydia B. Chilton , Jonathan Bragg , and Daniel S. Weld . 2016. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy . In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP) . Ryan Drapeau, Lydia B. Chilton, Jonathan Bragg, and Daniel S. Weld. 2016. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP) .
  17. 10.5555/2584544 · doi:10.5555/2584544
  18. Gowri Gopalakrishna , Miranda W Langendam , Rob JPM Scholten , Patrick MM Bossuyt, and Mariska MG Leeflang. 2013 . Guidelines for guideline developers: a systematic review of grading systems for medical tests. Implementation Science , Vol. 8 , 1 (12 2013), 78. https://doi.org/10.1186/1748--5908--8--78 Gowri Gopalakrishna, Miranda W Langendam, Rob JPM Scholten, Patrick MM Bossuyt, and Mariska MG Leeflang. 2013. Guidelines for guideline developers: a systematic review of grading systems for medical tests. Implementation Science, Vol. 8, 1 (12 2013), 78. https://doi.org/10.1186/1748--5908--8--78 · doi:10.1186/1748--5908--8--78
  19. 10.1145/2818048.2820071 · DOI
  20. Melody Guan , Varun Gulshan , Andrew Dai , and Geoffrey Hinton . 2018 . Who said what: Modeling individual labelers improves classification . In AAAI Conference on Artificial Intelligence. https://arxiv.org/pdf/1703 .08774.pdf Melody Guan, Varun Gulshan, Andrew Dai, and Geoffrey Hinton. 2018. Who said what: Modeling individual labelers improves classification. In AAAI Conference on Artificial Intelligence. https://arxiv.org/pdf/1703.08774.pdf · arXiv
  21. Danna Gurari , Kun He , Bo Xiong , Jianming Zhang , Mehrnoosh Sameki , Suyog Dutt Jain , Stan Sclaroff, Margrit Betke, and Kristen Grauman. 2017 . Predicting Foreground Object Ambiguity and Efficiently Crowdsourcing the Segmentation( s). (4 2017). http://arxiv.org/abs/1705.00366 Danna Gurari, Kun He, Bo Xiong, Jianming Zhang, Mehrnoosh Sameki, Suyog Dutt Jain, Stan Sclaroff, Margrit Betke, and Kristen Grauman. 2017. Predicting Foreground Object Ambiguity and Efficiently Crowdsourcing the Segmentation(s). (4 2017). http://arxiv.org/abs/1705.00366 · arXiv
  22. 10.1061/(ASCE)0887-3801(1995)9:4(244) · doi:10.1061/(asce)0887-3801(1995)9:4(244
  23. Conrad Iber , Sonia Ancoli-Israel , Andrew L Cheeson Jr., and Stuart F Quan . 2007 . The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications. American Academy of Sleep Medicine . Conrad Iber, Sonia Ancoli-Israel, Andrew L Cheeson Jr., and Stuart F Quan. 2007. The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications. American Academy of Sleep Medicine.
  24. 10.1145/2818048.2820016 · DOI
  25. Jonathan Krause , Varun Gulshan , Ehsan Rahimy , Peter Karth , Kasumi Widner , Greg S. Corrado , Lily Peng , and Dale R . Webster . 2018 . Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy. Ophthalmology ( 3 2018). https://doi.org/10.1016/j.ophtha.2018.01.034 Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S. Corrado, Lily Peng, and Dale R. Webster. 2018. Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy. Ophthalmology (3 2018). https://doi.org/10.1016/j.ophtha.2018.01.034 · DOI · Paper page
  26. 10.3115/v1/W15-0516 · DOI
  27. John Lawrence and Chris Reed . 2016 . Argument Mining using Argumentation Scheme Structures . Proceedings of the 6th International Conference on Computational Models of Argument (COMMA 2016 ), Vol. 0 (2016), 379 -- 390. https://doi.org/10.3233/978--1--61499--686--6--379 John Lawrence and Chris Reed. 2016. Argument Mining using Argumentation Scheme Structures. Proceedings of the 6th International Conference on Computational Models of Argument (COMMA 2016), Vol. 0 (2016), 379 -- 390. https://doi.org/10.3233/978--1--61499--686--6--379 · doi:10.3233/978--1--61499--686--6--379
  28. V K Chaithanya Manam and Alexander J Quinn . 2018 . WingIt: Efficient Refinement of Unclear Task Instructions. In The Sixth AAAI Conference on Human Computation and Crowdsourcing. 108--116 . https://www.aaai.org/ocs/index.php/HCOMP/HCOMP18/paper/view/17931 V K Chaithanya Manam and Alexander J Quinn. 2018. WingIt: Efficient Refinement of Unclear Task Instructions. In The Sixth AAAI Conference on Human Computation and Crowdsourcing. 108--116. https://www.aaai.org/ocs/index.php/HCOMP/HCOMP18/paper/view/17931
  29. 10.1080/135467896394500 · DOI
  30. 10.1002/acp.3213 · DOI
  31. Simon Parsons , Elizabeth Sklar , Jordan Salvit , Holly Wall , and Zimi Li . 2013 . ArgTrust: Decision Making with Information from Sources of Varying Trustworthiness . In Proceedings of the 2013 International Conference on Autonomous Agents and Multi-agent Systems (AAMAS '13) . International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1395--1396. http://dl.acm.org/citation.cfm?id=2484920.2485242 Simon Parsons, Elizabeth Sklar, Jordan Salvit, Holly Wall, and Zimi Li. 2013. ArgTrust: Decision Making with Information from Sources of Varying Trustworthiness. In Proceedings of the 2013 International Conference on Autonomous Agents and Multi-agent Systems (AAMAS '13). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 1395--1396. http://dl.acm.org/citation.cfm?id=2484920.2485242
  32. 10.5664/jcsm.2352 · DOI
  33. 10.1093/brain/awz030 · DOI · Paper page
  34. Stefan R"abiger, Gizem Gezici , Yücel Saygin , and Myra Spiliopoulou . 2018 . Predicting worker disagreement for more effective crowd labeling . In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA). IEEE, 179--188 . Stefan R"abiger, Gizem Gezici, Yücel Saygin, and Myra Spiliopoulou. 2018. Predicting worker disagreement for more effective crowd labeling. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA). IEEE, 179--188.
  35. Maithra Raghu , Katy Blumer , Rory Sayres , Ziad Obermeyer , Robert Kleinberg , Sendhil Mullainathan , and Jon Kleinberg . 2018. Direct Uncertainty Prediction for Medical Second Opinions. (7 2018 ). http://arxiv.org/abs/1807.01771 Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Robert Kleinberg, Sendhil Mullainathan, and Jon Kleinberg. 2018. Direct Uncertainty Prediction for Medical Second Opinions. (7 2018). http://arxiv.org/abs/1807.01771 · arXiv
  36. Pranav Rajpurkar , Awni Y. Hannun , Masoumeh Haghpanahi , Codie Bourn , and Andrew Y . Ng . 2017 . Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks . (7 2017). http://arxiv.org/abs/1707.01836 Pranav Rajpurkar, Awni Y. Hannun, Masoumeh Haghpanahi, Codie Bourn, and Andrew Y. Ng. 2017. Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks. (7 2017). http://arxiv.org/abs/1707.01836 · arXiv
  37. Chris Reed and Timothy Norman . 2004 . Argumentation Machines . Argumentation Library , Vol. 9 . Springer Netherlands, Dordrecht. https://doi.org/10.1007/978--94-017-0431--1 Chris Reed and Timothy Norman. 2004. Argumentation Machines. Argumentation Library, Vol. 9. Springer Netherlands, Dordrecht. https://doi.org/10.1007/978--94-017-0431--1 · doi:10.1007/978--94-017-0431--1
  38. Chris Reed and Doug Walton. 2005. Towards a Formal and Implemented Model of Argumentation Schemes in Agent Communication. 19--30. https://doi.org/10.1007/978--3--540--32261-0_2 Chris Reed and Doug Walton. 2005. Towards a Formal and Implemented Model of Argumentation Schemes in Agent Communication. 19--30. https://doi.org/10.1007/978--3--540--32261-0_2 · doi:10.1007/978-3-540-32261-0_2
  39. Richard S. Rosenberg and Steven van Hout . 2013 . The American Academy of Sleep Medicine Inter-scorer Reliability Program : Sleep Stage Scoring. Journal of Clinical Sleep Medicine (1 2013). https://doi.org/10.5664/jcsm.2350 Richard S. Rosenberg and Steven van Hout. 2013. The American Academy of Sleep Medicine Inter-scorer Reliability Program: Sleep Stage Scoring. Journal of Clinical Sleep Medicine (1 2013). https://doi.org/10.5664/jcsm.2350 · DOI · Paper page
  40. 10.1145/3308560.3317085 · DOI
  41. Mike Schaekermann , Graeme Beaton , Minahz Habib , Andrew Lim , Kate Larson , and Edith Law . 2019 . crowdEEG: A Platform for Structured Consensus Formation in Medical Time Series Analysis . In 8th Workshop on Interactive Systems in Healthcare (WISH) at CHI 2019. Glasgow, UK. Mike Schaekermann, Graeme Beaton, Minahz Habib, Andrew Lim, Kate Larson, and Edith Law. 2019. crowdEEG: A Platform for Structured Consensus Formation in Medical Time Series Analysis. In 8th Workshop on Interactive Systems in Healthcare (WISH) at CHI 2019. Glasgow, UK.
  42. 10.1145/3274423 · DOI
  43. Mike Schaekermann , Edith Law , Kate Larson , and Andrew Lim . 2018 b. Expert Disagreement in Sequential Labeling: A Case Study on Adjudication in Medical Time Series Analysis. In 1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing at HCOMP 2018. Zurich, Switzerland. Mike Schaekermann, Edith Law, Kate Larson, and Andrew Lim. 2018b. Expert Disagreement in Sequential Labeling: A Case Study on Adjudication in Medical Time Series Analysis. In 1st Workshop on Subjectivity, Ambiguity and Disagreement in Crowdsourcing at HCOMP 2018. Zurich, Switzerland.
  44. Mike Schaekermann , Edith Law , Alex C Williams , and William Callaghan . 2016 . Resolvable vs . Irresolvable Ambiguity: A New Hybrid Framework for Dealing with Uncertain Ground Truth. In 1st Workshop on Human-Centered Machine Learning at SIGCHI 2016. San Jose, CA. Mike Schaekermann, Edith Law, Alex C Williams, and William Callaghan. 2016. Resolvable vs. Irresolvable Ambiguity: A New Hybrid Framework for Dealing with Uncertain Ground Truth. In 1st Workshop on Human-Centered Machine Learning at SIGCHI 2016. San Jose, CA.
  45. 10.1111/j.2041-6962.2006.tb00028.x · DOI
  46. Miriam Solomon . 2007. The social epistemology of NIH consensus conferences . In Establishing medical reality . Springer , 167--177. Miriam Solomon. 2007. The social epistemology of NIH consensus conferences. In Establishing medical reality. Springer, 167--177.
  47. D Walton C Reed and F Macagno. 2008. Argumentation Schemes. Cambridge University Press. https://books.google.ca/books?id=qc3LCgAAQBAJ D Walton C Reed and F Macagno. 2008. Argumentation Schemes. Cambridge University Press. https://books.google.ca/books?id=qc3LCgAAQBAJ · DOI · Paper page

Cited by

See all 47

Metadata

Authors
Schaekermann, M., Beaton, G., Habib, M., Lim, A., Larson, K., & Law, E.
Affiliations
  • University of Waterloo, Waterloo, ON, Canada
  • University of Toronto, Toronto, ON, Canada
  • Sunnybrook Health Sciences Centre, Toronto, ON, Canada
Venue
Proceedings of the ACM on Human-Computer Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2019-11-07
Volume
3
Issue
CSCW
Pages
1-23
ISSN
2573-0142
DOI
10.1145/3359178
Cited by
47
References
5
Language
en
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Funders
  • Canadian Institutes of Health Research (CPG-140200)
  • Natural Sciences and Engineering Research Council of Canada (CHRP 478468-15)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Understanding Expert Disagreement in Medical Data Analysis through Structured Adjudication. https://app.cochat.ai/papers/10.1145/3359178
The paper
Schaekermann, M., Beaton, G., Habib, M., Lim, A., Larson, K., & Law, E. (2019). Understanding Expert Disagreement in Medical Data Analysis through Structured Adjudication. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3359178

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.