CoChat
Sign Up
Cited by 47Mobile Crowdsensing and CrowdsourcingMachine Learning and AlgorithmsData Stream Mining Techniques

The DOI matches its Crossref record; the references it cites are not checked yet.

Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learning

Open access
Pandey, Rahul, Purohit, Hemant, Castillo, Carlos, Shalin, Valerie L.·2022·International Journal of Human-Computer Studies
DOI 10.1016/j.ijhcs.2022.102772
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.ijhcs.2022.102772

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

References 78 · Cited by 47

Research works printed but not linked: 29

  1. Almeida, 2018, Adapting dynamic classifier selection for concept drift, Expert Syst. Appl., 104, 67, 10.1016/j.eswa.2018.03.021 · DOI
  2. Anderson, 1991, Reflections of the environment in memory, Psychol. Sci., 2, 396, 10.1111/j.1467-9280.1991.tb00174.x · DOI
  3. Brown, 1958, Some tests of the decay theory of immediate memory, Q. J. Exp. Psychol., 10, 12, 10.1080/17470215808416249 · DOI
  4. Bröder, 2017, On a problematic procedure to manipulate response biases in recognition experiments: the case of implied base rates, Memory, 25, 736, 10.1080/09658211.2016.1214735 · DOI
  5. Ebbinghaus, 1913, vol. 3
  6. Gama, 2006, Decision trees for mining data streams, Intell. Data Anal., 10, 23, 10.3233/IDA-2006-10103 · DOI
  7. Grant, 2017, An evaluation of interrater reliability measures on binary tasks using d-prime, Appl. Psychol. Meas., 41, 264, 10.1177/0146621616684584 · DOI
  8. Hansen, 2013, Quality control mechanisms for crowdsourcing: peer review, arbitration, & expertise at familysearch indexing, 649
  9. Helmbold, 1994, Tracking drifting concepts by minimizing disagreements, Mach. Learn., 14, 27, 10.1023/A:1022694620923 · DOI
  10. Ikonomovska, 2011, Learning model trees from evolving data streams, Data Min. Knowl. Discov., 23, 128, 10.1007/s10618-010-0201-y · DOI
  11. Imran, M., Lykourentzou, I., Naudet, Y., Castillo, C., 2013. Engineering crowdsourced stream processing systems. arXiv preprint arXiv:1310.5463. · arXiv
  12. Klinkenberg, 2004, Learning drifting concepts: example selection vs. example weighting, Intell. Data Anal., 8, 281, 10.3233/IDA-2004-8305 · DOI
  13. Koren, 2010, Collaborative filtering with temporal dynamics, Commun. ACM, 53, 89, 10.1145/1721654.1721677 · DOI
  14. Koychev, I., 2000. Gradual Forgetting for Adaptation to Concept DriftURL: https://research.uni-sofia.bg/handle/10506/57. Accepted: 2008-01-18T15:13:52Z Publisher: Proceedings of ECAI 2000 Workshop on Current Issues in Spatio-Temporal Reasoning.
  15. Lofi, 2014, Design patterns for hybrid algorithmic-crowdsourcing workflows, vol. 1, 1
  16. Loftus, 1985, Evaluating forgetting curves, J. Exp. Psychol., 11, 397
  17. Mackworth, 1948, The breakdown of vigilance during prolonged visual search, Q. J. Exp. Psychol., 1, 6, 10.1080/17470214808416738 · DOI
  18. Melton, 1963, Implications of short-term memory for a general theory of memory, J. Verbal Learn. Verbal Behav., 2, 1, 10.1016/S0022-5371(63)80063-8 · DOI · Paper page
  19. Peterson, 1959, Short-term retention of individual verbal items, J. Exp. Psychol., 58, 193, 10.1037/h0049234 · DOI · Paper page
  20. Reason, 2000, Human error: models and management, BMJ, 320, 768, 10.1136/bmj.320.7237.768 · DOI · Paper page
  21. Sutton, 2015, Terse messaging and public health in the midst of natural disasters: the case of the boulder floods, Health Commun., 30, 135, 10.1080/10410236.2014.974124 · DOI
  22. Tulving, 1966, Availability versus accessibility of information in memory for words, J. Verbal Learn. Verbal Behav., 5, 381, 10.1016/S0022-5371(66)80048-8 · DOI
  23. Vitter, 1985, Random sampling with a reservoir, ACM Trans. Math. Softw., 11, 37, 10.1145/3147.3165 · DOI
  24. Widmer, 1996, Learning in the presence of concept drift and hidden contexts, Mach. Learn., 23, 69, 10.1023/A:1018046501280 · DOI · Paper page
  25. Wiener, 1987, Application of vigilance research: rare, medium, or well done?, Hum. Factors, 29, 725, 10.1177/001872088702900611 · DOI
  26. Zhang, 2004, A cognitive taxonomy of medical errors, J. Biomed. Inform., 37, 193, 10.1016/j.jbi.2004.04.004 · DOI
  27. Zhou, 2021, Characterizing the dissemination of misinformation on social media in health emergencies: an empirical study based on COVID-19, Inf. Process. Manag., 58, 102554, 10.1016/j.ipm.2021.102554 · DOI
  28. Žliobaitė, 2011, Combining similarity in time and space for training set formation under concept drift, Intell. Data Anal., 15, 589, 10.3233/IDA-2011-0484 · DOI
  29. Žliobaitė, 2014, Active learning with drifting streaming data, IEEE Trans. Neural Netw. Learn. Syst., 25, 27, 10.1109/TNNLS.2012.2236570 · DOI

Metadata

Authors
Pandey, R., Purohit, H., Castillo, C., & Shalin, V. L.
Affiliations
  • George Mason University (US)
  • Institució Catalana de Recerca i Estudis Avançats (ES)
  • Wright State University (US)
Corresponding author
Pandey, Rahul
ORCID
Venue
International Journal of Human-Computer Studies
Publisher
Elsevier BV
Published
2022-04
Volume
160
Pages
102772
ISSN
1071-5819 (ISSN-L)
DOI
10.1016/j.ijhcs.2022.102772
OpenAlex
W3041011740
Cited by
47
References
78
Citations by year
2026: 11 · 2025: 15 · 2024: 9 · 2023: 8 · 2022: 4
FWCI
6.95
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Human-Computer Studies (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • European Commission
  • Joint Research Centre
  • National Science Foundation (1815459, LCF/PR/PR16/51110009, 100010434, IIS-1657379)
  • Fundación La Caixa (LCF/PR/PR16/51110009)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Data Stream Mining Techniques
Keywords
human-in-the-loop learning, stream processing, active learning (machine learning), crowdsourcing, social media analytics, disaster response, digital volunteers, news filtering, human memory

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review