Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 39

The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.

“Discovering the sweet spot of human–computer configurations: a case study in information extraction,”

Mackeprang, Maximilian, Müller-Birn, Claudia, Stauss, Maximilian Timo·2019·Proceedings of the ACM on Human-Computer Interaction
DOI 10.1145/3359297
Open this paper in CoChat — free account

Abstract

Interactive intelligent systems, i.e., interactive systems that employ AI technologies, are currently present in many parts of our social, public and political life. An issue reoccurring often in the development of these systems is the question regarding the level of appropriate human and computer contributions. Engineers and designers lack a way of systematically defining and delimiting possible options for designing such systems in terms of levels of automation. In this paper, we propose, apply and reflect on a method for human-computer configuration design. It supports the systematic investigation of the design space for developing an interactive intelligent system. We illustrate our method with a use case in the context of collaborative ideation. Here, we developed a tool for information extraction from idea content. A challenge was to find the right level of algorithmic support, whereby the quality of the information extraction should be as high as possible, but, at the same time, the human effort should be low. Such contradicting goals are often an issue in system development; thus, our method proposed helped us to conceptualize and explore the design space. Based on a critical reflection on our method application, we want to offer a complementary perspective to the value-centered design of interactive intelligent systems. Our overarching goal is to contribute to the design of so-called hybrid systems where humans and computers are partners.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3359297

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 9 references

Research works printed but not linked: 49

  1. Bruce W Arden . 1983. What Can Be Automated?: Computer Science and Engineering Research Study . MiT Press . Bruce W Arden. 1983. What Can Be Automated?: Computer Science and Engineering Research Study .MiT Press.
  2. Emily M Bender and Batya Friedman . 2018. Data Statements for NLP: Toward Mitigating System Bias and Enabling Better Science . (March 2018 ). Emily M Bender and Batya Friedman. 2018. Data Statements for NLP: Toward Mitigating System Bias and Enabling Better Science . (March 2018).
  3. Osvald M Bjelland and Robert Chapman Wood . 2008. An inside view of IBM's' Innovation Jam'. MIT Sloan management review , Vol. 50 , 1 ( 2008 ), 32--40. Osvald M Bjelland and Robert Chapman Wood. 2008. An inside view of IBM's' Innovation Jam'. MIT Sloan management review , Vol. 50, 1 (2008), 32--40.
  4. Andre Breitenfeld , Maximilian Mackeprang , and Ming-Tung Hong . 2017. Enabling Structured Data Generation by Nontechnical Experts. Mensch und Computer 2017-Tagungsband: Spielend einfach interagieren , Vol. 17 ( Sept. 2017 ), 181--192. Andre Breitenfeld, Maximilian Mackeprang, and Ming-Tung Hong. 2017. Enabling Structured Data Generation by Nontechnical Experts. Mensch und Computer 2017-Tagungsband: Spielend einfach interagieren , Vol. 17 (Sept. 2017), 181--192.
  5. 10.1145/3059454.3059455 · DOI
  6. Jacob Cohen . 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement , Vol. 20 , 1 ( 1960 ), 37--46. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement , Vol. 20, 1 (1960), 37--46.
  7. Joachim Daiber , Max Jakob , Chris Hokamp , and Pablo N. Mendes . 2013. Improving Efficiency and Accuracy in Multilingual Entity Extraction . In Proceedings of the 9th International Conference on Semantic Systems (I-Semantics) . Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N. Mendes. 2013. Improving Efficiency and Accuracy in Multilingual Entity Extraction. In Proceedings of the 9th International Conference on Semantic Systems (I-Semantics) .
  8. 10.1080/00140139.2018.1457725 · DOI · Paper page
  9. Sidney W. A. Dekker and David D . Woods . 2002 . MABA-MABA or Abracadabra? Progress on HumantextendashAutomation Coordination . , Vol. 4 , 4 (Nov. 2002), 240--244. Sidney W. A. Dekker and David D. Woods. 2002. MABA-MABA or Abracadabra? Progress on HumantextendashAutomation Coordination . , Vol. 4, 4 (Nov. 2002), 240--244.
  10. Gianluca Demartini , Djellel Eddine Difallah , Ujwal Gadiraju, and Michele Catasta. 2017 . An introduction to hybrid human-machine information systems. Foundations and Trends® in Web Science , Vol. 7 , 1 (2017), 1--87. Gianluca Demartini, Djellel Eddine Difallah, Ujwal Gadiraju, and Michele Catasta. 2017. An introduction to hybrid human-machine information systems. Foundations and Trends® in Web Science , Vol. 7, 1 (2017), 1--87.
  11. 10.1518/001872095779064555 · DOI · Paper page
  12. Umer Farooq and Jonathan Grudin . 2016. Human-computer integration . interactions , Vol. 23 , 6 ( Oct. 2016 ), 26--32. Umer Farooq and Jonathan Grudin. 2016. Human-computer integration . interactions , Vol. 23, 6 (Oct. 2016), 26--32.
  13. Batya Friedman . 1996. Value-sensitive design . interactions , Vol. 3 , 6 ( Dec. 1996 ), 16--23. Batya Friedman. 1996. Value-sensitive design . interactions , Vol. 3, 6 (Dec. 1996), 16--23.
  14. Timnit Gebru , Jamie Morgenstern , Briana Vecchione , Jennifer Wortman Vaughan , Hanna Wallach, Hal Daumeé III, and Kate Crawford. 2018 . Datasheets for Datasets . (2018). Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumeé III, and Kate Crawford. 2018. Datasheets for Datasets. (2018).
  15. R. Stuart Geiger and Aaron Halfaker . 2017 . Operationalizing Conflict and Cooperation Between Automated Software Agents in Wikipedia: A Replication and Expansion of 'Even Good Bots Fight '. Proc. ACM Hum.-Comput. Interact. , Vol. 1 , CSCW, Article 49 (Dec. 2017), bibinfonumpages 33 pages. R. Stuart Geiger and Aaron Halfaker. 2017. Operationalizing Conflict and Cooperation Between Automated Software Agents in Wikipedia: A Replication and Expansion of 'Even Good Bots Fight'. Proc. ACM Hum.-Comput. Interact. , Vol. 1, CSCW, Article 49 (Dec. 2017), bibinfonumpages33 pages.
  16. 10.1145/1718918.1718941 · DOI · Paper page
  17. 10.1145/3173574.3173695 · DOI
  18. 10.1145/3025453.3025464 · DOI
  19. Marco Grassi , Christian Morbidoni , Michele Nucci , Simone Fonda , and Francesco Piazza . 2013. Pundit: augmenting web contents with semantics. Literary and linguistic computing , Vol. 28 , 4 ( 2013 ), 640--659. Marco Grassi, Christian Morbidoni, Michele Nucci, Simone Fonda, and Francesco Piazza. 2013. Pundit: augmenting web contents with semantics. Literary and linguistic computing , Vol. 28, 4 (2013), 640--659.
  20. Aaron Halfaker and John Riedl . 2012 . Bots and Cyborgs : Wikipedia's Immune System. IEEE Computer , Vol. 45 , 3 (2012), 79 -- 82 . Aaron Halfaker and John Riedl. 2012. Bots and Cyborgs: Wikipedia's Immune System. IEEE Computer , Vol. 45, 3 (2012), 79--82.
  21. Sandra G Hart and Lowell E Staveland . 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research . In Advances in psychology . Vol. 52 . Elsevier , 139--183. Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology . Vol. 52. Elsevier, 139--183.
  22. 10.1145/302979.303030 · DOI · Paper page
  23. Eric Horvitz . 1999 b. Uncertainty, Action, and Interaction: In Pursuit of Mixed-Initiative Computing . Intelligent Systems 6 (Sept . 1999 ), 17--20. Eric Horvitz. 1999 b. Uncertainty, Action, and Interaction: In Pursuit of Mixed-Initiative Computing . Intelligent Systems 6 (Sept. 1999), 17--20.
  24. 10.1145/2030365.2030366 · DOI
  25. Matthew Johnson , Jeffrey M Bradshaw , and Paul J Feltovich . 2017 . Tomorrowtextquoterights HumantextendashMachine Design Tools: From Levels of Automation to Interdependencies : . Journal of Cognitive Engineering and Decision Making , Vol. 12 , 1 (Oct. 2017), 77--82. Matthew Johnson, Jeffrey M Bradshaw, and Paul J Feltovich. 2017. Tomorrowtextquoterights HumantextendashMachine Design Tools: From Levels of Automation to Interdependencies: . Journal of Cognitive Engineering and Decision Making , Vol. 12, 1 (Oct. 2017), 77--82.
  26. 10.1109/MIS.2011.47 · DOI
  27. 10.1037/h0043729 · DOI · Paper page
  28. 10.1177/107118139704100147 · DOI
  29. Ece Kamar , Severin Hacker , and Eric Horvitz . 2012 . Combining Human and Machine Intelligence in Large-scale Crowdsourcing . In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1 (AAMAS '12). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 467--474. Ece Kamar, Severin Hacker, and Eric Horvitz. 2012. Combining Human and Machine Intelligence in Large-scale Crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1 (AAMAS '12). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 467--474.
  30. Claire Liang , Julia Proft , Erik Andersen , and Ross A. Knepper . 2019. Implicit Communication of Actionable Information in Human-AI Teams . In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19) . ACM, New York, NY, USA, Article 95, bibinfonumpages13 pages. Claire Liang, Julia Proft, Erik Andersen, and Ross A. Knepper. 2019. Implicit Communication of Actionable Information in Human-AI Teams. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, New York, NY, USA, Article 95, bibinfonumpages13 pages.
  31. Joseph Carl Robnett Licklider . 1960. Man-Computer Symbiosis . IRE transactions on human factors in electronics , Vol. HFE-1 , 1 ( March 1960 ), 4--11. Joseph Carl Robnett Licklider. 1960. Man-Computer Symbiosis . IRE transactions on human factors in electronics , Vol. HFE-1, 1 (March 1960), 4--11.
  32. Maximilian Mackeprang , Abderrahmane Khiat , and Claudia Müller-Birn . 2018 a. Concept Validation During Collaborative Ideation and Its Effect on Ideation Outcome. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM , New York, NY, USA, Article LBW033, bibinfonumpages6 pages. Maximilian Mackeprang, Abderrahmane Khiat, and Claudia Müller-Birn. 2018a. Concept Validation During Collaborative Ideation and Its Effect on Ideation Outcome. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (CHI EA '18). ACM, New York, NY, USA, Article LBW033, bibinfonumpages6 pages.
  33. Maximilian Mackeprang , Abderrahmane Khiat , and Claudia Müller-Birn . 2018 b. Innovonto: An Enhanced Crowd Ideation Platform with Semantic Annotation (Hallway Test) (FU Technical Reports Serie B) , Vol. TR-B-18- 02 . Berlin . Maximilian Mackeprang, Abderrahmane Khiat, and Claudia Müller-Birn. 2018b. Innovonto: An Enhanced Crowd Ideation Platform with Semantic Annotation (Hallway Test) (FU Technical Reports Serie B), Vol. TR-B-18-02. Berlin.
  34. Jose L Martinez-Rodriguez , Aidan Hogan , and Ivan Lopez-Arevalo . 2018. Information extraction meets the semantic web: a survey. Semantic Web Preprint ( 2018 ), 1--81. Jose L Martinez-Rodriguez, Aidan Hogan, and Ivan Lopez-Arevalo. 2018. Information extraction meets the semantic web: a survey. Semantic Web Preprint (2018), 1--81.
  35. 10.1145/2063518.2063519 · DOI · Paper page
  36. Judith S Olson and Wendy A Kellogg . 2014. Ways of Knowing in HCI . Vol. 2 . Springer . Judith S Olson and Wendy A Kellogg. 2014. Ways of Knowing in HCI . Vol. 2. Springer.
  37. Heiko Paulheim . 2017. Knowledge graph refinement: A survey of approaches and evaluation methods. Semantic web , Vol. 8 , 3 ( 2017 ), 489--508. Heiko Paulheim. 2017. Knowledge graph refinement: A survey of approaches and evaluation methods. Semantic web , Vol. 8, 3 (2017), 489--508.
  38. 10.1145/2043932.2043962 · DOI · Paper page
  39. Ben Shneiderman and Pattie Maes . 1997. Direct Manipulation vs. Interface Agents . interactions , Vol. 4 , 6 ( Nov. 1997 ), 42--61. Ben Shneiderman and Pattie Maes. 1997. Direct Manipulation vs. Interface Agents . interactions , Vol. 4, 6 (Nov. 1997), 42--61.
  40. 10.1145/2675133.2675239 · DOI
  41. 10.1145/2984511.2984578 · DOI
  42. 10.1145/2757226.2757230 · DOI · Paper page
  43. 10.1145/1390085.1390096 · doi:10.1145/1390085.1390096
  44. Viswanath Venkatesh and Hillol Bala . 2008. Technology acceptance model 3 and a research agenda on interventions. Decision sciences , Vol. 39 , 2 ( 2008 ), 273--315. Viswanath Venkatesh and Hillol Bala. 2008. Technology acceptance model 3 and a research agenda on interventions. Decision sciences , Vol. 39, 2 (2008), 273--315.
  45. 10.1145/2838729 · DOI
  46. 10.1145/2361354.2361394 · doi:10.1145/2361354.2361394
  47. Langdon Winner . 1980. Do artifacts have politics? Daedalus ( 1980 ), 121--136. Langdon Winner. 1980. Do artifacts have politics? Daedalus (1980), 121--136.
  48. 10.1145/2804075.2804077 · DOI
  49. 10.1145/2470654.2470715 · DOI

Cited by

Metadata

Authors
Mackeprang, M., Müller-Birn, C., & Stauss, M. T.
Affiliations
Freie Universität Berlin, Berlin, Germany
Venue
Proceedings of the ACM on Human-Computer Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2019-11-07
Volume
3
Issue
CSCW
Pages
1-30
ISSN
2573-0142
DOI
10.1145/3359297
Cited by
39
References
9
Language
en
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.