CoChat
Sign Up
PreprintCited by 15Machine Learning and AlgorithmsRecommender Systems and TechniquesMobile Crowdsensing and Crowdsourcing

ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems

Open access
Simard, Patrice, Chickering, David, Lakshmiratan, Aparna, Charles, Denis, Bottou, Leon, Suarez, Carlos Garcia Jurado, Grangier, David, Amershi, Saleema +2 more·2014·arXiv
DOI 10.48550/arxiv.1409.4814arXiv 1409.4814
Open this paper in CoChat — free accountOpen PDF

Abstract

Quick interaction between a human teacher and a learning machine presents numerous benefits and challenges when working with web-scale data. The human teacher guides the machine towards accomplishing the task of interest. The learning machine leverages big data to find examples that maximize the training value of its interaction with the teacher. When the teacher is restricted to labeling examples selected by the machine, this problem is an instance of active learning. When the teacher can provide additional information to the machine (e.g., suggestions on what examples or predictive features should be used) as the learning task progresses, then the problem becomes one of interactive learning. To accommodate the two-way communication channel needed for efficient interactive learning, the teacher and the machine need an environment that supports an interaction language. The machine can access, process, and summarize more examples than the teacher can see in a lifetime. Based on the machine's output, the teacher can revise the definition of the task or make it more precise. Both the teacher and the machine continuously learn and benefit from the interaction. We have built a platform to (1) produce valuable and deployable models and (2) support research on both the machine learning and user interface challenges of the interactive learning problem. The platform relies on a dedicated, low-latency, distributed, in-memory architecture that allows us to construct web-scale learning machines with quick interaction speed. The purpose of this paper is to describe this architecture and demonstrate how it supports our research efforts. Preliminary results are presented as illustrations of the architecture but are not the primary focus of the paper.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1409.4814

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Interfaces that facilitate seamless transitions between iML activities result in fewer errors and outputs that better match users' expectations.

    “Interfaces that facilitate seamless transitions between these activities result in fewer errors and outputs that better match users' expectations [38, 46].”

    From Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (2024) · cites this paper as [46]

    Section 2.2 (p. 3) says ICE integrates labeling and feature editing in one interface and lets teachers inspect model errors to decide what to do next. Section 2.3 (p. 4) describes tools for reviewing errors and revising labels. These passages support the integrated workflow, but do not establish that seamless transitions result in fewer errors or outputs that better match users’ expectations. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 21

  1. [1][1] Adomavicius, G. and Tuzhilin, A. 2005. Towards the Next Generation of Recommender Systems: A Survey of the Stateof-the-Art and Possible Extensions. IEEE Transactions on Knowledge and Data Engineering 17 , 6, 734-749. -
  2. [2][2] Attenberg, J., Melville, P., and Provost, F. 2010. A Unified Approach to Active Dual Supervision for Labeling Features and Examples. 2010. Proceedings of the European Conference on Machine Learning (ECML-10). -
  3. [3][3] Cesa-Bianchi, N. and Lugosi, G. 2006. Prediction, learning, and games . Cambridge University Press. -
  4. [4][4] Chaiken, R., Jenkins, B., Larson, P.-A., Ramsey, B., Shakib, D., Weaver, S. and Zhou, J., SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets, Proceedings of VLDB, 1(2), 2008, 1265-1276. -
  5. [5][5] Cheng-Tao Chu, Sang Kyun Kim, Yi-An Lin, YuanYuan Yu, Gary Bradski, Andrew Y. Ng, Kunle Olukotun Map-Reduce for Machine Learning on Multicore. 2006. In Proceedings of Neural Information Processing Systems (NIPS) -
  6. [6][6] Druck G. and A. McCallum A. 2011. Toward interactive training and evaluation, Proceedings of the ACM Conference on Information Knowledge Managment (CIKM'11) . -
  7. [7][7] Dean, J. and Ghemawat, S. 2004. MapReduce: Simplified Data Processing on Large Clusters. Sixth Symposium on Operating System Design and Implementation (OSDI'04) , 137-150. -
  8. [8][8] Fails, J.A. and Olsen Jr., D.R. 2003. Interactive Machine Learning. In Proceedings of the International Conference on Intelligent User Interfaces. IUI 2003 . ACM, 39-45. -
  9. [9][9] Fiebrink, R., Cook, P.R., and Trueman, D. 2011. Human Model Evaluation in Interactive Supervised Learning. In 10. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . CHI 2011. ACM, 147-156. -
  10. [10][10] Fogarty, J., Tan, D., Kapoor, A., and Windor, S. 2008. CueFlik: Interactive Concept Learning in Image Search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . CHI 2008. ACM, 29-38. -
  11. [11][11] Godbole, S., Harpale, A., Sarawagi S., and Chakrabarti, S. 2004. Document classification through interactive supervision of document and term labels. I n Proceedings of ECML/PKDD '04. -
  12. [12][12] Kapoor, A., Lee, B., Tan, D., and Horvitz, E. 2010. Interactive Optimization for Steering Machine Classification. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI 2010. ACM, 1343-1352. -
  13. [14][14] Langford, J., Smola, A.J. and Zinkevich, 2009. M. Slow learners are fast. In Proceedings of Neural Information Processing Systems (NIPS) -
  14. [15][15] Manning, C., Raghavan P., Schutze, H., an Introduction to Information Retrieval, Cambridge University Press, 2009. -
  15. [16][16] Montaner, M., Lopez, B., de la Rosa, J.L. 2003. A Taxonomy for Recommender Agents on the Internet. Artificial Intelligence Review 19 , 285-330. -
  16. [18][18] Salton, G. and Buckley, C. 1990. Improving Retrieval Performance by Relevance Feedback. Journal of the American Society of Information Science 41 , 288-297. -
  17. [20][20] Settles, B. 2011. Closing the Loop: Fast, Interactive SemiSupervised Annotation with Queries on Features and Instances. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP) -
  18. [21][21] Azari Soufiani, H., Chickering, D., Charles, D., and Parkes, D. 2014. Approximating the Shapely Value via Multi-Issue Decompositions. In Proceedings of the 13 th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2014). -
  19. [22][22] Talbot, J., Lee, B., Kapoor, A., and Tan, D.S. 2009. Ensemble Matrix: Interactive Visualization to Support Machine Learning with Multiple Classifiers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI 2009. ACM, 1283-1292. -
  20. [23][23] Zaharia, M., Chowdhury, M., Franklin, M., Shenker, S. and Stoica, I., Spark: Cluster Computing with Working Sets, 2 nd USENIX Workshop on Hot Topics in Cloud Computing (HotCloud'10), 2010. -
  21. [24][24] Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M., Shenker, S., Stoica, I., Resilient distributed datasets: a fault-tolerant abstraction for inmemory cluster computing, 9 th USENIX Symposium on Networked Systems Design and Implementation , 2012.

Cited by

Sorted by citations

Authors
Simard, P., Chickering, D., Lakshmiratan, A., Charles, D., Bottou, L., Suarez, C. G. J., Grangier, D., Amershi, S., Verwey, J., & Suh, J.
Affiliations
Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2014
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1409.4814
arXiv
1409.4814
OpenAlex
W1558632383
Cited by
15
References
0
Citations by year
2023: 3 · 2021: 1 · 2020: 2 · 2019: 1 · 2018: 1 · 2016: 4 · 2015: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
Keywords
interactive learning, active learning (machine learning), human-in-the-loop learning, web-scale data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review