Calendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop
Open accessAbstract
Although we may complain about meetings, they are an essential part of an information worker's work life. Consequently, busy people spend a significant amount of time scheduling meetings. We present Calendar.help, a system that provides fast, efficient scheduling through structured workflows. Users interact with the system via email, delegating their scheduling needs to the system as if it were a human personal assistant. Common scheduling scenarios are broken down using well-defined workflows and completed as a series of microtasks that are automated when possible and executed by a human otherwise. Unusual scenarios fall back to a trained human assistant executing an unstructured macrotask. We describe the iterative approach we used to develop Calendar.help, and share the lessons learned from scheduling thousands of meetings during a year of real-world deployments. Our findings provide insight into how complex information tasks can be broken down into repeatable components that can be executed efficiently to improve productivity.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Today's chatbot design practice follows a well-established workflow of identifying functionality or persona, creating a dialogue flow template, filling the template with supervised NLP models, and iterating.
“Today's chatbot design practice-i.e., designing multi-turn conversational interactions-follows a well-established workfow [12,13,24,25,39,44].”
From Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts (2023) · cites this paper as [13]
The Discussion and Tier 1: Automatic Microtask Execution sections describe Calendar.help’s iterative development of workflows and use of ML and NLP. They do not establish a well-established chatbot-design workflow involving persona selection, dialogue-flow templates, and supervised models. Page numbers are unavailable in our copy. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 29 · Cited by 106
Research works printed but not linked: 5
- Gabor Angeli, Christopher D Manning, and Daniel Jurafsky. 2012. Parsing time: Learning to interpret time expressions. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 446-455.
- Yoav Artzi, Nicholas FitzGerald, and Luke S Zettlemoyer. 2013. Semantic Parsing with Combinatory Categorial Grammars. ACL (Tutorial Abstracts) 3 (2013).
- Angel X. Chang and Christopher D. Manning. 2012. SUTime: A library for recognizing and normalizing time expressions. In LREC. 3735-3740.
- Ece Kamar. 2016. Hybrid Intelligence and the Future of Work. In Productivity Decomposed: Getting Big Things Done with Little Microtasks Workshop (CHI 2016). http://research.microsoft.com/en-us/um/people/eckamar/ papers/HybridIntelligence.pdf
- Ece Kamar, Severin Hacker, and Eric Horvitz. 2012. Combining human and machine intelligence in large-scale crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems-Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 467-474.
- Authors
- Cranshaw, J., Elwany, E., Newman, T., Kocielnik, R., Yu, B., Soni, S., Teevan, J., & Monroy-Hernández, A.
- Affiliations
- Microsoft (United States) (US)
- University of Washington, Seattle, WA, USA
- University of Minnesota (US)
- Georgia Institute of Technology (US)
- Microsoft Research, Redmond, WA, USA
- Venue
- Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2017-05-02
- Pages
- 2382-2393
- OpenAlex
- W4297823417
- Cited by
- 106
- References
- 29
- Citations by year
- 2026: 8 · 2025: 16 · 2024: 9 · 2023: 11 · 2022: 13 · 2021: 16 · 2020: 9 · 2019: 10 · 2018: 14
- Type
- paratext
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Personal Information Management and User Behavior
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Sociology and Political Science › Information Systems Theories and Implementation
- Keywords
- meeting scheduling, calendar management, workflow automation, human-in-the-loop automation, task decomposition, information professionals, productivity, personal assistants
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
