CoChat
Sign Up
PreprintCited by 1

Choice-75: A Dataset on Decision Branching in Script Learning

Hou, Zhaoyi Joey, Zhang, Li, Callison-Burch, Chris·2023·arXiv
DOI 10.48550/arxiv.2309.11737arXiv 2309.11737
Open this paper in CoChat — free account

Abstract

Script learning studies how stereotypical events unfold, enabling machines to reason about narratives with implicit information. Previous works mostly consider a script as a linear sequence of events while ignoring the potential branches that arise due to people's circumstantial choices. We hence propose Choice-75, the first benchmark that challenges intelligent systems to make decisions given descriptive scenarios, containing 75 scripts and more than 600 scenarios. We also present preliminary results with current large language models (LLM). Although they demonstrate overall decent performance, there is still notable headroom in hard scenarios.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2309.11737

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Choice-75 was introduced to evaluate daily decision-making capabilities of intelligent systems.

    “To evaluate the daily decision-making capabilities of intelligent systems, Hou et al. [75] introduced Choice-75.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [75]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 26

  1. JinYeong Bak and Alice Oh. 2018. Conversational decision-making model for predicting the king's de- cision in the annals of the Joseon dynasty. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 956-961, Brussels, Belgium. Association for Computational Linguistics. · DOI
  2. Nathanael Chambers and Dan Jurafsky. 2008. Unsuper- vised learning of narrative event chains. In Proceed- ings of ACL-08: HLT, pages 789-797, Columbus, Ohio. Association for Computational Linguistics.
  3. Nathanael Chambers and Dan Jurafsky. 2010. A database of narrative schemas. In Proceedings of the Seventh International Conference on Language Re- sources and Evaluation (LREC'10), Valletta, Malta. European Language Resources Association (ELRA).
  4. Bhavana Dalvi, Niket Tandon, Antoine Bosselut, Wen- tau Yih, and Peter Clark. 2019. Everything happens for a reason: Discovering the purpose of actions in procedural text. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4496-4505, Hong Kong, China. Association for Computational Linguistics. · DOI
  5. Xinya Du, Zixuan Zhang, Sha Li, Pengfei Yu, Hongwei Wang, Tuan Lai, Xudong Lin, Ziqi Wang, Iris Liu, Ben Zhou, Haoyang Wen, Manling Li, Darryl Han- nan, Jie Lei, Hyounghun Kim, Rotem Dror, Haoyu Wang, Michael Regan, Qi Zeng, Qing Lyu, Charles Yu, Carl Edwards, Xiaomeng Jin, Yizhu Jiao, Ghaza- leh Kazeminejad, Zhenhailong Wang, Chris Callison- Burch, Mohit Bansal, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, Martha Palmer, and Heng Ji. 2022. RESIN-11: Schema-guided event predic- tion for 11 newsworthy scenarios. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies: System Demonstra- tions, pages 54-63, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics. · DOI
  6. Raquel Fernández, Matthew Frampton, Patrick Ehlen, Matthew Purver, and Stanley Peters. 2008. Mod- elling and detecting decisions in multi-party dialogue. In Proceedings of the 9th SIGdial Workshop on Dis- course and Dialogue, pages 156-163, Columbus, Ohio. Association for Computational Linguistics.
  7. Georgi Karadzhov, Tom Stafford, and Andreas Vlachos. 2022. What makes you change your mind? an em- pirical investigation in online group decision-making conversations. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 552-563, Edinburgh, UK. As- sociation for Computational Linguistics.
  8. Katherine Keith and Amanda Stent. 2019. Modeling financial analysts' decision making via the pragmat- ics and semantics of earnings calls. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 493-503, Florence, Italy. Association for Computational Linguistics. · DOI
  9. Manling Li, Qi Zeng, Ying Lin, Kyunghyun Cho, Heng Ji, Jonathan May, Nathanael Chambers, and Clare Voss. 2020. Connecting the dots: Event graph schema induction with path language modeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 684-695, Online. Association for Computa- tional Linguistics. · DOI
  10. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and AI collabora- tion for natural language inference dataset creation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826-6847, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics. · DOI
  11. Qing Lyu, Li Zhang, and Chris Callison-Burch. 2021. Goal-oriented script construction. In Proceedings of the 14th International Conference on Natural Lan- guage Generation, pages 184-200, Aberdeen, Scot- land, UK. Association for Computational Linguistics.
  12. Naoki Otani, Jun Araki, HyeongSik Kim, and Eduard Hovy. 2023. A textual dataset for situated proactive response selection. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 3856-3874, Toronto, Canada. Association for Computational Lin- guistics. · DOI
  13. Liang Qiu, Yizhou Zhao, Yuan Liang, Pan Lu, Weiyan Shi, Zhou Yu, and Song-Chun Zhu. 2022. Towards socially intelligent agents with mental state transition and human value. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 146-158, Edinburgh, UK. As- sociation for Computational Linguistics.
  14. G Rücker, T Schimek-Jasch, and U Nestle. 2012. Mea- suring inter-observer agreement in contour delin- eation of medical imaging in a dummy run using fleiss' kappa. Methods of information in medicine, 51(06):489-494. · DOI
  15. Rachel Rudinger, Vered Shwartz, Jena D. Hwang, Chan- dra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like a skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 4661-4675, Online. Association for Computational Linguistics. · DOI
  16. Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, and Yejin Choi. 2021. proScript: Partially ordered scripts generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2138-2149, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics. · DOI
  17. Roger C. Schank. 1977. Scripts, plans, goals, and un- derstanding : an inquiry into human knowledge struc- tures / . · DOI
  18. Niket Tandon, Keisuke Sakaguchi, Bhavana Dalvi, Dheeraj Rajagopal, Peter Clark, Michal Guerquin, Kyle Richardson, and Eduard Hovy. 2020. A dataset for tracking entities in open domain procedural text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6408-6417, Online. Association for Computa- tional Linguistics. · DOI
  19. Serra Sinem Tekiroglu, Yi-Ling Chung, and Marco Guerini. 2020. Generating counter narratives against online hate speech: Data and strategies. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1177-1190, On- line. Association for Computational Linguistics. · DOI
  20. Jingjing Wang, Changlong Sun, Shoushan Li, Jiancheng Wang, Luo Si, Min Zhang, Xiaozhong Liu, and Guodong Zhou. 2019. Human-like decision making: Document-level aspect sentiment classification via hierarchical reinforcement learning. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5581-5590, Hong Kong, China. Association for Computational Linguistics. · DOI
  21. Zhaowei Wang, Quyet V. Do, Hongming Zhang, Jiayao Zhang, Weiqi Wang, Tianqing Fang, Yangqiu Song, Ginny Wong, and Simon See. 2023. COLA: Con- textualized commonsense causal reasoning from the causal inference perspective. In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 5253-5271, Toronto, Canada. Association for Com- putational Linguistics. · DOI
  22. Li Zhang, Qing Lyu, and Chris Callison-Burch. 2020. Reasoning about goals, steps, and temporal ordering with WikiHow. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 4630-4639, Online. As- sociation for Computational Linguistics. · DOI
  23. Li Zhang, Hainiu Xu, Yue Yang, Shuyan Zhou, Weiqiu You, Manni Arora, and Chris Callison-Burch. 2023a. Causal reasoning of entities and events in procedural texts. In Findings of the Association for Compu- tational Linguistics: EACL 2023, pages 415-431, Dubrovnik, Croatia. Association for Computational Linguistics. · DOI
  24. Tianyi Zhang, Isaac Tham, Zhaoyi Hou, Jiaxuan Ren, Leon Zhou, Hainiu Xu, Li Zhang, Lara Martin, Rotem Dror, Sha Li, Heng Ji, Martha Palmer, Su- san Windisch Brown, Reece Suchocki, and Chris Callison-Burch. 2023b. Human-in-the-loop schema induction. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguis- tics (Volume 3: System Demonstrations), pages 1-10, Toronto, Canada. Association for Computational Lin- guistics. · DOI
  25. Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2021. Temporal reasoning on implicit events from distant supervision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1361-1371, Online. Association for Computa- tional Linguistics. · DOI
  26. Shuyan Zhou, Li Zhang, Yue Yang, Qing Lyu, Pengcheng Yin, Chris Callison-Burch, and Graham Neubig. 2022. Show me more details: Discover- ing hierarchies of procedures from semi-structured · DOI

Cited by

Sorted by citations

Authors
Hou, Z. J., Zhang, L., & Callison-Burch, C.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2309.11737
arXiv
2309.11737
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review