COSMO: A Large-Scale E-commerce Common Sense Knowledge Generation and Serving System at Amazon
Abstract
Applications of large-scale knowledge graphs in the e-commerce platforms can improve shopping experience for their customers. While existing e-commerce knowledge graphs (KGs) integrate a large volume of concepts or product attributes, they fail to discover user intentions, leaving the gap with how people think, behave, and interact with the surrounding world. In this work, we present COSMO, a scalable system to mine user-centric commonsense knowledge from massive behaviors and construct industry-scale knowledge graphs to empower diverse online services. In particular, we describe a pipeline for collecting high-quality seed knowledge assertions that are distilled from large language models (LLMs) and further refined by critic classifiers trained over human-in-the-loop annotated data.Since those generations may not always align with human preferences and contain noises, we then describe how we adopt instruction tuning to finetune an efficient language model~(COSMO-LM) for faithful e-commerce commonsense knowledge generation at scale. COSMO-LM effectively expands our knowledge graph to 18 major categories at Amazon, producing millions of high-quality knowledge with only 30k annotated instructions. Finally COSMO has been deployed in Amazon search applications such as search navigation. Both offline and online A/B experiments demonstrate our proposed system achieves significant improvement. Furthermore, these experiments highlight the immense potential of commonsense knowledge extracted from instruction-finetuned large language models.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Factorizing personalized Markov chains for next-basket recommendation
Rendle, S., Freudenthaler, C., & Schmidt-Thieme, L. (2010) · International World Wide Web Conference (WWW) · doi:10.1145/1772690.1772773 · Cited by 2,766 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer
Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., & Jiang, P. (2019) · Proceedings of the 28th ACM International Conference on Information and Knowledge Management · doi:10.1145/3357384.3357895 · Cited by 2,558 · see citing works
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
Speer, R., Chin, J., & Havasi, C. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11164 · Cited by 2,312 · see citing works
Session-Based Recommendation with Graph Neural Networks
Wu, S., Tang, Y., Zhu, Y., Wang, L., Xie, X., & Tan, T. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301346 · Cited by 1,903 · see citing works
Modern Information Retrieval : A Brief Overview
Singhal, A. (2001) · Cited by 1,520 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
Self-citationSelf-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243 · see citing works
Liu, Q., Zeng, Y., Mokhosi, R., & Zhang, H. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3219950 · Cited by 962 · see citing works
Self-citationLearning semantic representations using convolutional neural networks for web search
Shen, Y., He, X., Gao, J., Deng, L., & Mesnil, G. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2567948.2577348 · Cited by 677 · see citing works
Graph Contextualized Self-Attention Network for Session-based Recommendation
Xu, C., Zhao, P., Liu, Y., Sheng, V. S., Xu, J., Zhuang, F., Fang, J., & Zhou, X. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/547 · Cited by 667 · see citing works
Global Context Enhanced Graph Neural Networks for Session-based Recommendation
Wang, Z., Wei, W., Cong, G., Li, X.-L., Mao, X.-L., & Qiu, M. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401142 · Cited by 641 · see citing works
A Collaborative Session-based Recommendation Approach with Parallel Memory Modules
Wang, M., Ren, P., Mei, L., Chen, Z., Ma, J., & de Rijke, M. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331210 · Cited by 284 · see citing works
A Survey on Trajectory Data Management, Analytics, and Learning
Wang, S., Bao, Z., Culpepper, J. S., & Cong, G. (2021) · ACM Computing Surveys · doi:10.1145/3440207 · Cited by 271 · see citing works
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.340 · Cited by 247 · see citing works
Generate Neural Template Explanations for Recommendation
Li, L., Zhang, Y., & Chen, L. (2020) · Proceedings of the 29th ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3411992 · Cited by 149 · see citing works
Nigam, P., Song, Y., Mohan, V., Lakshman, V., Ding, W. (., Shingavi, A., Teo, C. H., Gu, H., & Yin, B. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330759 · Cited by 127 · see citing works
Self-citationIntention, Emotion, and Action: A Neural Theory Based on Semantic Pointers
Schröder, T., Stewart, T. C., & Thagard, P. R. (2013) · Cognitive Science · doi:10.1111/cogs.12100 · Cited by 113 · see citing works
Elastic Machine Learning Algorithms in Amazon SageMaker
Liberty, E., Karnin, Z. S., Xiang, B., Rouesnel, L., Coşkun, B., Nallapati, R. M., Delgado, J., Sadoughi, A., Astashonok, Y., Das, P., et al. (2020) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3318464.3386126 · Cited by 103 · see citing works
Li, F.-L., Chen, H., Xu, G., Qiu, T., Ji, F., Zhang, J., & Chen, H. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3412685 · Cited by 72 · see citing works
SCOTT: Self-Consistent Chain-of-Thought Distillation
Wang, P., Wang, Z., Li, Z., Gao, Y., Yin, B., & Ren, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.304 · Cited by 68 · see citing works
Self-citationAliCoCo: Alibaba E-commerce Cognitive Concept Net
Luo, X., Liu, L., Yang, Y., Bo, L., Cao, Y., Wu, J., Li, Q., Yang, K., & Zhu, K. Q. (2020) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3318464.3386132 · Cited by 67 · see citing works
AliCG: Fine-grained and Evolvable Conceptual Graph Construction for Semantic Search at Alibaba
Zhang, N., Jia, Q., Deng, S., et al. (2021) · doi:10.1145/3447548.3467057 · Cited by 59 · see citing works
Spatio-temporal Contrastive Learning-enhanced GNNs for Session-based Recommendation
Wan, Z., Liu, X., Wang, B., Qiu, J., Li, B., Guo, T., Chen, G., & Wang, Y. (2023) · ACM Transactions on Information Systems · doi:10.1145/3626091 · Cited by 38 · see citing works
Self-citationMultilingual Knowledge Graph Completion with Self-Supervised Adaptive Graph Alignment
Huang, Z., Li, Z., Jiang, H., et al. (2022) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/2022.acl-long.36 · Cited by 34 · see citing works
Self-citationAll You Need to Know to Build a Product Knowledge Graph
Zalmout, N., Zhang, C., Li, X., et al. (2021) · doi:10.1145/3447548.3470825 · Cited by 30 · see citing works
RETE: Retrieval-Enhanced Temporal Event Forecasting on Unified Query Product Evolutionary Graph
Wang, R., Li, Z., Zhang, D., Yin, Q., Zhao, T., Yin, B., & Abdelzaher, T. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511974 · Cited by 22 · see citing works
Self-citationShopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search
Reddy, C. K., Màrquez, L., Valero, F., Rao, N., Zaragoza, H., Bandyopadhyay, S., Biswas, A., Xing, A., & Subbian, K. (2022) · arXiv · doi:10.48550/arxiv.2206.06588 · Cited by 17 · see citing works
Zhang, D., Li, Z., Cao, T., et al. (2021) · Proceedings of the ... ACM International Conference on Information & Knowledge Management · doi:10.1145/3459637.3481946 · Cited by 16 · see citing works
Self-citationQuery Attribute Recommendation at Amazon Search
Luo, C., Headden, W. P., Avudaiappan, N., Jiang, H., Cao, T., Yin, Q., Gao, Y., Li, Z., Goutam, R., Zhang, H., et al. (2022) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3523227.3547395 · Cited by 15 · see citing works
Self-citationAliCoCo2: Commonsense Knowledge Extraction, Representation and Application in E-commerce
Luo, X., Bo, L., Wu, J., Li, L., Luo, Z., Yang, Y., & Yang, K. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467203 · Cited by 14 · see citing works
Query Rewriting in TaoBao Search
Li, S., Lv, F., Jin, T., Li, G., Zheng, Y., Zhuang, T., Liu, Q., Zeng, X., Kwok, J. T.-Y., & Ma, Q. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557068 · Cited by 12 · see citing works
Wang, W., Fang, T., Li, C., Shi, H., Ding, W., Xu, B., Wang, Z., Bai, J., Liu, X., Cheng, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.07286 · Cited by 1 · see citing works
Self-citationEnhancing User Intent Capture in Session-Based Recommendation with Attribute Patterns
Liu, X., Li, Z., Gao, Y., Yang, J., Cao, T., Wang, Z., Yin, B., & Song, Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1344
Self-citationCommonsense Knowledge Salience Evaluation with a Benchmark Dataset in E-commerce
Qu, Y., Zhang, N., Chen, H., Dai, Z., Xu, Z., Wang, C., Wang, X., Chen, Q., & Chen, H. (2022) · arXiv · doi:10.48550/arxiv.2205.10843
Construction and Applications of Billion-Scale Pre-Trained Multimodal Business Knowledge Graph
Deng, S., Wang, C., Li, Z., Zhang, N., Dai, Z., Chen, H., Xiong, F., Yan, M., Chen, Q., Chen, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.15214
Self-citation
Flagged first, then by citations
Research works printed but not linked: 18
- Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional AI: Harmlessness from AI Feedback. arXiv preprint arXiv:2212.08073 (2022). · arXiv
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. NeurIPS (2020), 1877--1901.
- Pengcheng He Xiaodong Liu Jianfeng Gao and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. In ICLR.
- Balázs Hidasi Alexandros Karatzoglou Linas Baltrunas and Domonkos Tikk. 2016. Session-based Recommendations with Recurrent Neural Networks. In ICLR.
- Jie Huang, Yifan Gao, Zheng Li, Jingfeng Yang, Yangqiu Song, Chao Zhang, Zining Zhu, Haoming Jiang, Kevin Chen-Chuan Chang, and Bing Yin. 2023. Ccgen: Explainable complementary concept generation in e-commerce. arXiv preprint arXiv:2305.11480 (2023). · arXiv
- Wei Jin Haitao Mao Zheng Li Haoming Jiang Chen Luo Hongzhi Wen Haoyu Han Hanqing Lu Zhengyang Wang Ruirui Li et al. 2024. Amazon-m2: A multilingual multi-locale shopping session dataset for recommendation and text generation. Advances in Neural Information Processing Systems 36 (2024).
- Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 (2021). · arXiv
- Jianmo Ni Jiacheng Li and Julian McAuley. 2019. Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects. In EMNLP. 188--197.
- Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 (2023). · arXiv
- Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Scao, Arun Raja, et al. 2022. Multitask Prompted Training Enables Zero-Shot Task Generalization. In International Conference on Learning Representations.
- Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In the AAAI. 3027--3035.
- William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802 (2022). · arXiv
- Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). · arXiv
- Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, AndrewMDai, and Quoc V Le. 2021. Finetuned Language Models are Zero-Shot Learners. In ICLR.
- Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic Knowledge Distillation: from General Language Models to Commonsense Models. In NAACL. 4602--4625.
- Fanyou Wu, Yang Liu, Rado Gazo, Benes Bedrich, and Xiaobo Qu. 2022. Some Practice for Improving the Search Results of E-commerce. arXiv preprint arXiv:2208.00108 (2022). · arXiv
- Changlong Yu, Weiqi Wang, Xin Liu, Jiaxin Bai, Yangqiu Song, Zheng Li, Yifan Gao, Tianyu Cao, and Bing Yin. 2023. FolkScope: Intention Knowledge Graph Construction for Discovering E-commerce Commonsense. In Findings of the Association for Computational Linguistics: ACL 2023.
- Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv:2205.01068 (2022). · arXiv
Cited by
Still being checked.
- Authors
- Yu, C., Liu, X., Maia, J., Li, Y., Cao, T., Gao, Y., Song, Y., Goutam, R., Zhang, H., Yin, B., & Li, Z.
- Affiliations
- Amazon (United States) (US)
- Hong Kong University of Science and Technology (HK)
- ORCID
- Venue
- Companion of the 2024 International Conference on Management of Data
- Publisher
- ACM
- Published
- 2024-06-09
- Pages
- 148-160
- ISSN
- 0730-8078 (ISSN-L)
- OpenAlex
- W4398233570
- Semantic Scholar
- 5f9195769eb8ea3ac9ae3d513b4c3123551842ea
- Cited by
- 13
- References
- 38
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 3
- FWCI
- 3.75
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGMOD International Conference on Management of Data (SIGMOD) (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Semantic Web and Ontologies
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- large language models, instruction tuning, human-in-the-loop annotation, knowledge graph construction, Amazon
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
