CoChat
Sign Up
PreprintCited by 9Advanced Database Systems and QueriesTopic ModelingNatural Language Processing Techniques

Semantic Operators: A Declarative Model for Rich, AI-based Data Processing

Open access
Patel, Liana, Jha, Siddharth, Pan, Melissa, Gupta, Harshit, Asawa, Parth, Guestrin, Carlos, Zaharia, Matei·2024·arXiv
DOI 10.48550/arxiv.2407.11418arXiv 2407.11418
Open this paper in CoChat — free accountOpen PDF

Abstract

The semantic capabilities of large language models (LLMs) have the potential to enable rich analytics and reasoning over vast knowledge corpora. Unfortunately, existing systems either empirically optimize expensive LLM-powered operations with no performance guarantees, or serve a limited set of row-wise LLM operations, providing limited robustness, expressiveness and usability. We introduce semantic operators, the first formalism for declarative and general-purpose AI-based transformations based on natural language specifications (e.g., filtering, sorting, joining or aggregating records using natural language criteria). Each operator opens a rich space for execution plans, similar to relational operators. Our model specifies the expected behavior of each operator with a high-quality gold algorithm, and we develop an optimization framework that reduces cost, while providing accuracy guarantees with respect to a gold algorithm. Using this approach, we propose several novel optimizations to accelerate semantic filtering, joining, group-by and top-k operations by up to $1,000\times$. We implement semantic operators in the LOTUS system and demonstrate LOTUS' effectiveness on real, bulk-semantic processing applications, including fact-checking, biomedical multi-label classification, search, and topic analysis. We show that the semantic operator model is expressive, capturing state-of-the-art AI pipelines in a few operator calls, and making it easy to express new pipelines that match or exceed quality of recent LLM-based analytic systems by up to $170\%$, while offering accuracy guarantees. Overall, LOTUS programs match or exceed the accuracy of state-of-the-art AI pipelines for each task while running up to $3.6\times$ faster than the highest-quality baselines. LOTUS is publicly available at https://github.com/lotus-data/lotus.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2407.11418

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The strong relative performance of DeepScholar-ref likely reflects the efficiency of the data-processing semantic operators [37] that DeepScholar-ref uses to perform LLM-based filtering, ranking and summarization of sources to generate its report.

    “Overall, the strong relative performance of DeepScholar-ref likely reflects the efficiency of the dataprocessing semantic operators [37] that DeepScholar-ref uses to perform LLM-based filtering, ranking and summarization of sources to generate its report.”

    From DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis (2025) · cites this paper as [37]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 54

  1. Griffin Adams, Alexander Fabbri, Faisal Ladhak, Eric Lehman, and Noémie El- hadad. 2023. From Sparse to Dense: GPT-4 Summarization with Chain of Density Prompting. http://arxiv.org/abs/2309.04269 arXiv:2309.04269 [cs]. · arXiv
  2. Eric Anderson, Jonathan Fritz, Austin Lee, Bohou Li, Mark Lindblad, Henry Lindeman, Alex Meyer, Parth Parmar, Tanvi Ranade, Mehul A. Shah, Benjamin Sowell, Dan Tecuci, Vinayak Thapliyal, and Matt Welsh. 2024. The Design of an LLM-powered Unstructured Analytics System. https://doi.org/10.48550/arXiv. 2409.00847 arXiv:2409.00847 [cs]. · DOI
  3. Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, and Christopher Ré. 2023. Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes. http: //arxiv.org/abs/2304.09433 arXiv:2304.09433 [cs]. · arXiv
  4. Asim Biswal, Liana Patel, Siddarth Jha, Amog Kamsetty, Shu Liu, Joseph E. Gonzalez, Carlos Guestrin, and Matei Zaharia. 2024. Text2SQL is Not Enough: Unifying AI and Databases with TAG. https://doi.org/10.48550/arXiv.2408.14717 arXiv:2408.14717 [cs]. · DOI
  5. Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. BooookScore: A systematic exploration of book-length summarization in the era of LLMs. http: //arxiv.org/abs/2310.00785 arXiv:2310.00785 [cs]. · arXiv
  6. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. http: //arxiv.org/abs/2305.05176 arXiv:2305.05176 [cs]. · arXiv
  7. I.-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. 2023. FacTool: Factu- ality Detection in Generative AI -A Tool Augmented Framework for Multi- Task and Multi-Domain Scenarios. https://doi.org/10.48550/arXiv.2307.13528 arXiv:2307.13528 [cs]. · DOI · Paper page
  8. E F Codd. 1970. A Relational Model of Data for Large Shared Data Banks. 13, 6 (1970).
  9. Hanjun Dai, Bethany Yixin Wang, Xingchen Wan, Bo Dai, Sherry Yang, Azade Nova, Pengcheng Yin, Phitchaya Mangpo Phothilimthana, Charles Sutton, and Dale Schuurmans. 2024. UQE: A Query Engine for Unstructured Databases. https://doi.org/10.48550/arXiv.2407.09522 arXiv:2407.09522 [cs]. · DOI
  10. Shrey Desai and Greg Durrett. 2020. Calibration of Pre-trained Transformers. https://arxiv.org/abs/2003.07892v3 · arXiv
  11. Karel D'Oosterlinck, Omar Khattab, François Remy, Thomas Demeester, Chris Develder, and Christopher Potts. 2024. In-Context Learning for Extreme Multi- Label Classification. https://doi.org/10.48550/arXiv.2401.12178 arXiv:2401.12178 [cs]. · DOI
  12. Karel D'Oosterlinck, François Remy, Johannes Deleu, Thomas Demeester, Chris Develder, Klim Zaporojets, Aneiss Ghodsi, Simon Ellershaw, Jack Collins, and Christopher Potts. 2023. BioDEX: Large-Scale Biomedical Adverse Drug Event Extraction for Real-World Pharmacovigilance. https://doi.org/10.48550/arXiv. 2305.13395 arXiv:2305.13395 [cs]. · DOI
  13. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss library. https://doi.org/10.48550/arXiv.2401.08281 arXiv:2401.08281 [cs]. · DOI
  14. Andrew Drozdov, Honglei Zhuang, Zhuyun Dai, Zhen Qin, Razieh Rahimi, Xuan- hui Wang, Dana Alon, Mohit Iyyer, Andrew McCallum, Donald Metzler, and Kai Hui. 2023. PaRaDe: Passage Ranking using Demonstrations with Large Language Models. https://doi.org/10.48550/arXiv.2310.14408 arXiv:2310.14408 [cs]. · DOI
  15. Joe Hellerstein, Christopher Ré, Florian Schoppmann, Daisy Zhe Wang, Eugene Fratkin, Aleksander Gorajek, Kee Siong Ng, Caleb Welton, Xixuan Feng, Kun Li, and Arun Kumar. 2012. The MADlib Analytics Library or MAD Skills, the SQL. http://arxiv.org/abs/1208.4165 arXiv:1208.4165 [cs]. · arXiv
  16. C. A. R. Hoare. 1961. Algorithm 65: find. Commun. ACM 4, 7 (July 1961), 321-322. https://doi.org/10.1145/366622.366647 · DOI
  17. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with GPUs. http://arxiv.org/abs/1702.08734 arXiv:1702.08734 [cs]. · arXiv
  18. Daniel Kang, Peter Bailis, and Matei Zaharia. 2019. BlazeIt: Optimizing Declara- tive Aggregation and Limit Queries for Neural Network-Based Video Analytics. http://arxiv.org/abs/1805.01046 arXiv:1805.01046 [cs]. · arXiv
  19. Daniel Kang, John Emmons, Firas Abuzaid, Peter Bailis, and Matei Zaharia. 2017. NoScope: Optimizing Neural Network Queries over Video at Scale. http: //arxiv.org/abs/1703.02529 arXiv:1703.02529 [cs]. · arXiv
  20. Daniel Kang, Edward Gan, Peter Bailis, Tatsunori Hashimoto, and Matei Zaharia. 2020. Approximate selection with guarantees using proxies. Proceedings of the VLDB Endowment 13, 12 (Aug. 2020), 1990-2003. https://doi.org/10.14778/ 3407790.3407804 · DOI
  21. Daniel Kang, Edward Gan, Peter Bailis, Tatsunori Hashimoto, and Matei Zaharia. 2022. Approximate Selection with Guarantees using Proxies. https://doi.org/10. 48550/arXiv.2004.00827 arXiv:2004.00827 [cs]. · DOI
  22. Daniel Kang, John Guibas, Peter Bailis, Tatsunori Hashimoto, Yi Sun, and Matei Zaharia. 2021. Accelerating approximate aggregation queries with expensive predicates. Proceedings of the VLDB Endowment 14, 11 (July 2021), 2341-2354. https://doi.org/10.14778/3476249.3476285 · DOI
  23. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav San- thanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. https://arxiv.org/abs/2310.03714v1 · arXiv
  24. Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. https://arxiv.org/abs/ 2004.12832v2
  25. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. https://arxiv.org/abs/2309.06180v1 · arXiv
  26. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. https://doi.org/10.48550/arXiv. 2005.11401 arXiv:2005.11401 [cs]. · DOI · Paper page
  27. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michi- hiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic Evaluation of Language Models. https://arxiv.org/abs/2211.09110v2 · arXiv
  28. Yiming Lin, Madelon Hulsebos, Ruiying Ma, Shreya Shankar, Sepanta Zeigham, Aditya G. Parameswaran, and Eugene Wu. 2024. Towards Accurate and Efficient Document Analytics with Large Language Models. http://arxiv.org/abs/2405. 04674 arXiv:2405.04674 [cs]. · arXiv
  29. Chunwei Liu, Matthew Russo, Michael Cafarella, Lei Cao, Peter Baille Chen, Zui Chen, Michael Franklin, Tim Kraska, Samuel Madden, and Gerardo Vitagliano. 2024. A Declarative System for Optimizing AI Workloads. http://arxiv.org/abs/ 2405.14696 arXiv:2405.14696 [cs]. · arXiv
  30. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the Middle: How Language Models Use Long Contexts. https://doi.org/10.48550/arXiv.2307.03172 arXiv:2307.03172 [cs]. · DOI · Paper page
  31. Shu Liu, Asim Biswal, Audrey Cheng, Xiangxi Mo, Shiyi Cao, Joseph E. Gonzalez, Ion Stoica, and Matei Zaharia. 2024. Optimizing LLM Queries in Relational Workloads. http://arxiv.org/abs/2403.05821 arXiv:2403.05821 [cs]. · arXiv
  32. Shicheng Liu, Jialiang Xu, Wesley Tjangnaka, Sina J. Semnani, Chen Jie Yu, and Monica S. Lam. 2024. SUQL: Conversational Search over Structured and Unstructured Data with Large Language Models. https://doi.org/10.48550/arXiv. 2311.09818 arXiv:2311.09818 [cs]. · DOI
  33. Yao Lu, Aakanksha Chowdhery, Srikanth Kandula, and Surajit Chaudhuri. 2018. Accelerating Machine Learning Inference with Probabilistic Predicates. In Pro- ceedings of the 2018 International Conference on Management of Data (SIGMOD '18). Association for Computing Machinery, New York, NY, USA, 1493-1508. https://doi.org/10.1145/3183713.3183751 · DOI
  34. Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero-Shot Listwise Document Reranking with a Large Language Model. https://arxiv.org/ abs/2305.02156v1 · arXiv
  35. Liana Patel, Peter Kraft, Carlos Guestrin, and Matei Zaharia. 2024. ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Struc- tured Data. Proceedings of the ACM on Management of Data 2, 3 (May 2024), 120:1-120:27. https://doi.org/10.1145/3654923 · DOI
  36. Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023. RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models. http://arxiv.org/abs/2309.15088 arXiv:2309.15088 [cs]. · arXiv
  37. Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023. RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze! http://arxiv.org/ abs/2312.02724 arXiv:2312.02724 [cs]. · arXiv
  38. Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Ben- dersky. 2024. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting. https://doi.org/10.48550/arXiv.2306.17563 arXiv:2306.17563 [cs]. · DOI · Paper page
  39. Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. Improving Passage Retrieval with Zero-Shot Question Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 3781-3797. https://doi.org/10. 18653/v1/2022.emnlp-main.249 · DOI · Paper page
  40. Nihar B. Shah and Martin J. Wainwright. 2016. Simple, Robust and Opti- mal Ranking from Pairwise Comparisons. http://arxiv.org/abs/1512.08949 arXiv:1512.08949 [cs, math, stat]. · arXiv
  41. Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G. Parameswaran, and Eugene Wu. 2024. DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing. https://doi.org/10.48550/arXiv.2410.12189 arXiv:2410.12189 [cs]. · DOI
  42. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT Good at Search? Investi- gating Large Language Models as Re-Ranking Agents. https://arxiv.org/abs/ 2304.09542v2
  43. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. https://doi.org/10.48550/arXiv.2104.08663 arXiv:2104.08663 [cs]. · DOI · Paper page
  44. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Marilyn Walker, Heng Ji, and Amanda Stent (Eds.). Association for Computational Linguistics, New Orleans, Louisiana, 809-819. https://doi.org/10. 18653/v1/N18-1074 · DOI · Paper page
  45. P. Viola and M. Jones. 2001. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, Vol. 1. IEEE Comput. Soc, Kauai, HI, USA, I-511-I-518. https://doi.org/10.1109/CVPR.2001.990517 · DOI · Paper page
  46. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2024. Text Embeddings by Weakly-Supervised Contrastive Pre-training. http://arxiv.org/abs/2212.03533 arXiv:2212.03533 [cs]. · arXiv
  47. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively Summarizing Books with Human Feedback. https://doi.org/10.48550/arXiv.2109.10862 arXiv:2109.10862 [cs]. · DOI · Paper page
  48. Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis N. Ioannidis, Karthik Subbian, James Zou, and Jure Leskovec. 2024. STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases. https://arxiv.org/abs/2404.13207v2 · arXiv
  49. Navid Yaghmazadeh, View Profile, Yuepeng Wang, View Profile, Isil Dillig, View Profile, Thomas Dillig, and View Profile. 2017. SQLizer: query synthesis from natural language. Proceedings of the ACM on Programming Languages 1, OOPSLA (Oct. 2017), 1-26. https://doi.org/10.1145/3133887 Publisher: Association for Computing Machinery. · DOI
  50. Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. 2018. TypeSQL: Knowledge-based Type-Aware Neural Text-to-SQL Generation. https://doi.org/ 10.48550/arXiv.1804.09769 arXiv:1804.09769 [cs]. · DOI
  51. Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. 2024. Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning. http://arxiv.org/abs/2310.03094 arXiv:2310.03094 [cs]. · arXiv
  52. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? https://doi.org/10. 48550/arXiv.1905.07830 arXiv:1905.07830 [cs]. · DOI
  53. Bin Zhang, Yuxiao Ye, Guoqing Du, Xiaoru Hu, Zhishuai Li, Sun Yang, Chi Harold Liu, Rui Zhao, Ziyue Li, and Hangyu Mao. 2024. Benchmarking the Text-to-SQL Capability of Large Language Models: A Comprehensive Evaluation. https: //doi.org/10.48550/arXiv.2403.02951 arXiv:2403.02951 [cs]. · DOI
  54. Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, and Michael Bendersky. 2024. Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels. http://arxiv.org/abs/2310.14122 arXiv:2310.14122 [cs]. · arXiv

Cited by

Sorted by citations

Authors
Patel, L., Jha, S., Pan, M., Gupta, H., Asawa, P., Guestrin, C., & Zaharia, M.
Affiliations
  • University of California, Berkeley (US)
  • Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2407.11418
arXiv
2407.11418
OpenAlex
W4403754277
Cited by
9
References
0
Citations by year
2025: 9
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Cisco Systems
  • VMware
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Advanced Database Systems and Queries
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
semantic operators, large language models, natural language specifications, query optimization, semantic filtering, execution plans, guaranteed accuracy, fact-checking, topic analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review