CoChat
Sign Up
PreprintCited by 4Machine Learning and AlgorithmsDomain Adaptation and Few-Shot LearningTopic Modeling

The DOI matches its DataCite record.

Cache & Distil: Optimising API Calls to Large Language Models

Open access
Ramírez, Guillem, Lindemann, Matthias, Birch, Alexandra, Titov, Ivan·2023·arXiv
DOI 10.48550/arxiv.2310.13561arXiv 2310.13561
Open this paper in CoChat — free accountOpen PDF

Abstract

Large-scale deployment of generative AI tools often depends on costly API calls to a Large Language Model (LLM) to fulfil user queries. To curtail the frequency of these calls, one can employ a smaller language model -- a student -- which is continuously trained on the responses of the LLM. This student gradually gains proficiency in independently handling an increasing number of user requests, a process we term neural caching. The crucial element in neural caching is a policy that decides which requests should be processed by the student alone and which should be redirected to the LLM, subsequently aiding the student's learning. In this study, we focus on classification tasks, and we consider a range of classic active learning-based selection criteria as the policy. Our experiments suggest that Margin Sampling and Query by Committee bring consistent benefits across tasks and budgets.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.13561

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Batching, caching and other optimization techniques are routinely used as a means to improve the scaling of AI, allowing more users to interact with ever more powerful technologies.

    “In fact, batching, caching and other optimization techniques are routinely used as a means to improve the scaling of AI, allowing more users to interact with ever more powerful technologies [93, 141].”

    From From Efficiency Gains to Rebound Effects: The Problem of Jevons' Paradox in AI's Polarized Environmental Debate (2025) · cites this paper as [93]

    The Introduction describes neural caching as a way to reduce costly LLM API calls while a smaller model handles an increasing number of requests. The paper does not establish that batching is routinely used, or that these techniques allow more users to access increasingly powerful AI; its evidence is narrower than the citing sentence. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 53

  1. Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. 2020. Pseudo- labeling and confirmation bias in deep semi- supervised learning. In 2020 International Joint Con- ference on Neural Networks (IJCNN). IEEE.
  2. Maria-Florina Balcan, Andrei Broder, and Tong Zhang. 2007. Margin Based Active Learning. In Learning Theory, Lecture Notes in Computer Science, pages 35-50, Berlin, Heidelberg. Springer. · DOI
  3. Fu Bang. 2023. GPTCache: An open-source semantic cache for LLM applications enabling faster answers and cost savings. In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Soft- ware (NLP-OSS 2023), pages 212-218, Singapore, Singapore. Empirical Methods in Natural Language Processing.
  4. Anson Bastos and Manohar Kaul. 2021. ALLWAS: active learning on language models in wasserstein space. CoRR, abs/2109.01691. · arXiv
  5. Cenk Baykal, Khoa Trinh, Fotis Iliopoulos, Gaurav Menghani, and Erik Vee. 2023. Robust active distil- lation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  6. Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020. Continual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Confer- ence on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 6523-6541. International Committee on Com- putational Linguistics. · DOI
  7. Albert Bifet and Ricard Gavaldà. 2007. Learning from time-changing data with adaptive windowing. In Proceedings of the Seventh SIAM International Con- ference on Data Mining, April 26-28, 2007, Min- neapolis, Minnesota, USA, pages 443-448. SIAM. · DOI
  8. Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Proceedings of the Twelfth ACM SIGKDD Interna- tional Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, August 20-23, 2006, pages 535-541. ACM. · DOI · Paper page
  9. Robert Burbidge, Jem J. Rowland, and Ross D. King. 2007. Active Learning for Regression Based on Query by Committee. In Intelligent Data Engineer- ing and Automated Learning -IDEAL 2007, Lecture Notes in Computer Science, pages 209-218, Berlin, Heidelberg. Springer. · DOI
  10. Davide Cacciarelli and Murat Kulahci. 2023. A survey on online active learning. CoRR, abs/2302.08893. · DOI
  11. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. CoRR, abs/2006.14799. · arXiv
  12. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. CoRR, abs/2305.05176. · DOI · Paper page
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics. · DOI · Paper page
  14. Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020. Active learning for BERT: an empirical study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 7949- 7962. Association for Computational Linguistics. · DOI
  15. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-11 November, 2021, pages 6894- 6910. Association for Computational Linguistics. · DOI · Paper page
  16. Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop.
  17. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  18. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mix- tral of experts. CoRR, abs/2401.04088. · DOI · Paper page
  19. Seokhwan Kim, Yu Song, Kyungduk Kim, Jeong-Won Cha, and Gary Geunbae Lee. 2006. MMR-based Ac- tive Machine Learning for Bio Named Entity Recog- nition. In Proceedings of the Human Language Tech- nology Conference of the NAACL, Companion Vol- ume: Short Papers, pages 69-72, New York City, USA. Association for Computational Linguistics.
  20. Kevin J. Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. 2021. Mixkd: Towards efficient distillation of large- scale language models. In 9th International Confer- ence on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net.
  21. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin A Raf- fel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems, volume 35, pages 1950-1965.
  22. Lu Liu and T. Tan. 2021. Certainty driven consistency loss on multi-teacher networks for semi-supervised learning. Pattern Recognition, 120:108140. · DOI
  23. Tong Luo, K. Kramer, S. Samson, A. Remsen, D.B. Goldgof, L.O. Hall, and T. Hopkins. 2004. Active learning to recognize multiple types of plankton. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 3, pages 478-481 Vol.3. ISSN: 1051-4651. · DOI
  24. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct elec- tricity? A new dataset for open book question an- swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 -November 4, 2018, pages 2381-2391. Association for Computational Linguistics. · DOI
  25. Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the As- sociation for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1864-1874. Association for Computational Linguistics. · DOI
  26. Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 188-197, Hong Kong, China. Association for Computational Lin- guistics. · DOI
  27. Bo Pang and Lillian Lee. 2005. Seeing Stars: Exploit- ing Class Relationships for Sentiment Categorization with Respect to Rating Scales. In Proceedings of the 43rd Annual Meeting of the Association for Compu- tational Linguistics (ACL'05), pages 115-124, Ann Arbor, Michigan. Association for Computational Lin- guistics. · DOI
  28. Edoardo Maria Ponti, Alessandro Sordoni, Yoshua Ben- gio, and Siva Reddy. 2023. Combining parameter- efficient modules for task-level generalisation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 687-702, Dubrovnik, Croatia. Associ- ation for Computational Linguistics. · DOI
  29. Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B. Gupta, Xiaojiang Chen, and Xin Wang. 2022. A survey of deep active learn- ing. ACM Comput. Surv., 54(9):180:1-180:40. · DOI · Paper page
  30. Carlos Riquelme. 2017. Online Decision Making for Statistical Model Fitting. Ph.d. thesis, Stanford Uni- versity. Submitted to the Department of Mathemati- cal and Computational Engineering.
  31. Dan Roth and Kevin Small. 2006. Margin-based active learning for structured output spaces. In Machine Learning: ECML 2006, 17th European Conference on Machine Learning, Berlin, Germany, September 18-22, 2006, Proceedings, volume 4212 of Lecture Notes in Computer Science, pages 413-424. Springer. · DOI
  32. Nicholas Roy and Andrew McCallum. 2001. Toward op- timal active learning through sampling estimation of error reduction. In Proceedings of the Eighteenth In- ternational Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 -July 1, 2001, pages 441-448.
  33. Tobias Scheffer, Christian Decomain, and Stefan Wro- bel. 2001. Active Hidden Markov Models for Infor- mation Extraction. In Advances in Intelligent Data Analysis, Lecture Notes in Computer Science, pages 309-318, Berlin, Heidelberg. Springer. · DOI
  34. Greg Schohn and David Cohn. 2000. Less is more: Active learning with support vector machines. In Proceedings of the Seventeenth International Confer- ence on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 -July 2, 2000, pages 839-846.
  35. Morgan Kaufmann.
  36. Christopher Schröder, Andreas Niekler, and Martin Pot- thast. 2022. Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2194-2203, Dublin, Ireland. Association for Computational Linguistics. · DOI
  37. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020. Green AI. Commun. ACM, 63(12):54- 63.
  38. Ozan Sener and Silvio Savarese. 2018. Active learning for convolutional neural networks: A core-set ap- proach. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018, Conference Track Proceed- ings. OpenReview.net.
  39. H. S. Seung, M. Opper, and H. Sompolinsky. 1992. Query by committee. In Proceedings of the fifth annual workshop on Computational learning theory, pages 287-294, Pittsburgh Pennsylvania USA. ACM. · DOI
  40. Bo Shao, Lorna Doucet, and David R. Caruso. 2015. Universality Versus Cultural Specificity of Three Emotion Domains: Some Evidence Based on the Cascading Model of Emotional Intelligence. Jour- nal of Cross-Cultural Psychology, 46(2):229-251. Publisher: SAGE Publications Inc. · DOI
  41. Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. 2023. Large language model routing with benchmark datasets. CoRR, abs/2309.15789. · DOI
  42. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep mod- els for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIG- DAT, a Special Interest Group of the ACL, pages 1631-1642.
  43. ACL.
  44. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. In Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  45. Ilias Stogiannidis, Stavros Vassos, Prodromos Malaka- siotis, and Ion Androutsopoulos. 2023. Cache me if you can: an online cost-aware teacher-student frame- work to reduce the calls to large language models. In Findings of the Association for Computational Lin- guistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 14999-15008. Association for Computa- tional Linguistics.
  46. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 809-819. Association for Computational Linguistics. · DOI · Paper page
  47. Marija Šakota, Maxime Peyrard, and Robert West. 2023. Fly-swat or cannon? cost-effective language model choice via meta-modeling. CoRR, abs/2308.06077. · DOI
  48. Guodong Xu, Ziwei Liu, and Chen Change Loy. 2023. Computation-efficient knowledge distillation via uncertainty-aware mixup. Pattern Recognit., 138:109338. · DOI
  49. Xiangkai Zeng, Sarthak Garg, Rajen Chatterjee, Ud- hyakumar Nallasamy, and Matthias Paulik. 2019. Empirical Evaluation of Active Learning Techniques for Neural MT. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 84-93, Hong Kong, China. Association for Computational Linguistics. · DOI
  50. Xueying Zhan, Qingzhong Wang, Kuan-Hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B. Chan. 2022. A comparative survey of deep active learning. CoRR, abs/2203.13450. · DOI
  51. Zhisong Zhang, Emma Strubell, and Eduard H. Hovy. 2022. A survey of active learning for natural lan- guage processing. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 6166-6190. Association for Computational Linguistics. · DOI
  52. Fan Zhou and Chengtai Cao. 2021. Overcoming catas- trophic forgetting in graph neural networks with ex- perience replay. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 4714-4722. AAAI Press. · DOI
  53. Banghua Zhu, Ying Sheng, Lianmin Zheng, Clark W. Barrett, Michael I. Jordan, and Jiantao Jiao. 2023. On optimal caching and model multiplexing for large model inference. CoRR, abs/2306.02003. · DOI

Cited by

Sorted by citations

Metadata

Authors
Ramírez, G., Lindemann, M., Birch, A., & Titov, I.
Affiliations
  • University of Edinburgh (GB)
  • University of Amsterdam (NL)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
11838-11853
DOI
10.48550/arxiv.2310.13561
arXiv
2310.13561
OpenAlex
W4402670353
Cited by
4
References
0
Citations by year
2026: 2 · 2025: 1 · 2024: 1
FWCI
1.15
Type
conference paper
Language
en
Open access
Licence
Locations
  • Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • UvA-DARE (University of Amsterdam) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (100010434)
  • UK Research and Innovation (EP/T022221/1, EP/S022481/1, EP/W032244/1)
  • European Commission (100010434, 101070631, ID 100010434)
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek (VI.C.212.053, 100010434)
  • Fundación Bancaria Caixa d'Estalvis i Pensions de Barcelona (ID 100010434, 100010434)
  • HORIZON EUROPE Framework Programme
  • Engineering and Physical Sciences Research Council (100010434, EP/S022481/1, EP/P020267/1, EP/W032244/1, EP/T022221/1)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, student model, active learning (machine learning), margin sampling, query-by-committee, classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review