Lazy Batching: An SLA-aware Batching System for Cloud Machine Learning Inference
Open accessAbstract
In cloud ML inference systems, batching is an essential technique to increase throughput which helps optimize total-cost-of-ownership. Prior graph batching combines the individual DNN graphs into a single one, allowing multiple inputs to be concurrently executed in parallel. We observe that the coarse-grained graph batching becomes suboptimal in effectively handling the dynamic inference request traffic, leaving significant performance left on the table. This paper proposes LazyBatching, an SLA-aware batching system that considers both scheduling and batching in the granularity of individual graph nodes, rather than the entire graph for flexible batching. We show that LazyBatching can intelligently determine the set of nodes that can be efficiently batched together, achieving an average 15×, 1.5×, and 5.5 × improvement than graph batching in terms of average response time, throughput, and SLA satisfaction, respectively.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Serving-related challenges are some of the ones most often reported in the literature, usually from a performance perspective.
“Serving-related challenges are some of the ones most often reported in the literature [5-7, 9, 11, 12, 16, 20, 28, 34, 44, 53, 54], usually from a performance perspective.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [11]
An adaptive batching scheme specifically for neural networks was proposed, reducing latency during low-traffic scenarios.
“while [11] proposed an adaptive batching scheme specifically for neural networks, reducing latency during low-traffic scenarios.”
From Operationalizing machine learning models: a systematic literature review (2022) · cites this paper as [11]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · see citing works
Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks
Chen, Y.-H., Krishna, T., Emer, J. S., & Sze, V. (2017) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2016.2616357 · Cited by 3,247 · see citing works
EIE: efficient inference engine on compressed deep neural network
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., & Dally, W. J. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001163 · Cited by 1,937 · see citing works
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
Chen, T., Li, M., Li, Y., Lin, M., Wang, N., Wang, M., Xiao, T., Xu, B., Zhang, C., & Zhang, Z. (2015) · arXiv · doi:10.48550/arxiv.1512.01274 · Cited by 1,819 · see citing works
DaDianNao: A Machine-Learning Supercomputer
Chen, Y., Luo, T., Liu, S., Zhang, S., He, L., Wang, J., Li, L., Chen, T., Xu, Z., Sun, N., et al. (2014) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2014.58 · Cited by 1,532 · see citing works
PipeDream: generalized pipeline parallelism for DNN training
Narayanan, D., Harlap, A., Phanishayee, A., Seshadri, V., Devanur, N. R., Ganger, G., Gibbons, P. B., & Zaharia, M. (2019) · Symposium on Operating Systems Principles · doi:10.1145/3341301.3359646 · Cited by 1,362 · see citing works
Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., & Temam, O. (2014) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/2541940.2541967 · Cited by 1,314 · see citing works
Du, Z., Fasthuber, R., Chen, T., Ienne, P., Li, L., Luo, T., Feng, X., Chen, Y., & Temam, O. (2015) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/2749469.2750389 · Cited by 928 · see citing works
Parashar, A., Rhu, M., Mukkara, A., Puglielli, A., Venkatesan, R., Khailany, B., Emer, J., Keckler, S. W., & Dally, W. J. (2017) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3079856.3080254 · Cited by 899 · see citing works
Self-citation14.5 Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks
Chen, Y., Krishna, T., Emer, J., & Sze, V. (2016) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc.2016.7418007 · Cited by 713 · see citing works
Albericio, J., Judd, P., Hetherington, T., Aamodt, T. M., Jerger, N. E., & Moshovos, A. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001138 · Cited by 614 · see citing works
Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective
Hazelwood, K., Bird, S., Brooks, D., Chintala, S., Diril, U., Dzhulgakov, D., Fawzy, M., Jia, B., Jia, Y., Kalro, A., et al. (2018) · 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2018.00059 · Cited by 590 · see citing works
Gated Feedback Recurrent Neural Networks
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.02367 · Cited by 583 · see citing works
Optimus: an efficient dynamic resource scheduler for deep learning clusters
Peng, Y.-H., Bao, Y., Chen, Y.-R., Wu, C., & Guo, C.-X. (2018) · European Conference on Computer Systems · doi:10.1145/3190508.3190517 · Cited by 544 · see citing works
A Configurable Cloud-Scale DNN Processor for Real-Time AI
Fowers, J., Ovtcharov, K., Papamichael, M. K., Massengill, T., Liu, M., Lo, D., Alkalay, S., Haselman, M., Adams, L., Ghandi, M., et al. (2018) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2018.00012 · Cited by 529 · see citing works
Massive Exploration of Neural Machine Translation Architectures
Britz, D., Goldie, A., Luong, M.-T., & Le, Q. V. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1151 · Cited by 487 · see citing works
Massive Exploration of Neural Machine Translation Architectures
Britz, D., Goldie, A., Luong, M.-T., & Le, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.03906 · Cited by 487 · see citing works
From high-level deep neural models to FPGAs
Sharma, H., Park, J., Mahajan, D., Amaro, E., Kim, J. K., Shao, C., Mishra, A., & Esmaeilzadeh, H. (2016) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783720 · Cited by 487 · see citing works
Can FPGAs Beat GPUs in Accelerating Next-Generation Deep Neural Networks?
Nurvitadhi, E., Venkatesh, G., Sim, J., Marr, D., Huang, R., Ong Gee Hock, J., Liew, Y. T., Srivatsan, K., Moss, D., Subhaschandra, S., et al. (2017) · Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays · doi:10.1145/3020078.3021740 · Cited by 467 · see citing works
Minerva: enabling low-power, highly-accurate deep neural network accelerators
Reagen, B., Whatmough, P., Adolf, R., Rama, S., Lee, H., Lee, S. K., Hernández-Lobato, J. M., Wei, G.-Y., & Brooks, D. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001165 · Cited by 446 · see citing works
ATLAS: A scalable and high-performance scheduling algorithm for multiple memory controllers
Kim, Y., Han, D., Mutlu, O., & Harchol‐Balter, M. (2010) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2010.5416658 · Cited by 433 · see citing works
Self-citationGao, M., Pu, J., Yang, X., Horowitz, M., & Kozyrakis, C. (2017) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3037697.3037702 · Cited by 421 · see citing works
vDNN: Virtualized deep neural networks for scalable, memory-efficient neural network design
Rhu, M., Gimelshein, N. E., Clemons, J., Zulfiqar, A., & Keckler, S. W. (2016) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783721 · Cited by 416 · see citing works
Self-citationThread Cluster Memory Scheduling: Exploiting Differences in Memory Access Behavior
Kim, Y., Papamichael, M. K., Mutlu, O., & Harchol‐Balter, M. (2010) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2010.51 · Cited by 408 · see citing works
Self-citationJointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services
Eshratifar, A. E., Abrishami, M. S., & Pedram, M. (2019) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2019.2947893 · Cited by 383 · see citing works
Stripes: Bit-serial deep neural network computing
Judd, P., Albericio, J., Hetherington, T., Aamodt, T. M., & Moshovos, A. (2016) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro.2016.7783722 · Cited by 355 · see citing works
Parallelism-Aware Batch Scheduling: Enhancing both Performance and Fairness of Shared DRAM Systems
Mutlu, O., & Moscibroda, T. (2008) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2008.7 · Cited by 346 · see citing works
Kim, D.-H., Kung, J., Chai, S. M., Yalamanchili, S., & Mukhopadhyay, S. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001178 · Cited by 342 · see citing works
Listen, Attend and SpellPreprint
Chan, W., Jaitly, N., Le, Q. V., & Vinyals, O. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.01211 · Cited by 300 · see citing works
Beyond Data and Model Parallelism for Deep Neural NetworksPreprint
Jia, Z., Zaharia, M., & Aiken, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.05358 · Cited by 297 · see citing works
Bit-pragmatic deep neural network computing
Albericio, J., Delmás, A., Judd, P., Sharify, S., O’Leary, G., Genov, R., & Moshovos, A. (2017) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3123939.3123982 · Cited by 254 · see citing works
Liu, D., Chen, T., Liu, S., Zhou, J., Zhou, S., Teman, O., Feng, X., Zhou, X., & Chen, Y. (2015) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/2694344.2694358 · Cited by 247 · see citing works
Liu, S., Du, Z., Tao, J., Han, D. S., Luo, T., Xie, Y., Chen, Y., & Chen, T. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001179 · Cited by 243 · see citing works
Shen, H., Chen, L., Jin, Y., Zhao, L., Kong, B., Philipose, M., Krishnamurthy, A., & Sundaram, R. (2019) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3341301.3359658 · Cited by 238 · see citing works
Kwon, Y., Lee, Y., & Rhu, M. (2019) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3352460.3358284 · Cited by 233 · see citing works
Self-citationFlexSC: flexible system call scheduling with exception-less system calls
Soares, L., & Stumm, M. (2010) · doi:10.5555/1924943.1924946 · Cited by 225 · see citing works
SCALE-Sim: Systolic CNN Accelerator SimulatorPreprint
Samajdar, A., Zhu, Y., Whatmough, P. N., Mattina, M., & Krishna, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.02883 · Cited by 213 · see citing works
LiKamWa, R., Hou, Y., Gao, J., Polansky, M. G., & Zhong, L. (2016) · ACM SIGARCH Computer Architecture News · doi:10.1145/3007787.3001164 · Cited by 189 · see citing works
Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks
Rhu, M., O’Connor, M., Chatterjee, N., Pool, J., Kwon, Y., & Keckler, S. W. (2018) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2018.00017 · Cited by 183 · see citing works
Self-citationWhatmough, P. N., Lee, S. K., Lee, H., Rama, S., Brooks, D. M., & Wei, G.-Y. (2017) · Digest of technical papers/Digest of technical papers - IEEE International Solid-State Circuits Conference · doi:10.1109/isscc.2017.7870351 · Cited by 165 · see citing works
TABLA: A unified template-based framework for accelerating statistical machine learning
Mahajan, D., Park, J., Amaro, E., Sharma, H., Yazdanbakhsh, A., Kim, J. K., & Esmaeilzadeh, H. (2016) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca.2016.7446050 · Cited by 156 · see citing works
Kannan, R. S., Subramanian, L., Raju, A., Ahn, J., Mars, J., & Tang, L. (2019) · European Conference on Computer Systems · doi:10.1145/3302424.3303958 · Cited by 152 · see citing works
PREMA: A Predictive Multi-Task Scheduling Algorithm For Preemptible Neural Processing Units
Choi, Y., & Rhu, M. (2020) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca47549.2020.00027 · Cited by 152 · see citing works
Self-citationMegaPipe: a new programming interface for scalable network I/O
Han, S., Marshall, S., Chun, B.-G., & Ratnasamy, S. (2012) · doi:10.5555/2387880.2387894 · Cited by 139 · see citing works
Shenango: Achieving High {CPU} Efficiency for Latency-sensitive Datacenter Workloads
Ousterhout, A., Fried, J., Behrens, J., Belay, A., & Balakrishnan, H. (2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 139 · see citing works
Accelerating Deep Convolutional Networks using low-precision and sparsity
Venkatesh, G., Nurvitadhi, E., & Marr, D. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952679 · Cited by 123 · see citing works
Centaur: A Chiplet-based, Hybrid Sparse-Dense Accelerator for Personalized Recommendations
Hwang, R., Kim, T., Kwon, Y., & Rhu, M. (2020) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca45697.2020.00083 · Cited by 104 · see citing works
Self-citationBatch-Aware Unified Memory Management in GPUs for Irregular Workloads
Kim, H., Sim, J., Gera, P., Hadidi, R., & Kim, H. (2020) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3373376.3378529 · Cited by 91 · see citing works
Du, Z., Rubin, D. B. D., Chen, Y., He, L., Chen, T., Zhang, L., Wu, C., & Temam, O. (2015) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/2830772.2830789 · Cited by 81 · see citing works
A Customizable Matrix Multiplication Framework for the Intel HARPv2 Xeon+FPGA Platform
Moss, D. J. M., Krishnan, S., Nurvitadhi, E., Ratuszniak, P., Johnson, C., Sim, J., Mishra, A., Marr, D., Subhaschandra, S., & Leong, P. H. W. (2018) · ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA) · doi:10.1145/3174243.3174258 · Cited by 78 · see citing works
Low latency RNN inference with cellular batching
Gao, P., Yu, L., Wu, Y., & Li, J. (2018) · Proceedings of the Thirteenth EuroSys Conference · doi:10.1145/3190508.3190541 · Cited by 66 · see citing works
Scheduling Page Table Walks for Irregular GPU Applications
Shin, S., Cox, G., Oskin, M., Loh, G. H., Solihin, Y., Bhattacharjee, A., & Basu, A. (2018) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca.2018.00025 · Cited by 61 · see citing works
Tensor Casting: Co-Designing Algorithm-Architecture for Personalized Recommendation Training
Kwon, Y., Lee, Y., & Rhu, M. (2021) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca51647.2021.00029 · Cited by 42 · see citing works
Self-citationA case for user-level dynamic page migration
Nikolopoulos, D. S., Papatheodorou, T. S., Polychronopoulos, C. D., Labarta, J., & Ayguadé, E. (2000) · International Conference on Supercomputing · doi:10.1145/335231.335243 · Cited by 35 · see citing works
Hyun, B., Kwon, Y., Choi, Y., Kim, J., & Rhu, M. (2020) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3373376.3378494 · Cited by 28 · see citing works
Self-citationA Case for Memory-Centric HPC System Architecture for Training Deep Neural Networks
Kwon, Y., & Rhu, M. (2018) · IEEE Computer Architecture Letters · doi:10.1109/lca.2018.2823302 · Cited by 21 · see citing works
Self-citationDelmás, A., Judd, P., Stuart, D. M., Poulos, Z., Mahmoud, M., Sharify, S., Nikolić, M., & Moshovos, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.03688 · Cited by 20 · see citing works
A Disaggregated Memory System for Deep Learning
Kwon, Y., & Rhu, M. (2019) · IEEE Micro · doi:10.1109/mm.2019.2929165 · Cited by 20 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
4 of the 69 references couldn’t be matched to a known paper.
Research works printed but not linked: 23
- J. Albericio, P. Judd, T. Hetherington, T. Aamodt, N. E. Jerger, and A. Moshovos, "Cnvlutin: Ineffectual-Neuron-Free Deep Convolutional Neural Network Computing," in Proceedings of the International Symposium on Computer Architecture (ISCA), June 2016.
- D. Amodei, S. Ananthanarayanan, R. Anubhai, J. Bai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, Q. Cheng, G. Chen et al., "Deep Speech 2: End-to-end Speech Recognition in English and Mandarin," in International conference on machine learning, 2016, pp. 173-182.
- T. Chen, Z. Du, N. Sun, J. Wang, C. Wu, Y. Chen, and O. Temam, "DianNao: A Small-footprint High-throughput Accelerator for Ubiqui- tous Machine-learning," in Proceedings of the International Conference on Architectural Support for Programming Languages and Operation Systems (ASPLOS), March 2014.
- P. Chi, S. Li, C. Xu, T. Zhang, J. Zhao, Y. Liu, Y. Wang, and Y. Xie, "A Novel Processing-in-memory Architecture for Neural Network Computation in ReRAM-based Main Memory," in Proceedings of the International Symposium on Computer Architecture (ISCA), June 2016.
- --, "PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible Neural Processing Units," in Proceedings of the In- ternational Symposium on High-Performance Computer Architecture (HPCA), 2020.
- Z. Du, R. Fasthuber, T. Chen, P. Ienne, L. Li, T. Luo, X. Feng, Y. Chen, and O. Temam, "ShiDianNao: Shifting Vision Processing Closer to the Sensor," in Proceedings of the International Symposium on Computer Architecture (ISCA), June 2015.
- M. Gao, J. Pu, X. Yang, M. Horowitz, and C. Kozyrakis, "TETRIS: Scalable and Efficient Neural Network Acceleration with 3D Memory," in Proceedings of the International Conference on Architectural Sup- port for Programming Languages and Operation Systems (ASPLOS), 2017.
- B. Hyun, Y. Kwon, Y. Choi, J. Kim, and M. Rhu, "NeuMMU: Architectural Support for Efficient Address Translations in Neural Processing Units," in Proceedings of the International Conference on Architectural Support for Programming Languages and Operation Systems (ASPLOS), 2020.
- R. S. Kannan, L. Subramanian, A. Raju, J. Ahn, J. Mars, and L. Tang, "GrandSLAm: Guaranteeing SLAs for Jobs in Microservices Execution Frameworks," in Proceedings of the Fourteenth EuroSys Conference 2019, 2019, pp. 1-16.
- D. Kim, J. Kung, S. Chai, S. Yalamanchili, and S. Mukhopadhyay, "Neurocube: A Programmable Digital Neuromorphic Architecture with High-Density 3D Memory," in Proceedings of the International Sym- posium on Computer Architecture (ISCA), June 2016.
- Y. Kwon, Y. Lee, and M. Rhu, "TensorDIMM: A Practical Near- Memory Processing Architecture for Embeddings and Tensor Opera- tions in Deep Learning," in Proceedings of the International Symposium on Microarchitecture (MICRO), 2019.
- --, "Beyond the Memory Wall: A Case for Memory-Centric HPC System for Deep Learning," in Proceedings of the International Sym- posium on Microarchitecture (MICRO), 2018.
- R. LiKamWa, Y. Hou, M. Polansky, Y. Gao, and L. Zhong, "RedEye: Analog ConvNet Image Sensor Architecture for Continuous Mobile Vision," in Proceedings of the International Symposium on Computer Architecture (ISCA), June 2016.
- D. Liu, T. Chen, S. Liu, J. Zhou, S. Zhou, O. Temam, X. Feng, X. Zhou, and Y. Chen, "PuDianNao: A Polyvalent Machine Learning Accelerator," in Proceedings of the International Conference on Archi- tectural Support for Programming Languages and Operation Systems (ASPLOS), April 2015.
- S. Liu, Z. Du, J. Tao, D. Han, T. Luo, Y. Xie, Y. Chen, and T. Chen, "Cambricon: An Instruction Set Architecture for Neural Networks," in Proceedings of the International Symposium on Computer Architecture (ISCA), 2016.
- MIT Technology Review, "Why Facebook Want to Design Its Own AI Chips," 2018.
- D. Moss, E. Nurvitadhi, J. Sim, A. Mishra, D. Marr, S. Subhaschandra, and P. Leong, "High Performance Binary Neural Networks on the Xeon+FPGA Platform," in Proceedings of the International Conference on Field Programmable Logic and Applications (FPL), 2017.
- A. Parashar, M. Rhu, A. Mukkara, A. Puglielli, R. Venkatesan, B. Khailany, J. Emer, S. W. Keckler, and W. J. Dally, "SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks," in Proceedings of the International Symposium on Computer Architecture (ISCA), June 2017.
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language Models Are Unsupervised Multitask Learners," OpenAI Blog, vol. 1, no. 8, 2019.
- A. Shafiee, A. Nag, N. Muralimanohar, R. Balasubramonian, J. P. Strachan, M. Hu, R. S. Williams, and V. Srikumar, "ISAAC: A Convo- lutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars," in Proceedings of the International Symposium on Computer Architecture (ISCA), June 2016.
- H. Shen, L. Chen, Y. Jin, L. Zhao, B. Kong, M. Philipose, A. Kr- ishnamurthy, and R. Sundaram, "Nexus: A GPU Cluster Engine for Accelerating DNN-based Video Analysis," in Proceedings of the 27th ACM Symposium on Operating Systems Principles. ACM, 2019, pp. 322-337.
- P. Whatmough, S. Lee, N. Mulholland, P. Hansen, S. Kodali, D. Brooks, and G. Wei, "DNN ENGINE: A 16nm Sub-uJ Deep Neural Network Inference Accelerator for the Embedded Masses," in Hot Chips: A Symposium on High Performance Chips, August 2017.
- S. Zhang, Z. Du, L. Zhang, H. Lan, S. Liu, L. Li, Q. Guo, T. Chen, and Y. Chen, "Cambricon-X: An Accelerator for Sparse Neural Networks," in Proceedings of the International Symposium on Microarchitecture (MICRO), October 2016.
Cited by
Sorted by citations
OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization
Guo, C., Tang, J., Hu, W., Leng, J., Zhang, C., Yang, F., Liu, Y., Guo, M., & Zhu, Y. (2023) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3579371.3589038 · Cited by 162 · see citing works
ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization
Guo, C., Zhang, C., Leng, J., Liu, Z., Yang, F., Liu, Y., Guo, M., & Zhu, Y. (2022) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro56248.2022.00095 · Cited by 116 · see citing works
Resource Allocation for Multiuser Edge Inference With Batching and Early Exiting
Liu, Z., Lan, Q., & Huang, K. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2023.3242724 · Cited by 82 · see citing works
Beyond the Cloud: Edge Inference for Generative Large Language Models in Wireless Networks
Zhang, X., Nie, J.-T., Huang, Y., Xie, G.-C., Xiong, Z.-H., Liu, J., Niyato, D., & Shen, X. (2025) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2024.3497923 · Cited by 104 · see citing works
Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service
Li, B., Samsi, S., Gadepally, V. N., & Tiwari, D. (2023) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3581784.3607034 · Cited by 65 · see citing works
Liu, Z., Leng, J., Zhang, Z., Chen, Q., Li, C., & Guo, M. (2022) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3503222.3507752 · Cited by 54 · see citing works
Multiuser Co-Inference With Batch Processing Capable Edge Server
Shi, W., Zhou, S., Niu, Z., Jiang, M., & Geng, L. (2022) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2022.3192613 · Cited by 51 · see citing works
Gao, H., Qiu, B., Wang, Y., Si, Y., Xu, Y., & Wang, X. (2023) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2023.3339633 · Cited by 49 · see citing works
Tensor Casting: Co-Designing Algorithm-Architecture for Personalized Recommendation Training
Kwon, Y., Lee, Y., & Rhu, M. (2021) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca51647.2021.00029 · Cited by 42 · see citing works
Self-citationFA2: Fast, Accurate Autoscaling for Serving Deep Learning Inference with SLA Guarantees
Razavi, K., Luthra, M., Koldehofe, B., Mühlhäuser, M., & Wang, L. (2022) · IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS) · doi:10.1109/rtas54340.2022.00020 · Cited by 39 · see citing works
Liu, W., Geng, J., Zhu, Z., Zhao, Y., Ji, C., Li, C., Lian, Z., & Zhou, X. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3314388 · Cited by 36 · see citing works
Operationalizing machine learning models: a systematic literature review
Kolltveit, A. B., & Li, J. (2022) · Proceedings of the 1st Workshop on Software Engineering for Responsible AI · doi:10.1145/3526073.3527584 · Cited by 34 · see citing works
PArtNNer: Platform-Agnostic Adaptive Edge-Cloud DNN Partitioning for Minimizing End-to-End Latency
Ghosh, S. K., Raha, A., Raghunathan, V., & Raghunathan, A. (2023) · ACM Transactions on Embedded Computing Systems · doi:10.1145/3630266 · Cited by 35 · see citing works
Shen, S., Ren, Y., Ju, Y., Wang, X., Wang, W., & Leung, V. C. M. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3229444 · Cited by 33 · see citing works
Proteus: A High-Throughput Inference-Serving System with Accuracy Scaling
Ahmad, S., Guan, H., Friedman, B. D., Williams, T., Sitaraman, R. K., & Woo, T. L. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3617232.3624849 · Cited by 31 · see citing works
Machine learning inference serving models in serverless computing: a survey
Aslani, A., & Ghobaei‐Arani, M. (2025) · Computing · doi:10.1007/s00607-024-01377-9 · Cited by 31 · see citing works
Guo, C., Zhang, R., Xu, J., Leng, J., Liu, Z., Huang, Z., Guo, M., Wu, H., Zhao, S., Zhao, J., et al. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3620665.3640423 · Cited by 32 · see citing works
InSS: An Intelligent Scheduling Orchestrator for Multi-GPU Inference With Spatio-Temporal Sharing
Han, Z., Zhou, R., Xu, C., Zeng, Y., & Zhang, R. (2024) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2024.3430063 · Cited by 28 · see citing works
Demystifying BERT: System Design Implications
Pati, S., Aga, S., Jayasena, N., & Sinclair, M. D. (2022) · IEEE International Symposium on Workload Characterization · doi:10.1109/iiswc55918.2022.00033 · Cited by 24 · see citing works
Hercules: Heterogeneity-Aware Inference Serving for At-Scale Personalized Recommendation
Ke, L., Gupta, U., Hempstead, M., Wu, C.-J., Lee, H.-H. S., & Zhang, X. (2022) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca53966.2022.00019 · Cited by 26 · see citing works
CD-MSA: Cooperative and Deadline-Aware Scheduling for Efficient Multi-Tenancy on DNN Accelerators
Wang, C., Bai, Y., & Sun, D. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2023.3276759 · Cited by 18 · see citing works
Bang, J., Choi, Y., Kim, M., Kim, Y., & Rhu, M. (2024) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1109/micro61859.2024.00021 · Cited by 17 · see citing works
Self-citationSMSS: Stateful Model Serving in Metaverse With Serverless Computing and GPU Sharing
Cai, Z., Chen, Z., Ma, R., & Guan, H. (2023) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2023.3345401 · Cited by 14 · see citing works
TCB: Accelerating Transformer Inference Services with Request Concatenation
Fu, B., Chen, F., Li, P., & Zeng, D. (2022) · Proceedings of the International Conference on Parallel Processing · doi:10.1145/3545008.3545052 · Cited by 13 · see citing works
SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms
Xu, Y., Sun, J., Zhou, S., & Niu, Z. (2023) · International Conference on Communications · doi:10.1109/icc45041.2023.10278962 · Cited by 12 · see citing works
Li, B., Samsi, S., Gadepally, V. N., & Tiwari, D. (2023) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3588195.3592997 · Cited by 12 · see citing works
Multi-model Machine Learning Inference Serving with GPU Spatial PartitioningPreprint
Choi, S., Lee, S., Kim, Y., Park, J., Kwon, Y., & Huh, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.01611 · Cited by 11 · see citing works
Self-citationApparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving
Dai, Y., Pan, R., Iyer, A. P., Li, K., & Netravali, R. (2024) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3694715.3695963 · Cited by 9 · see citing works
Khan, M. A., Khan, S. M., & Subramaniam, S. K. (2023) · IEEE Access · doi:10.1109/access.2023.3339378 · Cited by 9 · see citing works
Razavi, K., Ghafouri, S., Mühlhäuser, M., Jamshidi, P., & Wang, L. (2024) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3642970.3655833 · Cited by 8 · see citing works
QoS-Aware Irregular Collaborative Inference for Improving Throughput of DNN Services
Fu, K., Shi, J., Chen, Q., Zheng, N., Zhang, W., Zeng, D., & Guo, M. (2022) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/sc41404.2022.00074 · Cited by 7 · see citing works
SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services
Xu, Y., Zhou, S., & Niu, Z. (2025) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2025.3526283 · Cited by 6 · see citing works
Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures
Vellaisamy, P., Labonte, T., Chakraborty, S., Turner, M., Sury, S., & Shen, J. P. (2025) · IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) · doi:10.1109/ispass64960.2025.00015 · Cited by 6 · see citing works
Efficient Adaptive Batching of DNN Inference Services for Improved Latency
Khan, O., Yu, J., Kim, Y., & Seo, E. (2024) · International Conference on Information Networking · doi:10.1109/icoin59985.2024.10572152 · Cited by 6 · see citing works
Self-citationGACER: Granularity-Aware ConcurrEncy Regulation for Multi-Tenant Deep Learning
Yu, Y., Yu, F., Tian, Z., & Chen, X. (2024) · doi:10.1145/3676536.3676718 · Cited by 5 · see citing works
CAMDNN: Content-Aware Mapping of a Network of Deep Neural Networks on Edge MPSoCs
Heidari, S., Ghasemi, M., Kim, Y. G., Wu, C.-J., & Vrudhula, S. B. K. (2022) · IEEE Transactions on Computers · doi:10.1109/tc.2022.3207137 · Cited by 5 · see citing works
Self-citationOnline Resource Provisioning and Batch Scheduling for AIoT Inference Serving in an XPU Edge Cloud
Liu, R., Wu, Y., Zhao, K., Zhou, Z., Gao, X., Lin, X., Zhang, X., Chen, X., & Lu, G. (2024) · IEEE Transactions on Emerging Topics in Computing · doi:10.1109/tetc.2024.3403874 · Cited by 5 · see citing works
Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism
Mo, Z., Liao, J., Xu, H., Zhou, Z., & Xu, C. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759784 · Cited by 4 · see citing works
Huang, J., Wu, J., Wu, Y., & Wu, J. (2025) · IEEE Transactions on Network Science and Engineering · doi:10.1109/tnse.2025.3551148 · Cited by 4 · see citing works
A Sample-Free Compilation Framework for Efficient Dynamic Tensor Computation
Zhou, Y., Zhu, H., Qiu, Q., Cui, W., Liu, Z., Chen, P., Wahib, M., Guo, C., Feng, S. M., Meng, J., et al. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759779 · Cited by 3 · see citing works
Resource Allocation for Batched Multiuser Edge Inference with Early Exiting
Liu, Z., Lan, Q., & Huang, K. (2023) · International Conference on Communications · doi:10.1109/icc45041.2023.10278846 · Cited by 3 · see citing works
Eco-Friendly AI: A Guide to Energy-Efficient Techniques Across the AI Life-Cycle
Meemken, B., & Poth, A. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-71142-8_3 · Cited by 3 · see citing works
PULSE: Using Mixed-Quality Models for Reducing Serverless Keep-Alive Cost
Sankaranarayanan, K., Roy, R. B., & Tiwari, D. (2024) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1109/scw63240.2024.00021 · Cited by 3 · see citing works
Carat: Unlocking Value-Level Parallelism for Multiplier-Free GEMMs
Pan, Z., Miguel, J. S., & Wu, D. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3620665.3640364 · Cited by 6 · see citing works
Provisioning high precision edge inference with runtime model reconfiguration
Sun, H., Qian, Z., Zhu, A., Zhang, S., Lu, S., Chu, X., & Luo, G. (2025) · Computer Networks · doi:10.1016/j.comnet.2025.111585 · Cited by 3 · see citing works
Characterizing and understanding deep neural network batching systems on GPUs
Yu, F., Zhang, H., Chen, A., Wang, X., Liang, X., Wang, S., Li, G., Cui, H., & Feng, X. (2023) · BenchCouncil Transactions on Benchmarks Standards and Evaluations · doi:10.1016/j.tbench.2024.100151 · Cited by 2 · see citing works
Zhang, Q., Liu, Y., Liu, T., & Qian, D. (2023) · The Journal of Supercomputing · doi:10.1007/s11227-023-05183-6 · Cited by 2 · see citing works
CBA Sketch: A Sketching Algorithm Mining Persistent Batches in Data Streams
Zhou, Q., Sun, Y.-E., Huang, H., & Han, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-0811-6_7 · Cited by 2 · see citing works
Efficient Batched Inference in Conditional Neural Networks
Selvam, S., Nagarajan, A., & Raghunathan, A. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3445263 · Cited by 2 · see citing works
Serving Transformer Models via Joint Requst Scheduling and Batching in the Network Edge
Fu, B., Chen, F., Li, P., & Zeng, D. (2025) · IEEE Transactions on Sustainable Computing · doi:10.1109/tsusc.2025.3528105 · Cited by 2 · see citing works
mSIRM: Cost-Efficient and SLO-aware ML Load Balancing on Fog and Multi-Cloud Network
Phalak, C., Chahal, D., Ramesh, M., & Singhal, R. (2023) · doi:10.1145/3589013.3596676 · Cited by 2 · see citing works
Adaptive Scheduling for Edge-Assisted DNN Serving
He, J., Yang, C., He, Z., Baig, G., & Qiu, L. (2023) · IEEE International Conference on Mobile Ad-Hoc and Sensor Systems (MASS) · doi:10.1109/mass58611.2023.00071 · Cited by 1 · see citing works
An Empirical Design and Implementation of Job Scheduling Enhancement for Kubernetes Clusters
Duong, B. V., Phung-Ha, D., Lee, J.-W., & Kim, Y. (2024) · International Conference on Information Networking · doi:10.1109/icoin59985.2024.10572101 · Cited by 1 · see citing works
Self-citationAuto Batching Scheme for Optimizing LSTM Inference on FPGA Platforms
Kim, B. J., & Chung, E.-Y. (2024) · IEEE Access · doi:10.1109/access.2024.3488033 · Cited by 1 · see citing works
QMLFQ: A Quantization-based Queue Scheduling Framework for Efficient LLM Serving
Wu, J., Dai, H., Wu, J., Jin, W., & Wang, Y. (2024) · IEEE Smart World Congress (SWC) · doi:10.1109/swc62898.2024.00208
Accordion: A malleable pipeline scheduling approach for adaptive SLO-aware inference serving
Soomro, P. N., Παπαδοπούλου, Ν., & Pericàs, M. (2025) · ACM International Conference on Computing Frontiers (CF) · doi:10.1145/3719276.3725190
Provably Efficient Online Batch Scheduling for Deep Learning Inference
Tang, B., Xie, Y., Chen, S., Zhou, S., & Ye, B. (2024) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc64274.2024.00115
TopServe: Task-Operator Co-scheduling for Efficient Multi-DNN Inference Serving on GPUs
Chen, A., Li, G., Yu, F., Wang, X., Zhao, J., Cui, H., Feng, X., & Xue, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-99857-7_21
BATCH-DNN: Adaptive and Dynamic Batching for Multi-DNN Accelerators
Ranawaka, P., & Stenström, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-99857-7_8
Joint Optimization of Offloading, Batching and DVFS for Multiuser Co-Inference
Xu, Y., Zhou, S., & Niu, Z. (2025) · International Conference on Communications · doi:10.1109/icc52391.2025.11161748
Compile-Time QoS Scheme for Deep Learning Inferences
Hong, S., Kim, H., & Han, H. (2025) · International Conference for High Performance Computing, Networking, Storage and Analysis (SC) · doi:10.1145/3712285.3759846
Sun, J., Hao, H., Shi, H., Su, Z., Li, H., & Zhang, W. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229166
ZipBatch: Multi-Tenant GPU Batching with Dual-Resource Regulation
Yu, H., Yao, S., & Wang, W. (2025) · ACM Symposium on Cloud Computing (SoCC) · doi:10.1145/3772052.3772229
HyDLR: Load-Aware Dynamic Rescheduling for Deep Learning Hybrid Deployment
Wang, D., Sun, X. Y., Si, S., Chen, S., & Zhang, W. (2025) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads67057.2025.11323176
SwiftSNNI: Optimized Scheduling for Secure Neural Network Inference (SNNI) on Multi-Core Systems
Batool, K., Anwar, S., Regazzoni, F., Pimentel, A. D., & Mann, Z. Á. (2026) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3777884.3797005
PARD: Enhancing Goodput for Inference Pipeline via Proactive Request Dropping
Zhao, Z., Hu, Y., Chen, S. H., Ji, M., Yang, W., Zhang, Y., Zhao, L., Li, W., Liu, X., Qu, W., et al. (2026) · European Conference on Computer Systems · doi:10.1145/3767295.3803581
GACER-M: Granularity-Aware and Memory-Centric Coordination for Multi-Tenant Deep Learning
Yu, Y., Yu, F., Yang, J., Cheng, X., Liu, C., & Chen, X. Y. (2026) · IEEE Transactions on Computers · doi:10.1109/tc.2026.3701443
SMOOTH: Scalable Multitask Offloading with Backbone Sharing
Geng, W., Su, X., Mohan, N., Ott, J., & Hui, P. (2026) · IFIP Networking Conference (IFIP Networking) · doi:10.23919/ifipnetworking70592.2026.11578439
Janus: Realizing Practical Operator Parallelism for Latency-Sensitive DNN Inference on GPUs
Zhang, Y., Ma, H., Long, Y., Gao, L., Zhang, W., & Wang, R. (2026) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3838190
- Authors
- Choi, Y., Kim, Y., & Rhu, M.
- Affiliations
- Korea Advanced Institute of Science and Technology (KR)
- Venue
- 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA)
- Publisher
- IEEE
- Published
- 2021-02
- Pages
- 493-506
- ISSN
- 1530-0897 (ISSN-L)
- OpenAlex
- W3104481311
- Cited by
- 69
- References
- 69
- Citations by year
- 2026: 5 · 2025: 16 · 2024: 19 · 2023: 16 · 2022: 11 · 2021: 2
- FWCI
- 9.95
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE International Symposium on High-Performance Computer Architecture (HPCA) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Research Foundation
- KAIST
- Topics
- Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Keywords
- deep neural network inference, throughput maximization, response time, total cost of ownership
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
