Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Agrawal, Amey, Kedia, Nitin, Panwar, Ashish · arXiv · 2024