When Bursty Traffic Makes LLM Inference Faster
Read the actual post on the Harvard Systems Group’s blog here.
Rather than just copying the content onto my own website, I thought it would be nice to instead expand on my personal thoughts during that project.
If you have comments, questions, or find typos on either this page or the blog post itself, it would be great to hear from you.
Discovery
As the writeup somewhat suggests, the investigation came about pretty much by accident. I was actually doing a screening test of sorts that Juncheng had sent over, where one of the tasks was benchmarking the runtime performance of various LLMs.
I’d learned a few months earlier that “burstier arrivals” (highly variable interarrival times) worsen latency in computer systems, so I basically included perf vs burstiness as one of the experiments. The hypothesis was a linearly increasing curve, based on Kingman’s formula, but instead I saw latency decrease.
At first, I thought I just plotted it wrong, but nope! There was actually some spooky stuff going on, and it needed explaining.
Implications
As the work progressed and we were increasingly convinced that burstier arrivals actually did reduce TPOT, I was thinking about why this actually matters.
Beyond making everyones lives easier, given that we don’t actually need to “deal with” bursty arrivals, it seems natural that we should also design our request routers to take advantage.
In “standard” queueing theoretic literature, the better routing policies tend to reduce the burstiness of arrivals, and also have a focus on load balancing.
As an example, Round Robin and Random, in expectation, send the same number of requests to each server, implying the loads are balanced as best as we can without knowing queue states or job sizes. However, the interarrival times generated by each, respectively, are Erlang-k and Exponential, where the former has lower variance at the same mean. Going back to Kingman’s formula, Round Robin should thus give us lower latency because it reduces the burstiness (variance) of the arrival patterns.
It follows that you might just default to Round Robin, and justifiably so. However, if burstiness reduces your latency, Random might turn out to be the better choice (now that I write this, I actually want to run that experiment soon).
In LLM inference, its true that most systems are smarter and take into account cache locality, as well as whatever info the router has regarding queue depth and load, but those can still produce Round Robin-like behavior, smoothing out an otherwise bursty arrival process and falling short of optimal.
This work would instead imply we should send short streams of requests to each server, rather than simply allocating them one at a time. Although the idea wasn’t pursued further given a lack of generality, there were some pretty nice results on a toy cluster, which are shown in the appendix of the blog.
Note to self, this section needs some revision, but the point is that if you’re in a situation where burstiness helps latency, autopilot decisions about your router may be leaving easy performance gains on the table, and the best choice might be counterintuitive.
Closing thoughts
As one of the first investigations I led, this was a lot of fun and smoother than I expected. It was a bit sad when the effects turned out to be caused by kernel artifacts. In fact, I’m not sure if I should consider myself super lucky or unlucky after coincidentally picking two model + GPU configs that just happened to “work”.
At the very least, I now know to triple check what attention backend I’m using, and got better at conducting thorough measurements etc. etc. insert more larp.
Thanks for reading!