I’m looking for a course or program that covers algorithmic efficiency and production scaling — kernel fusion, quantization-aware training, dynamic batching — targeting 8B–13B models on 4×A100 with p95 < 80 ms at about 2k RPS. If you’ve taken something in the last 6 months that went deep on profiling (Nsight/TensorBoard) and distributed inference stacks (vLLM or TensorRT-LLM), please point me to it.
I did NVIDIA DLI’s “Optimizing LLM Inference with TensorRT-LLM” recently; prebuilding engines for 512/1024/2048 max_seq_len and capturing CUDA Graphs per bucket was the single change that got us to p95 < 80 ms at about 2k RPS on 4×A100. Caveat: keep dynamic batching tightly bucketed — Nsight Systems made it obvious when overly wide buckets caused host stalls between graph replays; link: NVIDIA Deep Learning Institute.
One small thing that shaved our p95 on 4×A100: split prefill and decode onto separate CUDA streams and give prefill a higher “stream priority,” plus pin the H2D input buffers so copies don’t stall it at about 2k RPS. It pairs well with vLLM’s “chunked prefill,” but you’ll still want to cap max_num_seqs to avoid over-queuing. On TensorRT-LLM the same pattern helps with inflight batching, though it won’t rescue poor KV-cache locality.
On vLLM, turning on chunked prefill and capping each chunk to about 256 tokens kept us at ‘p95 < 80 ms’ on 8–13B when prompts got long; Nsight Systems showed the decode kernels backfilling between prefill chunks. If you try it, nudge gpu_memory_utilization up a bit so the extra KV pages don’t churn; docs are here: https://vllm.ai. It can slightly hurt very short prompts, so we gate it by prompt length.