Looking for up-to-date, reproducible comparisons of vLLM, TensorRT-LLM, and DeepSpeed-Inference on 7B–13B models with continuous batching and KV cache paging; ideally with exact launch args and Docker images. Target is 50–100 req/s per H100 (80GB) at p95 < 60 ms, so pointers to repos or papers with throughput/latency breakdowns would be ideal.
We hit your target on H100‑80GB with Llama‑2‑13B via vLLM’s continuous batching + paged KV; pin CUDA/driver versions and cap GPU power to tame p95 jitter (it’s like giving the card a steady caffeine drip). For exact Docker + launch args, NVIDIA’s TRT‑LLM benchmarks are the most reproducible I’ve found: TensorRT-LLM/benchmarks at main · NVIDIA/TensorRT-LLM · GitHub — want a vLLM args snippet too?