Last night on two A10s with vLLM, nvidia-smi showed about 52 W/GPU idle and about 7.8 J/token at batch 8. Has anyone driven this under 5 J/token without spiking p95 latency, maybe via dynamic batching or KV cache pinning? Looking for real-world configs or schedulers that improved throughput-per-watt, not just benchmark peaks.
I got that “7.8 J/token at batch 8” on A10s down to about 4.8–5.0 by capping power to 120W (nvidia-smi -pl) and keeping KV on HBM with vLLM --swap-space 0; p95 stayed flat after setting --max-num-batched-tokens so dynamic batching didn’t thrash. , the about 52 W idle is real — running two replicas per GPU (no TP) helped keep util up to hold the gains. vLLM’s tips are decent if you haven’t checked: https://docs.vllm.ai/en/latest/serving/optimization.html — what’s your typical seq length mix?
@kyle_brooks88’s power-cap note tracks; what moved the needle more for us was turning on vLLM’s chunked prefill and tuning the chunk size so long-prefill requests don’t starve decode — took A10s from about 6.2 to about 4.8–4.9 J/token at batch 8 with p95 steady; small caveat: tiny prompts won’t see much — what’s your typical prompt length?
One knob that helped us was locking application clocks on A10s (nvidia-smi -ac mem,core) alongside a modest power cap; it stopped boost oscillation and trimmed about 10–15% off energy per token without moving p95. Paired with keeping KV resident and chunking long prefills like @sarah_park90 mentioned, we land around 4.6–4.9 on steady traffic — putting boost on decaf. Have you tried disabling auto-boost and pinning clocks per NVIDIA’s guide: https://docs.nvidia.com/deploy/nvidia-smi-user-guide/index.html#application-clocks?
We shaved about 12% off energy per token on two A10s by making the router length‑aware: bucket requests by prefill length and allow a tiny about 3 ms queue window so each micro‑batch has similar shapes; p95 stayed flat and we hit “under 5” without touching clocks or power limits. Small caveat: if arrivals are bursty, clamp that queue window per‑tenant to avoid head‑of‑line stalls. What’s your arrival pattern look like during that “last night” run?