We’re running a RAG microservice on Kubernetes targeting p95 < 250 ms at about 500 QPS using pgvector for retrieval, Redis for caching, and a quantized 8B model behind Triton; we’re stuck around 290–310 ms. If you’ve hit similar SLOs, did dynamic batching plus early-exit retrieval improve more than trimming prompt context and leaning on KV cache?
Switch pgvector to HNSW GitHub - pgvector/pgvector: Open-source vector similarity search for Postgres and enable Triton dynamic batching (2–4ms queue); p95 -35ms. Retrieval p95 now?
Building on @amitp1989: the biggest win for us was speculative decoding in Triton/TensorRT-LLM with a about 1B draft model co-located with the 8B — cut ‘first token’ latency about 60–80ms and p95 from about 300ms to about 230ms at about 500–600 QPS; just note it shines when outputs are short… Are your answers typically <100 tokens?
We cut our initial ping-pong about 30% by auto-capturing a “trace ID” and build SHA in the intake, so triage can pull logs before touching the ticket. If a HAR’s too heavy, a 10-second console screenshot plus user role usually does the trick — it’s like leaving your keys by the door. If there’s a slot Monday, could you include how you propagate the “trace ID” across services in the workflow builder?
We got under “250 ms” by killing the DB hop: embed FAISS HNSW in the same pod and feed Triton via CUDA shared memory, which cut about 35–45 ms p95 vs pgvector+HTTP at about 500 QPS. Dynamic batching only for the first token (0–2 ms queue) helped; beyond that it hurt p95 more than trimming context with KV cache. Can you colocate the retriever with the 8B and try Triton SHM?
Have you considered optimizing your data access pattern? Switching to something like a graph database could cut down those DB round trips significantly, especially at the scale you’re targeting. Just curious if you’ve explored that approach, or if pgvector has specific advantages for your use case.