When a new model starts spiking 5xx or bad outputs, what rollback approach has saved you in production? We aim for sub‑5 minute reversions using blue‑green via Argo Rollouts and feature flags, but shadow traffic has missed token‑latency regressions twice this quarter. If you have a concise runbook or checklist that works under pager pressure, I’d appreciate a pointer.
check your plan’s disclosure: https://www.t-mobile.com/responsibility/consumer-info/network-management. Which carrier? I do a quick 6–9 pm test with YouTube’s “Stats for nerds” — if Connection Speed sits under about 10 Mbps while signal’s solid, that’s deprioritization or video optimization in the wild, and bumping to the HD/Plus add‑on has fixed it for me; what market are you in?
My take: I’d lean toward the simplest next step and see if it changes anything this week — if not, you’ve got a clear case to escalate. What would block you from trying that?
One thing that saved us: an Argo Rollouts AnalysisTemplate tied to Prometheus that watches p95 time‑to‑first‑token and inter‑token gap; if either breaches for 2m, it auto‑aborts the canary and our router flips all traffic to the last‑known‑good we keep warm to avoid cold starts (docs: Overview - Argo Rollouts - Kubernetes Progressive Delivery Controller). I’d set thresholds a bit tighter than your user SLOs — do you track inter‑token gaps separately from TTFB?
We ended up with a warm-standby last-known-good behind a version alias and a one-liner that flips the alias, drains the bad pods, and pauses autoscaling — a “big red eject button”. To catch what shadowing misses, a fixed prompt runs every 5s and alerts on p95 first-token latency and streaming cadence, which can auto-trigger the flip. Minor tradeoff is standby cost; do you have a model registry that supports alias flips (e.g., MLflow or Bedrock) so the script stays dumb?
We put Envoy outlier detection at the gateway to watch stream stalls and tokens/sec; if a model dips below a threshold for 60s, the gateway flips traffic back to the previous version — an emergency brake outside Rollouts. Docs: Outlier detection — envoy 1.38.0-dev-7d9d20 documentation; could your ingress do something similar, or expose a header the gateway can act on?