Last week, our forum was buzzing with discussions that cut across both technical and strategic aspects of AI. Members exchanged thoughts on selecting optimal AI stacks for emerging markets, a topic that sparked debates on scalability and localization. Another thread delved into autoscalers and their unexpected interactions with large language models, highlighting the complexities of AI infrastructure. Elsewhere, the community explored the role of upskilling in accelerating enterprise deals, revealing a keen interest in bridging technical expertise with business growth.
This Week’s Hot Topics
Picking an AI stack for new markets
A lively debate unfolded around choosing the right AI stack, focusing on balancing innovation with market-specific needs. It’s an insightful read for anyone entering new territories. Read more here
Autoscaler tried to outsmart the LLM
This discussion unpacked some surprising outcomes when autoscalers interact with LLMs, shedding light on the nuances of AI system management. Read more here
Trying a 5-minute AI warm-up
Members shared quick AI exercises to kickstart their day and boost productivity—perfect for those looking to integrate AI learning into their routine. Read more here
Using LLM copilots with process mining
A fascinating thread on how LLMs can enhance process mining, offering fresh perspectives on process optimization and efficiency. Read more here
Upskilling that moves enterprise deals
The conversation highlighted the impact of upskilling on closing deals, emphasizing the intersection of skill enhancement and business strategy. Read more here
Canary rules for LLM endpoint rollouts
This topic explored the best practices for rolling out LLM endpoints, providing valuable insights into risk management and implementation strategies. Read more here
Looking forward to another week of engaging conversations and shared learning. Stay curious and keep contributing.
But ran into the autoscaler/LLM weirdness last week: our K8s HPA kept oscillating because CPU didn’t reflect token-generation, so we scaled on a “requests-in-queue” metric and set min_replicas=2 to keep a warm pool. That dropped p95 from about 3.2s to about 1.1s with a small cost bump; if budgets are tight, schedule the warm pool for peak hours only. If you’re on vLLM, scraping /metrics for tokens/sec works too, but queue depth has been more reliable for us.
We stabilized HPA by exposing a custom vLLM metric for outgoing tokens/sec and scaling on its 30s EMA; it matched load way better than CPU and stopped the thrash. > for peak hours only. If you’re on vLLM, scraping /metrics for tokens/sec works too, but queue depth has been — agreed, though we still keep a tiny minReplicas floor to avoid cold starts on early spikes. @lauren_baker77 have you tried EMA smoothing instead of queue depth?
We’ve had better luck scaling on p95 time-to-first-token at the gateway with a damped target and minReadySeconds to prevent flapping — scale what users feel, not what GPUs brag about… Caveat: aggressive batching can hide spikes, so cap batch size or add an ‘active sequences’ gauge, @kyle_brooks88.
Pre-flight on each leaf: only run the overlay tasks if LLDP sees both spines, EVPN/BGP is established, and NTP’s sane; otherwise skip the node, flip BGP Graceful Shutdown (RFC 8326: RFC 8326 - Graceful BGP Session Shutdown), and open a ticket. When ‘zero-touch needs a human’, at least the fabric’s wearing a seatbelt — would you gate on different signals?
Ran into the “unexpected interactions” when retries doubled traffic during an emerging‑market launch… Adding idempotency keys at the gateway and a 200ms jittered retry cut dupes and kept the autoscaler from see-sawing; Stripe’s primer helped: https://stripe.com/blog/idempotency-keys. If you can’t change clients, a small per-user ingress rate limiter is a decent stopgap.
One thing that saved us during an emerging‑market launch was adding admission control at the gateway: when pending_requests × avg_prompt_tokens crossed a threshold, we diverted long contexts to a cheaper model and delayed them, which prevented autoscaler stampedes and the “unexpected interactions.” If you’re on K8s, expose that as a custom metric and scale with KEDA + Prometheus so you can tune it per region: https://keda.sh/. Caveat: set the threshold per tier, or you’ll accidentally throttle your top customers.