🔹 Feature: Load Aware Routing in Application Gateway for Containers
🔹 What It Does: Routes traffic to the backend pod reporting the lowest current CPU, memory, GPU (or any custom) utilization.
What Is It Giving You:
✅ Real Utilization, Not Just Request Count: Traditional least-request or round-robin don’t reflect what a request really costs. A GPU inference request can saturate a node while a small HTTP request costs almost nothing. Load-aware routing evaluates actual resource utilization and sends the next request to the endpoint with the most spare capacity.
✅ Standard-Based (ORCA): Backends emit an endpoint-load-metrics HTTP response header per the Open Request Cost Aggregation (ORCA) standard — in text or JSON. AGC consumes this header and strips it before responding to the client, so no internal metrics leak externally.
✅ Built-In Metrics: cpu_utilization, mem_utilization, application_utilization, rps_fractional, and eps are supported out-of-the-box. Include rps_fractional plus any utilization metric for a valid ORCA report.
✅ Custom Named Metrics for GPU / Queue / Anything: Use the named_metrics prefix to report any signal — GPU utilization, queue depth, VRAM pressure, model load state. Reference them in the namedMetrics list of the policy and AGC computes endpoint weight from the maximum of those signals. Perfect for AI inference or non‑CPU‑bound workloads.
✅ Configurable Warm-Up & Expiration: blackoutPeriod (default 10s) defines how long a new endpoint must report metrics before being trusted. metricExpirationPeriod (default 3 min) lets stale endpoints fall back to round‑robin weighting if they stop reporting.
✅ Error-Aware Weighting: errorUtilizationPenalty automatically down‑weights endpoints with higher error rates — calculated from rps_fractional and eps metrics — to protect overall reliability.
✅ Graceful Fallback: Backends without ORCA headers are treated with equal weight under standard round‑robin until they opt into metrics. Safe to enable across mixed workloads.
✅ Declarative Configuration: Apply via BackendLoadBalancingPolicy (Gateway API extension) on the target Kubernetes Service — clean, declarative, and GitOps‑friendly.
🌐 https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/load-balancing-strategies
