๐น Feature: Run:AI Model Streamer + Azure Blob Storage โ Eliminate LLM Cold Starts
๐น What It Does: Streams LLM weights directly from Azure Blob Storage through CPU memory into GPU memory โ skipping the local-disk copy step entirely โ so cold starts drop from minutes to seconds and fit inside a single autoscaler cycle.
The Problem It Solves
๐ธ Default loaders (vLLM, SGLang) run two sequential steps on every cold start: download to local disk, THEN read into GPU memory. The GPU sits idle through both, and local disk adds a bandwidth bottleneck.
๐ธ For a 232โฏGiB model, default vLLM needs ~3โ5 minutes of paid GPU time before serving a single request.
๐ธ Autoscaler cycles run every 30โ60s. A 3+ minute cold start causes cascading queues, 5xx errors, and overprovisioning from stale autoscaler signals.
What It Is Giving You
โ
Zero Local Disk Hop: Azure Blob Storage โ CPU memory โ GPU memory โ no SSD staging or local I/O bottleneck.
โ
Cold Starts in One Autoscaler Cycle: 37โฏs load time for 233โฏGiB fits 30โ60s cycles. Autoscaler sees the replica online, avoiding 5xx cascades.
โ
Speedup Grows With Model Size: ~4.3ร at 15โฏGiB โ ~6.1ร at 233โฏGiB.
โ
Saturates Network: Streamer holds 80โฏGbps; default loader peaks ~40โฏGbps and drops to ~10โฏGbps on large models.
โ
Natively in vLLM & SGLang: Accepts az://
โ
Trivial Install: uv pip install vllm[runai] or uv pip install “sglang[runai]”; set AZURE_STORAGE_ACCOUNT_NAME; serve.
โ
Standard Azure Auth: Uses DefaultAzureCredential โ az login, Managed Identity, or service principal env vars.
โ
Distributed Streaming + Tuning: Parallelize with –tensor-parallel-size 8 and bump RUNAI_STREAMER_CONCURRENCY to 32โ64 to saturate NICs.
โ
Open Source (Apache 2.0): First-class Azure Blob plugin co-developed by Run:AI and Microsoft.
Where This Matters
๐ธ Autoscaling LLM inference with unpredictable spikes.
๐ธ Spot VM fleets with recurrent reclaim events.
๐ธ Rolling deploys across many replicas.
๐ธ Multi-tenant serving platforms swapping models on demand.
๐ธ Any environment where โGPU minutes spent loading modelsโ is a real line item.
๐ https://devblogs.microsoft.com/azure-sdk/eliminate-llm-cold-starts-load-models-up-to-6x-faster-with-azure-blob-storage-and-runai-model-streamer/
๐ Repo: https://github.com/run-ai/runai-model-streamer
๐ vLLM docs: https://docs.vllm.ai/en/stable/models/extensions/runaimodelstreamer/
๐ SGLang docs: https://docs.sglang.io/advancedfeatures/objectstorage.html