Posted in

Run:AI Model Streamer + Azure Blob Storage โ€” Eliminate LLM Cold Starts

๐Ÿ”น Feature: Run:AI Model Streamer + Azure Blob Storage โ€” Eliminate LLM Cold Starts
๐Ÿ”น What It Does: Streams LLM weights directly from Azure Blob Storage through CPU memory into GPU memory โ€” skipping the local-disk copy step entirely โ€” so cold starts drop from minutes to seconds and fit inside a single autoscaler cycle.

The Problem It Solves
๐Ÿ”ธ Default loaders (vLLM, SGLang) run two sequential steps on every cold start: download to local disk, THEN read into GPU memory. The GPU sits idle through both, and local disk adds a bandwidth bottleneck.
๐Ÿ”ธ For a 232โ€ฏGiB model, default vLLM needs ~3โ€“5 minutes of paid GPU time before serving a single request.
๐Ÿ”ธ Autoscaler cycles run every 30โ€“60s. A 3+ minute cold start causes cascading queues, 5xx errors, and overprovisioning from stale autoscaler signals.

What It Is Giving You
โœ… Zero Local Disk Hop: Azure Blob Storage โ†’ CPU memory โ†’ GPU memory โ€” no SSD staging or local I/O bottleneck.
โœ… Cold Starts in One Autoscaler Cycle: 37โ€ฏs load time for 233โ€ฏGiB fits 30โ€“60s cycles. Autoscaler sees the replica online, avoiding 5xx cascades.
โœ… Speedup Grows With Model Size: ~4.3ร— at 15โ€ฏGiB โ†’ ~6.1ร— at 233โ€ฏGiB.
โœ… Saturates Network: Streamer holds 80โ€ฏGbps; default loader peaks ~40โ€ฏGbps and drops to ~10โ€ฏGbps on large models.
โœ… Natively in vLLM & SGLang: Accepts az:/// via –load-format runai_streamer. No sidecars or staging scripts.
โœ… Trivial Install: uv pip install vllm[runai] or uv pip install “sglang[runai]”; set AZURE_STORAGE_ACCOUNT_NAME; serve.
โœ… Standard Azure Auth: Uses DefaultAzureCredential โ€” az login, Managed Identity, or service principal env vars.
โœ… Distributed Streaming + Tuning: Parallelize with –tensor-parallel-size 8 and bump RUNAI_STREAMER_CONCURRENCY to 32โ€“64 to saturate NICs.
โœ… Open Source (Apache 2.0): First-class Azure Blob plugin co-developed by Run:AI and Microsoft.

Where This Matters
๐Ÿ”ธ Autoscaling LLM inference with unpredictable spikes.
๐Ÿ”ธ Spot VM fleets with recurrent reclaim events.
๐Ÿ”ธ Rolling deploys across many replicas.
๐Ÿ”ธ Multi-tenant serving platforms swapping models on demand.
๐Ÿ”ธ Any environment where โ€œGPU minutes spent loading modelsโ€ is a real line item.

๐ŸŒ https://devblogs.microsoft.com/azure-sdk/eliminate-llm-cold-starts-load-models-up-to-6x-faster-with-azure-blob-storage-and-runai-model-streamer/
๐ŸŒ Repo: https://github.com/run-ai/runai-model-streamer
๐ŸŒ vLLM docs: https://docs.vllm.ai/en/stable/models/extensions/runaimodelstreamer/
๐ŸŒ SGLang docs: https://docs.sglang.io/advancedfeatures/objectstorage.html

Microsoft Certified Trainer, Office 365, AWS, Azure and Cloud Expert-Architect. In the IT world for over than 20 years.

Apart from the main area of Microsoft Azure expert in the field of infrastructure servers Windows Server 2003-2019, Microsoft Active Directory, Hyper-V Private Cloud, IIS, System Center, SQL.

Private Cloud, System Center, Hyper-V, Open Stack Expert and all Microsoft products Expert. Linux Server administrator.

My Azure community projects:

https://mazeball.azurewebsites.net/
https://github.com/MariuszFerdyn?tab=repositories

More