Most production inference services run on Kubernetes, where declarative configuration helps coordinate distributed services. Running one inference server in a Slurm job is easy; running a production-style topology across multiple containers and nodes is not.

NVIDIA’s open-source srt-slurm (https://t.co/Q7BXRQwFeb) provides a YAML-based orchestration layer for these deployments. It coordinates sbatch and srun, GPU placement, networking, readiness checks, and cleanup for systems involving prefill/decode disaggregation, multiple workers, Dynamo frontends, KV-cache-aware routers, and KV-cache offloading services.

It’s useful for quickly deploying representative inference systems for testing or benchmarking on GPU clusters that already run Slurm.

Great work to @0xishand and the rest of the @NVIDIAAI team on this!