Who Decides What Runs Where

The scarce resource in an AI factory is accelerator time. Everything in the orchestration layer exists to allocate it. Two families of software do most of that allocation, and they come from different traditions.

The first is Slurm. It began at Lawrence Livermore National Laboratory as a resource manager for Linux clusters and grew into the scheduler that, by NVIDIA's count, runs on more than half of the top ten and of the top hundred systems on the TOP500 list of supercomputers. Its model is the queue. A researcher submits a job that asks for a number of nodes for a length of time. Slurm holds the job until the resources are free, launches it across every node at once, and releases the nodes when it ends. That model fits large model training well, because training is a batch job that needs many machines to start together and stay together. SchedMD, the company that developed and supported Slurm, was acquired by NVIDIA in December 2025. NVIDIA said at the time that Slurm would remain open-source and vendor-neutral.

The second family is Kubernetes. Google open-sourced it in 2014, and it became the standard way to run containerized services. Its model is not the queue but the desired state. An operator declares what should be running, and Kubernetes continually works to make reality match. That model fits inference well, because inference is a service that must stay up, scale with traffic and recover from failures without anyone submitting a new job.

The two traditions are converging. Kubernetes was not originally built to understand accelerators as anything more than a count: this node has eight GPUs, which is to say eight graphics processing units. That is too coarse for modern hardware, where devices differ in memory, interconnect and partitioning. The Kubernetes project addressed this with Dynamic Resource Allocation, which reached general availability in version 1.34 in 2025. It lets a workload describe the device it needs and leaves the scheduler to find a match. Meanwhile, batch-style scheduling has been layered onto Kubernetes by several projects, so training jobs can run there too.

The scheduler's decisions are only as good as its information. This is where accelerator telemetry enters. NVIDIA's Data Center GPU Manager, or DCGM, collects per-device utilization, power, temperature, memory error counts and interconnect status. Its documentation describes three modes of health checking: passive background monitoring while jobs run, short readiness checks before work is admitted, and longer active diagnostics when hardware is suspected. A companion exporter publishes the same metrics into the monitoring systems Kubernetes operators already use. The readiness check deserves attention. It means the control plane can refuse to place an expensive job on a node that is likely to fail, which is cheaper than discovering the failure an hour in.

Vendors now sell software that spans these layers. NVIDIA announced Mission Control at its GTC developer conference in March 2025, describing it as operations and orchestration software for its Blackwell-generation systems. According to NVIDIA, it incorporates Run:ai scheduling technology, automates recovery of failed jobs and manages power profiles. Treat the performance multiples in such announcements as vendor claims until an operator reports them. The structural fact is more durable than the numbers: the chip vendor now ships the scheduler, the telemetry and, through SchedMD, the dominant batch workload manager.

That concentration is the dependency to watch. When one company supplies the accelerator, the interconnect, the health monitoring and the scheduling software, the control plane becomes easier to integrate and harder to leave. An operator gains a tested stack. It also accepts that the rules for allocating its most expensive asset are written largely by its supplier. Neither outcome is wrong. Both should be chosen deliberately.

Now place the orchestration layer against the facility layer from the previous lesson. Slurm and Kubernetes know which jobs are running and how important each one is. The building management system knows how much cooling capacity remains. The electrical power monitoring system knows how close each feed is to its limit. Historically, none of that facility knowledge reached the scheduler. A scheduler would happily pack the hottest jobs into the rack with the weakest cooling, because nothing told it otherwise.

Faraday's read is that the next gains in AI factory output come less from faster chips than from schedulers that can see the building. The mechanism is simple. A scheduler that knows the thermal and electrical state of each row can place work where headroom exists, delay work where it does not and avoid the throttling that silently slows jobs. The software to do this exists in pieces. The integration is the unfinished work.

So the orchestration layer is where intent becomes allocation. It holds the queue, the priorities and the recovery logic. It also holds a blind spot, the building, which the rest of this course is about closing.