Your training dataset (47,832 examples) is divided into equal shards — one per provider. Each node trains independently on its shard using the same frozen base model, which is already cached on all selected nodes. Rather than syncing full model parameters after every step, Ordain uses DiLoCo: nodes run multiple local training steps, then exchange only compressed gradient updates every 50 steps. This reduces inter-node communication by ~98% versus standard distributed training, making geographic distribution across internet-connected hardware viable.

Because all three nodes train simultaneously, wall-clock completion time is approximately 2.5× faster than single-node execution — a Standard tier job that would take ~18 hours on one A100 completes in ~7 hours across three nodes. DiLoCo sync overhead accounts for the difference from a theoretical 3× speedup.

DiLoCo was designed for pre-training large models but its properties are particularly well-suited to fine-tuning: the base model fits on a single node, the adapter weights being trained are small, the dataset shards are bounded and clean, and the run is short enough that sync frequency can be optimized for efficiency. Ordain's Coordinator only routes to multiple nodes when job characteristics justify it — datasets above 20,000 examples and runs exceeding 2 hours. For smaller jobs single-node routing is the correct default.

Adding more nodes increases parallelism but with diminishing returns. Communication overhead, straggler effects, and fault tolerance complexity grow with each additional node. The practical optimum for internet-connected DiLoCo fine-tuning is 3 to 6 nodes. Beyond that, coordination overhead begins to erode the parallel benefit faster than additional nodes contribute. Ordain prices multi-node jobs accordingly — more nodes means faster completion at a proportionally higher cost.