Distributed Training¶
TorchEBM leaves parallelism to the standard PyTorch wrappers: components never require an initialized process group, and no default forward() path of a model, loss or sampler issues a collective. You wrap the energy network, use your own launcher and data sharding, and the components behave identically. Collectives exist only behind explicit opt-ins (process_group= arguments and explicit methods) where the math is batch-global.
FSDP2 (fully_shard with DTensor parameters) is the primary target and the path validated on multi-GPU NCCL hardware. DDP also runs, but drives no design decisions.
The shard-inside pattern¶
Shard the network inside your BaseModel subclass and keep the model object as the user-facing type. Samplers and losses stay unaware of sharding; every first-order path (energy calls, Langevin and HMC chains, CD, implicit EqM) runs through FSDP2's hooks unchanged.
The root call turns the parameters into sharded DTensors. Never wrap samplers or losses: losses see only model outputs and samplers call the model you pass them. TorchEBMModule resolves model.device and model.dtype correctly from DTensor parameters.
Training loop (torchrun)¶
Persistent CD on a DistributedSampler-sharded loader, launched with torchrun --standalone --nproc-per-node=4 train.py; the model is built shard-inside as above:
Three library-specific points, everything else is a standard FSDP2 loop:
- Pass a per-rank
generator. The library holds no hidden RNG state; with a shared seed every rank draws identical start points and runs identical chains, and you pay for \(N\) GPUs to get one GPU of sample diversity. Generators are device-bound: create them on the compute device. - Reduction semantics. Losses reduce with means over the local batch; with gradient averaging this equals the global-batch mean exactly when per-rank batch sizes are equal (keep
drop_last=True). -
k-step MCMC and resharding. With default reshard-after-forward, every Langevin step re-gathers the parameters. For sampling-only loops (not inside a training forward/backward), hold them gathered:
Training through the score¶
FSDP2 hooks cannot run the double backward that training through the score needs (score matching, explicit EqM energies, energy matching): the post-backward hook reshards parameter storage that the second-order graph still references, independent of reshard_after_forward. A prediction_type="score" or ("energy", "score") model with DTensor parameters therefore takes a hook-free functional path, selected by giving it a template at construction:
The template supplies the module structure for torch.func.functional_call and its own parameters are never used, so it lives on the meta device. The sharded DTensor parameters are injected per call and the double backward runs through DTensor's differentiable collectives. Gradients match an unsharded reference with global-batch-mean semantics and land with the parameter's own placement (the per-rank contributions are reduce-scattered at accumulation time), so standard optimizers step them directly. Without a template, a score call under grad mode fails fast with an actionable error when it sees DTensor parameters. Energy matching and EqM-E train through the same call, so they shard the same way.
One dtype caveat: a bf16 MixedPrecisionPolicy affects the hook path only. The functional path reads the fp32 sharded parameters directly, so its compute stays fp32 regardless of the policy.
Under DDP the hook path handles the double backward; build the energy head without an output bias (the score does not depend on it) so every parameter receives a gradient with the default static_graph=False.
PCD replay buffers across ranks¶
Persistent CD buffers are rank-local by design: each rank's ReplayBuffer keeps independent chains, so the world size multiplies chain diversity at zero cost, and no collective touches the buffer in the training step. To occasionally exchange chains between ranks, call the explicit collective between steps, on all ranks together:
It gathers the pooled chains, applies one shared permutation (drawn on rank 0 and broadcast, so per-rank generators cannot desynchronize it), and keeps the local shard: no chain is duplicated or lost.
Global-batch OT coupling¶
Minibatch OT couplings are biased toward the batch size. With a process group, SinkhornCoupling solves on the pooled global batch instead, shrinking that bias at fixed per-rank batch size:
Every rank gathers both batches, solves the identical pooled problem, and keeps its own targets, paired with sources from any rank; the per-target draw is broadcast from rank 0. Budget for the \((\text{world\_size} \times \text{batch})^2\) cost matrix on every rank. Exact and greedy couplings reject a process group (assignment solvers scale quadratically with the pooled batch); couple becomes a collective all ranks must enter together.
EMA¶
update_ema works on sharded parameters when the EMA model and the training model are sharded identically (same mesh, same policy); each local shard updates in place with no collective. Copy before sharding, then shard both the same way:
Checkpointing¶
Sharded state takes the distributed checkpoint (one collective save); the replay buffer is rank-local state (PCD chains differ per rank) and goes to one plain file per rank:
Do not put rank-local buffers into the DCP state dict: its planner deduplicates non-sharded tensors as replicated and silently keeps rank 0's chains. Restore rank-local files with the same world size, or re-initialize the buffer from noise (chains re-warm within a few hundred steps).
Large-scale recipe¶
For models too large to materialize on one device, compose the standard FSDP2 ingredients; every piece is validated with TorchEBM components:
The meta-device construction allocates no memory, to_empty materializes the local shards only, and the shared seed gives the same initialization on every rank. With a bf16 policy the score follows the input's dtype; set model.force_fp32_gradient = True if a low-precision model needs fp32-precision sampler gradients. Combine with the DCP checkpointing above.
Other frameworks¶
The shard-inside pattern is framework-agnostic: anything that wraps an nn.Module composes the same way.
- DDP:
model.net = DistributedDataParallel(model.net). Everything runs, including training through the score, through the hooks. - DeepSpeed ZeRO-3 / HF Accelerate / Lightning Fabric: wrap or
preparethe inner network, keep theBaseModelfacade outside. The FSDP2 caveat carries over wherever parameters are partitioned: training through the score needs the functional path, which supports DTensor only, or a replica-style engine (ZeRO-½, DDP). - Tensor parallel (DTensor): the functional score path requires a 1-D device mesh; TP meshes are untested territory.
Validation status¶
The distributed test suite (pytest tests/distributed) runs 2-process gloo/CPU in CI and validates the contract; the same suite runs under NCCL with one process per GPU via TORCHEBM_DIST_DEVICE=cuda, and benchmarks/distributed_fsdp2.py exercises full CD and functional-DSM training steps at hundred-million-parameter scale on multi-GPU hardware.