Performance and Benchmarking¶
Patterns for writing performance-sensitive code, and the two measurement tools that back any performance claim: the benchmark suite (how fast is it?) and the profiler (where does the time go?).
The four hot spots¶
- Sampler steps: iterative, run 100-1000x, dominate wall time.
- Score / energy gradients:
autograd.gradcalls are frequent and stack. - Loss forward + backward: called every training batch.
- Host-device traffic:
.item(),.cpu(), repeated.to()stall the GPU.
Optimise in that order.
Vectorise¶
Work in batch dimensions. A Python loop over samples launches one kernel per sample:
The batched call computes all (batch,) energies in one pass:
Sample many chains in parallel by putting the chain index in the leading dim.
Stay on device¶
Keep tensors on one device and dtype for the whole pipeline. Use self.device / self.dtype from TorchEBMModule inside the library; never hard-code "cuda".
.item(), .cpu(), .tolist(), and Python if tensor > 0: all trigger a full GPU sync. This loop stalls the GPU at every step:
Keeping the values on device and reading them once after the loop costs a single sync:
The GPU-first contract¶
Every new sampler, loss, or trainer honours these rules; the existing components already do:
- Normalise transfers once at entry, never per step. Move conditioning and inputs to the device a single time when a call begins, then reuse them.
TorchEBMModule._prepare_model_kwargsis the reference: it device-alignsmodel_kwargsonce and the per-step drift closures capture the result. - No
.item()/.cpu()/.tolist()/.numpy()inside a step loop. Diagnostics and metrics stay device tensors and are stacked, then synced once at a logging boundary (BaseTrainer.train_stepreturns device scalars;train_epochsyncs once per epoch). Preallocate trajectories/diagnostics on device and fill them in place. - Host scalars enter only through schedulers.
get_scheduled_valuereturns a Python float computed on the host, so it folds into the next kernel launch with no device→host sync. Do not read step sizes or temperatures back off a device tensor. - Respect the input dtype. The autograd score of a
prediction_type="score"model computes in the input's dtype (byte-identical for float32, no fp64 downcast); setmodel.force_fp32_gradient = Trueonly when a low-precision model needs an fp32-precision score. - Adaptive/implicit integrators carry one bounded sync per iteration (the step-acceptance and solver-convergence checks are data-dependent and must reach the host). This is inherent; fixed-step integrators are the sync-free path. Greedy OT coupling is likewise host-bound by nature - use Sinkhorn in the training loop.
To catch a regression, wrap a hot loop in torch.cuda.set_sync_debug_mode("error"): any accidental host sync raises with the offending op named.
Reuse memory¶
Pre-allocate buffers once and fill them in place inside loops. The samplers write a trajectory into one pre-allocated tensor:
In-place ops (x.add_, x.mul_) are safe outside autograd-tracked paths.
Mixed precision and compilation¶
Inside the library, wrap large matmul blocks with self.autocast_context(), which honours the user's configured dtype. At the benchmark/application layer, --amp and --compile apply the same transforms via benchmarks/registry.py::apply_mode, so eager, compiled, and mixed-precision results stay comparable.
Common pitfalls¶
- Implicit host-device copies:
torch.tensor(x_numpy, device=...)inside a loop. - Redundant
.to()calls:BaseLoss.__call__already moves inputs; subclassforward()must not. - Missing
torch.no_grad(): interpolation targets, momentum init, and random projections need no grad tracking. - Tiny batches on GPU: prefer one big step over many small ones.
isinstancechecks inside the inner loop: resolve once before the loop.
Benchmarks: detecting change¶
The suite (pytest-benchmark under benchmarks/) auto-discovers every component exported from torchebm.*.__init__ and times its standard workload at three scales. Regular pytest tests/ never runs them.
| Command | Runs |
|---|---|
bash benchmarks/run.sh --quick | a smoke run at small scale |
bash benchmarks/run.sh --module losses | one module |
bash benchmarks/run.sh --filter "ScoreMatching" | the benchmarks matching a name |
bash benchmarks/run.sh --baseline | the suite, saving the results as the baseline |
bash benchmarks/run.sh --compare | a comparison of the latest two runs |
bash benchmarks/run.sh --ci | the suite, failing if the geo-mean speedup is below 0.95x |
Components needing non-default construction get an entry in benchmarks/registry.py::COMPONENT_OVERRIDES; existing entries are the best reference. Modules and individual benchmarks can be excluded in benchmarks/benchmark.toml.
Publishing. Results and the dashboard live in the separate torchebm-benchmarks repository, deployed at soran-ghaderi.github.io/torchebm-benchmarks. After a run: copy the autosaved JSON from benchmarks/results/Linux-CPython-*/ into that repo and run bash scripts/publish.sh <path-to-json>; GitHub Pages auto-deploys.
Profiling: explaining change¶
benchmarks/profiler.py wraps torch.profiler around the same registry callables the benchmarks time. Profile only when an optimisation is non-trivial and evidence-driven: a dashboard regression to localise, a hot path rewrite to justify, or a suspected memory issue. Skip it for one-line fixes and cleanups.
The workflow is one profile before the change, one after it, and a diff:
Add --trace only when the top-N table is not enough (open in ui.perfetto.dev), --memory for allocator work (view at pytorch.org/memory_viz), --nvtx for Nsight Systems. Arbitrary callables profile via --callable module:factory. Outputs land under benchmarks/profiles/ (gitignored; profiles are local by design).
Division of labour: benchmarks detect a regression and track it across releases; the profiler explains it op by op on one run.