What features are there in batch preparation and sampling in DDP?

In DDP, each rank needs its own data shard; DDP does not automatically split the input batch. Use a distributed sampler and distinguish per-rank batch size from the effective global batch.

Call sampler.set_epoch(epoch) before each epoch's iterator so shuffling changes consistently. Handle padding or dropping tail samples explicitly: padding can duplicate examples, including during evaluation. For equal per-rank batches, effective batch size is local batch × world size × accumulation steps. Seed data-loader workers appropriately and aggregate metrics across ranks instead of averaging unequal groups blindly.

Источник: docs.pytorch.org.