View source for PyTorch FSDP vs DDP

From Ben's Personal Wiki

← PyTorch FSDP vs DDP

You do not have permission to edit this page, for the following reason:

---

You can view and copy the source of this page.

I observed that PyTorch's FSDP with NO_SHARD is substantially faster and slightly more memory-efficient in a multi-node setting than DDP. Apparently this comes down to differences in the bucketing strategy used. Another important note, regarding gradient accumulation: you need to use <code>no_sync</code> for all except the last batch (which is available for both FSDP and DDP). [[Category:Machine Learning]]

Return to PyTorch FSDP vs DDP.