What is FSDP?

FSDP, or Fully Sharded Data Parallel, is a data parallelism strategy used in deep learning to train large models that would otherwise not fit in the memory of a single GPU. FSDP shards the model parameters, optimizer states, and gradients across multiple GPUs, allowing for training models with billions or even trillions of parameters. During the forward and backward passes, the necessary shards are gathered to each GPU on demand, and then discarded, thus reducing the memory footprint.