Name: Torch.Compile for Autograd, DDP and FSDP - Will Feng & Chien-Chin Huang & Simon Fan, Meta
Start: 2024-09-19T14:30:00-0700
End: 2024-09-19T14:55:00-0700

September 18-19, 2024
San Francisco, California
View More Details & Registration
Note: The schedule is subject to change.

The Sched app allows you to build your schedule but is not a substitute for your event registration. You must be registered for PyTorch Conference 2024 to participate in the sessions. If you have not registered but would like to join us, please go to the event registration page to purchase a registration.

This schedule is automatically displayed in Pacific Daylight Time (UTC-7). To see the schedule in your preferred timezone, please select from the drop-down located at the bottom of the menu to the right.

IMPORTANT NOTE: Timing of sessions and room locations are subject to change.

Thursday September 19, 2024 2:30pm - 2:55pm PDT

Room A

In this talk, we will present the latest advancements in torch.compile for distributed training via DDP and FSDP. We will first introduce Compiled Autograd, a torch.compile mode to fully capture the backpropagation step, including the communication collective operators used in distributed. We will then cover the improvements this new approach brought to Compiled DDP/FSDP, notably by removing DDP/FSDP graph breaks which brings the potential of improving compute/communication overlap.

Speakers

Thursday September 19, 2024 2:30pm - 2:55pm PDT
Room A

Breakout Sessions

Audience Intermediate

PyTorch Conference 2024

Chien-Chin Huang

Simon Fan

Will Feng

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!