What is the reason for this call to synchronize?

[mitchellnw]

I was wondering if this call to synchronize is necessary: https://github.com/rwightman/pytorch-image-models/blob/master/train.py#L753 ?

I could not find it in, e.g., https://github.com/pytorch/examples/blob/main/imagenet/main.py and we also don't have in open_clip.

[rwightman]

@mitchellnw it's not necessary in this train script and could be removed since the only use of the output in logging is via an item (implicit synchronization), there might be a possible issue with the loss reduction for distributed being unreliable w/o? That should be checkd.

Removing it could slightu increase the throughput but would have to measure from start -> end of epoch and not rely on the avg_meter as it'd potentially make the batch-to-bach timing unreliable...

it is necessary in the bits_and_tpu branch where taking output of model and accumulating in another device tensor appears to cause a race:
https://github.com/rwightman/pytorch-image-models/blob/bits_and_tpu/train.py#L777-L787

[mitchellnw]

Thanks for the response. I'm curious about this:

there might be a possible issue with the loss reduction for distributed being unreliable w/o?

I would think this is not the case but curious if anyone knows better

[mitchellnw]

This seems to imply to me that sync automatically happens on backward: https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L937-L958

[rwightman]

@mitchellnw that's a different sort of 'sync' that's preventing the gradient from being synchronized (reduced) across all workers (expensive) when you're doing something like gradient accumulation .. vs this is making a call to block until all aysnc tensor ops are complete on device

[rwightman]

and when I say makes the loss reduction unreliable, that's only for logging purposes (computing the average loss across nodes), the actual mechanics of backward / grad reduction could not be harmed be removing that sync...

[mitchellnw]

Thanks!