Long step time (0.1s) for ResNet-18 training on Imagenet #1542
-
|
Hi! I want to train a ResNet-18 on Imagenet using timm on a cluster with a V100 GPU, pytorch 1.12. Using 10 CPUs, with the data stored on an SSD and having replaced Pillow with Pillow-SIMD, as per the recommendations of this disccusion, I obtain the following numbers for the beginning of my training: Even though the data time appears quite low, I think that it actually is the data loading that is problematic : indeed I know by timing it offline that the forward and backward passes of the resnet-18 together take 0.02s. I would like to know if this is expected and if not what I can do to accelerate the training. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 8 replies
-
|
@zaccharieramzi average data time still looks fairly high (number in brackets is the avg_, looks like there's a pretty slow start for the system and the avg throughput is slowly increase towards approx 1200im/sec, not sure what sort of machine it is, but the persistent SSD disks on typical cloud instances aren't very fast. A resnet18, even in float32 should be a lot faster than that, yes. I use TFDS for imagenet in most shared drive / cloud scenarios and only use raw folder/file datasets on local machines. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks so much for answering so diligently. So the SSD I have on my cluster is supposed to have a 500Gb/s throughput. To use TFDS with the |
Beta Was this translation helpful? Give feedback.
-
|
You do need to install tensorflow and tfds though.... I usually do somethin lik which exact version of tensorflow is a good fit changes over time, often the latest latest ver in pypi conflicts on some libs... I think 2.8.x was working last time I setup, and the cpu ver prevents tensorflow from taking over the GPU... |
Beta Was this translation helpful? Give feedback.
-
|
you also need to build the shards with the tfds builder (to get the tfrecord) if you don't have those somewhere |
Beta Was this translation helpful? Give feedback.
-
|
I got the following error with So it seems that I don't have the tfrecords in the expected format : it should be 2 big tfrecords one for train and one for val right? (asking bc it's pre-prepared on my cluster). |
Beta Was this translation helpful? Give feedback.
-
|
So actually not 2 big tf records at all, I managed to find the correct expected format ( /path/to/tfds/root/imagenet2012/5.1.0/ and then a lot of tfrecords). It did reduce the data loading time from 0.08s to 0.008s and the average step time from 0.2s to 0.1s: But the total time per step is still at 0.1s which seems enormous. Can you think of any other reason why this problem would appear? |
Beta Was this translation helpful? Give feedback.
-
|
@zaccharieramzi k, so dataloading all good I don't have a v100 handy, it broke :( but I have a Titan RTX which is similar but a bit slower... So yeah, the numbers you see are normal I think. I get 1000 im/sec with your settings and no loader bottleneck. Not sure if you intended to train with these settings, but if you want to increase throughput you should up the batch size, use amp, and enable channels_last
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks so much for providing all this information. I did not intend to train with all these settings necessarily, increasing the batch size was definitely on my list. One thing that I don't understand is that while it seems from the numbers that there is no data bottleneck, if I do an offline test of the time it takes to do the forward and the backward passes, it is more on the order of 0.02-0.04s while the step time currently is more like 0.1s. This is why I was thinking that the CPU -> GPU transfer was actually the bottleneck. Do you have any thoughts on this? Thanks again so much for taking the time to answer all of these questions it's so useful for me. |
Beta Was this translation helpful? Give feedback.
-
|
So I think my main problem here is actually that I am measuring the time it takes to perform the forward and the backward passes in the wrong way. I write the content here as well: from torchvision.models import resnet18
import torch
X = torch.rand(128, 3, 224, 224, device='cuda')
y = torch.ones(128, dtype=torch.uint8, device='cuda')
model = resnet18().cuda()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
loss = torch.nn.CrossEntropyLoss().cuda()
#warm-up
optimizer.zero_grad(set_to_none=True)
y_pred = model(X)
l = loss(y_pred, y)
l.backward()
optimizer.step()
################
%%time
optimizer.zero_grad(set_to_none=True)
y_pred = model(X)
l = loss(y_pred, y)
l.backward()
optimizer.step()
################
# outputs something like
# CPU times: user 17 ms, sys: 0 ns, total: 17 ms
# Wall time: 23.2 ms
################
%%time
for _ in range(4):
optimizer.zero_grad(set_to_none=True)
y_pred = model(X)
l = loss(y_pred, y)
l.backward()
optimizer.step()
################
# outputs something like
# CPU times: user 221 ms, sys: 218 ms, total: 439 ms
# Wall time: 440 ms
################
%%timeit
optimizer.zero_grad(set_to_none=True)
y_pred = model(X)
l = loss(y_pred, y)
l.backward()
optimizer.step()
#################
# outputs something like
# 324 ms ± 1.82 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)I don't understand why there is such a discrepancy depending on whether you chain the passes or not. Does the GPU need some cooling down? |
Beta Was this translation helpful? Give feedback.
-
|
Ok so turns out I was measuring the time wrong : indeed the GPU needs some "cooling down", i.e. all the kernels must have finished, and apparently this is not the case when you do |
Beta Was this translation helpful? Give feedback.
@zaccharieramzi average data time still looks fairly high (number in brackets is the avg_, looks like there's a pretty slow start for the system and the avg throughput is slowly increase towards approx 1200im/sec, not sure what sort of machine it is, but the persistent SSD disks on typical cloud instances aren't very fast. A resnet18, even in float32 should be a lot faster than that, yes.
I use TFDS for imagenet in most shared drive / cloud scenarios and only use raw folder/file datasets on local machines.