Different results after forward_features if size of input changes

[orion512]

This completely confusses me. Depending on the input into forward features the tensor result changes.
The change isn't large I believe only rounding (maybe grad_fn=SelectBackward0). Yet still different.

Below I simply run forward features 3x times.

• once with first 10 images
• once with first 15 images
• once with first 30 images

keep in mind the first 10 images are always the same.
Then when I compare the results to each other the batch with 30 images produces different tensors.

# model init
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = create_model("resnet50", pretrained=True, num_classes=1000)
model.to(device)
model.eval()
config = resolve_data_config({}, model=model)
tfms = create_transform(**config)

# Preprocess 10 images
read_images = [PILImage.open(img) for img in all_demo_paths][:10]
imges_list = [tfms(img) for img in read_images]
inputs = torch.stack(imges_list)
inputs = inputs.to(device)
features_10 = model.forward_features(inputs)

# Preprocess 15 images
read_images = [PILImage.open(img) for img in all_demo_paths][:15]
imges_list = [tfms(img) for img in read_images]
inputs = torch.stack(imges_list)
inputs = inputs.to(device)
features_15 = model.forward_features(inputs)

# Preprocess 30 images
read_images = [PILImage.open(img) for img in all_demo_paths][:30]
imges_list = [tfms(img) for img in read_images]
inputs = torch.stack(imges_list)
inputs = inputs.to(device)
features_30 = model.forward_features(inputs)

for i in range(10):
    print(f"[10-15] Comparing tensor {i}: {torch.eq(features_10[i], features_15[i]).all()}")
    print(f"[10-30] Comparing tensor {i}: {torch.eq(features_10[i], features_30[i]).all()}")
    print("---")

the output is

[10-15] Comparing tensor 0: True
[10-30] Comparing tensor 0: False
---
[10-15] Comparing tensor 1: True
[10-30] Comparing tensor 1: False
---
[10-15] Comparing tensor 2: True
[10-30] Comparing tensor 2: False
---
[10-15] Comparing tensor 3: True
[10-30] Comparing tensor 3: False
---
[10-15] Comparing tensor 4: True
[10-30] Comparing tensor 4: False
---
[10-15] Comparing tensor 5: True
[10-30] Comparing tensor 5: False
---
[10-15] Comparing tensor 6: True
[10-30] Comparing tensor 6: False
---
[10-15] Comparing tensor 7: True
[10-30] Comparing tensor 7: False
---
[10-15] Comparing tensor 8: True
...
---
[10-15] Comparing tensor 9: True
[10-30] Comparing tensor 9: False
---

[rwightman]

@orion512 typically I wouldn't spending anytime worrying about this if it is with typical floating point rounding differences

This (https://pytorch.org/docs/stable/notes/randomness.html) is worth a read, and might want to see if it still happens if you disable benchmarking and use deterministic algorithms

[rwightman]

Does it do that if it runs on CPU?

[orion512]

Yes I have only been running it on a CPU.

For now I avoid the issue by running batches of 15 images.
That approach is also way faster than throwing a larger pack of images into model.forward_features(inputs).
Not sure if that is because I am using a CPU instead of a GPU but the speed difference is significant.

[orion512]

@rwightman As I mentioned above I was able to control reproducibility (on my mac laptop) by processing only 15 tensors in a batch.

The problem I am facing now is that I get different results locally than on the build machine (ubuntu) while using GitHub actions.

The difference again is in rounding (below example of the first element of the vector)

mac:         0.0161567535251379
ubuntu:    0.01615680381655693

as mention I only use a CPU and not a GPU and have also tried setting the below:

torch.manual_seed(1993)
np.random.seed(1993)
random.seed(1993)
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False

Are you aware of anything else I can try to have reproducible results.

[rwightman]

@orion512 that is not something to worry about, you're pretty much at the float32 epsilon there...