Training reproducibility?

[alexander-soare]

Tried training the same model twice and noticed different figures each time. I do see this random_seed(args.seed, args.rank) in the training script. Any other reasons I'm not getting reproducibility?

I had a look at random_seed and found that changing it to this did the trick:

def random_seed(seed=42, rank=0):
    # torch.manual_seed(seed + rank)
    # np.random.seed(seed + rank)
    # random.seed(seed + rank)
    random.seed(seed + rank)
    os.environ['PYTHONHASHSEED'] = str(seed + rank)
    np.random.seed(seed + rank)
    torch.manual_seed(seed + rank)
    torch.cuda.manual_seed(seed + rank)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

I don't know which bit does it, and I got that from some Kaggle competition ages ago :)

[rwightman]

@alexander-soare I've never wanted to sacrifice performance for complete determinism so have never gone all the way, and seems to suffer regressions in PyTorch/vida ... But I would accept a PR with a --deterministic flag