Repeated Augmentation and its effects

[vbvg2008]

Hi all,

I wonder if anyone has given any thought about repeated augmentation and its effect on the training.

From a look at the code, if using repeated augmentation (say 3), then the number of samples we train per epoch is extended 3 times. In other words, each epoch will become 3 times longer.

As a result, when setting --aug-repeats 3 and train for 300 epochs (such as the A2 in Resnet Strikes Back paper), we are in fact training 900 effective epochs.

Can anyone comment on whether the benefit of repeated augmentation is because of itself, or simply because it is training n times longer?

[hankyul2]

Hi @vbvg2008

Benefit of repeated augmentation is from itself. Repeated augmentation does not extend training epoch. RepeatAugSampler only select 1/aug_repeat samples from whole dataset and repeat each sample for aug_repeat times so that each gpu could train model on same samples.

In this code, the number of selected samples for each gpu is len(dataset)/num_gpu, which means that the number of samples does not change at all.

For better understanding, see below code with comments.

class RepeatAugSampler(Sampler):
    def __init__(self, ...):
        ...
        selected_ratio = dist.get_worl_size() # selected_ratio = number of gpu
        self.num_selected_samples = len(self.datasets) / selected_ratio # num_selected_samples = len(dataset) / (number of gpu)
    def __iter__(self):
        ...
        return iter(indices[:self.num_selected_samples]) # whatever indices is, it is sliced as long as num_selected_samples

I want this answer can help you.

Thank you.

hankyul

[vbvg2008]

@hankyul2 Hi, thank you so much for the answer.

I see, so despite the total number of samples are extended n times by repeated augmentation , but the self.num_selected_samples ensures the epoch length to be the same.

To further understand it, does this mean some samples may not be selected at all for a given epoch?

[hankyul2]

@vbvg2008 Yes, RepeatAugSampler only selects 1/aug_repeat samples and repeats selected sample aug_repeat times for each epoch, which means that unselected samples are not used in that epoch.