Train a 2D VAE with dual channel images

[postmarone96]

Hi, i have a question about the visual autoencoder, what if the input images have 2 channels or even 3.

I get a tensor shape mismatch error in the the following line :
p_loss = perceptual_loss(reconstruction.float(), images.float())
- The Shape of the reconstruction.float() : torch.Size([1, 2, 256, 256])
- The shape of images.float() : torch.Size([1, 2, 256, 256])

here are the parameters i used for the AutoencoderKL :
autoencoderkl = AutoencoderKL(
spatial_dims=2,
in_channels=2,
out_channels=2,
num_channels=(128, 128, 256),
latent_channels=3, num_res_blocks=2,
attention_levels=(False, False, False),
with_encoder_nonlocal_attn=False,
with_decoder_nonlocal_attn=False
)
discriminator = PatchDiscriminator(
spatial_dims=2,
num_layers_d=3,
num_channels=64,
in_channels=2,
out_channels=2
)
Thank very much in advance !!

here is the error log : Error_log.pdf

[marksgraham]

Hi,

Can you verify you don't get this error for images/reconstructions with a single channel?

[marksgraham]

Ah yes - the percepual loss expects grayscale ( 1 channel) or RGB (3 channels). It won't deal with 2 channels. What do the two channels represent? If they are seperate grayscale images you could split them out and run the perceptual loss on each channel separately.

[postmarone96]

Thank you very much for your help! Each channel represents the same image at different wave lengths. Each highlights different features and structures in the scanned body part. Would actually creating a third channel, maybe a duplicate of one of the channels also solve the problem ?

[marksgraham]

Yes that could work, too

[postmarone96]

Hi mark, thank you very much for your help, I have another question. do channels have an impact on the training and the output of the other channels or are each channel handled separately in the training process?

[marksgraham]

The input channels will interact during training, as convolutions work across channels.