Attention Mask for T5 / T2V Attention

[karan-dalal]

System Info / 系統信息

N/A

Information / 问题信息

• The official example scripts / 官方的示例脚本
• My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

N/A

Expected behavior / 期待表现

Hi. I have a few questions on the way text was integrated in the T2V model.

T5 Model

When producing the text embeddings, you pad the tokens to the self.max_length first. Then, you forward through the T5 encoder:

CogVideo/sat/sgm/modules/encoders/modules.py

Line 276 in bbe909d

outputs = self.transformer(input_ids=tokens)

How come you don't include an attention mask? Right now, the T5 encoder is able to attend to padded tokens during the encoding process.

The better thing to do would be:
outputs = model(input_ids=inputs['input_ids'], attention_mask=inputs['attention_mask'])

Attention in DiT

When training the DiT, you concat the text to the video embedding and pass it through attention. But your attention mask is set to all ones:

CogVideo/sat/dit_video_concat.py

Line 855 in bbe909d

kwargs["input_ids"] = kwargs["position_ids"] = kwargs["attention_mask"] = torch.ones((1, 1)).to(x.dtype)

Because your text is right padded, you are doing attention over [Text, Pad Tokens, Video Embedding] – why do you not use an attention mask?