Adding Differential Diffusion to Kolors, Auraflow, HunyuanDiT

[a-r-r-o-w]

Diffusers recently added support for the following models:

• Kolors (@tuanh123789)
• AuraFlow
• HunyuanDiT (@MnCSSJ4x)

A few weeks ago, we also added community pipelines for Differential Diffusion utilizing SDXL and SD3. You can search for "diff diff" comments in the PR files to find the required changes.

We would like to extend DiffDiff support for Kolors, AuraFlow and Hunyuan. Feel free to pick and iterate on one of the models that has not been assigned to someone else already 🤗

• You will have to create a community pipeline in examples/community folder.
• The pipeline may need to be an Image-to-Image variant of the original Text-to-Image implementation to start off, since it would then be easier to add the required changes for DiffDiff. It should, hopefully, not be too difficult and can be created by following the changes in our Img2Img pipelines. For example, Kolors and KolorsImg2Img. Feel free to ping me for help regarding anything and mentioning what you tried
• Add the pipeline name, description, reference link, colab (if any), and your name to the Community README. Also, add a section with the necessary inference code and your cool image showcases 😎
• Create a PR posting a reproducible code example or link to a colab notebook. Also post a few generation results with all required input images for the code to be runnable.

When opening a PR, you can tag me and @DN6. For a more critical review of your generations, you can also tag @asomoza.

Make sure to read the Diffusers contributing guide if you're a first-time contributor.

You can find some more informational content here:

• https://huggingface.co/docs/diffusers/main/en/using-diffusers/custom_pipeline_overview
• https://huggingface.co/docs/diffusers/main/en/using-diffusers/contribute_pipeline

Keep diffusing 🧨

Edit: If you're working on this, it is better to follow the implementation of Stable Diffusion 3. Make sure to not add any additional pre-processing code to the pipelines using external libraries or torchvision. You can follow the changes in this commit. Please make sure to go through all the links shared here

Thanks to @MnCSSJ4x for adding support to HunyuanDiT!

[tuanh123789]

@a-r-r-o-w can i take Kolors

[a-r-r-o-w]

@a-r-r-o-w can i take Kolors

awesome, go for it!

[devesh-2002]

@a-r-r-o-w I would like to take AuraFlow

[MnCSSJ4x]

@a-r-r-o-w I would like to take HunyuanDIT. Please guide me how to get started and what to refer!

[a-r-r-o-w]

Thanks for taking these up! Feel free to pick up and PR :) The changes can be made as follows:

• Copy Auraflow/Hunyuan pipelines
• Create their Img2Img pipeline variant. This would include adding an image and strength parameter and calling prepare_latents using it. The easiest way to see how to convert a Text2Img pipeline to Img2Img is to take a look at the diff of any of our current pipelines, for example, Kolors (linked in Issue description), or StableDiffusionPipeline and StableDiffusionImg2ImgPipeline. These changes would be minimal.
• Look through the DiffDiff implementation for SDXL/SD3 linked in Issue description, and search for the strings "diff diff" to see all the relevant changes needed. These are not more than a few lines.
• You can set up your environment with Diffusers however you'd like following the install instructions, although having a reproducible colab notebook would be cool. You can get a feel for using the library by the demo examples in our pipeline docs.

Let me know if I can be of more help or any specific questions

[saqlain2204]

@a-r-r-o-w I would like to take AuraFlow

Hey @devesh-2002 . Are you working on it?

[devesh-2002]

@a-r-r-o-w I would like to take AuraFlow

Hey @devesh-2002 . Are you working on it?

Yes. I am working on it.

[asomoza]

@a-r-r-o-w maybe you can add that we need to remove the need for preprocessing the images (and torchvision) from the SDXL one, like in this commit if someone from the community wants to do it.

And it's better to follow the example for SD3 than the SDXL one in the meantime.

[chuck-ma]

@a-r-r-o-w maybe you can add that we need to remove the need for preprocessing the images (and torchvision) from the SDXL one, like in this commit if someone from the community wants to do it.

And it's better to follow the example for SD3 than the SDXL one in the meantime.

I want to know why Kolors doesn't need to preprocess images like SDXL Differential Pipe but instead preprocesses images like SD3. I looked at the SDXL Differential Pipe code and it seems there is no image preprocessing code related to SD3.