Wan Speech2Video

[continue-revolution]

Single GPU and multi-GPU cases have been tested, all examples work properly.
TODO:

• update moviepy
• consider replace decord with imageio
• get rid of mediapipe -- maybe completely get rid of fixed bbox generation. let user choose their own. also, we probably don't take any path, but move all loading of video / audio input into the example.
• probably move some examples to lfs
• write some testcases
• edit parallel logic according to Refactor/parallel #157

[gemini-code-assist]

Summary of Changes

Hello @continue-revolution, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a comprehensive Sound2Video capability, enabling the generation of video content driven by audio input. It includes new models for audio encoding and a modified Diffusion Transformer architecture to integrate audio and motion information, alongside a complete pipeline for orchestrating the generation process.

Highlights

• New Sound2Video Pipeline: Introduced the WanSound2VideoPipeline to enable the generation of video content from audio inputs, integrating advanced audio and video processing capabilities.
• Dedicated Audio Encoder Integration: Incorporated a Wav2Vec2Model for robust audio feature extraction, crucial for driving the video generation process based on sound.
• Specialized Diffusion Transformer: Developed WanS2VDiT, a modified Diffusion Transformer architecture that includes mechanisms for audio injection and sophisticated motion handling, allowing for dynamic video synthesis.
• Comprehensive Supporting Infrastructure: Added new configuration files, utility functions for image and video processing (e.g., resize_and_center_crop, read_n_frames, save_video_with_audio), and updated dependencies to support the new Sound2Video functionality.
• Example Usage Scripts: Provided new example scripts (wan_rs2v, wan_rsp2v, wan_rs2v_multi_people) to demonstrate how to effectively use the WanSound2VideoPipeline for various video generation scenarios.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new Sound-to-Video pipeline, WanSound2VideoPipeline, along with the necessary model implementations, configurations, and an example. The changes are substantial and add significant new functionality. My review has identified a critical issue in the example code that appears to mask an underlying bug, as well as several high and medium severity issues related to code correctness, maintainability, and performance. These should be addressed to ensure the stability and quality of the new pipeline.

[continue-revolution]

@akaitsuki-ii 应该没啥问题了，可以再看看