variable (per frame) IP Adapter weights in video

[eps696]

is there a (planned or existing) way to have variable IP Adapter weights for videos (e.g. with AnimateDiff)?
that means setting different values for different frames, as both scaling and masking currently seem to work with the whole generation at once (be it video or still image).

[a-r-r-o-w]

Thanks for the interesting question! There's no such plan AFAIK to add per-frame IP-adapter weighting. An example video demonstrating the potential use-case of such a technique and its effectiveness would be nice. If you can point us to an existing implementation that is well known in the community or have an implementation of your own that produces great results, please do let us know

[eps696]

this may be an example https://thecvf-art.com/project/turbulence
here the text prompts are interpolated, while the image reference set is kept the same. first part (before 3:30) is with ip adapter, second one (after 3:40) - without. i would love to be able to go betweent the two within one video.
i've got ip adapter references interpolation in my repo https://github.com/eps696/SDfu but it can only vary images, keeping adapter fully active all the time - which is what i'd love to get changed.

[a-r-r-o-w]

This is a really interesting and cool idea now that I've looked around and found more information. I think it does make sense to support this for advanced controllability especially with FreeNoise for long generations, so I'll start thinking about how we could add support. I have seen atleast one well-known Comfy YTer make use of this for text animations and interpolation, which I found really cool. Would also be awesome if you're interested in working on this and potentially opening a PR, or we could colab together to cook something up.

cc @DN6

[eps696]

yes, this is promising tech for lengthy generations with time-based scenario. it would be cool to keep it disentangled from FreeNoise implementation though, since there is also (less resource-demanding) sliding window technique from comfyui which i'd love to keep using.
i'm not sure about opening a PR (if i knew how to implement it, i'd do it), but will be more than happy to colab on this.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.