HOW TO USE AUTOMATIC PREFIX CACHING FOR sliding window attention based models ? #3881

Unanswered

vin136 asked this question in Q&A

vin136
Apr 6, 2024

if enable_caching and sliding_window is not None:
            raise NotImplementedError(
                "Sliding window is not allowed with prefix caching enabled!")

Currently any model with sliding window(mistral and friends) is not allowing me to enable_caching. Any work around or are we waiting on that?

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment