server: is there a way to get peak actual kv cache size?

[okuvshynov]

Good afternoon,

is there a way to get peak kv cache usage? I would like to understand how large the kv cache needs to be for my realistic workloads.

There's a mention of llamacpp:kv_cache_tokens in server docs, but it doesn't seem to be exported at the moment.

If there's no way currently, would it be ok to add tracking of peak size and export it to /metrics?

Thank you!

[okuvshynov]

#15361 is this something we could add?

[okuvshynov]

Merged