RWKV.cpp

[Pinucci0]

I propose that Local-AI implements rwkv.cpp as a backend, along with the latest multilingual models based on it.

RWKV.cpp is a port of BlinkDL's RWKV-LM that leverages GGML to provide efficient inference capabilities. Unlike traditional transformer-based models with O(n²) attention complexity, RWKV uses a linear attention mechanism that only requires state from the previous step, making it particularly efficient for CPU inference with long context lengths.

The project offers:

• Support for multiple model precisions (FP32, FP16, quantized INT4, INT5, INT8)
• CPU-focused inference with optional GPU acceleration via cuBLAS, hipBLAS, and Metal
• A C library interface (rwkv.h) with Python bindings
• Support for RWKV architecture versions v4 through v7
• LoRA fine-tuning integration

Currently, the only orchestrator (as far as I know) for rwkv.cpp is RWKV-Runner, which requires Python 3.10 in linux. I am convinced that Local-AI could help facilitate the use of this backend and allow for exploring its full potential.