llama-server as CLI

[ngxson]

@ericcurtin I'm creating a dedicated discussion in order not to make the thread become off-topic

Part of me wishes we had a llama-client that interacted with llama-server (with a CLI interface like llama-run). Although rather than C++, python possibly makes more sense as one can use the openai client library.

FYI my recent refactor #10691 is aimed to (somewhat) doing this. The idea is that in the future, we can expose llama-server as an internal library. A downstream program (like llama-client as you mentioned) can make calls internally to llama-server without ever touching HTTP stack.

Not sure if this is what you're looking for, feel free to discuss more.

[ericcurtin]

I think something that can make calls without touching HTTP is useful, you can get that extra performance. But, it would still be nice to have a HTTP-based OpenAI one, since OpenAI is more popular and standard.

[ngxson]

Hmm if you want that, then it's far too simple. Just use something like rust or go, compiled to single binary. I can make one in under 30 mn.

But I'm doubt if anyone will gonna need that, because they will need to spin up 2 processes (server+client) just to get the job done.

[ngxson]

Obviously someone already made that: https://github.com/picatz/openai