Support for more LLMs? Soon!

[RamiAwar]

Update September, 2024: We've investigated this, and it seems like none of the local models were good enough to use with tool calling and DataLine's specific toolkit. Since this was unusable, we decided not to continue working on it as there is no point. We'll revisit this every 3 months and test again.

As for custom provider support, we also weren't too happy with other providers. Got a lot of structured output issues, tool calling issues, different API formats, etc. We've decided not to invest more of our time in this for now. Again, revisit in around 3 months. Hard to stay up to date with everything without wasting all of your time, but it seems like the APIs are converging to the OpenAI api, albeit slowly.

Update August, 2024: Work has begun! #267

Phase 1 🚧

@anthony2261 is working on adding a custom LLMProvider architecture to replace what we have now. This will later be populated from the frontend, allowing custom base URLs.

Phase 2 📝

Support LLMProvider configuration from settings page

Phase 3 📝

Add more LLMProvider implementations (Anthropic, Ollama, etc.) supporting more models as well as local models

---

We've heard this from many users already: "We want more models! Not just GPT-3.5-Turbo!"

We hear you, and our vision is aligned with that! We've been experimenting with new models as they come out, and we noticed something: Sometimes models really decrease the quality of results when you're using DataLine.

There is no point in supporting a model that performs so bad that you won't use it.

Why aren't we releasing this yet?

Our goal isn't to just "support" lots of models, our goal is to allow users to pick among high quality options that all work. There is a minimum bar we want to maintain here, otherwise someone might come and try DataLine with GPT4o and decide that DataLine is bad (meanwhile the model itself is the problem). It's in our collective best interest to prevent that from happening 🙂

Ok when will it be out?

We're building an evaluation pipeline to help us judge whether a model is 'good enough'. That should be done in the upcoming week or two. After that, we'll be able to add support for more models (including local ones!).