Automated feedback loop

[luarss]

Think about an automatic mechanism in which we can use the feedback we get from the users back in our evaluation dataset.

Use it as part of our dataset ground truth (good or bad )

Tasks

• DeepEval using explicit feedback https://docs.confident-ai.com/docs/confident-ai-human-feedback

For this, we need to institute more metrics (tracked in #18 )

Explicit feedback

• Thumbs up/ thumbs down. But this is rare.

Implicit feedback

• Inferred from user behavior, such as response time, follow-up questions, or even actions like copying and pasting responses. Implicit feedback is far more abundant and can provide valuable insights into how well the LLM is performing

1. https://towardsdatascience.com/how-to-make-the-most-out-of-llm-production-data-simulated-user-feedback-843c444febc7
2. https://www.nebuly.com/blog/llm-feedback-loop

[Kannav02]

Hey @luarss!

how would you want the feedback to be stored, do you want a feeder pipeline, that feeds back into the evaluation dataset and then based on that each subsequent prompt answer is influenced by these metrics

do you also want the thumbs up and thumbs down feature to be implemented, I believe the functionality would be similar to how LLMs like chatGPT take the feedback, like once about 100 prompts maybe?

If those are the requirements can I work on this issue?

thank you!

[luarss]

Hi @Kannav02, thanks for your interest.

We have some implementation of the thumbs up/down interface in this PR: #61, done by Aviral (@error9098x). That should be merged soon.

What we need is a convenient way to store and access this feedback programmatically. Currently we use Google Sheets but that isn't the most programming friendly database. This feedback could be thumbs up/down, or in the form of actual text (suggestions). Happy to hear your suggestions on this.

We would need to store minimally: Question, Response, Sources, Date/Time, Thumbs up/down, Feedback, Conversation history

[Kannav02]

Since the data is pretty structured, I believe something like this could be stored within A SQL based database, maybe PostgreSQL,

I would go with MongoDB as well since we would consider scalability considering the amount of prompt feedback we will have, but the idea again is it's schemaless, I mean we could still work around this by enforcing it in the code itself,

Initially, when I saw this problem I thought maybe Redis could be a game-changer given the fast-query capabilities, but again the fields here are more and I believe it would rather just complicate our stuff

In the end I believe it boils down to PostgreSQL and MongoDB, if it is just for the prompt services and feeding it back to AI, I would go with MongoDB but if you still have its use in other cases we might have to consider the consistency in queries across the codebase, so then maybe PostgreSQL

[luarss]

I think MongoDB might be preferable since our feedback might evolve over time and schemas could change. A nice checkpoint could be DB hosting and basic web UI for filtering, querying and sorting.

I know postgres is relatively easy to set up, but have no experience with mongo. How is the setup and maintenance like?

[Kannav02]

Postgres is good when we have laid down schemas however we can have migrations in that as well meaning even if you change the fields or columns, an ORM like 'prisma' would help us with migrations , but again seeing the requirements of this feature now I believe MongoDB would be the right choice

If you say so I can start working on the hosting part as for the UI, what would you expect from that, do you want an interactive dashboard kind of UI to interact with MongoDB?

Thank you!

[luarss]

Some milestones:

1. Interactive web UI
2. Integration with ORAssistant. Pending PR merge, or you can try to do a small prototype with our existing streamlit code in frontend/
3. Export dataset using python

That would be a good amount for a PoC, then we can decide to go further with mongodb/postgres or not.

Maybe this will work? https://github.com/mongo-express/mongo-express

[Kannav02]

So right now you wouldn't want this issue to focus on DB integration before the milestones right?

And as for the Web UI, what would you want in that, is this related to the DB operations?

I can start working on these milestones if you want me to

Thank you!

[luarss]

DB integration can be done in hand with the web UI, otherwise the web UI will not show any meaningful information. I would suggest actually spending more time in the DB integration, and use something pre-built frontend like mongo-express that we can run as a docker container.

Yes, you can start working on it :)

[Kannav02]

sure i will start working on this, I will keep on posting regular updates on this issue

[Kannav02]

Hey @luarss,

Hope you're doing well,

I just had a question regarding the GOOGLE_APPLICATION_CREDENTIALS key in the backend folder, what is this supposed to do, is it the one that provides access to service accounts, is there a guide for the same that I could follow ?

Thank you!

[error9098x]

Hi @Kannav02, currently we are using GOOGLE_APPLICATION_CREDENTIALS to access a service account that grants us access to use Gemini without the rate limit that is imposed when using AI Studio platform. I hope this clarifies things.

[Kannav02]

That's perfect, I appreciate your help

Thank you!

[Kannav02]

Just a quick update on this, I am still integrating the DB, I have a follow up for the same, would you want a single Big PR comprising all the features or Individual PR's for different features referencing this issue?

[luarss]

Individual, feature-based PRs will be preferred and easier to review