Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a script for evaluation in Enterprise #3541

Open
chloedia opened this issue Jan 2, 2025 — with Linear · 1 comment
Open

Create a script for evaluation in Enterprise #3541

chloedia opened this issue Jan 2, 2025 — with Linear · 1 comment
Labels
area: scripts Related to ad-hoc scripts, SQL migrations, or under the /scripts directory

Comments

@chloedia Linear
Copy link
Collaborator

chloedia commented Jan 2, 2025

The scripts

  • takes a config and spawn supporting services (S3/DB)
  • Split the RAG in 3 tasks (Parsing, Retrieval, Generation)
  • Get specific datasets for evaluating each tasks from the Exp tracker
  • Run the task on their dataset
  • Compute the metrics based on GT and predicted results (This can be really simple logic at first and will iteratively grow complex)
  • Save Metrics in Exp Tracker
@linear Linear
Copy link

linear bot commented Jan 2, 2025

CORE-323 Create a script for evaluation in Enterprise

@dosubot dosubot bot added the area: scripts Related to ad-hoc scripts, SQL migrations, or under the /scripts directory label Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area: scripts Related to ad-hoc scripts, SQL migrations, or under the /scripts directory
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@chloedia