Why is "add_question_sql" storing vectors as question+sql? SQL isn't natural language, so it wouldn't affect the semantic matching of input queries?

[qingwu11]

Why is "add_question_sql" storing vectors as question+sql? SQL isn't natural language, so it wouldn't affect the semantic matching of input queries?

[erickrribeiro]

Yes, @qingwu11. I also wondered the same thing when I saw how this id is created. I think that only the question field should be used to generate the unique code and not the question+sql

[zainhoda]

In our testing we actually did find better search by embedding the SQL query particularly if it contains specific terminology that may not be contained in the question.

However, to @erickrribeiro 's point -- I think that the ID should only be based on the question

[erickrribeiro]

Thanks @zainhoda for considering the ID suggestion based on the question. Continuing with the subject, I separated three advantages of using the ID in this way:

1. Vanna will not have two SQLs pointing to the same question. When two sqls point to the same question there may possible that one of them is incorrect, when this occurs it creates ambiguity in the system.

2. ID based on question will facilitate the process of updating the SQL of an existing document. For example, in ChromaDb you can use upsert

3. It will be easy to create a feature to update a document

[watertianyi]

@erickrribeiro Is training these ddl, sql and document files mainly for offline vector storage operations? Do these files need to be defined according to their own databases? Is each input query question converted into a vector and then the similarity calculated with the three file vectors in the vector library? Then combine it into prompt and send it to llm? Is this the workflow?