Improve GenerationStats to Handle Summation of embed_stats
and generation_stats
in RAGConvert.convert()
#131
Labels
invalid
This doesn't seem right
Summing the
embed_stats
andgeneration_stats
leads to correct calculations of operator-record, operator, plan, and execution stats.However, if one were to analyze our stats in post-processing -- they would have no way of parsing which e.g.
total_input_tokens
came from the embedding step and which came from the generation step.The text was updated successfully, but these errors were encountered: