Skip to content

add bench eval pipeline (string match and semantic match) #238

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Sep 10, 2025
Merged

add bench eval pipeline (string match and semantic match) #238

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
c94b46f
add bench eval pipeline
scuuy Aug 31, 2025
39ad340
eval module
scuuy Sep 4, 2025
fcffe2e
0905 eval
scuuy Sep 5, 2025
e2d7951
fix get_desc
scuuy Sep 5, 2025
f060cc9
remove useless prompt
scuuy Sep 5, 2025
3e9d04d
rename reasoning ops
scuuy Sep 5, 2025
a09cd68
fix some unchanged rename bug in pipeline
scuuy Sep 5, 2025
62c351c
fix name bug in init
scuuy Sep 5, 2025
00c477d
fix name bug in init
scuuy Sep 5, 2025
e19dba9
fix name bug in init
scuuy Sep 5, 2025
367dadd
rename the core op, BenchDatasetEvaluator
scuuy Sep 8, 2025
3643989
add timestamp suffix
scuuy Sep 8, 2025
a413bc1
using folder and timestamp to save result
scuuy Sep 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dataflow/agent/toolkits/pipeline_processor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -494,5 +494,5 @@ def local_tool_for_update_operators_info():
# "/mnt/h_h_public/lh/lz/DataFlow/dataflow/agent/toolkits/resources/Operator.json",
# save_as="/mnt/h_h_public/lh/lz/DataFlow/dataflow/agent/toolkits/resources/Operator_patched.json"
# )
# pre_result = {'edges': [{'source': 'node0', 'target': 'node1'}, {'source': 'node1', 'target': 'node2'}, {'source': 'node2', 'target': 'node3'}, {'source': 'node3', 'target': 'node4'}, {'source': 'node4', 'target': 'node5'}, {'source': 'node5', 'target': 'node6'}, {'source': 'node6', 'target': 'node7'}, {'source': 'node7', 'target': 'node8'}], 'reason': "The pipeline is designed to process mathematics and science content with a focus on prediction without deduplication. Starting with QuestionGenerator to create new questions, followed by QuestionFilter to ensure correctness. Then, QuestionDifficultyClassifier and QuestionCategoryClassifier are used to categorize and label the questions. AnswerPipelineRoot separates data with and without answers. PseudoAnswerGenerator generates candidate answers, and AnswerFormatterFilter ensures the answers meet format requirements. Finally, AnswerTokenLengthFilter checks for appropriate answer lengths. This sequence ensures comprehensive processing while adhering to the user's requirements.", 'outputs': [], 'nodes': [{'name': 'QuestionGenerator', 'type': 'generator', 'description': '基于现有的问题数据，每个问题合成1-5个新问题', 'command': {'init': [{'name': 'num_prompts', 'default': 1, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'llm_serving', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': 'GPU', 'id': 'node1'}, {'name': 'QuestionFilter', 'type': 'filter', 'description': '检查每个问题的正确性', 'command': {'init': [{'name': 'system_prompt', 'default': 'You are a helpful assistant.', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'llm_serving', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': 'math_problem', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': 'GPU', 'id': 'node2'}, {'name': 'QuestionDifficultyClassifier', 'type': 'generator', 'description': '为每个问题确定一个难度分数标签', 'command': {'init': [{'name': 'llm_serving', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'output_key', 'default': 'difficulty_score', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': 'GPU', 'id': 'node3'}, {'name': 'QuestionCategoryClassifier', 'type': 'generator', 'description': '将所有问题分类到7个大类别，以及每个大类别下的若干的小类别', 'command': {'init': [{'name': 'llm_serving', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': 'instruction', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'output_key', 'default': 'question_category', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': 'GPU', 'id': 'node4'}, {'name': 'AnswerPipelineRoot', 'type': 'generator', 'description': '用于检查数据是否包含Answer、groundtruth，并分离有答案和没答案的数据，方便后续分别处理', 'command': {'init': [], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_answer_key', 'default': 'output', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_gt_key', 'default': 'golden_answer', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': '', 'id': 'node5'}, {'name': 'PseudoAnswerGenerator', 'type': 'generator', 'description': '该算子能够生成多个候选答案，并通过统计方法选择最优解，从而实现伪答案的生成。其输入参数包括：输入文件路径（input_file）、输出文件路径（output_file）、最大生成次数（max_times）以及统计选择模式（selection_mode，可选frequency或consistency）。输出参数包括：最终选择的答案字段（final_answer）和候选答案列表字段（candidate_answers）。该算子能够生成多个候选答案，并通过统计方法选择最优解，从而实现伪答案的生成。其输入参数包括：输入文件路径（input_file）、输出文件路径（output_file）、最大生成次数（max_times）以及统计选择模式（selection_mode，可选frequency或consistency）。输出参数包括：最终选择的答案字段（final_answer）和候选答案列表字段（candidate_answers）。', 'command': {'init': [{'name': 'llm_serving', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'max_times', 'default': 3, 'kind': 'POSITIONAL_OR_KEYWORD'}], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': 'instruction', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'output_key_answer', 'default': 'pseudo_answers', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'output_key_answer_value', 'default': 'pseudo_answer_value', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'output_key_solutions', 'default': 'pseudo_solutions', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'output_key_correct_solution_example', 'default': 'pseudo_correct_solution_example', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': 'GPU', 'id': 'node6'}, {'name': 'AnswerFormatterFilter', 'type': 'processor', 'description': '按照给定的格式，基于规则过滤掉不符合格式要求的数据', 'command': {'init': [], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': 'generated_cot', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': '', 'id': 'node7'}, {'name': 'AnswerTokenLengthFilter', 'type': 'processor', 'description': '过滤掉Answer长度不合适的数据', 'command': {'init': [{'name': 'max_answer_token_length', 'default': 8192, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'tokenizer_dir', 'default': 'Qwen/Qwen2.5-0.5B-Instruct', 'kind': 'POSITIONAL_OR_KEYWORD'}], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': 'generated_cot', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': '', 'id': 'node8'}], 'name': 'fcfa5825-d28b-4eba-ad17-7a46d7cc2fc7_pipeline'}
# pre_result = {'edges': [{'source': 'node0', 'target': 'node1'}, {'source': 'node1', 'target': 'node2'}, {'source': 'node2', 'target': 'node3'}, {'source': 'node3', 'target': 'node4'}, {'source': 'node4', 'target': 'node5'}, {'source': 'node5', 'target': 'node6'}, {'source': 'node6', 'target': 'node7'}, {'source': 'node7', 'target': 'node8'}], 'reason': "The pipeline is designed to process mathematics and science content with a focus on prediction without deduplication. Starting with QuestionGenerator to create new questions, followed by QuestionFilter to ensure correctness. Then, ReasoningDifficultyDatasetEvaluator and ReasoningCategoryDatasetEvaluator are used to categorize and label the questions. AnswerPipelineRoot separates data with and without answers. PseudoAnswerGenerator generates candidate answers, and AnswerFormatterFilter ensures the answers meet format requirements. Finally, AnswerTokenLengthFilter checks for appropriate answer lengths. This sequence ensures comprehensive processing while adhering to the user's requirements.", 'outputs': [], 'nodes': [{'name': 'QuestionGenerator', 'type': 'generator', 'description': '基于现有的问题数据，每个问题合成1-5个新问题', 'command': {'init': [{'name': 'num_prompts', 'default': 1, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'llm_serving', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': 'GPU', 'id': 'node1'}, {'name': 'QuestionFilter', 'type': 'filter', 'description': '检查每个问题的正确性', 'command': {'init': [{'name': 'system_prompt', 'default': 'You are a helpful assistant.', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'llm_serving', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': 'math_problem', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': 'GPU', 'id': 'node2'}, {'name': 'ReasoningDifficultyDatasetEvaluator', 'type': 'generator', 'description': '为每个问题确定一个难度分数标签', 'command': {'init': [{'name': 'llm_serving', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'output_key', 'default': 'difficulty_score', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': 'GPU', 'id': 'node3'}, {'name': 'ReasoningCategoryDatasetEvaluator', 'type': 'generator', 'description': '将所有问题分类到7个大类别，以及每个大类别下的若干的小类别', 'command': {'init': [{'name': 'llm_serving', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': 'instruction', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'output_key', 'default': 'question_category', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': 'GPU', 'id': 'node4'}, {'name': 'AnswerPipelineRoot', 'type': 'generator', 'description': '用于检查数据是否包含Answer、groundtruth，并分离有答案和没答案的数据，方便后续分别处理', 'command': {'init': [], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_answer_key', 'default': 'output', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_gt_key', 'default': 'golden_answer', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': '', 'id': 'node5'}, {'name': 'PseudoAnswerGenerator', 'type': 'generator', 'description': '该算子能够生成多个候选答案，并通过统计方法选择最优解，从而实现伪答案的生成。其输入参数包括：输入文件路径（input_file）、输出文件路径（output_file）、最大生成次数（max_times）以及统计选择模式（selection_mode，可选frequency或consistency）。输出参数包括：最终选择的答案字段（final_answer）和候选答案列表字段（candidate_answers）。该算子能够生成多个候选答案，并通过统计方法选择最优解，从而实现伪答案的生成。其输入参数包括：输入文件路径（input_file）、输出文件路径（output_file）、最大生成次数（max_times）以及统计选择模式（selection_mode，可选frequency或consistency）。输出参数包括：最终选择的答案字段（final_answer）和候选答案列表字段（candidate_answers）。', 'command': {'init': [{'name': 'llm_serving', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'max_times', 'default': 3, 'kind': 'POSITIONAL_OR_KEYWORD'}], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': 'instruction', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'output_key_answer', 'default': 'pseudo_answers', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'output_key_answer_value', 'default': 'pseudo_answer_value', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'output_key_solutions', 'default': 'pseudo_solutions', 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'output_key_correct_solution_example', 'default': 'pseudo_correct_solution_example', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': 'GPU', 'id': 'node6'}, {'name': 'AnswerFormatterFilter', 'type': 'processor', 'description': '按照给定的格式，基于规则过滤掉不符合格式要求的数据', 'command': {'init': [], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': 'generated_cot', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': '', 'id': 'node7'}, {'name': 'AnswerTokenLengthFilter', 'type': 'processor', 'description': '过滤掉Answer长度不合适的数据', 'command': {'init': [{'name': 'max_answer_token_length', 'default': 8192, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'tokenizer_dir', 'default': 'Qwen/Qwen2.5-0.5B-Instruct', 'kind': 'POSITIONAL_OR_KEYWORD'}], 'run': [{'name': 'storage', 'default': None, 'kind': 'POSITIONAL_OR_KEYWORD'}, {'name': 'input_key', 'default': 'generated_cot', 'kind': 'POSITIONAL_OR_KEYWORD'}]}, 'required': True, 'depends_on': [], 'mode': '', 'id': 'node8'}], 'name': 'fcfa5825-d28b-4eba-ad17-7a46d7cc2fc7_pipeline'}
# print(local_tool_for_execute_the_recommended_pipeline(pre_result,dry_run=True))
10 changes: 5 additions & 5 deletions dataflow/agent/toolkits/resources/Operator_patched.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5541,7 +5541,7 @@
},
{
"node": 134,
"name": "QuestionCategoryClassifier",
"name": "ReasoningCategoryDatasetEvaluator",
"description": "该算子用于对用户问题进行多级分类（主分类和子分类）。通过大语言模型对输入问题进行语义分析，输出分类编码结果。\n\n输入参数：\n- db_port/db_name/table_name：数据库连接参数（存储模式）\n- input_file/output_file：文件路径（文件模式）\n- input_key：输入数据中问题字段的键名\n- generator_type：模型调用方式（aisuite/request）\n\n输出参数：\n- classification_result：包含主分类和子分类的编码结果",
"command": {
"init": [
Expand Down Expand Up @@ -5575,8 +5575,8 @@
},
{
"node": 135,
"name": "QuestionDifficultyClassifier",
"description": "该算子用于评估问题的难度等级。通过大语言模型分析问题复杂度，输出1-10级的难度评分。\n\n输入参数：\n- eval_stage：评估阶段标识\n- read_min/max_score：分数过滤阈值\n- 其他参数同QuestionCategoryClassifier\n\n输出参数：\n- difficulty_score：数值型难度评分（1-10）",
"name": "ReasoningDifficultyDatasetEvaluator",
"description": "该算子用于评估问题的难度等级。通过大语言模型分析问题复杂度，输出1-10级的难度评分。\n\n输入参数：\n- eval_stage：评估阶段标识\n- read_min/max_score：分数过滤阈值\n- 其他参数同ReasoningCategoryDatasetEvaluator\n\n输出参数：\n- difficulty_score：数值型难度评分（1-10）",
"command": {
"init": [
{
Expand Down Expand Up @@ -9512,7 +9512,7 @@
},
{
"node": 3,
"name": "QuestionDifficultyClassifier",
"name": "ReasoningDifficultyDatasetEvaluator",
"type": "generator",
"description": "为每个问题确定一个难度分数标签",
"input": [
Expand Down Expand Up @@ -9562,7 +9562,7 @@
},
{
"node": 4,
"name": "QuestionCategoryClassifier",
"name": "ReasoningCategoryDatasetEvaluator",
"type": "generator",
"description": "将所有问题分类到7个大类别，以及每个大类别下的若干的小类别",
"input": [
Expand Down
Loading
Loading