- ⌚️ Overview
- 📦 Project Framework
- ⚡ Getting Started
- 📊 Data Preparation
- 🛠️ Custom Tools
- 📜 SciToolEval
- 🖥️ Website
- 📝️ Cite
- [2024-12]: The SciToolAgent project is now available on GitHub.
SciToolAgent is a powerful agent framework designed to integrate diverse scientific tools with large language models (LLMs) to address the limitations of existing systems in scientific research. By combining LLMs as Planners, Executors, and Summarizers with a comprehensive scientific tool knowledge graph (SciToolKG), SciToolAgent autonomously plans, executes, and summarizes workflows for solving complex scientific tasks across multiple domains.
- 500+ Tools: Access to a vast array of tools, including web APIs, machine learning models, Python functions, knowledge databases, and custom tools for tasks in various scientific domains.
- SciToolKG: A comprehensive knowledge graph that models relationships among hundreds of scientific tools from biology, chemistry, and materials science. It encodes tool dependencies, prerequisites, and compatibility, enabling informed tool selection and combination.
- LLM-based Planner: Utilizes SciToolKG to autonomously plan tool sequences for problem-solving.
- LLM-based Executor: Executes the planned tools in sequence, retrying where necessary to ensure accurate results.
- LLM-based Summarizer: Compiles and synthesizes outputs from multiple tools, generating a final solution while assessing the process for improvements.
- Safety Checking: A built-in safety system that monitors tool execution to prevent harmful outcomes and ensures responsible research.
/SciToolAgent
├── app # Application directory
├── data # Data storage directory
├── KG # SciToolKG related files
├── scripts # Scripts for running SciToolAgent
├── test # Testing scripts
├── SciToolEval # SciToolEval related files
└── tools # Tool service directory
├── DataFiles # Data file storage directory (e.g., cif, csv, md, pdb, pdf, sdf, etc.)
├── LogFiles # Log files storage directory
├── TempFiles # Temporary files directory
├── TestCode # Tool test code directory
├── ToolsFuns # Core functional modules directory
├── utils # Common utility functions for tools
├── README.md # Project overview documentation
├── requirements.txt # Python dependency list
├── run.sh # One-click execution script for tool service
├── struct.md # Project structure documentation of tool service
├── config.py # Configuration file of tool service
├── example.env # Environment variable configuration file of tool service
└── tool_runner.py# Tool execution entry script
-
Clone the repository
First, clone the project to your local machine:git clone https://github.com/HICAI-ZJU/SciToolAgent.git cd SciToolAgent
-
Create and activate a virtual environment
Set up a new virtual environment using Conda and activate it:conda create -n SciToolAgent python=3.10 conda activate SciToolAgent
-
Install project dependencies Install the necessary dependencies for the project:
pip install -r requirements.txt
Optional: If you only need to use the agent part of SciToolAgent and do not require the pre-configured tools, you can install the dependencies from the requirement_agent.txt file instead:
pip install -r requirement_agent.txt
Resolve package conflicts: If you encounter any package conflicts, you can install the project without its dependencies using the following command:
pip install --no-deps -e .
You need to modify the example.env
and ToolsAgent/example.env
files to set your API_KEY
and API_BASE
.
*
OPENAI_API_BASE = your_api_base OPENAI_API_KEY = your_api_key
-
Run the Tool service
cd tools bash run.sh
For some AI model-driven tools, you also need to configure model files, paths, and environment information. You can find the corresponding tool code in
ToolsAgent/ToolsFuns
for specific modifications. -
Run the SciToolAgent
cd ../test PYTHONPATH=. python test_run_SciToolAgent.py
We also provide four case in Cases.ipynb
, which you can find and run in the root directory.
-
Fill in the tool content:
- Open the
data/your_tool_KG.xlsx
file. - Fill in the relevant information of the tool according to the format in the file, including tool name, category, function, input, output, safety, etc.
- Open the
-
Build the Knowledge Graph (KG):
- Ensure the configuration in the
scripts/generate_kg_index.py
file is correct. - Run the
scripts/generate_kg_index.py
script to build the knowledge graph. - The main function of the
scripts/generate_kg_index.py
file is to load data from the Excel file, create triplets, and build the knowledge graph. The main steps of the file are as follows: - Load data:
df = load_data(Config().DATA_FILE_PATH)
- Create triplets:
triplets = create_triplets(df)
- Build and save the knowledge graph:
build_knowledge_graph(triplets, Config().PERSIST_DIR)
- Run the following command in the terminal:
python scripts/generate_kg_index.py
- Ensure the configuration in the
By following the above steps, you can successfully build the knowledge graph.
You can add custom tools by following these steps:
-
Create a new tool in Python under
ToolsAgent/ToolsFuns
. You can follow the existing tools as examples. Format:def custom_tool(parameter: str): # Your code here return result
-
Add the tool name to
tool_name_dict.py
andtools_dict.py
. -
Restart the tool service to apply the changes.
{
"question": "",
"final_answer": ""
}
You can use eval/eval_accuracy.py
to evaluate the accuracy of the results and eval/eval_tool_path.py
to evaluate the tool path.
python eval_accuracy.py
--input_file example_input.jsonl
--standard_file example_standard_answers.jsonl
--output_file example_accuracy_evaluation_results.jsonl
python eval_tool_path.py
--input_file example_input.jsonl
--standard_file example_standard_answers.jsonl
--tool_description_file example_tools_dict.json
--output_file example_toolpath_evaluation_results.jsonl
input_file
is the result of your agent's answer, standard_file
is the standard answer file containing answer or tool_path, tool_description_file
is the tool description file, and output_file
is the evaluation output result file.
We provide an online service for SciToolAgent, you can access it through the following link: http://scitoolagent.scimind.ai:8080/
@article{
title={SciToolAgent: Knowledge Graph-Driven Scientific Agent for Multi-Tool Integration},
author={Keyan Ding, Jing Yu, Junjie Huang, Yuchen Yang, Qiang Zhang, Huajun Chen},
journal={bioRxiv}
year={2024},
}