[BUG] dpgen is stucked in the stage of "data stating... (this step may take long time)"

[LingjunWu92]

Bug summary

Dear Developers,

I found that strangely my dpgen run is stucked at the stage of "data stating... (this step may take long time)", and sometimes lasted for even 20 hours (I don't know what will happen after 20 hours as I just killed the job then), and while it stucked, no outputs were produced, also no errors were reported either. However, the init_bulk stage is totally fine, and I have also tried the examples downloaded from the deepmd website, which turned out absolutely successful either in the init_bulk stage or in the run stage, so it seems very strange why my task just stucked at the run stage. I uploaded my input and ouput data as attached, could you please give me some advice on what might go wrong? Thank you so much.

Best wishes,
Lingjun

DeePMD-kit Version

3.1.1

Backend and its version

TensorFlow 2.19.1

How did you download the software?

Offline packages

Input Files, Running Commands, Error Log, etc.

issue.zip

The attached zip file includes all related files for the dpgen task.

Steps to Reproduce

I just used "dpgen init_bulk param.json machine.json" for the init_bulk stage, and "dpgen run param.json machine.json" for the run stage. The param.json and machine.json files for each stage are uploaded before in the zip file.

Further Information, Files, and Links

No response

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Possible Duplicates

• data stating takes too long #4454

Related Issues

• ValueError: need at least one array to concatenate #4373
• the MD running is only working when using CPU #5092
• FileNotFoundError when Train DPA-2 model from scratch，but the file actually exists #4422
• Encounter program crash when using the deepmdkit and dpgen #4462

👤 Suggested Assignees

• cjxxjc729
• niuniu800
• LingjunWu92
• zhubro321
• zhanghuang103031

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[dosubot]

Hi @LingjunWu92! I'm Dosu and I’m helping the deepmd-kit team.

This kind of hang at the "data stating..." stage is often caused by issues in your input data or configuration. Based on similar reports, the most common causes are:

• A mismatch between the type_map in your param.json and the actual atom types present in your input data (like POSCAR/CONTCAR files). If your data contains atom types not listed in type_map, or the order doesn't match, it can cause hangs or TensorFlow shape errors during this stage. Double-check that every atom type in your data is listed in type_map and in the correct order. See this discussion and this one.

• Problems with your dataset, such as systems with no frames or mixing different systems in one element in sys_configs, can also cause hangs or silent failures. Make sure all referenced systems contain valid frames and are not empty, and that you don't mix different systems in a single element of sys_configs (reference).

• Leftover files from previous runs can sometimes cause dpgen to hang. Clean your working directory of any files from previous attempts before restarting (reference).

• Outdated versions of dpdata can cause issues with empty systems. Upgrading dpdata to the latest version is recommended (reference).

If you check these points and still have issues, try enabling verbose or debug logging in dpgen to get more detailed output during the "data stating..." stage. If you can, paste the contents of your param.json and machine.json here, along with a directory listing, so it's easier to spot any configuration issues.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

  

[LingjunWu92]

Dear dosu @dosu, I tried all your suggestions, but still none of these worked. Could you please help me on what might be wrong? I have attached the new documents below. Many thanks.

Best wishes,
Lingjun

issue2.zip

[dosubot]

Based on similar cases, the most common reasons for dpgen hanging at the "data stating..." stage are issues with your input data or configuration—especially with sys_configs paths, type_map mismatches, or empty/missing data folders. If the confs folder inside 01.model_devi is empty, or if sys_configs points to non-existent or misformatted files, dpgen can hang without errors. Double-check that every path in sys_configs is correct and accessible, and that all referenced data files (like POSCARs) exist and match your type_map. Also, make sure to clean out any files from previous runs before restarting, as leftover files can cause silent hangs source source.

If everything looks correct, try running a minimal test with just one valid system in sys_configs to see if the problem persists. If it does, enabling verbose or debug logging in dpgen may help pinpoint where the process is getting stuck.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

  

[LingjunWu92]

Dear dosu @dosu, I tried to run with only one valid system in sys_configs, however, it still doesn't work. Could you please tell me what might be wrong and how to enable verbose or debug logging in dpgen? Many thanks.

Best wishes,
Lingjun

[dosubot]

To enable debug logging in dpgen, you need to manually change the log level in the source code. In your dpgen installation, open the file dpgen/__init__.py and look for the line where the logger dlog is configured:

dlog.setLevel(logging.INFO)

Change it to:

dlog.setLevel(logging.DEBUG)

Then rerun your dpgen workflow. This will produce much more detailed output in the dpgen.log file in your working directory. Watch for the last log entry before the process hangs, any warnings or errors, or repeated operations—these can help pinpoint where things are getting stuck source.

Common causes for hanging at "data stating..." include: mismatches between type_map and your input atom types, empty or missing data files, incorrect paths in sys_configs, or leftover files from previous runs. Make sure all referenced files exist and match your configuration, and clean out any old files before retrying.

If the debug log doesn't reveal the issue, share the relevant portion of dpgen.log and a minimal reproducible example for further diagnosis.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

  

[LingjunWu92]

Dear dosu @dosu, I tried to change the init.py file, however I found that there is no extra output in the dpgen.log file after this change, still no outputs and seems got stucked. I have attached the dpgen.log and the related other files in zip, please can you give me some advice? Many thanks.

issue3.zip

dpgen.log

[dosubot]

The lack of extra output in dpgen.log after changing the log level is likely due to how logging is configured in dpgen. A recent update sets the root logger to WARNING, which can override your changes and prevent DEBUG messages from appearing in the log. Also, the logger disables propagation to the root logger, so changing the log level in init.py alone may not be enough to get more verbose output details.

As for the hang at "data stating...", the most common causes are: leftover files from previous runs, mismatches between type_map and your input atom types, empty or missing data files, or invalid sys_configs paths. Make sure to fully clean your working directory before restarting, double-check all referenced files and paths, and confirm that your input data matches your configuration. Upgrading dpdata can also help if your systems contain no frames reference.

If you want to force more verbose logging, try adding logging.basicConfig(level=logging.DEBUG) at the very top of your dpgen/init.py, before any other logger configuration. This may help ensure DEBUG messages are written to dpgen.log.

If the issue persists, please share the contents of your param.json, machine.json, and a directory listing, so I can help spot any configuration or data issues.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

  

[LingjunWu92]

Dear dosu @dosu, I tried to change the init.py file, however I found that there is no extra output in the dpgen.log file after this change, still no outputs and seems got stucked.

[dosubot]

Changing the log level in __init__.py alone won't give you more output because dpgen's logging config sets the root logger to WARNING and disables propagation, so DEBUG messages are filtered out before they reach your log file — this is a known behavior in recent versions (details).

The hang at "data stating..." is almost always due to a data or configuration issue, not a logging problem. The most common causes are: a mismatch between type_map and the actual atom types in your data, empty or missing frames, invalid or missing files in sys_configs, or leftover files from previous runs (example).

Here's what you can do:

• Double-check that every atom type in your input files is listed in type_map and in the correct order.
• Make sure all paths in sys_configs are correct and accessible, and that all referenced files exist and are not empty.
• Clean your working directory of any files from previous runs before restarting.
• Ensure all element symbols in your input files are valid and supported.

If you want to force debug output, add logging.basicConfig(level=logging.DEBUG) at the very top of dpgen/__init__.py (before any other logging setup). If that still doesn't help, try inserting print() statements directly in the code around the "data stating..." step to see where it hangs.

If you can, paste the contents of your param.json, machine.json, and a directory listing here — I can help review them for common issues. If the problem persists, try updating both dpgen and dpdata to the latest versions, as some related bugs have been fixed recently.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

  

[suntaozhi]

Hi, may I ask whether you are using an NVIDIA RTX 5090 GPU?
I’m asking because I’ve encountered a very similar issue on my side, and I’m wondering if it could be related to the GPU model.