Unexpected high memory usage in AppWorld Env leading to Ray OOM

[jingyi-xioa]

Hi ,
I have been studying your paper AgentEvolver and I find it to be a very impressive and valuable piece of work.
However, I encountered a issue while reproducing the code with the AppWorld environment. The memory usage on my node spiked to 1.4 TB, triggering Ray's 0.95 threshold and causing the worker to be killed.
It appears to be a memory leak. The logs show that two single RemoteEnv processes consumed over 500 GB of memory each, which is abnormal.
Logs are shown below:
Memory on the node ... was 1406.36GB / 1480.00GB (0.950242)
Top 10 memory users:
PID MEM(GB) COMMAND
70473 536.45 ray::RemoteEnv.step <-- !!!
70355 533.25 ray::RemoteEnv.step <-- !!!
Have you encountered this excessive memory consumption during your experiments? Could you please advise on which part of the code I should inspect to fix this leak?

[Kanaricc]

Glad to hear that our work is useful for you!

To look into this, we'll need a bit more context:

• All hyperparamters
• The script to launch the training
• Logs for this particular session (AgentEvolver and EnvService)

[jingyi-xioa]

Thanks for your reply and here is the context:

1.Hyperparameters I used the configuration provided in your repo (examples/overall.yaml), so the hyperparameters are identical to the defaults.

2.Launch Script I followed the instructions in the README:python launcher.py --conf examples/overall.yaml --with-appworld --with-reme

3.Here is the full error log:
Memory on the node (IP: 172.18.36.176, ID: 83299264eaf36ac7d7d247abfc897540aad18ed8a07c54d9a14c6423) where the task (actor ID: 548426920cfe97542d2892fe01000000, name=async_llm_server_0:AsyncvLLMServer.init, pid=40721, memory used=0.69GB) was running was 1406.36GB / 1480.00GB (0.950242), which exceeds the memory usage threshold of 0.95. Ray killed this worker (ID: 06ccd434c83002cc9c8ab972414613077f27556a448db16c11115cd2) because it was the most recently scheduled task; to see more information about memory usage on this node, use ray logs raylet.out -ip 172.18.36.176. To see the logs of the worker, use ray logs worker-06ccd434c83002cc9c8ab972414613077f27556a448db16c11115cd2*out -ip 172.18.36.176. Top 10 memory users: PID MEM(GB) COMMAND 70473 536.45 ray::RemoteEnv.step 70355 533.25 ray::RemoteEnv.step 38784 34.46 ray::WorkerDict 38791 34.19 ray::WorkerDict 38790 34.18 ray::WorkerDict 38789 34.15 ray::WorkerDict 38786 33.77 ray::WorkerDict 38788 33.76 ray::WorkerDict 38553 33.71 ray::WorkerDict 38787 33.70 ray::WorkerDict Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. Set max_restarts and max_task_retries to enable retry when the task crashes due to OOM. To adjust the kill threshold, set the environment variable RAY_memory_usage_thresholdwhen starting Ray. To disable worker killing, set the environment variableRAY_memory_monitor_refresh_ms` to zero.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Found success string 'Starting server on' in output
Successfully launched ['APPWORLD_PORT=8081 APPWORLD_HOST=127.0.0.1 source /dfs/share-groups/letrain/install/glm/miniconda/etc/profile.d/conda.sh && conda activate appworld && bash appworld.sh'] with PID 12387
Script is already running, skipping launch
Running command

Thanks for your assistance

[Kanaricc]

Thanks for the details. Here are some of my initial thoughts:

Given that AppWorld is an external benchmark that executes LLM-generated scripts, it is susceptible to occasional performance issues like memory leaks or infinite loops. We are planning to implement resource constraints to suppress these types of issues and improve system stability.

In the meantime, if you're willing to provide more information, it would be very helpful to know:

1. Does this issue persist if you kill the EnvService and restart the training from scratch?
2. Could you share the complete logs to help us pinpoint the root cause?

[TonyHuBD]

(TaskRunner pid=1400931) ERROR:2025-12-29 11:47:21,721:chat completion failed with exception: Cannot connect to host 10.75.75.
23:42657 ssl:default [Connect call failed ('10.75.75.23', 42657)]                                                             
(TaskRunner pid=1400931) NoneType: None                                                                                       
Epoch test.1.0: Collecting rollouts: 100%|██████████| 456/456 [10:21:07<00:Epoch test.1.0: Collecting rollouts: 100%|█████████
█| 456/456 [10:21:07<00:00, 81.73s/it]                                                                                        
(TaskRunner pid=1400931) Rollout progress (0.00 tokens/s): [finished]:456 threads                                             
(TaskRunner pid=1400931) ==========end validate rollout==========                                                             
Error executing job with overrides: []                                                                                        
Traceback (most recent call last):                                                                                            
  File "/mnt/share/hbd/AgentEvolver/agentevolver/main_ppo.py", line 144, in main                                              
    run_ppo(config)  # ⭐ Executes the PPO training process with the given configuration                                      
    ^^^^^^^^^^^^^^^                                                                                                           
  File "/mnt/share/hbd/AgentEvolver/agentevolver/main_ppo.py", line 170, in run_ppo                                           
    ray.get(runner.run.remote(config))  # ⭐ Start the PPO training process by calling the run method on the TaskRunner actor 
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                        
  File "/home/gpt/anaconda3/envs/agentevolver/lib/python3.11/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_i
nit_wrapper                                                                                                                   
    return fn(*args, **kwargs)                                                                                                
           ^^^^^^^^^^^^^^^^^^^                                                                                                
  File "/home/gpt/anaconda3/envs/agentevolver/lib/python3.11/site-packages/ray/_private/client_mode_hook.py", line 104, in wra
pper                                                                                                                          
    return func(*args, **kwargs)                                                                                              
           ^^^^^^^^^^^^^^^^^^^^^                                                                                              
  File "/home/gpt/anaconda3/envs/agentevolver/lib/python3.11/site-packages/ray/_private/worker.py", line 2849, in get         
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)                                            
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                            
  File "/home/gpt/anaconda3/envs/agentevolver/lib/python3.11/site-packages/ray/_private/worker.py", line 937, in get_objects  
    raise value.as_instanceof_cause()                                                                                         
ray.exceptions.RayTaskError(OutOfMemoryError): ray::TaskRunner.run() (pid=1400931, ip=10.75.75.23, actor_id=36f0f7034e694f59b7
be6c2e01000000, repr=<main_ppo.TaskRunner object at 0x7ed959636910>)                                                          
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                               
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                    
  File "/mnt/share/hbd/AgentEvolver/agentevolver/main_ppo.py", line 336, in run                                               
    trainer.fit()  # ⭐ Start the training process                                                                            
    ^^^^^^^^^^^^^              
  File "/mnt/share/hbd/AgentEvolver/agentevolver/module/trainer/ae_ray_trainer.py", line 1055, in fit
    self.initialize_exp_pool()                                 
  File "/mnt/share/hbd/AgentEvolver/agentevolver/module/trainer/ae_ray_trainer.py", line 1019, in initialize_exp_pool
    self.async_rollout_manager.sleep()                                                                                        
  File "/home/gpt/anaconda3/envs/agentevolver/lib/python3.11/site-packages/verl/workers/rollout/async_server.py", line 187, in
 sleep                         
    ray.get([server.sleep.remote() for server in self.async_llm_servers])                                                     
           ^^^^^^^^^^^^^^^^^^^                                 
           ^^^^^^^^^^^^^^^^^^^^^                               
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ray.exceptions.OutOfMemoryError: Task was killed due to the node running low on memory.
Memory on the node (IP: 10.75.75.23, ID: 0d30a01d3b05bfeb1d988648bc0fe2f2219098dbb71281c67aafd6ad) where the task (actor ID: 2
d56aeacfb6c038d4489fe3801000000, name=async_llm_server_0:AsyncvLLMServer.__init__, pid=1417258, memory used=0.62GB) was runnin
g was 957.14GB / 1007.51GB (0.950003), which exceeds the memory usage threshold of 0.95. Ray killed this worker (ID: beae187a3
52383cf0804baf6ac3f5946e24c2411dcca92d4546b142f) because it was the most recently scheduled task; to see more information abou
t memory usage on this node, use `ray logs raylet.out -ip 10.75.75.23`. To see the logs of the worker, use `ray logs worker-be
ae187a352383cf0804baf6ac3f5946e24c2411dcca92d4546b142f*out -ip 10.75.75.23. Top 10 memory users:
PID     MEM(GB) COMMAND                                        
2050661 329.32  ray::RemoteEnv.step                            
1408295 68.45   ray::WorkerDict                                
1408900 68.24   ray::WorkerDict                                
1408902 67.32   ray::WorkerDict                                
1408898 67.21   ray::WorkerDict                                
1408901 66.96   ray::WorkerDict                                
1408903 66.93   ray::WorkerDict                                
1408897 66.89   ray::WorkerDict                                
1408899 66.89   ray::WorkerDict                                
1392315 1.07    /home/gpt/anaconda3/envs/agentevolver/lib/python3.11/site-packages/ray/core/src/ray/gcs/gcs_server -...
Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oo
m-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per tas
k. Set max_restarts and max_task_retries to enable retry when the task crashes due to OOM.

Same issue.