在使用多进程+hf datasets.map做tokenize的时候会报no space left on device

[rangehow]

大家碰到过这个问题吗，我的磁盘空间非常充足，不知道这个space指得是什么

12:34:06.420 tokenize (num_proc=90):  60%|██████    | 565187/939344 [03:38<38:29, 162.03 examples/s] [rank3]: multiprocess.pool.RemoteTraceback: 
12:34:06.420 [rank3]: """
12:34:06.420 [rank3]: Traceback (most recent call last):
12:34:06.420 [rank3]:   File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/ruanjunhao04/env/rjh/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3534, in _map_single
12:34:06.420 [rank3]:     writer.write_batch(batch)
12:34:06.420 [rank3]:   File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/ruanjunhao04/env/rjh/lib/python3.12/site-packages/datasets/arrow_writer.py", line 609, in write_batch
12:34:06.420 [rank3]:     self.write_table(pa_table, writer_batch_size)
12:34:06.420 [rank3]:   File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/ruanjunhao04/env/rjh/lib/python3.12/site-packages/datasets/arrow_writer.py", line 627, in write_table
12:34:06.420 [rank3]:     self.pa_writer.write_table(pa_table, writer_batch_size)
12:34:06.420 [rank3]:   File "pyarrow/ipc.pxi", line 529, in pyarrow.lib._CRecordBatchWriter.write_table
12:34:06.420 [rank3]:   File "pyarrow/error.pxi", line 89, in pyarrow.lib.check_status
12:34:06.420 [rank3]:   File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/ruanjunhao04/env/rjh/lib/python3.12/site-packages/fsspec/implementations/local.py", line 389, in write
12:34:06.420 [rank3]:     return self.f.write(*args, **kwargs)
12:34:06.420 [rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
12:34:06.420 [rank3]: OSError: [Errno 28] No space left on device
12:34:06.420 
12:34:06.420 [rank3]: During handling of the above exception, another exception occurred:
12:34:06.420 
12:34:06.420 [rank3]: Traceback (most recent call last):
12:34:06.420 [rank3]:   File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/ruanjunhao04/env/rjh/lib/python3.12/site-packages/multiprocess/pool.py", line 125, in worker
12:34:06.420 [rank3]:     result = (True, func(*args, **kwds))
12:34:06.420 [rank3]:                     ^^^^^^^^^^^^^^^^^^^
12:34:06.420 [rank3]:   File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/ruanjunhao04/env/rjh/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
12:34:06.420 [rank3]:     for i, result in enumerate(func(**kwargs)):
12:34:06.420 [rank3]:   File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/ruanjunhao04/env/rjh/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3546, in _map_single
12:34:06.420 [rank3]:     writer.finalize()
12:34:06.420 [rank3]:   File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/ruanjunhao04/env/rjh/lib/python3.12/site-packages/datasets/arrow_writer.py", line 644, in finalize
12:34:06.420 [rank3]:     self.stream.close()
12:34:06.420 [rank3]:   File "/mnt/dolphinfs/hdd_pool/docker/user/hadoop-aipnlp/ruanjunhao04/env/rjh/lib/python3.12/site-packages/fsspec/implementations/local.py", line 407, in close
12:34:06.420 [rank3]:     return self.f.close()
12:34:06.420 [rank3]:            ^^^^^^^^^^^^^^
12:34:06.420 [rank3]: OSError: [Errno 28] No space left on device
12:34:06.420 [rank3]: """