ArrowInvalid: Column 4 named images expected length 360 but got length 352

[DhruvaBansal00]

Reminder

• I have read the README and searched the existing issues.

System Info

• llamafactory version: 0.9.1
• Platform: Linux-5.15.0-1048-oracle-x86_64-with-glibc2.31
• Python version: 3.11.9
• PyTorch version: 2.3.1+cu121 (GPU)
• Transformers version: 4.46.1
• Datasets version: 3.1.0
• Accelerate version: 1.0.1
• PEFT version: 0.12.0
• TRL version: 0.9.6
• GPU type: NVIDIA A100-SXM4-80GB
• DeepSpeed version: 0.14.4
• Bitsandbytes version: 0.45.0

Reproduction

### model
model_name_or_path: Qwen/Qwen2-VL-72B-Instruct

### method
stage: sft
do_train: true
finetuning_type: full
freeze_vision_tower: true  # choices: [true, false]
train_mm_proj_only: false  # choices: [true, false]
deepspeed: examples/deepspeed/ds_z3_offload_config.json  # choices: [ds_z0_config.json, ds_z2_config.json, ds_z3_config.json]
use_adam_mini: true

### dataset
dataset: mllm_demo,identity,alpaca_en_demo,slimorca
template: qwen2_vl
cutoff_len: 128000
max_samples: 25000
overwrite_cache: true
preprocessing_num_workers: 16
sequence_parallel_size: 4

### output
output_dir: saves/qwen2_vl-72b/full/sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 1
learning_rate: 1.0e-5
num_train_epochs: 30.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
packing: true
enable_liger_kernel: false
flash_attn: fa2
use_unsloth_gc: true

### eval
val_size: 0.1
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 500

### logging
report_to: wandb
run_name: qwen2vl-72b-full-sft-1

^config file for training

Stack trace:

[rank7]: ╭───────────────────── Traceback (most recent call last) ──────────────────────╮
[rank7]: │ /360-LLaMA-Factory/src/train.py:28 in <module>                               │
[rank7]: │                                                                              │
[rank7]: │   25                                                                         │
[rank7]: │   26                                                                         │
[rank7]: │   27 if __name__ == "__main__":                                              │
[rank7]: │ ❱ 28 │   main()                                                              │
[rank7]: │   29                                                                         │
[rank7]: │                                                                              │
[rank7]: │ /360-LLaMA-Factory/src/train.py:19 in main                                   │
[rank7]: │                                                                              │
[rank7]: │   16                                                                         │
[rank7]: │   17                                                                         │
[rank7]: │   18 def main():                                                             │
[rank7]: │ ❱ 19 │   run_exp()                                                           │
[rank7]: │   20                                                                         │
[rank7]: │   21                                                                         │
[rank7]: │   22 def _mp_fn(index):                                                      │
[rank7]: │                                                                              │
[rank7]: │ /360-LLaMA-Factory/src/llamafactory/train/tuner.py:50 in run_exp             │
[rank7]: │                                                                              │
[rank7]: │    47 │   if finetuning_args.stage == "pt":                                  │
[rank7]: │    48 │   │   run_pt(model_args, data_args, training_args, finetuning_args,  │
[rank7]: │    49 │   elif finetuning_args.stage == "sft":                               │
[rank7]: │ ❱  50 │   │   run_sft(model_args, data_args, training_args, finetuning_args, │
[rank7]: │    51 │   elif finetuning_args.stage == "rm":                                │
[rank7]: │    52 │   │   run_rm(model_args, data_args, training_args, finetuning_args,  │
[rank7]: │    53 │   elif finetuning_args.stage == "ppo":                               │
[rank7]: │                                                                              │
[rank7]: │ /360-LLaMA-Factory/src/llamafactory/train/sft/workflow.py:47 in run_sft      │
[rank7]: │                                                                              │
[rank7]: │    44 │   tokenizer_module = load_tokenizer(model_args)                      │
[rank7]: │    45 │   tokenizer = tokenizer_module["tokenizer"]                          │
[rank7]: │    46 │   template = get_template_and_fix_tokenizer(tokenizer, data_args)    │
[rank7]: │ ❱  47 │   dataset_module = get_dataset(template, model_args, data_args, trai │
[rank7]: │    48 │   model = load_model(tokenizer, model_args, finetuning_args, trainin │
[rank7]: │    49 │                                                                      │
[rank7]: │    50 │   if getattr(model, "is_quantized", False) and not training_args.do_ │
[rank7]: │                                                                              │
[rank7]: │ /360-LLaMA-Factory/src/llamafactory/data/loader.py:279 in                    │
[rank7]: │ sequence_parallel_processor                                                  │
[rank7]: │                                                                              │
[rank7]: │   276 │   │   │   │   if data_args.shuffle_for_sequence_parallel:            │
[rank7]: │   277 │   │   │   │   │   dataset = dataset.shuffle(seed=training_args.seed) │
[rank7]: │   278 │   │   │   │   padded_dataset = dataset.map(pad_sequence, batched=Tru │
[rank7]: │ ❱ 279 │   │   │   │   sp_dataset = padded_dataset.map(sp_split, batched=True │
[rank7]: │   280 │   │   │   │   dataset_module[k] = sp_dataset                         │
[rank7]: │   281 │   │                                                                  │
[rank7]: │   282 │   │   else:                                                          │
[rank7]: │                                                                              │
[rank7]: │ /usr/lib/python3/dist-packages/datasets/arrow_dataset.py:560 in wrapper      │
[rank7]: │                                                                              │
[rank7]: │    557 │   │   │   "output_all_columns": self._output_all_columns,           │
[rank7]: │    558 │   │   }                                                             │
[rank7]: │    559 │   │   # apply actual function                                       │
[rank7]: │ ❱  560 │   │   out: Union["Dataset", "DatasetDict"] = func(self, *args, **kw │
[rank7]: │    561 │   │   datasets: List["Dataset"] = list(out.values()) if isinstance( │
[rank7]: │    562 │   │   # re-apply format to the output                               │
[rank7]: │    563 │   │   for dataset in datasets:                                      │
[rank7]: │                                                                              │
[rank7]: │ /usr/lib/python3/dist-packages/datasets/arrow_dataset.py:3055 in map         │
[rank7]: │                                                                              │
[rank7]: │   3052 │   │   │   │   │   total=pbar_total,                                 │
[rank7]: │   3053 │   │   │   │   │   desc=desc or "Map",                               │
[rank7]: │   3054 │   │   │   │   ) as pbar:                                            │
[rank7]: │ ❱ 3055 │   │   │   │   │   for rank, done, content in Dataset._map_single(** │
[rank7]: │   3056 │   │   │   │   │   │   if done:                                      │
[rank7]: │   3057 │   │   │   │   │   │   │   shards_done += 1                          │
[rank7]: │   3058 │   │   │   │   │   │   │   logger.debug(f"Finished processing shard  │
[rank7]: │                                                                              │
[rank7]: │ /usr/lib/python3/dist-packages/datasets/arrow_dataset.py:3481 in _map_single │
[rank7]: │                                                                              │
[rank7]: │   3478 │   │   │   │   │   │   │   ):                                        │
[rank7]: │   3479 │   │   │   │   │   │   │   │   writer.write_table(batch.to_arrow())  │
[rank7]: │   3480 │   │   │   │   │   │   │   else:                                     │
[rank7]: │ ❱ 3481 │   │   │   │   │   │   │   │   writer.write_batch(batch)             │
[rank7]: │   3482 │   │   │   │   │   │   num_examples_progress_update += num_examples_ │
[rank7]: │   3483 │   │   │   │   │   │   if time.time() > _time + config.PBAR_REFRESH_ │
[rank7]: │   3484 │   │   │   │   │   │   │   _time = time.time()                       │
[rank7]: │                                                                              │
[rank7]: │ /usr/lib/python3/dist-packages/datasets/arrow_writer.py:608 in write_batch   │
[rank7]: │                                                                              │
[rank7]: │   605 │   │   │   │   arrays.append(pa.array(typed_sequence))                │
[rank7]: │   606 │   │   │   │   inferred_features[col] = typed_sequence.get_inferred_t │
[rank7]: │   607 │   │   schema = inferred_features.arrow_schema if self.pa_writer is N │
[rank7]: │ ❱ 608 │   │   pa_table = pa.Table.from_arrays(arrays, schema=schema)         │
[rank7]: │   609 │   │   self.write_table(pa_table, writer_batch_size)                  │
[rank7]: │   610 │                                                                      │
[rank7]: │   611 │   def write_table(self, pa_table: pa.Table, writer_batch_size: Optio │
[rank7]: │                                                                              │
[rank7]: │ in pyarrow.lib.Table.from_arrays:4868                                        │
[rank7]: │                                                                              │
[rank7]: │ in pyarrow.lib.Table.validate:4214                                           │
[rank7]: │                                                                              │
[rank7]: │ in pyarrow.lib.check_status:92                                               │
[rank7]: ╰──────────────────────────────────────────────────────────────────────────────╯
[rank7]: ArrowInvalid: Column 4 named images expected length 360 but got length 352

Expected behavior

Training should proceed for Qwen 2.5 VL 72b normally

Others

mllm_demo is a dataset with images. Has this repo been tested with multimodal datasets yet?

[DhruvaBansal00]

Ran the same config through without the mllm_demo dataset and training succeeded.

I am hoping to train on multimodal datasets with sequence parallelism - would love advice on how we could enable training on image datasets too.

[HaoshengZou]

Thanks for your interest!
Multimodal SP is our next internal milestone - the current release hasn't been tested with multimodal data but we are playing with them already. Stay tuned for a tested multimodal SP support!

[DhruvaBansal00]

Thanks for the update!

What timelines are you tracking internally for releasing multimodal SP? And any way I could help with adding support for multimodal SP? This is relatively high on my priority list at the moment!

[DhruvaBansal00]

@HaoshengZou following up here - curious to hear how your team is thinking about integrating this!

[HaoshengZou]

@DhruvaBansal00 Hi! Thanks for your interest! We are now at Chinese New Year and expect to get on this in two weeks.
In the meantime, feel free to contribute! We haven't had much experience with multi-modal data yet. I'm thinking that we have to pay extra attention to the sequence split for SP.

[HaoshengZou]

@DhruvaBansal00 Sorry we haven't had man power recently as focus is shifted to R1 models for now. We'll update later when we have man power.