How to recover stage quantization from finetuning stage after an error

[jiangjiadi]

After executing examples/quantization_2of4_sparse_w4a16/llama7b_sparse_w4a16.py, I obtained three models at different stages: stage_sparsity, stage_finetuning, and stage_quantization. My question is, if an error occurs while running the stage_quantization phase, how can I resume the process from the stage_finetuning model? Given that both the stage_sparsity and stage_finetuning phases are resource-intensive, restarting from the beginning would be inefficient and time-consuming.

[dsikka]

Hi @jiangjiadi if you update your recipe with the remaining quantization stage and start with the finetuned model, you should be able to run quantization without repeating the other stages. Did you run into an issue while loading the finetuned model?

[jiangjiadi]

@dsikka When I start with the stage_finetuning model, I get this error.

The recipe I used only contains quantization_stage:

quantization_stage:
  run_type: oneshot
  quantization_modifiers:
    GPTQModifier:
      ignore: ["lm_head"]
      dampening_frac: 0.01
      config_groups:
        group_0:
          weights:
            num_bits: 4
            type: "int"
            symmetric: true
            strategy: "channel"
          targets: ["Linear"]

[dsikka]

Hi @jiangjiadi if you update to use transformers main or the latest release, this should fix the issue.

[jiangjiadi]

@dsikka I upgraded the version of transformers to 4.47.0 and found that the model fails to load the weights.

[dsikka]

Hi @jiangjiadi can you update your transformers to main and also share the recipe you applied and what the final config of the model looks like?