[Bug]: bad inference result after gptq using llm-compressor on Qwen2.5-72b model

[yoonlee888]

⚙️ Your current environment

The output of python collect_env.py

Your output of `python collect_env.py` here

🐛 Describe the bug

I want to quantize Qwen2.5-72b model using llm-compressor. After quantizing the model using config

recipe = GPTQModifier(ignore=["lm_head"], config_groups={
        "group_0": QuantizationScheme(
            targets=["Linear"],
            weights=QuantizationArgs(num_bits=4, strategy="group", group_size=64),
        )
    },
)

data processing:

DATASET_ID = "/workspace/dataset/quant/qwen2-72b-chat/common_26000_plus_3000lc_extracted_3000_4096.jsonl"
DATASET_SPLIT = "train"

# Select number of samples. 512 samples is a good place to start.
# Increasing the number of samples can improve accuracy.
NUM_CALIBRATION_SAMPLES = 300
MAX_SEQUENCE_LENGTH = 4096

# Load dataset and preprocess.
ds = load_dataset("json", data_files=DATASET_ID, split=f"{DATASET_SPLIT}[:{NUM_CALIBRATION_SAMPLES}]")
# ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
# ds = ds.shuffle(seed=42)


def preprocess(example):
    return {
        "text": tokenizer.apply_chat_template(
            example["messages"],
            tokenize=False,
            add_generation_prompt=True
        )
    }


ds = ds.map(preprocess)


# Tokenize inputs.
def tokenize(sample):
    return tokenizer(
        sample["text"],
        padding=False,
        max_length=MAX_SEQUENCE_LENGTH,
        truncation=True,
        add_special_tokens=False,
    )


ds = ds.map(tokenize, remove_columns=ds.column_names)

The inference result is always strange when the output token length are longer than 400. The result is like this, normal output followed by abnormal output:

If i use auto-gptq, this problem will not exist, the result is always normal. Any idea on this ?

🛠️ Steps to reproduce

No response

[brian-dellabetta]

Hi @yoonlee888 , hmm that's strange. Some ideas that may help figure out where the issue is coming from:

1. Does the same issue appear with GPTQModifier(ignore=["lm_head"], targets=["Linear"], scheme="W4A16")?
2. Does the same issue appear with "HuggingFaceH4/ultrachat_200k" dataset?
3. Does increasing MAX_SEQUENCE_LENGTH help at all?

[yoonlee888]

Hi @brian-dellabetta, thank you for your suggestions. I edited my quant script following your suggestions:

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer

from compressed_tensors.quantization import QuantizationArgs, QuantizationScheme
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.transformers import oneshot
from llmcompressor.utils import dispatch_for_generation

# Select model and load it.
model_id = "/workspace/models/q25-72b_sft-2nd-new-data-v4_wi-geo-gt_mixed_sample-rag"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

NUM_CALIBRATION_SAMPLES = 512
MAX_SEQUENCE_LENGTH = 8192  #4096 to 8192

# Load dataset and preprocess.
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")
ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))

def preprocess(example):
    return {
        "text": tokenizer.apply_chat_template(
            example["messages"],
            tokenize=False,
            add_generation_prompt=True
        )
    }

ds = ds.map(preprocess)

# Tokenize inputs.
def tokenize(sample):
    return tokenizer(
        sample["text"],
        padding=False,
        max_length=MAX_SEQUENCE_LENGTH,
        truncation=True,
        add_special_tokens=False,
    )

ds = ds.map(tokenize, remove_columns=ds.column_names)

recipe = GPTQModifier(ignore=["lm_head"], targets=["Linear"], scheme="W4A16")

# Apply algorithms.
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

# Save to disk compressed.
SAVE_DIR = model_id.rstrip("/").split("/")[-1] + "-W4A16-ultrachat"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)

I still got strange inference output with user instruction "Write an essay of at least 900 words on the topic of navigable freeway. Make sure the entire response is in English and no capital letters are used.":

By the way, my llmcompressor version is v0.6.0, I will update to the latest to see if it works.

[brian-dellabetta]

Hi @yoonlee888 , yes upgrading might help. I also noticed that you're using an SFT-tuned checkpoint

model_id = "/workspace/models/q25-72b_sft-2nd-new-data-v4_wi-geo-gt_mixed_sample-rag"

Does the same problem occur if you use the original qwen instruct model?

[yoonlee888]

I upgraded my llmcompressor to 0.8.1 and tested two cases:

1. GPTQ on official Qwen 2.5 72b model downloaded from Huggingface, the inference result after quantization is normal.
2. GPTQ on the stf-tuned model downloaded from Huggingface, the inference result after quantization is abnormal as above. (However this model is good with auto-gptq)

[brian-dellabetta]

Hi @yoonlee888 , I'm not sure what would cause that, tbh. We can see if anyone in the community responds, but a few ideas:

1. You can compare the loss reported in the logs (there should be a line with "error" reported after each linear layer is processed in gptq.
2. You can also try QuantizationModifier or AWQModifier to see if they both similarly don't work well with the sft-tuned model.
3. Ensure the calibration dataset size (both num_samples and max_sequence_length) matches what you are using in auto-gptq.