Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

return none when use torch.load to load a quantize_model #126

Open
ev-day opened this issue Sep 20, 2024 · 1 comment
Open

return none when use torch.load to load a quantize_model #126

ev-day opened this issue Sep 20, 2024 · 1 comment

Comments

@ev-day
Copy link

ev-day commented Sep 20, 2024

Describe the bug
while I use quantize_model.py(in script folder) to quantize ChatGLM3 model, it can be saved successfully, but when I try to load the model by torch.load(), it was return None.

To Reproduce
Steps to reproduce the behavior:
1.run 'python quantize_model.py -m .\chatglm3-6b -b 4 -o .\chatglm3-6b' to quantize the model, it will export below files
Screenshot 2024-09-20 030909

  1. try to load model
Screenshot 2024-09-20 031329
  1. return none
Screenshot 2024-09-20 031423

Except:
It should return a model object

@ev-day
Copy link
Author

ev-day commented Sep 20, 2024

By another way, just I can quantize the model use 'model = intel_npu_acceleration_library.compile(model, dtype=int4) ', however, it will need much time to quantize and compile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant