add dataset disposal of b-mc2/sql-create-context for codegen and fix zero3 lora save issue

[sywangyi]

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[sywangyi]

@regisss could you help review the PR, also model save is fixed, the state_dict needs "to('cpu')" before being saved, so I use save_model api instead.

[regisss]

@regisss could you help review the PR, also model save is fixed, the state_dict needs "to('cpu')" before being saved, so I use save_model api instead.

Does it save the whole model or just the adapter weights?

[sywangyi]

@regisss could you help review the PR, also model save is fixed, the state_dict needs "to('cpu')" before being saved, so I use save_model api instead.

Does it save the whole model or just the adapter weights?

only adapter weights is saved

[sywangyi]

@mandy-li mandy, could you help review the PR? this PR also fix same issue in model save for deepspeed zero3

[mandy-li]

LGTM

[sywangyi]

@regisss could you help merge it?

[regisss]

Just a couple of minor comments to address and then I merge 🙂

[regisss]

LGTM!