LLaMA support

[michaelroyzen]

Given existing support for GPT-J and its rotary embeddings, is LLaMA supported as well? Huggingface just shipped their implementation: huggingface/transformers@464d420

@byshiue

[byshiue]

Can you introduce what's difference between GPT-J and LLaMA?

[teknium1]

+1 for this

[michaelroyzen]

They look very similar. In HuggingFace's doc page, they say that the implementation is based on the GPT-NeoX codebase, which seems to be supported by FasterTransformer: https://huggingface.co/docs/transformers/main/model_doc/llama.

Do you think it'll work?

[yuikns]

+1

@byshiue According to our investigation, it is not difficult to portal this model to Megatron as well. But I am not sure will one convert script works.

[byshiue]

Thank you for the suggestion and discussion. We may not have time to work on that issue right now. If you are interesting, you can try to support it.
It is welcome to ask question if you encounter any question, and merge back into our repo if you can support it.

[Hap-Zhang]

+1 for this

[michaelroyzen]

It seems to be quite a simple implementation @byshiue. All that needs to be done is implement RMS layer norm in GPT-NeoX, as well as support the SILU activation. It seem that both of these features are already implemented elsewhere in FasterTransformer.

I'd be happy to take the lead if you can help me with the general steps.

[ZZR0]

+1 for this

[troycheng]

+1 for this

[moonscar]

I compared the GPT-j and llama models in huggingface, they have the same attention layer. There are some differences in FFN, llama uses 3 weights, and the forward function is as follows

    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

I checked the relevant code of the ffn layer in the source code, and it seems that there is no similar structure. Or such a layer already exists in the current code and I have not found it, I hope to get some tips.
@byshiue

[byshiue]

I compared the GPT-j and llama models in huggingface, they have the same attention layer. There are some differences in FFN, llama uses 3 weights, and the forward function is as follows

    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

I checked the relevant code of the ffn layer in the source code, and it seems that there is no similar structure. Or such a layer already exists in the current code and I have not found it, I hope to get some tips. @byshiue

It looks like a standard gated silu. Can you explain what difference do you think?

[moonscar]

Thanks for the reminder, I missed this part.
I will try to make this work

[michaelroyzen]

Wow, thank you @moonscar. Want any help? What's the status of your PR?

[AnShengqiang]

need this too