Add Esm #2244

pass-lin · 2025-05-03T07:40:33Z

from #2177
Achieved a smaller error with hf.

import os
os.environ["KERAS_BACKEND"] = "torch"
os.environ["HF_ENDPOINT"] = "https://hf-mirror.com"

from keras import ops
from transformers.models.esm.modeling_esm import EsmAttention as hf_EsmSelfAttention
from transformers import EsmConfig
from esm2.esm2_layers import EsmSelfAttention
import numpy as np
import keras
from transformers.models.esm.modeling_esm import EsmModel
weights_path = "facebook/esm2_t6_8M_UR50D"
hf_model = EsmModel.from_pretrained(weights_path)
hf_model.cuda().eval()
hf_model.embeddings.token_dropout = False


from keras_hub.src.models.esm.esm_backbone import (
    ESMBackbone,
)


keras_model =  ESMBackbone.from_preset('hf://'+weights_path)
keras_model.summary()


x = ops.array([[1,2,3,4,5]])+1
hf_out = hf_model(x,ops.ones_like(x))[0]
keras_out = keras_model({'token_ids': x})

print(ops.all(ops.isclose(hf_out, keras_out,atol=1e-4)))

ESM Checkpoint Conversion and Numerics Verification Demo (across multiple backends): Notebook Link

Train Demo: Notebook Link

pass-lin · 2025-05-03T07:56:28Z

ruff.....................................................................Passed
ruff-format..............................................................Passed
Error: Process completed with exit code 1.

Please help me figure out how to solve this problem.

mattdangerw · 2025-05-06T18:35:36Z

Probably an issue with generating the API symbols. Looks like you need to sync with the latest changes on master, then you could try running ./shell/api_gen.sh

sachinprasadhs · 2025-05-09T17:15:23Z

ruff.....................................................................Passed
ruff-format..............................................................Passed
Error: Process completed with exit code 1.

Please help me figure out how to solve this problem.

You can rebase it to latest master code
and then run - pre-commit run --all-files
pip install -u namex

pass-lin · 2025-05-10T13:13:18Z

keras_hub/src/layers/modeling/reversible_embedding_test.py::ReversibleEmbeddingTest::test_quantize_dtype_argument_tie_weights - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/layers/modeling/reversible_embedding_test.py::ReversibleEmbeddingTest::test_quantize_dtype_argument_untie_weights - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/layers/modeling/reversible_embedding_test.py::ReversibleEmbeddingTest::test_quantize_int8_tie_weights - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/layers/modeling/reversible_embedding_test.py::ReversibleEmbeddingTest::test_quantize_int8_untie_weights - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/albert/albert_backbone_test.py::AlbertBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/bart/bart_backbone_test.py::BartBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/bert/bert_backbone_test.py::BertBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/bloom/bloom_backbone_test.py::BloomBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/clip/clip_backbone_test.py::CLIPBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/deberta_v3/deberta_v3_backbone_test.py::DebertaV3BackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/distil_bert/distil_bert_backbone_test.py::DistilBertBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/electra/electra_backbone_test.py::ElectraBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/f_net/f_net_backbone_test.py::FNetBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/falcon/falcon_backbone_test.py::FalconBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/gemma/gemma_backbone_test.py::GemmaBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/gemma/gemma_backbone_test.py::Gemma2BackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/gpt2/gpt2_backbone_test.py::GPT2BackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/gpt_neo_x/gpt_neo_x_backbone_test.py::GPTNeoXBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/llama/llama_backbone_test.py::LlamaTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/mistral/mistral_backbone_test.py::MistralBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/opt/opt_backbone_test.py::OPTBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/pali_gemma/pali_gemma_backbone_test.py::PaliGemmaBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/pali_gemma/pali_gemma_backbone_test.py::PaliGemma2BackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/phi3/phi3_backbone_test.py::Phi3Test::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/phi3/phi3_backbone_test.py::Phi3Test::test_backbone_basics_with_su_rotary - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/roberta/roberta_backbone_test.py::RobertaBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/siglip/siglip_backbone_test.py::SigLIPBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/siglip/siglip_backbone_test.py::SigLIP2BackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/t5/t5_backbone_test.py::T5BackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/whisper/whisper_backbone_test.py::WhisperBackboneTest::test_backbone_basics - TypeError: _int8_build() takes 2 positional arguments but 3 were given
FAILED keras_hub/src/models/xlm_roberta/xlm_roberta_backbone_test.py

@mattdangerw @sachinprasadhs
Is it a problem with the test environment? Why are there so many errors that don't belong to me?

sachinprasadhs · 2025-05-12T17:44:15Z

It's not related to your code, looks like some issue with the JAX backend, we will look into it.

sachinprasadhs

Thanks fro the PR, I have added my comments, also add checkpoints conversion under: keras-hub/tools/checkpoint_conversion

keras_hub/src/models/esm/esm_backbone.py

sachinprasadhs · 2025-05-15T22:02:55Z

keras_hub/src/models/esm/esm_backbone.py

+        intermediate_dim: int. The output dimension of the first Dense layer in
+            a two-layer feedforward network for each transformer.
+        dropout: float. Dropout probability for the Transformer encoder.
+        layer_norm_eps:bool.Should we use ln after embedding?


Didn't get the point here, are you asking our input or it's the arg detail, if it is the arg details, it needs to be repharsed, avoid question marks and the argument name is emb_layer_norm_before

layer_norm_eps discription needs to be updated.

keras_hub/src/models/esm/esm_backbone.py

keras_hub/src/models/esm/esm_classifier_test.py

keras_hub/src/models/esm/esm_masked_plm_test.py

keras_hub/src/models/esm/esm_masked_plm.py

keras_hub/src/models/esm/esm_classifier.py

keras_hub/src/utils/transformers/convert_esm.py

pass-lin · 2025-05-17T18:13:05Z

@sachinprasadhs @mattdangerw
Can anybody review my code?

pass-lin · 2025-06-02T18:11:23Z

@mattdangerw @sachinprasadhs
Please check my code, thank you.

sachinprasadhs

Added few more comments and few of the previous review comments still needs to be addressed

keras_hub/src/models/esm/esm_backbone.py

sachinprasadhs · 2025-06-02T18:37:34Z

keras_hub/src/models/esm/esm_backbone.py

+    Disclaimer: Pre-trained models are provided on an "as is" basis, without
+    warranties or conditions of any kind.
+
+    Args:


Still activation and max_wavelength description is missing!

keras_hub/src/models/esm/esm_backbone.py

sachinprasadhs · 2025-06-02T18:42:21Z

keras_hub/src/models/esm/esm_backbone.py

+    Disclaimer: Pre-trained models are provided on an "as is" basis, without
+    warranties or conditions of any kind.
+
+    Args:


add arg description for pad_token_id as well

keras_hub/src/models/esm/esm_backbone.py

sachinprasadhs · 2025-06-02T18:46:33Z

keras_hub/src/models/esm/esm_backbone.py

+        position_embedding_type:esm1 use abs position embeding,esm2 use rope.
+            so this parameter is only except for absolute and rotary.


This still needs to be changed to:

position_embedding_type: str. The position embedding type to use. One of "absolute" and "rotary". Use "absolute" for ESM1. Use "rotary" for ESM2. Defaults to "rotary".

keras_hub/src/models/esm/esm_backbone_test.py

sachinprasadhs · 2025-06-02T18:50:08Z

keras_hub/src/models/esm/esm_classifier_preprocessor.py

+
+
+@keras_hub_export("keras_hub.models.ESMProteinClassifierPreprocessor")
+class ESMProteinClassifierPreprocessor(BertTextClassifierPreprocessor):


Pending change here which should be subclassed from TextClassifierPreprocessor instead of BertTextClassifierPreprocessor

sachinprasadhs · 2025-06-02T18:55:23Z

keras_hub/src/models/esm/esm_backbone.py

+        max_sequence_length=1024,
+        max_wavelength=10000,
+        layer_norm_eps=1e-12,
+        emb_layer_norm_before=False,


pending change, instead emb_layer_norm_before --> use_pre_layer_norm

sachinprasadhs · 2025-06-02T18:56:17Z

keras_hub/src/models/esm/esm_classifier.py

+
+
+@keras_hub_export("keras_hub.models.ESMProteinClassifier")
+class ESMProteinClassifier(RobertaTextClassifier):


pending change.
You can subclass TextClassifier and make the same changes as RobertaTextClassifier instead of subclassing from another model.

sachinprasadhs · 2025-06-02T19:13:43Z

Once you address all the comments, add end to end working colab along with the checkpoints conversion under: keras-hub/tools/checkpoint_conversion

pass-lin · 2025-06-11T05:00:14Z

Thanks, few minor comments.

Also, need more details specific to Keras 3.6 older version issue.

Finally in the PR description, add the colab notebook to show end to end working of the model, numerics verification. you can follow the PR description template from the recent PR.

How to add a Colab notebook? Can you give me give a demo?

sachinprasadhs · 2025-06-11T18:14:09Z

Thanks, few minor comments.
Also, need more details specific to Keras 3.6 older version issue.
Finally in the PR description, add the colab notebook to show end to end working of the model, numerics verification. you can follow the PR description template from the recent PR.

How to add a Colab notebook? Can you give me give a demo?

Adding from one of the recent PR which got merged, you can do something like this

DeiT Checkpoint Conversion and Numerics Verification Demo (across multiple backends): Notebook Link
DeiT End-to-End Demo (zero-shot and finetuning): Notebook Link
Here are the converted DeiT presets from Hugging Face checkpoints for reference.

pass-lin · 2025-06-12T07:20:20Z

Thanks, few minor comments.
Also, need more details specific to Keras 3.6 older version issue.
Finally in the PR description, add the colab notebook to show end to end working of the model, numerics verification. you can follow the PR description template from the recent PR.

How to add a Colab notebook? Can you give me give a demo?

Adding from one of the recent PR which got merged, you can do something like this

DeiT Checkpoint Conversion and Numerics Verification Demo (across multiple backends): Notebook Link

DeiT End-to-End Demo (zero-shot and finetuning): Notebook Link

Here are the converted DeiT presets from Hugging Face checkpoints for reference.

Hello, I've already added the Colab demo of tools/checkpoint_conversion/convert_esm_checkpoints.py in the PR description. I think this is enough, and we can refer to BERT for the rest.
Can we merge now?

sachinprasadhs · 2025-06-12T17:58:24Z

We don't have access to view the notebook, can you make it public. Thanks

pass-lin · 2025-06-12T18:36:32Z

We don't have access to view the notebook, can you make it public. Thanks

OK,It has been enable sharing

sachinprasadhs · 2025-06-13T20:05:11Z

Hi, The intention of the notebook is to verify the correctness of the model including, backbone, tasks with the usage details and the expected outcome and to verify the numerics stablity after weights transfer to the Keras architecture, with wither forward pass.

pass-lin · 2025-06-14T05:37:10Z

Hi, The intention of the notebook is to verify the correctness of the model including, backbone, tasks with the usage details and the expected outcome and to verify the numerics stablity after weights transfer to the Keras architecture, with wither forward pass.

Okay, I've added another notebook, which is a demo for predicting the suitable pH of enzymes using ESM.

sachinprasadhs · 2025-06-16T21:20:01Z

You can remove the esm2_t6_8M directory, that will be generated using the conversion script you have provided and will be uploaded to Kaggle.

The notebook which you have provided doesn't have predict method,
take any sample suitable input and display the output with predict.

Also in your conversion script, you have mentioned atol=1e-3, what would be the error percentage when the atol=1e-04 and we need following things in your notebook

Numerics verification, load the original ESM model and do forward pass, and do the same forward pass to Keras-Hub ESM implementation and compare the numerics layer by layer to show if numerics are matching(preferably to the 1e-4 precision)
Demonstrating usage of proprocessor, Tokenizer and other functionalities of ESM

I have provided the reference notebooks, please refer those.

You can keep only ESM changes in this PR, you can create a new PR for roformer which also needs checkpoint conversion script, so that we can maintain the latest weight in Kaggle by generating the new weights with the script with any future changes to Keras Hub model specific.

pass-lin · 2025-06-17T06:24:17Z

You can remove the esm2_t6_8M directory, that will be generated using the conversion script you have provided and will be uploaded to Kaggle.

The notebook which you have provided doesn't have predict method, take any sample suitable input and display the output with predict.

Also in your conversion script, you have mentioned atol=1e-3, what would be the error percentage when the atol=1e-04 and we need following things in your notebook

Numerics verification, load the original ESM model and do forward pass, and do the same forward pass to Keras-Hub ESM implementation and compare the numerics layer by layer to show if numerics are matching(preferably to the 1e-4 precision)

Demonstrating usage of proprocessor, Tokenizer and other functionalities of ESM

I have provided the reference notebooks, please refer those.

You can keep only ESM changes in this PR, you can create a new PR for roformer which also needs checkpoint conversion script, so that we can maintain the latest weight in Kaggle by generating the new weights with the script with any future changes to Keras Hub model specific.

OK, I have modified the notebook, please check. In addition, roformerV2 does not need to convert scripts, it is a native keras model. I just modified the keras2 api

pass-lin · 2025-06-28T07:25:10Z

@sachinprasadhs plz check my notebook

sachinprasadhs · 2025-07-09T21:03:22Z

Hi, Still your notebook does not demonstrate the actual use case example demonstrations like https://huggingface.co/docs/transformers/en/model_doc/esm#transformers.EsmForSequenceClassification.forward.example or https://huggingface.co/docs/transformers/en/model_doc/esm#transformers.EsmForProteinFolding.forward.example or
https://huggingface.co/docs/transformers/en/model_doc/esm#transformers.EsmForTokenClassification.forward.example, please include it.

pass-lin · 2025-07-10T02:08:45Z

Hi, Still your notebook does not demonstrate the actual use case example demonstrations like https://huggingface.co/docs/transformers/en/model_doc/esm#transformers.EsmForSequenceClassification.forward.example or https://huggingface.co/docs/transformers/en/model_doc/esm#transformers.EsmForProteinFolding.forward.example or https://huggingface.co/docs/transformers/en/model_doc/esm#transformers.EsmForTokenClassification.forward.example, please include it.

We've included a training demo for ESM. As for ESMFold, that's another brand new pr. So can you just click and tell me what demo to add? Sorry for the trouble.

sachinprasadhs · 2025-07-10T22:15:31Z

Any demo with the implementation you have which predicts the actual data or the sample input data and display the output in the existing colab, and remove the folder/directory named esm2_t6_8M in your code, rest all it looks good.
Thanks for all the work.

divyashreepathihalli · 2025-07-11T00:08:35Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for ESM models, including the backbone, classifier, and masked protein language modeling tasks, along with their corresponding preprocessors, tokenizers, and tests. I've identified several areas for improvement, including fixing a critical bug in an exception raise, correcting several documentation examples and descriptions that could mislead users, and addressing inconsistencies in model configuration and weight conversion. Addressing the feedback will improve the quality and robustness of the new ESM model support.

gemini-code-assist · 2025-07-11T00:11:23Z

keras_hub/src/models/esm/esm_attention.py

+        if self.use_rotary:
+            qw, kw = self.rotary_embedding_layer(qw, kw)
+        if version.parse(keras.__version__) < version.parse("3.6"):
+            raise ("Please make sure your Keras version is >=3.6.")


Raising a string or a tuple does not work as intended in Python 3 and will result in a TypeError. You should raise an instance of an exception class, such as ValueError.

raise ValueError("Please make sure your Keras version is >=3.6.")

gemini-code-assist · 2025-07-11T00:11:23Z

keras_hub/src/models/esm/esm_classifier.py

+        "token_ids": np.ones(shape=(2, 12), dtype="int32"),
+        "segment_ids": np.array([[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0]] * 2),
+        "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2),
+    }


The example for preprocessed integer data includes "segment_ids" and "padding_mask" in the input features. However, the ESMBackbone and ESMProteinClassifierPreprocessor only expect "token_ids". This example is misleading and will not work as written. Please update it to only include "token_ids".

Suggested change

"token_ids": np.ones(shape=(2, 12), dtype="int32"),

"segment_ids": np.array([[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0]] * 2),

"padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2),

}

features = {

"token_ids": np.ones(shape=(2, 12), dtype="int32"),

}

gemini-code-assist · 2025-07-11T00:11:23Z

keras_hub/src/utils/transformers/convert_esm.py

+            hf_weight_key=f"{hf_prefix}{i}.LayerNorm.weight",
+        )
+        loader.port_weight(
+            keras_variable=block.feedforward_norm.beta,
+            hf_weight_key=f"{hf_prefix}{i}.LayerNorm.bias",


The Hugging Face weight key for the feedforward layer normalization seems to be incorrect. Based on the transformers implementation of EsmLayer, the LayerNorm is part of the output block. The key should likely be encoder.layer.{i}.output.LayerNorm.weight instead of encoder.layer.{i}.LayerNorm.weight.

Suggested change

hf_weight_key=f"{hf_prefix}{i}.LayerNorm.weight",

)

loader.port_weight(

keras_variable=block.feedforward_norm.beta,

hf_weight_key=f"{hf_prefix}{i}.LayerNorm.bias",

hf_weight_key=f"{hf_prefix}{i}.output.LayerNorm.weight",

)

loader.port_weight(

keras_variable=block.feedforward_norm.beta,

hf_weight_key=f"{hf_prefix}{i}.output.LayerNorm.bias",

This is an illusion, and it is recommended to ignore it

gemini-code-assist · 2025-07-11T00:11:24Z

keras_hub/src/models/esm/esm_backbone.py

+        num_heads=4,
+        hidden_dim=256,
+        intermediate_dim=512,
+        head_size = 64,


The ESMBackbone constructor does not accept a head_size argument. This parameter is calculated internally as hidden_dim // num_heads. Including it in the example will cause an error for users who copy-paste the code. Please remove this line from the example.

gemini-code-assist · 2025-07-11T00:11:24Z

keras_hub/src/utils/transformers/convert_esm.py

+            keras_variable=backbone.get_layer("embeddings_layer_norm").gamma,
+            hf_weight_key="embeddings.layer_norm.weight",
+        )
+        loader.port_weight(
+            keras_variable=backbone.get_layer("embeddings_layer_norm").beta,


The LayerNormalization layer for embeddings in ESMBackbone is named "emb_layer_norm", but the conversion script is trying to access it with the name "embeddings_layer_norm". This will cause a ValueError when trying to get the layer.

Suggested change

keras_variable=backbone.get_layer("embeddings_layer_norm").gamma,

hf_weight_key="embeddings.layer_norm.weight",

)

loader.port_weight(

keras_variable=backbone.get_layer("embeddings_layer_norm").beta,

keras_variable=backbone.get_layer("emb_layer_norm").gamma,

hf_weight_key="embeddings.layer_norm.weight",

)

loader.port_weight(

keras_variable=backbone.get_layer("emb_layer_norm").beta,

gemini-code-assist · 2025-07-11T00:11:25Z

keras_hub/src/models/esm/esm_backbone.py

+        self.dropout = dropout
+        self.max_wavelength = max_wavelength
+        self.head_size = head_size
+        self.dropout = dropout


The self.dropout attribute is assigned twice in the __init__ method (lines 195 and 198). The second assignment is redundant and can be removed.

gemini-code-assist · 2025-07-11T00:11:25Z

keras_hub/src/models/esm/esm_tokenizer.py

+    ```python
+    # Unbatched input.
+    tokenizer = keras_hub.models.ESMTokenizer.from_preset(
+        "roformer_v2_base_zh",


The example for ESMTokenizer uses a roformer_v2_base_zh preset, which is confusing. It would be better to use an actual ESM preset to avoid misleading users.

Suggested change

"roformer_v2_base_zh",

"hf://facebook/esm2_t6_8M_UR50D",

gemini-code-assist · 2025-07-11T00:11:26Z

keras_hub/src/models/esm/esm_backbone.py

+            raise ValueError(
+                '`position_embedding_type` must be either `"rotary"`, or '
+                '`"absolute"`. Received '
+                "position_embedding_type={position_embedding_type}."


The string in the ValueError is intended to be an f-string to include the value of position_embedding_type, but it's missing the f prefix. This will result in the literal string {position_embedding_type} being part of the error message.

Suggested change

"position_embedding_type={position_embedding_type}."

f"position_embedding_type={position_embedding_type}."

gemini-code-assist · 2025-07-11T00:11:26Z

keras_hub/src/models/esm/esm_backbone.py

+    """A ESM2 and ESM encoder network.
+
+    This class implements a bi-directional Transformer-based encoder as
+    described in ["Roformer"](https://github.com/facebookresearch/esm).


The docstring mentions "Roformer" but links to the ESM repository. To avoid confusion, the link text should be updated to "ESM" to match the model being implemented.

Suggested change

described in ["Roformer"](https://github.com/facebookresearch/esm).

described in ["ESM"](https://github.com/facebookresearch/esm).

gemini-code-assist · 2025-07-11T00:11:26Z

keras_hub/api/models/__init__.py

+from keras_hub.src.models.esm.esm_masked_plm import (
+    ESMMaskedPLM as ESM2MaskedPLM,
+)
+from keras_hub.src.models.esm.esm_masked_plm import ESMMaskedPLM as ESMMaskedPLM


This import of ESMMaskedPLM is redundant. The name ESMMaskedPLM is already available from the import on lines 200-202. Removing this line will improve code clarity. Even though this file is autogenerated, it's good practice to address such issues in the source generator if possible.

pass-lin · 2025-07-11T08:34:30Z

Any demo with the implementation you have which predicts the actual data or the sample input data and display the output in the existing colab, and remove the folder/directory named esm2_t6_8M in your code, rest all it looks good. Thanks for all the work.

I’m not sure what you mean by “delete the esm2_t6_8M directory.”

Looking at the demo notebook, all it does is install the environment, change the OS, and then run:

python tools/checkpoint_conversion/convert_deit_checkpoints.py --preset deit-base-distilled-patch16-384

In my notebook I did exactly the same thing: installed the environment, changed the OS, and then ran

python tools/checkpoint_conversion/convert_esm_checkpoints.py --preset esm2_t6_8M

Could you give a more precise and detailed description of which notebook has the problem and what it is missing compared to the reference notebook?
In the reference notebook, what exactly shows that the esm2_t6_8M directory should be removed?

Further, in another notebook I explicitly provide demonstrations of predict, fit, and evaluate. What exactly is still missing?

A clear description would be greatly appreciated—thank you for your help!
And sorry for the extra work caused by adding a detailed description to you.

pass-lin · 2025-07-11T08:53:16Z

/gemini review

Thanks, I fixed some error with reference to gemini's review.

sachinprasadhs · 2025-07-14T21:02:10Z

In your code commit, there is the files with checkpoint files generated, we don't keep these files in our github, we upload the checkpoints to kaggle/ Hugging face, and the files will be generated by running the conversion script you have provided, you don't need to provide the converted checkpoints here in your commit

pass-lin · 2025-07-15T05:36:34Z

In your code commit, there is the files with checkpoint files generated, we don't keep these files in our github, we upload the checkpoints to kaggle/ Hugging face, and the files will be generated by running the conversion script you have provided, you don't need to provide the converted checkpoints here in your commit

I'm so sorry, this is an intermediate product when running the test before. I didn't notice his existence. Thanks very much for your reminder.

pass-lin added 2 commits May 3, 2025 01:28

add esm

f9ff098

add esm2

cc4123b

pass-lin added 4 commits May 3, 2025 17:35

fix

d3f598d

fix

737a147

format

140207b

fix test

cc9a11c

divyashreepathihalli requested a review from sachinprasadhs May 5, 2025 17:09

format

f8da784

pass-lin and others added 2 commits May 10, 2025 19:26

renew

72e9829

Merge branch 'keras-team:master' into esm

16bb9f2

pass-lin force-pushed the esm branch from a66ee78 to 19f4b1f Compare May 10, 2025 11:33

format

5cbf577

pass-lin force-pushed the esm branch from 19f4b1f to 5cbf577 Compare May 10, 2025 12:05

format

6e9f817

pass-lin mentioned this pull request May 10, 2025

_int8_build() bug from keras-nightly keras-team/keras#21272

Closed

sachinprasadhs reviewed May 16, 2025

View reviewed changes

pass-lin added 3 commits May 17, 2025 12:18

update

2815e9c

update

79e738c

update

20d5051

sachinprasadhs reviewed Jun 2, 2025

View reviewed changes

pass-lin added 2 commits June 3, 2025 17:59

update

fb18c98

add new tool

7609ab4

pass-lin added 2 commits June 6, 2025 18:24

update

11f56fd

update

bd7380a

pass-lin added 3 commits June 12, 2025 14:44

fix

0f6261c

fix

95bb704

fix

c2a2787

fix tool

cc69d43

sachinprasadhs added the kokoro:force-run Runs Tests on GPU label Jul 9, 2025

kokoro-team removed the kokoro:force-run Runs Tests on GPU label Jul 9, 2025

gemini-code-assist bot reviewed Jul 11, 2025

View reviewed changes

Merge branch 'keras-team:master' into esm

baff84c

update

4577f83

delete esm weight

2549396

		position_embedding_type:esm1 use abs position embeding,esm2 use rope.
		so this parameter is only except for absolute and rotary.



		@keras_hub_export("keras_hub.models.ESMProteinClassifierPreprocessor")
		class ESMProteinClassifierPreprocessor(BertTextClassifierPreprocessor):



		@keras_hub_export("keras_hub.models.ESMProteinClassifier")
		class ESMProteinClassifier(RobertaTextClassifier):

	"position_embedding_type={position_embedding_type}."
	f"position_embedding_type={position_embedding_type}."

	described in ["Roformer"](https://github.com/facebookresearch/esm).
	described in ["ESM"](https://github.com/facebookresearch/esm).

Add Esm #2244

Are you sure you want to change the base?

Add Esm #2244

Conversation

pass-lin commented May 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pass-lin commented May 3, 2025

Uh oh!

mattdangerw commented May 6, 2025

Uh oh!

sachinprasadhs commented May 9, 2025

Uh oh!

pass-lin commented May 10, 2025

Uh oh!

sachinprasadhs commented May 12, 2025

Uh oh!

sachinprasadhs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sachinprasadhs May 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pass-lin commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pass-lin commented Jun 2, 2025

Uh oh!

sachinprasadhs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sachinprasadhs Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sachinprasadhs Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sachinprasadhs Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sachinprasadhs Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sachinprasadhs Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sachinprasadhs Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sachinprasadhs commented Jun 2, 2025

Uh oh!

pass-lin commented Jun 11, 2025

Uh oh!

sachinprasadhs commented Jun 11, 2025

Uh oh!

pass-lin commented Jun 12, 2025

Uh oh!

sachinprasadhs commented Jun 12, 2025

Uh oh!

pass-lin commented Jun 12, 2025

Uh oh!

sachinprasadhs commented Jun 13, 2025

Uh oh!

pass-lin commented May 3, 2025 •

edited

Loading

pass-lin commented May 17, 2025 •

edited

Loading

sachinprasadhs Jun 2, 2025 •

edited

Loading

sachinprasadhs Jun 2, 2025 •

edited

Loading

sachinprasadhs Jun 2, 2025 •

edited

Loading

sachinprasadhs Jun 2, 2025 •

edited

Loading

pass-lin commented Jul 11, 2025 •

edited

Loading

pass-lin commented Jul 11, 2025 •

edited

Loading