PaliGemma #1636
Merged
PaliGemma #1636
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: divyashreepathihalli <[email protected]>
* update image size arg thourght paligemma * update tests
During generate, and in Gemma itself, we scale all text embeddings by the sqrt of the hidden dim. We should update our PaliGemmaBackbone to do the same.
* Add cli arguments to speed up conversion * Formatting * Update checkpoint conversion to image_classifier
* Add preset * move location * address review comments
…ength * Update pali_gemma_causal_lm_preprocesor.py * Update pali_gemma_causal_lm_preprocesor.py * Update pali_gemma_causal_lm_preprocesor.py * code reformat
* replicated gemma tokenizer * Different fix * Similar fix for causal lm * code reformat --------- Co-authored-by: Varun Singh <[email protected]>
I am not sure if we want to force the users to pass the image embeddings themselves, it might be more frienly to allow the raw image input. Anyway, since this is unused and untested, and we should check with the LIT team anyway on how to set this up, let's just remove it for now.
* Added docstrings for paligemma decoder and backbone * Add causal lm docstring * add vit docstring * add dtype arg * update docstring * Line wrapping and small nits * added vit docstrings * cleaned up with Matt's comments --------- Co-authored-by: divyashreepathihalli <[email protected]>
Spell out "zero" not "0" for api consistency. Remove include rescaling, the code looked very broken (wrong scale, output unused). Fix formatting.
* Add a response_mask input * Improved attention logic to account for response mask (#87) * Improved attention logic to account for response mask * Addressed several comments * remove vit_num_classes arg from pali_gemma_backbone * fix backbone test * Try simplifying the masking code * Updated tests for thoroughness * Comments and one fix * update preset version * deleted test using unused code path * Added cast to solve bool issues * update presets path * code reformat# modified: keras_nlp/src/models/pali_gemma/pali_gemma_decoder_block_test.py * remove changes to backbone args --------- Co-authored-by: divyashreepathihalli <[email protected]> Co-authored-by: Matt Watson <[email protected]> --------- Co-authored-by: Varun Singh <[email protected]> Co-authored-by: divyashreepathihalli <[email protected]>
Otherwise the symbol would have mismatched code examples
* More consistent defaults for PaliGemma In general, we do not copy the hyper parameters for a specific pre-trained model into the init args. Do the same here for consistency. Also, use as small test models as possible, so our unit testing stays somewhat reasonable. * add basica nd saved model test --------- Co-authored-by: divyashreepathihalli <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial implementation of PaliGemma for KerasNLP.