Adding own normalization layer #1062
-
|
What is the best way to add own normalization layer for research purpose? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
@MichaelMonashev I've been working w/ custom norm layers quite a bit lately, there's a branch with some fairly noteworthy changes to make handling more uniform across a number of common models, it's done via combined norm + act layers though, since many of the norm layers I was looking at also combine the non-linearity. A number of the models on this branch now support use of custom norm + act layers.. see the factory below And some examples of new model defs that have been updated (or already) use norm-act layers and have new non-BN configs:
The changes here are non-trivial though, and I'm pretty busy so can't get into more details (I'm trying to get more models trained and tested before I merge this to master, it's currently merged onto the bits_and_tpu branch though since most tests have been done on TPU so far) |
Beta Was this translation helpful? Give feedback.
@MichaelMonashev I've been working w/ custom norm layers quite a bit lately, there's a branch with some fairly noteworthy changes to make handling more uniform across a number of common models, it's done via combined norm + act layers though, since many of the norm layers I was looking at also combine the non-linearity. A number of the models on this branch now support use of custom norm + act layers.. see the factory below
https://github.com/rwightman/pytorch-image-models/blob/norm_norm_norm/timm/models/layers/create_norm_act.py
And some examples of new model defs that have been updated (or already) use norm-act layers and have new non-BN configs: