Skip to content

Weight initialization #204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Weight initialization #204

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
368f686
weight_initialization: added xavier and he
OneAdder Feb 17, 2025
ace6246
weight_initialization: free efficiency boost for dense layer
OneAdder Feb 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions src/nf/nf_dense_layer_submodule.f90
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

use nf_activation, only: activation_function
use nf_base_layer, only: base_layer
use nf_random, only: random_normal
use nf_random, only: random_normal, random_xavier, random_he

implicit none

Expand Down Expand Up @@ -125,8 +125,18 @@ module subroutine init(self, input_shape)
! Weights are a 2-d array of shape previous layer size
! times this layer size.
allocate(self % weights(self % input_size, self % output_size))
call random_normal(self % weights)
self % weights = self % weights / self % input_size
if (&
self % activation_name == 'relu' &
.or. self % activation_name == 'leaky_relu' &
.or. self % activation_name == 'celu' &
) then
call random_he(self % weights, self % input_size)
elseif (self % activation_name == 'sigmoid' .or. self % activation_name == 'tanhf') then
call random_xavier(self % weights, self % input_size)
Comment on lines +128 to +135
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these be as default? Or should the user be able to choose for another pseudo-random generator?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like how it's done here: #151
In the DL Framework of my dreams, I would have an option to pass the algorithm of weights initialization into a layer's constructor. So:

  • What I want: Initializers stub #151 with Kaiming weights by default but it requires a lot of refactoring
  • Why I made this PR-draft: it is correct from the mathematical standpoint, Xavier for S-shaped and He for .*elu. Will probably resolve CNN training on MNIST does not converge #145 if added to Conv layer
  • How Torch does it: Kaiming for everything. Not ideal, but covers vast majority of cases

else
call random_normal(self % weights)
self % weights = self % weights / self % input_size
end if

! Broadcast weights to all other images, if any.
#ifdef PARALLEL
Expand Down
20 changes: 19 additions & 1 deletion src/nf/nf_random.f90
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ module nf_random
implicit none

private
public :: random_normal
public :: random_normal, random_he, random_xavier

real, parameter :: pi = 4 * atan(1.d0)

Expand All @@ -23,4 +23,22 @@ impure elemental subroutine random_normal(x)
x = sqrt(- 2 * log(u(1))) * cos(2 * pi * u(2))
end subroutine random_normal

impure elemental subroutine random_he(x, n_prev)
!! Kaiming weight initialization
real, intent(in out) :: x
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
real, intent(in out) :: x
real, intent(out) :: x

integer, intent(in) :: n_prev
call random_number(x)
x = x * sqrt(2. / n_prev)
end subroutine random_he

impure elemental subroutine random_xavier(x, n_prev)
!! Kaiming weight initialization
real, intent(in out) :: x
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
real, intent(in out) :: x
real, intent(out) :: x

integer, intent(in) :: n_prev
real :: lower, upper
lower = -(1. / sqrt(real(n_prev)))
upper = 1. / sqrt(real(n_prev))
Comment on lines +39 to +40
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
lower = -(1. / sqrt(real(n_prev)))
upper = 1. / sqrt(real(n_prev))
upper = 1. / sqrt(real(n_prev))
lower = -upper

call random_number(x)
x = lower + x * (upper - lower)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this correct if lower == -upper?

end subroutine random_xavier
end module nf_random
Loading