add support for Float8Tensor in Linear layer#1

Open

sudhakarsingh27 wants to merge 27 commits intomainfrom float8tensor_experiments

+13,012-115

Commits on Jul 5, 2023

add support for Float8Tensor in Linear layer
sudhakarsingh27
committed

Commits on Jul 10, 2023

fp8 poc with te.linear (grads getting updated but not scale)
sudhakarsingh27
committed

Commits on Jul 17, 2023

Commits on Jul 24, 2023

python script
sudhakarsingh27
committed

Commits on Jul 26, 2023

Updates and fixes
ksivaman
committed

Commits on Aug 8, 2023

add transpose op and expose creating fp8 weights in init as option
sudhakarsingh27
committed

Commits on Aug 10, 2023

test poc with a real network with amax_reduce=False
sudhakarsingh27
committed

Commits on Aug 14, 2023

Commits on Aug 15, 2023

make non primary_weights_in_fp8 mode work as well
sudhakarsingh27
committed

Commits on Sep 1, 2023

add generic way to handle inplace ops
sudhakarsingh27
committed

Commits on Sep 7, 2023

version that works with NeMo MainParamsWrapper optimizer
sudhakarsingh27
committed

Commits on Sep 11, 2023

clean up Float8Tensor and enable layernorm_linear to run with fp8 weights
sudhakarsingh27
committed

Commits on Sep 12, 2023