Skip to content

Commit

Permalink
Prevent divide by zero error in loss computation.
Browse files Browse the repository at this point in the history
This occurs when trying to export a model.

Contains no functional changes to running code.

PiperOrigin-RevId: 645521424
  • Loading branch information
T5X Team authored and t5-copybara committed Jun 21, 2024
1 parent 645af9e commit f975a01
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion t5x/losses.py
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,10 @@ def compute_weighted_cross_entropy(
)
vocab_size = logits.shape[-1]
confidence = 1.0 - label_smoothing
low_confidence = (1.0 - confidence) / (vocab_size - 1)
if vocab_size == 1:
low_confidence = 1.0 - confidence
else:
low_confidence = (1.0 - confidence) / (vocab_size - 1)
normalizing_constant = -(
confidence * jnp.log(confidence)
+ (vocab_size - 1) * low_confidence * jnp.log(low_confidence + 1e-20)
Expand Down

0 comments on commit f975a01

Please sign in to comment.