Merge pull request labmlai#226 from MrYxJ/patch-1

Fix a typo in the formula of ALiBi.

Author: vpj

labml_nn/transformers/alibi/__init__.py

@@ -19,7 +19,7 @@
 
 \begin{align}
 \mathbf{a}_i
-&= \text{softmax} \bigg( \mathbf{q}_i \mathbf{K}^\top + m \cdot \big[-(i-1), \dots, 1, 0 \big] \bigg) \\
+&= \text{softmax} \bigg( \mathbf{q}_i \mathbf{K}^\top + m \cdot \big[-(i-1), \dots, -1, 0 \big] \bigg) \\
 &= \text{softmax} \bigg( \mathbf{q}_i \mathbf{K}^\top + m \cdot \big[0, 1, \dots, (i - 1) \big] \bigg)
 \end{align}