Encoder-only Next Token Prediction

Link to full paper: https://arxiv.org/abs/2410.01600

Link to website: https://sites.google.com/wisc.edu/entp/home

AWS Settings

• amazon/Deep Learning Base AMI (Amazon Linux 2) Version 57.8

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh
exec $SHELL
git clone https://github.com/ethanewer/encoder-addition.git
conda env create -f encoder-addition/environments/cuda.yml