Cerebras's internal version of EleutherAI's DeeperSpeed library with bug fixes and patches made for our own version of GPT-Neox at https://github.com/CerebrasResearch/gpt-neox.
DeeperSpeed is a fork of Microsoft's Deepspeed library that is tailor-made for the GPT-NeoX by EleutherAI.
Prior to 3/9/2023, DeeperSpeed was based on an old version of DeepSpeed (0.3.15). In order to migrate to the latest upstream DeepSpeed version while allowing users to access the old versions of GPT-NeoX and DeeperSpeed, we have introduced two versioned releases for both libraries:
- Version 1.0 of GPT-NeoX and DeeperSpeed maintain snapshots of the old stable versions that GPT-NeoX-20B and the Pythia Suite were trained on.
- Version 2.0 of GPT-NeoX and DeeperSpeed are the latest versions built on the latest DeepSpeed, and will be maintained going forward.
To create a conda environment with this version of DeeperSpeed to support our version of GPT-Neox:
conda env create -f environment.yml