Geneformer

[laraPPr]

• link to support ticket: #2024120560001104
• website: https://huggingface.co/ctheodoris/Geneformer
• installation docs: https://geneformer.readthedocs.io/en/latest/index.html
• toolchain: foss/2023a
• easyblock to use: PythonBundle
• required dependencies:
  • ...
• optional dependencies:
  • ...
• notes:
  • ...
• effort: (TBD)
• other install methods
  • conda: yes (link?) / no
  • container image: yes (link?) / no
  • pre-built binaries (RHEL8 Linux x86_64): yes (link?) / no
  • easyconfig outside EasyBuild: yes / no

[PetrKralCZ]

• {ai,bio}[foss/2023a] Geneformer v0.1.0-20241204, accelerate v0.33.0 easybuilders/easybuild-easyconfigs#21994

[boegel]

@PetrKralCZ Can you summarize here what the problem is with the draft easyconfig for Geneformer?

[boegel]

@PetrKralCZ We should look into implementing support for source_dependencies, so we can make sure that git-lfs is available when the source tarball for Geneformer is being created. If not, the model.safetensors file will not be the actual contents of the file, but a placeholder with metadata that tells git-lfs where to get the actual file...

Maybe it's better to use source_deps as name for this easyconfig parameter, since we'll soon start migrating towards a more consistent set of easyconfig parameter names, see also easybuilders/easybuild-framework#4464

Implementing support for source_deps can be done by:

• adding a new general easyconfig parameter source_deps to easybuild/framework/easyconfig/default.py, see the "DEPENDENCIES easyconfig parameters" section around line 168;
• picking up on source_deps in the fetch_step method in easybuild/framework/easyblock.py, something like:

def fetch_step(...):

    source_deps = self.cfg['source_deps']
    pre_fetch_env = None
    # load modules for source dependencies (if any)
    if source_deps:
        pre_fetch_env = copy.deepcopy(os.environ)
        source_deps_mod_names = ...
        self.modules_tool.load(source_deps_mod_names)

    ...  # current body of fetch_step

    # restore environment (unload source dependency modules)
    if pre_fetch_env:
        restore_env(pre_fetch_env)

To facilitate getting the module names for the source dependencies, it's probably easiest to update the parse method in easybuild/framework/easyconfig/easyconfig.py to also parse source_deps, as is already done for dependencies, builddependencies, etc.

Do let me know if you need any further help with this!

[boegel]

Temporary workaround to get Geneformer installed while support for source_deps is not installed yet:

eb git-lfs-3.5.1.eb -r
module load git-lfs/3.5.1
eb --fetch --from-pr 21994
eb --from-pr 21994 -r

[boegel]

PR for CUDA version:

• {bio}[foss/2023a] Geneformer v0.1.0-20241204 w/ CUDA 12.1.1 easybuilders/easybuild-easyconfigs#22200

[boegel]

@PetrKralCZ How much time did you spend on this more-or-less one prior to looking into implementing support for source_deps in framework?

[PetrKralCZ]

I think it took 2 days.

[boegel]

PR to add support for source_deps:

• add support for specifying dependencies required to obtain source files via source_deps easyconfig parameter easybuilders/easybuild-framework#4766