Skip to content

add support for specifying dependencies required to obtain source files via source_deps easyconfig parameter #4766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 6 commits into
base: develop
Choose a base branch
from
Draft

add support for specifying dependencies required to obtain source files via source_deps easyconfig parameter #4766

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e7a27c9
Implement `source_deps`
PetrKralCZ Feb 18, 2025
ab95e29
undo unwanted changes
PetrKralCZ Feb 18, 2025
64c539b
Merge branch 'easybuilders:develop' into 20250218121129_new_pr_hFGIwk…
PetrKralCZ Mar 12, 2025
f3b6e32
Update easyconfig.py
PetrKralCZ Mar 12, 2025
3c97d1e
Merge branch 'develop' into 20250218121129_new_pr_hFGIwkIBFr
boegel Apr 22, 2025
0db8fd2
add source_deps to DEPENDENCY_PARAMETERS to make sure it's processed …
boegel Apr 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions easybuild/framework/easyblock.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2574,6 +2574,14 @@ def fetch_step(self, skip_checksums=False):
raise EasyBuildError("EasyBuild-version %s is newer than the currently running one. Aborting!",
easybuild_version)

source_deps = self.cfg['source_deps']
pre_fetch_env = None
# load modules for source dependencies (if any)
if source_deps:
pre_fetch_env = copy.deepcopy(os.environ)
source_deps_mod_names = [d['short_mod_name'] for d in source_deps]
self.modules_tool.load(source_deps_mod_names)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PetrKralCZ Before trying to load modules, we need to take into account that self.modules_tool may be a NoModulesTool instance (that will be the case when eb --fetch is used).

If so, then we need to re-initialize self.modules_tool to a proper ModulesTool instance (via the modules_tool function available in tools/config.py).

This will need changes in options.py too though, because there we're now overwriting self.options.modules_tool with None, we also need to keep track of the original value so we can then pass it to the modules_tool function (which will also need to be adjusted)...


start_progress_bar(PROGRESS_BAR_DOWNLOAD_ALL, self.cfg.count_files())

if self.dry_run:
Expand Down Expand Up @@ -2659,6 +2667,9 @@ def fetch_step(self, skip_checksums=False):

stop_progress_bar(PROGRESS_BAR_DOWNLOAD_ALL)

if pre_fetch_env:
restore_env(pre_fetch_env)

def checksum_step(self):
"""Verify checksum of sources and patches, if a checksum is available."""
for fil in self.src + self.patches:
Expand Down
1 change: 1 addition & 0 deletions easybuild/framework/easyconfig/default.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,7 @@
'multi_deps_load_default': [True, "Load module for first version listed in multi_deps by default", DEPENDENCIES],
'osdependencies': [[], "OS dependencies that should be present on the system", DEPENDENCIES],
'moddependpaths': [None, "Absolute path(s) to prepend to MODULEPATH before loading dependencies", DEPENDENCIES],
'source_deps': [[], "source dependencies that should be present befory getting the sources", DEPENDENCIES],

# LICENSE easyconfig parameters
'accept_eula': [False, "Accepted End User License Agreement (EULA) for this software", LICENSE],
Expand Down
7 changes: 7 additions & 0 deletions easybuild/framework/easyconfig/easyconfig.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -816,6 +816,13 @@ def remove_false_versions(deps):
builddeps = [self._parse_dependency(dep, build_only=True) for dep in builddeps]
self['builddependencies'] = remove_false_versions(builddeps)

source_deps = self['source_deps']
if source_deps and all(isinstance(x, (list, tuple)) for b in source_deps for x in b):
source_deps = [[self._parse_dependency(dep, build_only=True) for dep in x] for x in source_deps]
else:
source_deps = [self._parse_dependency(dep, build_only=True) for dep in source_deps]
self['source_deps'] = remove_false_versions(source_deps)

# keep track of parsed multi deps, they'll come in handy during sanity check & module steps...
self.multi_deps = self.get_parsed_multi_deps()

Expand Down
2 changes: 1 addition & 1 deletion easybuild/framework/easyconfig/format/format.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
FORMAT_VERSION_REGEXP = re.compile(r'^#\s+%s\s*(?P<major>\d+)\.(?P<minor>\d+)\s*$' % FORMAT_VERSION_KEYWORD, re.M)
FORMAT_DEFAULT_VERSION = EasyVersion('1.0')

DEPENDENCY_PARAMETERS = ['builddependencies', 'dependencies', 'hiddendependencies']
DEPENDENCY_PARAMETERS = ['builddependencies', 'dependencies', 'hiddendependencies', 'source_deps']

# values for these keys will not be templated in dump()
EXCLUDED_KEYS_REPLACE_TEMPLATES = ['description', 'easyblock', 'exts_default_options', 'exts_list',
Expand Down