Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to Conda Build 3 (finally) and Move to Azure DevOps #177

Merged
merged 69 commits into from
Jan 9, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
49f5d60
Start conversion to Conda Build 3
Lnaden Oct 10, 2018
c507f45
Update docker script to get newer conda and conda-build
Lnaden Oct 10, 2018
4e87c0f
Finalize the draft of the Conda Build script
Lnaden Jan 15, 2019
9d6003c
Merge in recent changes from john
Lnaden Jan 15, 2019
aa4eb58
Actually commit the other files...
Lnaden Jan 15, 2019
9a59fa1
I think its done...
Lnaden Feb 6, 2019
c99369b
Some last cleanup, may need to make another change based on image
Lnaden Feb 7, 2019
9e8bb42
yay brew....
Lnaden Feb 7, 2019
5813ca3
Merge branch 'master' of github.com:omnia-md/conda-dev-recipes
Lnaden Nov 26, 2019
e64dcba
See if cuda8 osx_image is also broken
jchodera Feb 8, 2019
93b2821
Try xcode 8.3 image
jchodera Mar 16, 2019
bc1efba
Try not using sudo for osx
jchodera Mar 16, 2019
e33e064
Keep tinkering with osx build
jchodera Mar 16, 2019
7378dff
Don't use -C in tar
jchodera Mar 16, 2019
da461f6
Remove comments
jchodera Mar 17, 2019
d2e2556
Fix typo
jchodera Mar 17, 2019
1e0bb3f
lol why
jchodera Mar 17, 2019
77b9d2b
Don't restore modification times when untarring
jchodera Mar 17, 2019
b1afe91
Might as well try -p too
jchodera Mar 17, 2019
b7584d6
Try something else
jchodera Mar 18, 2019
0ce1a76
Making a starting test build of the Azure side of things
Lnaden Nov 25, 2019
db3b00b
fix yaml typo
Lnaden Nov 25, 2019
0cb7b67
Fix )
Lnaden Nov 25, 2019
c236b0c
Docker cleanup
Lnaden Nov 25, 2019
36be1e4
Remove -it from docker flag
Lnaden Nov 25, 2019
096b5b9
fix path
Lnaden Nov 25, 2019
def5ddd
fix set(none)
Lnaden Nov 25, 2019
0592e68
Fix another set(none) issue and add new --scheduled-only flag and
Lnaden Nov 25, 2019
0036361
Try to fix schedule
Lnaden Nov 25, 2019
9e844e7
Fix typo in schedule
Lnaden Nov 25, 2019
f31f9c0
Start Testing Windows builds
Lnaden Nov 25, 2019
59176c5
Fix Cache var
Lnaden Nov 25, 2019
7d32826
Some cache and conda issues
Lnaden Nov 25, 2019
4840189
Fix indentation issue
Lnaden Nov 25, 2019
96d705f
Simplify the CUDA steps
Lnaden Nov 25, 2019
bf0c2b6
Try to cleanup the CUDA steps
Lnaden Nov 25, 2019
51e5039
Because Null is cast to '' for REASONS!
Lnaden Nov 25, 2019
58767ca
One more fix to the windows conditions...
Lnaden Nov 25, 2019
19b27ce
Try doing a conda build now
Lnaden Nov 25, 2019
2c86af8
Lets try switching to powershell
Lnaden Nov 25, 2019
19a3471
Lets throw in OSX and keep working on windows
Lnaden Nov 25, 2019
6219d73
So close now, more small cleanups
Lnaden Nov 25, 2019
7d06134
More windows, more OSX for CUDA cleanup (may need to cache this)
Lnaden Nov 25, 2019
0a9f878
A few more paths fixed.
Lnaden Nov 25, 2019
6753291
Fixed OSX path, fixed jinja var typo, added some debug lines
Lnaden Nov 25, 2019
5f79407
Try to fix windows
Lnaden Nov 25, 2019
67476e5
MORE CONDA AND VARIABLES!, MOAR
Lnaden Nov 25, 2019
a6dc2c9
fix yaml
Lnaden Nov 25, 2019
f77e6ad
Try another windows fix, fix typo in recipe processor
Lnaden Nov 25, 2019
c1e3c5a
Another debug thing
Lnaden Nov 25, 2019
d57b0f6
Another possible fix
Lnaden Nov 25, 2019
95851ee
Fix cba for cycle-packages, start caching the osx nvidia CUDA install,
Lnaden Nov 26, 2019
4ee15c9
Yaml in osx fix, more debug lines
Lnaden Nov 26, 2019
6503f20
Fix some more pipelines, cache the actual driver install for osx
Lnaden Nov 26, 2019
845a8ec
Fix OSX cache paths, more debug in windows
Lnaden Nov 26, 2019
96fbfa3
Fix open string in windows, try more loud osx download
Lnaden Nov 26, 2019
4677299
Another pass, actually fix windows string, change nightly flags,
Lnaden Nov 26, 2019
dc9c8d1
Drop CUDA from OSX, try removing lines in windows
Lnaden Nov 26, 2019
a01c094
try commenting a couple lines for good measure
Lnaden Nov 26, 2019
9478553
Restructure script to add channels later
Lnaden Nov 26, 2019
fc6a1cd
Remove debug lines, further split up the conda commands
Lnaden Nov 26, 2019
a60da97
Try 1 last debug thing, also check for empty recipe list early
Lnaden Nov 26, 2019
6d0ff54
Finalize configs.
Lnaden Nov 26, 2019
b75b2da
Remove old debug line, change variable assembly in windows to be Bash
Lnaden Nov 26, 2019
a618d39
Change downloads to be local rather than network (otherwise cache mak…
Lnaden Nov 26, 2019
8a27e50
Die Python 2.7, die die die, die. Kthx, -Levi
Lnaden Nov 26, 2019
77b1295
Fix some more issues in the conda-build all
Lnaden Dec 4, 2019
854a8ac
Pushing my changes to try and get OpenMM to build
Lnaden Dec 4, 2019
7af426e
Revert some OpenMM changes and make the yaml explicitly add CUDA_VERSION
Lnaden Jan 7, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 136 additions & 0 deletions .azure-pipelines/azure-pipelines-linux.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
jobs:
- job: linux
pool:
vmImage: ubuntu-16.04
# timeoutInMinutes: 360
strategy:
maxParallel: 8
matrix:
linux_python3.6:
CONFIG: python3.6
DOCKER_IMAGE: condaforge/linux-anvil-comp7
EXCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.7:
CONFIG: python3.7
DOCKER_IMAGE: condaforge/linux-anvil-comp7
EXCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.8:
CONFIG: python3.8
DOCKER_IMAGE: condaforge/linux-anvil-comp7
EXCLUDE_JINJA: CUDA_SHORT_VERSION

linux_python3.6_cuda8.0:
CONFIG: python3.6_cuda8.0
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda80"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.6_cuda9.0:
CONFIG: python3.6_cuda9.0
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda90"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.6_cuda9.1:
CONFIG: python3.6_cuda9.1
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda91"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.6_cuda9.2:
CONFIG: python3.6_cuda9.2
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda92"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.6_cuda10.0:
CONFIG: python3.6_cuda10.0
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda100"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.6_cuda10.1:
CONFIG: python3.6_cuda10.1
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda101"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.6_cuda10.2:
CONFIG: python3.6_cuda10.2
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda102"
INCLUDE_JINJA: CUDA_SHORT_VERSION

linux_python3.7_cuda8.0:
CONFIG: python3.7_cuda8.0
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda80"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.7_cuda9.0:
CONFIG: python3.7_cuda9.0
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda90"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.7_cuda9.1:
CONFIG: python3.7_cuda9.1
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda91"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.7_cuda9.2:
CONFIG: python3.7_cuda9.2
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda92"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.7_cuda10.0:
CONFIG: python3.7_cuda10.0
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda100"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.7_cuda10.1:
CONFIG: python3.7_cuda10.1
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda101"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.7_cuda10.2:
CONFIG: python3.7_cuda10.2
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda102"
INCLUDE_JINJA: CUDA_SHORT_VERSION

linux_python3.8_cuda8.0:
CONFIG: python3.8_cuda8.0
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda80"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.8_cuda9.0:
CONFIG: python3.8_cuda9.0
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda90"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.8_cuda9.1:
CONFIG: python3.8_cuda9.1
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda91"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.8_cuda9.2:
CONFIG: python3.8_cuda9.2
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda92"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.8_cuda10.0:
CONFIG: python3.8_cuda10.0
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda100"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.8_cuda10.1:
CONFIG: python3.8_cuda10.1
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda101"
INCLUDE_JINJA: CUDA_SHORT_VERSION
linux_python3.8_cuda10.2:
CONFIG: python3.8_cuda10.2
DOCKER_IMAGE: "omniamd/omnia-linux-anvil:condaforge-texlive19-cuda102"
INCLUDE_JINJA: CUDA_SHORT_VERSION

steps:
- script: |
echo "##vso[task.setvariable variable=NIGHTLY]true"
condition: eq(variables['Build.Reason'], 'Schedule')
displayName: Set dev trigger if scheduled

- script: |
CBA_FLAGS="-vvv --cycle-packages --dry-run"
if [ -n "$INCLUDE_JINJA" ]; then
CBA_FLAGS="$CBA_FLAGS --build-only-jinja $INCLUDE_JINJA"
fi
if [ -n "$EXCLUDE_JINJA" ]; then
CBA_FLAGS="$CBA_FLAGS --no-build-jinja $EXCLUDE_JINJA"
fi
if [ ! -z "$NIGHTLY"]; then
CBA_FLAGS="$CBA_FLAGS --dev --scheduled-only --upload omnia-dev"
fi
echo "##vso[task.setvariable variable=CBA_FLAGS]$CBA_FLAGS"
displayName: Assemble Variables

- script: |
DOCKER_FLAGS="--rm"
DOCKER_DIRS="-v `pwd`:/io"
DOCKER_DIRS="$DOCKER_DIRS -v `pwd`/.conda_configs:/conda_configs"
DOCKER_VARS="-e CBA_FLAGS -e CONFIG"
DOCKER_COMMAND="bash /io/devtools/docker-build.sh"
docker run ${DOCKER_FLAGS} ${DOCKER_DIRS} ${DOCKER_VARS} ${DOCKER_IMAGE} ${DOCKER_COMMAND}
displayName: Run Docker Build
61 changes: 61 additions & 0 deletions .azure-pipelines/azure-pipelines-osx.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
jobs:
- job: osx
pool:
vmImage: macOS-10.13
# timeoutInMinutes: 360
strategy:
maxParallel: 8
matrix:
osx_python3.6:
CONFIG: python3.6
EXCLUDE_JINJA: CUDA_SHORT_VERSION
osx_python3.7:
CONFIG: python3.7
EXCLUDE_JINJA: CUDA_SHORT_VERSION
osx_python3.8:
CONFIG: python3.8
EXCLUDE_JINJA: CUDA_SHORT_VERSION

steps:
- script: |
echo "##vso[task.setvariable variable=NIGHTLY]true"
condition: eq(variables['Build.Reason'], 'Schedule')
displayName: Set dev trigger if scheduled

- bash: |
echo "##vso[task.prependpath]$CONDA/bin"
sudo chown -R $USER $CONDA
displayName: Add conda to PATH

- script: |
source activate base
conda install -n base -c conda-forge --quiet --yes conda-build
conda update --yes --quiet -c conda-forge -c defaults --all
displayName: Add and update conda-forge

- script: |
CBA_FLAGS="-vvv --cycle-packages --dry-run"
if [ -n "$INCLUDE_JINJA" ]; then
CBA_FLAGS="$CBA_FLAGS --build-only-jinja $INCLUDE_JINJA"
fi
if [ -n "$EXCLUDE_JINJA" ]; then
CBA_FLAGS="$CBA_FLAGS --no-build-jinja $EXCLUDE_JINJA"
fi
if [ ! -z "$NIGHTLY"]; then
CBA_FLAGS="$CBA_FLAGS --dev --scheduled-only --upload omnia-dev"
fi
export CBA_FLAGS=$CBA_FLAGS
echo "##vso[task.setvariable variable=CBA_FLAGS]$CBA_FLAGS"
displayName: Assemble Variables

- script: |
source activate base
conda config --add channels omnia
conda config --add channels conda-forge

conda install --yes conda conda-build anaconda-client

chmod +x $(Build.SourcesDirectory)/conda-build-all

python $(Build.SourcesDirectory)/conda-build-all $CBA_FLAGS -m $(Build.SourcesDirectory)/.conda_configs/$(CONFIG).yaml -- $(Build.SourcesDirectory)/*
displayName: Run Conda Build
Loading