MechDriver is the main driver for executing AutoMech workflows.
MechDriver can be installed by your favorite conda package manager from the
auto-mech channel, with
dependencies from the conda-forge channel.
We would highly recommend the Pixi package manager, which you can install in
under a minute with this command.
The Pixi command to install MechDriver is as follows.
pixi global install mechdriver -c conda-forge -c auto-mech
This "global install" command makes the code available across all environments. Like Pixi itself, however, it only uses your home directory and does not require special permissions, so you can do this anywhere. If you prefer, Pixi also allows you to install the code into a local workspace environment.
Important
Until you complete the developer set-up, you will need to add the --frozen
flag to all Pixi commands.
To get started, you will need to have installed Pixi and forked the repositories as described in Appendix A. You will also need to have configured your Git username as described in Appendix B.
You can then complete your developer set-up by cloning this repository and running the following command inside it.
pixi run --frozen dev-setup
Follow the prompts to configure the set-up to your liking. If you run into issues, see Appendix C for manual set-up instructions.
After re-starting your shell, you can test that the setup worked by activating the development environment and running the help command.
mechenv
mechdriver --help
The mechenv command is a bash function for activating MechDriver environments
that was added to your .bashrc configuration by the developer set-up.
It activates the dev environment by default, but you can also pass it the name
of a another environment, such as mechenv test to activate the test
environment.
For more information on activating Pixi environments, see
here.
The easiest way get started with MechDriver is to try out the examples
in the examples/ directory.
The quick example is a good place to start.
It can be run as follows.
cd examples/quick
mechdriver run &> out.log &
Note that you will need the g16 executable for Gaussian 16 in your path and,
if you are not using a globally installed version of the code, you will need to
activate the environment.
Developers can do this with the mechenv command.
Tip
This testing workflow uses the experimental "subtasks" module in MechDriver to parallelize subtask execution with HyperQueue. See below for more information on subtask parallelization.
The main MechDriver test workflow must be run locally on a cluster with access to Gaussian 16. The archived data from this local test run are then used for automated testing on GitHub Actions.
Assuming you have completed the developer set-up, you can initiate the test workflow as follows.
# Activate test environment
mechenv test
# Run test workflow via SLURM ...
pixi run test local -m slurm -f "--partition=<partition name>"
# ... OR run test workflow via PBS
pixi run test local -m pbs -f "-q <queue name> -A <account name>"
The above will auto-configure HyperQueue to execute the testing workflow. In some cases, you may wish to configure HyperQueue to run on particular nodes. This can be done as described in Appendix D.
Important
The test environment uses the published conda packages for autochem,
autoio, autofile, and mechanalyzer specified in pyproject.toml.
These can be updated
as described below.
To test against your local changes to these repositories before updating the
conda packages, you can activate the dev environment.
In this case, the archive for GitHub Actions will not be generated.
You can check the progress of a local test workflow as follows.
pixi run test status
You can kill a local test workflow by stopping the HyperQueue server as follows.
hq server stop
The above local test run is required before submitting a pull request with any
significant changes.
The local test data is archived and used by the GitHub Actions workflow, which also
checks the signature.yaml file to make sure the local tests were run against
the incoming version of the code.
If you are submitting a pull request with minor changes, such as editing documentation, that will not affect the tests, you can circumvent the local test workflow by "signing off" on the tests with the following command.
pixi run test sign
You will be prompted to affirm that the commits since the last local test
workflow are all minor and will not break the tests.
This list of untested commits will be recorded in the signature.yaml file, which will
override the commit hash check on GitHub Actions and allow your tests to pass.
Warning
Versions are now automatically handled by the BumpVer versioning tool.
Therefore, do not manually change the version of a given package in its
pyproject.toml. Instead, the version will be automatically updated by
triggering the release workflow as described below.
To update the conda package for one of the AutoMech repositories, you can
trigger its release workflow by following these steps:
- Submit a pull request as usual.
- In the pull request UI, click "Labels" and add the label
release:patch. - If and only if the tests pass, merge the pull request.
The release:patch label is the signal that tells GitHub Actions to run the
release workflow.
This will (1.) bump the version number, (2.) publish the conda package to the
auto-mech channel, and (3.) create an
associated GitHub release.
For now, if you have updated the conda package for a lower-level module such as
autoio and want these changes to take effect in MechDriver, you will also need
to update the following two tables in the MechDriver pyproject.toml.
[tool.pixi.package.run-dependencies]
...
autoio = "==<new version number>"
[tool.pixi.dependencies]
...
autoio = "==<new version number>"
These two dependency tables must match exactly to ensure that the packaged version of the code matches the tested version.
As an experimental feature in MechDriver, you can parallelize across subtasks in your workflow with HyperQueue. If you are not using a developer set-up, you will first need to install the HyperQueue executable and the HyperQueue Python API.
To try it out, you can test the following steps on the same "quick" example that you ran above.
Setup. You can set up these subtask jobs as follows:
mechdriver subtasks setup
This will parse your inp/ directory and create individual subdirectories for running each individual task for each individual species or reaction. These directories will go in a folder called subtasks/.
Run. You can run these subtask jobs as follows.
# For SLURM:
mechdriver subtasks run -f "--partition=<partition name>"
# For PBS:
mechdriver subtasks run -f "-q <queue name> -A <account name>"
The -f flag allows you to pass additional flags to SLURM or PBS. This would be
anything beyond basic resources (memory, CPUs, etc.) that you are required to put in
your sbatch or qsub scripts.
Check status. To check the progress of your subtask run, you can use the following command:
mechdriver subtasks status
This will print a color-coded table showing which tasks have failed for which species/reactions. It will also generate a check.log file with the paths to log files that have have not completed successfully or have a warning.
To get started as a new developer, log into GitHub (or create an account) and follow these instructions to fork the following five repositories:
The first time getting set up on a new machine, you will need to configure your
username with git.
git config --global user.name "<username>"
Make sure this matches your username on GitHub.
If you run into issues with the automated developer setup described above, you can get set up manually as follows.
- On GitHub, follow
these instructions
to make sure all of your forks are synced.
Then clone your fork for each of repository into the parent directory containing
this
mechdriver/repository.
cd ..
git clone [email protected]:<username>/autochem.git
git clone [email protected]:<username>/autoio.git
git clone [email protected]:<username>/autofile.git
git clone [email protected]:<username>/mechanalyzer.git
- Download the HyperQueue executable to a directory of your choosing.
wget https://github.com/It4innovations/hyperqueue/releases/download/v0.20.0/hq-v0.20.0-linux-x64.tar.gz
tar -zxvf hq-v0.20.0-linux-x64.tar.gz -C $HOME/bin
- In your
.bashrcfile, add the directory with the HyperQueue executable to your path, set an environment variable to turn off Python buffering, and add a bash function for easily activating this Pixi environment from anywhere on your system.
# In .bashrc:
export PATH="$HOME/bin:PATH"
export PYTHONUNBUFFERED=1
mechenv() {
local env_name="${1:-dev}"
eval "$(pixi shell-hook -e "$env_name" --manifest-path "/path/to/mechdriver")"
}
On clusters where users frequently run without the Slurm or PBS workload manager, you may need to instead configure HyperQueue to run the test workflow on particular nodes. This can be done as follows.
hq server start &> server.log &
pixi run test local &> test.log &
mechdriver subtasks start-worker -o <node 1 name> -f "<PBS/Slurm flags>"
mechdriver subtasks start-worker -o <node 2 name> -f "<PBS/Slurm flags>"
...