pull changes from develop

CHANGELOG.md

@@ -15,6 +15,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - `RedisServerManager`: context to help with starting/stopping a redis server for tests
   - `CeleryWorkersManager`: context to help with starting/stopping workers for tests
 - Ability to copy and print the `Config` object from `merlin/config/__init__.py`
+- Conflict handler option to the `dict_deep_merge` function in `utils.py`
+- Ability to add module-specific pytest fixtures
+- Added fixtures specifically for testing status functionality
+- Added tests for reading and writing status files, and status conflict handling
+- Added tests for the `dict_deep_merge` function
+- Pytest-mock as a dependency for the test suite (necessary for using mocks and fixtures in the same test)
+
+### Changed
+
+### Fixed
+- Bugfix for output of `merlin example openfoam_wf_singularity`
+
 
 ## [1.12.1]
 ### Added

Makefile

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.12.0.
+# This file is part of Merlin, Version: 1.12.1.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

docs/tutorial/4_run_simulation.md

@@ -72,8 +72,8 @@ In the `openfoam_wf_singularity` directory you should see the following:
   <figcaption>Fig 3. openfoam_wf Directory Structure</figcaption>
 </figure>
 
-- `openfoam_wf.yaml` -- this spec file is partially blank. You will fill in the gaps as you follow this module's steps.
-- `openfoam_wf_template.yaml` -- this is a complete spec file. You can always reference it as an example.
+- `openfoam_wf_singularity.yaml` -- this spec file is partially blank. You will fill in the gaps as you follow this module's steps.
+- `openfoam_wf_singularity_template.yaml` -- this is a complete spec file. You can always reference it as an example.
 - `scripts` -- This directory contains all the necessary scripts for this module.
     - We'll be exploring these scripts as we go with the tutorial.
 - `requirements.txt` -- this is a text file listing this workflow's python dependencies.
@@ -87,22 +87,22 @@ We are going to build a spec file that produces this DAG:
   <figcaption>Fig 4. OpenFOAM DAG</figcaption>
 </figure>
 
-**To start, open** `openfoam_wf.yaml` **using your favorite text editor.**
+**To start, open** `openfoam_wf_singularity.yaml` **using your favorite text editor.**
 
 It should look something like this:
 
 ???+ abstract "Initial Contents of the Spec"
 
     <!--codeinclude-->
-    [openfoam_wf.yaml](../../merlin/examples/workflows/openfoam_wf_singularity/openfoam_wf.yaml)
+    [openfoam_wf_singularity.yaml](../../merlin/examples/workflows/openfoam_wf_singularity/openfoam_wf_singularity.yaml)
     <!--/codeinclude-->
 
 ### Variables
 
 First we specify some variables to make our life easier. Locate the `env` block in our yaml spec:
 
 <!--codeinclude-->
-[](../../merlin/examples/workflows/openfoam_wf_singularity/openfoam_wf.yaml) lines:9-15
+[](../../merlin/examples/workflows/openfoam_wf_singularity/openfoam_wf_singularity.yaml) lines:9-15
 <!--/codeinclude-->
 
 The `OUTPUT_PATH` variable is set to tell Merlin where you want your output directory to be written. The default is the current working directory.
@@ -254,7 +254,7 @@ nonsimworkers:
 
 ### Putting It All Together
 
-By the end, your `openfoam_wf.yaml` should look like the template version in the same directory:
+By the end, your `openfoam_wf_singularity.yaml` should look like the template version in the same directory:
 
 ???+ abstract "Complete Spec File"
 
@@ -273,21 +273,21 @@ Now that you are done with the Specification file, use the following commands fr
 Create the DAG and send tasks to the server with:
 
 ```bash
-merlin run openfoam_wf.yaml
+merlin run openfoam_wf_singularity.yaml
 ```
 
 Open a new terminal window, then start the workers that will consume the tasks we just queued by using:
 
 ```bash
-merlin run-workers openfoam_wf.yaml
+merlin run-workers openfoam_wf_singularity.yaml
 ```
 
 But wait! We realize that 10 samples is not enough to train a good model. We would like to restart with 100 samples instead of 10 (should take about 6 minutes):
 
 After sending the workers to start on their queues, we need to first stop the workers:
 
 ```bash
-merlin stop-workers --spec openfoam_wf.yaml
+merlin stop-workers --spec openfoam_wf_singularity.yaml
 ```
 
 !!! tip
@@ -297,25 +297,25 @@ merlin stop-workers --spec openfoam_wf.yaml
 We stopped these tasks from running but if we were to run the workflow again (with 100 samples instead of 10), we would continue running the 10 samples first! This is because the queues are still filled with the previous attempt's tasks. This can be seen with:
 
 ```bash
-merlin status openfoam_wf.yaml
+merlin status openfoam_wf_singularity.yaml
 ```
 
 We need to purge these queues first in order to repopulate them with the appropriate tasks. This is where we use the `merlin purge` command:
 
 ```bash
-merlin purge openfoam_wf.yaml
+merlin purge openfoam_wf_singularity.yaml
 ```
 
 Now we are free to repopulate the queues with the 100 samples. In our terminal window that's not designated for our workers, we'll queue up tasks again, this time with 100 samples:
 
 ```bash
-merlin run openfoam_wf.yaml --vars N_SAMPLES=100
+merlin run openfoam_wf_singularity.yaml --vars N_SAMPLES=100
 ```
 
 Then in our window for workers, we'll execute:
 
 ```bash
-merlin run-workers openfoam_wf.yaml
+merlin run-workers openfoam_wf_singularity.yaml
 ```
 
 To see your results, look inside the `learn` output directory. You should see a png that looks like this:

docs/user_guide/configuration/merlin_server.md

@@ -0,0 +1,177 @@
+# Merlin Server Configuration
+
+!!! warning
+
+    It's recommended that you read through the [Configuration Overview](./index.md) page before proceeding with this module.
+
+The merlin server command allows users easy access to containerized broker and results servers for Merlin workflows. This allows users to run Merlin without a dedicated external server.
+
+The main configuration will be stored in the subdirectory called `server/` by default in the main Merlin configuration directory `~/.merlin`. However, different server images can be created for different use cases or studies by simplying creating a new directory to store local configuration files for Merlin server instances.
+
+This module will walk through how to initalize the server, start it, and ensure it's linked to Merlin.
+
+## Initializing the Server
+
+First create and navigate into a directory to store your local Merlin configuration for a specific use case or study:
+
+```bash
+mkdir study1/ ; cd study1/
+```
+
+Afterwards you can instantiate Merlin server in this directory by running:
+
+```bash
+merlin server init
+```
+
+A main server configuration will be created in the `~/.merlin/server/` directory. This will have the following files:
+
+- docker.yaml
+- merlin_server.yaml
+- podman.yaml
+- singularity.yaml
+
+The main configuration in `~/.merlin/server/` deals with defaults and technical commands that might be used for setting up the Merlin server local configuration and its containers. Each container has their own configuration file to allow users to be able to switch between different containerized services freely.
+
+In addition to the main server configuration, a local server configuration will be created in your current working directory in a folder called `merlin_server/`. This directory will contain:
+
+- `redis.conf`: The Redis configuration file that contains all of the settings to be used for our Redis server
+- `redis.pass`: A password for the Redis server that we'll start up next
+- `redis.users`: A file defining the users that are allowed to access the Redis server and their permissions
+- `redis_latest.sif`: A singularity file that contains the latest Redis Docker image that was pulled behind the scenes by Merlin
+
+The local configuration `merlin_server/` folder contains configuration files specific to a certain use case or run. In the case above you can see that we have a Redis singularity container called `redis_latest.sif` with the Redis configuration file called `redis.conf`. This Redis configuration will allow the user to configure Redis to their specified needs without have to manage or edit the Redis container. When the server is run this configuration will be dynamically read, so settings can be changed between runs if needed.
+
+Once the Merlin server has been initialized in the local directory the user will be allowed to run other Merlin server commands such as `start`, `status`, and `stop` to interact with the Merlin server. A detailed list of commands can be found in the [Merlin Server](../command_line.md#server-merlin-server) section of the [Command Line](../command_line.md) page.
+
+!!! note
+
+    Running `merlin server init` again will *not* override any exisiting configuration that the users might have set or edited. By running this command again any missing files will be created for the users with exisiting defaults. *However,* it is highly advised that users back up their configuration in case an error occurs where configuration files are overriden.
+
+## Starting the Server and Linking it to Merlin
+
+!!! bug
+
+    For LC users, servers cannot be started outside your home (`~/`) directory.
+
+!!! warning
+
+    Newer versions of Redis have started requiring a global variable `LC_ALL` to be set in order for this to work. To set this properly, run:
+
+    ```bash
+    export LC_ALL="C"
+    ```
+
+    If this is not set, the `merlin server start` command may seem to run forever until you manually terminate it.
+
+After initializing the server, starting the server is as simple as running:
+
+```bash
+merlin server start
+```
+
+You can check that the server was started properly with:
+
+```bash
+merlin server status
+```
+
+The `merlin server start` command will add new files to the local configuration `merlin_server/` folder:
+
+- `merlin_server.pf`: A process file containing information regarding the Redis process
+- `app.yaml`: A new app.yaml file configured specifically for the containerized Redis server that we just started
+
+To have Merlin read this server configuration:
+
+=== "Copy Configuration to CWD"
+
+    ```bash
+    cp merlin_server/app.yaml .
+    ```
+
+=== "Make This Server Configuration Your Main Configuration"
+
+    If you're going to use the server configuration as your main configuration, it's a good idea to make a backup of your current server configuration (if you have one):
+
+    ```bash
+    mv ~/.merlin/app.yaml ~/.merlin/app.yaml.bak
+    ```
+
+    From here, simply copy the server configuration to your `~/.merlin/` folder:
+
+    ```bash
+    cp merlin_server/app.yaml ~/.merlin/app.yaml
+    ```
+
+You can check that Merlin recognizes the containerized server connection with:
+
+```bash
+merlin info
+```
+
+If your servers are running and set up properly, this should output something similar to this:
+
+???+ success
+
+    ```bash
+           *      
+       *~~~~~                                       
+      *~~*~~~*      __  __           _ _       
+     /   ~~~~~     |  \/  |         | (_)      
+         ~~~~~     | \  / | ___ _ __| |_ _ __  
+        ~~~~~*     | |\/| |/ _ \ '__| | | '_ \ 
+       *~~~~~~~    | |  | |  __/ |  | | | | | |
+      ~~~~~~~~~~   |_|  |_|\___|_|  |_|_|_| |_|
+     *~~~~~~~~~~~                                    
+       ~~~*~~~*    Machine Learning for HPC Workflows                                 
+
+
+
+    Merlin Configuration
+    -------------------------
+
+    config_file        | /path/to/app.yaml
+    is_debug           | False
+    merlin_home        | /path/to/.merlin
+    merlin_home_exists | True
+    broker server      | redis://default:******@127.0.0.1:6379/0
+    broker ssl         | False
+    results server     | redis://default:******@127.0.0.1:6379/0
+    results ssl        | False
+
+    Checking server connections:
+    ----------------------------
+    broker server connection: OK
+    results server connection: OK
+
+    Python Configuration
+    -------------------------
+
+    $ which python3
+    /path/to/python3
+
+    $ python3 --version
+    Python x.y.z
+
+    $ which pip3
+    /path/to/pip3
+
+    $ pip3 --version
+    pip x.y.x from /path/to/pip (python x.y)
+
+    "echo $PYTHONPATH"
+    ```
+
+## Stopping the Server
+
+Once you're done using your containerized server, it can be stopped with:
+
+```bash
+merlin server stop
+```
+
+You can check that it's no longer running with:
+
+```bash
+merlin server status
+```

merlin/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.12.0.
+# This file is part of Merlin, Version: 1.12.1.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -38,7 +38,7 @@
 import sys
 
 
-__version__ = "1.12.0"
+__version__ = "1.12.1"
 VERSION = __version__
 PATH_TO_PROJ = os.path.join(os.path.dirname(__file__), "")
 

merlin/ascii_art.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.12.0.
+# This file is part of Merlin, Version: 1.12.1.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/celery.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.12.0.
+# This file is part of Merlin, Version: 1.12.1.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.12.0.
+# This file is part of Merlin, Version: 1.12.1.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/abstracts/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.12.0.
+# This file is part of Merlin, Version: 1.12.1.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/abstracts/enums/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.12.0.
+# This file is part of Merlin, Version: 1.12.1.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/dumper.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.12.0
+# This file is part of Merlin, Version: 1.12.1
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/openfilelist.py

@@ -8,7 +8,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.12.0.
+# This file is part of Merlin, Version: 1.12.1.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/opennpylib.py

@@ -8,7 +8,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.12.0.
+# This file is part of Merlin, Version: 1.12.1.
 #
 # For details, see https://github.com/LLNL/merlin.
 #