Use a .env file for local dev instance config (#728)

+ Install dotenv Gradle plugin to source instance configs as env vars, reading a `.env` file for local development launches via `bootRun`. 
+ This replaces the prior use of a YAML file in `/etc/hoist/conf` for Toolbox.
+ Include `.env.template` to enumerate required vars.
+ Update README to document.
+ NOTE - requires new `build.gradle` file to be created at root directory of wrapper projects (Toolbox + hoist-core checked out together).  Documented w/required file contents in the README.

Author: amcclain

Diff for: .env.template

@@ -0,0 +1,33 @@
+# Duplicate this file in your local repo with filename `.env`. It will be ignored by git, but loaded
+# by Gradle via a dotenv Gradle plugin and passed to Toolbox to use as instance configs.
+
+# Note that any values provided in this template will never be read - they are there to provide
+# example values only. Any uncommented vars in this file *must* be set in a local .env file.
+# See the README for more information.
+
+# Required variables - must be set in .env, build will throw if missing.
+APP_TOOLBOX_ENVIRONMENT=Development
+APP_TOOLBOX_DB_HOST=localhost
+APP_TOOLBOX_DB_SCHEMA=toolbox
+APP_TOOLBOX_DB_USER=toolbox
+APP_TOOLBOX_DB_PASSWORD=toolbox
+
+# Additional DB options
+#APP_TOOLBOX_USE_H2=true
+#APP_TOOLBOX_DB_CREATE=update
+
+# Bootstrap admin
+# The _USER var can be set on its own to grant an OAuth (Auth0) sourced user admin rights in local
+# development. This is supported by Toolbox's use of Hoist Core DefaultRoleService.
+#APP_TOOLBOX_BOOTSTRAP_ADMIN_USER=
+
+# If the _PASSWORD var is also set, Toolbox will create a password-enabled user in its user
+# database that can be used when Auth0 is not available. Pair this with the env below to disable
+# the OAuth flow entirely and present a form-based login.
+#APP_TOOLBOX_BOOTSTRAP_ADMIN_PASSWORD=
+#APP_TOOLBOX_USE_OAUTH=false
+
+# Email support
+#APP_TOOLBOX_SMTP_HOST=
+#APP_TOOLBOX_SMTP_USER=
+#APP_TOOLBOX_SMTP_PASSWORD=

Diff for: .gitignore

@@ -1,5 +1,4 @@
 # IDE Files (mostly)
-###################
 .idea/*
 !/.idea/scopes/
 !/.idea/copyright/
@@ -8,8 +7,11 @@
 .project
 .classpath
 
-# Compiled source #
-###################
+# Local instance configs w/secrets
+toolbox.yml
+.env
+
+# Compiled source
 .gradle/
 build/
 bin/
@@ -18,14 +20,11 @@ out/
 /client-app/node_modules/
 /client-app/tsconfig.tsbuildinfo
 
-# Logs #
-########
+# Logs
 toolbox-logs/
 *.log
-toolbox.yml
 
-# OS generated files #
-######################
+# OS generated files
 .DS_Store
 .DS_Store?
 ehthumbs.db

Diff for: README.md

@@ -23,60 +23,36 @@ applications, which typically use whatever enterprise database is already in pla
 infrastructure, but it provides a common and easy-to-run DB for local development and our AWS-based
 deployments.
 
-* For initial/test usage, Toolbox is configured to  use an in-memory H2 database that will be 
-  rebuilt at startup of the app. This is governed by the `useH2` instance configuration (see below).
-* For persistent deployments, Toolbox is designed to work with MySQL 5.x.*.  If you don't already 
-  have it installed and are on a Mac, we recommend installing via [Homebrew](https://brew.sh/) with 
- `brew install [email protected]`.
-* Create a new empty database named `toolbox`, being sure to use a UTF8 charset. Alternatively, use
-  an export of the deployed toolbox DB with `CREATE DATABASE` included (Anselm can provide).
-* For local development, use of the `root` account is fine, or you can create a local user and
-  password dedicated to Toolbox. If using a non-root account, ensure that the user has DBO rights on
-  the new database.
+* For initial/test usage, Toolbox is can run with a transient in-memory database (H2) that will be
+  rebuilt at startup of the app. This is enabled via the `useH2` instance configuration (see below).
+* For persistent deployments, Toolbox is designed to work with MySQL. If you don't already have it
+  installed and are on a Mac, we recommend installing via [Homebrew](https://brew.sh/) with
+  `brew install mysql`.
+* Create a new empty database named `toolbox`, being sure to use a UTF8 charset (fortunately this is
+  the default for newer versions of MySQL). Alternatively, use an export of the deployed toolbox DB
+  with `CREATE DATABASE` included (Anselm can provide).
+* For local development, use of the `root` account is fine, or (better) you can create a local user
+  and password dedicated to Toolbox. If using a non-root account, ensure that the user has DBO
+  rights on the new database. Database credentials are provided to the app via instance
+  configuration (see below).
 * If the server is started against an empty database, Grails will auto-create the required schemas
   on first run as long as a suitable value is provided for the `dbCreate` data source parameter. See
-  `grails-app/conf/runtime.groovy` for where this is set - we leave toolbox on `update` to allow for
-  automatic schema changes as needed.
+  the `DBConfig` class for where this is set - we leave toolbox on `update` to allow for automatic
+  schema changes as needed.
 
 ## Instance Configuration
 
-Hoist applications can read low-level, instance-specific information from a YML configuration file
-on the local machine where they run. This is used primarily to set database credentials, which we
-don't wish to check in to source control but which are required to connect to the DB and read all
-_other_ data-driven app configurations.
-
-* Create a new instance config file - the default location is `/etc/hoist/conf/toolbox.yml`.
-  * If you don't wish to create that directory structure under `/etc/`, you can place the same file
-    elsewhere and point the server there with a JavaOpt - see the hoist-core provided
-    [`InstanceConfigUtils.groovy`](https://github.com/xh/hoist-core/blob/develop/src/main/groovy/io/xh/hoist/util/InstanceConfigUtils.groovy)
-    for details.
-* The contents of this file (for Toolbox) will typically be as follows:
-
-```
-environment: Development
-serverURL: http://localhost
-
-# The following are for use in early runs of the project before you have granted roles to any users,
-# or in cases where Auth0 isn't acting as expected, or you're hosting the app on a device. 
-# The bootstrapAdminUser will be available for forms based login and will be granted the role 
-# needed (HOIST_ROLE_MANAGER) to grant access to other users.
-useOAuth: false
-bootstrapAdminUser: [email protected]
-bootstrapAdminPassword: "your password"
-
-# Enable in memory h2 database option. When ready, configure proper DB below and set to false
-useH2: true
-
-# MySql DB config - provide either root or a dedicated local account, if using.
-dbHost: localhost:3306
-dbSchema: toolbox
-dbUser: root
-dbPassword: "your database user password" 
-```
-
-* When running the Toolbox server, look for a message along the lines of "Loaded 10 instanceConfigs
-  from /etc/hoist/conf/toolbox.yml" to be logged to the console early on in the startup process.
-  This will indicate that Hoist has successfully read your config.
+Hoist applications require low-level "instance configuration" properties to provide settings such
+as database credentials and other environment-specific settings that should not be checked into
+source control. These properties can be set in one of two primary ways: a YAML file on the local
+filesystem, or environment variables.
+
+Toolbox is configured to source these properties from *environment variables*, and uses a Gradle
+plugin to read them from a `.env` file in the project root. That file should never be pushed to git
+(it's listed in .gitignore accordingly), but an `.env.template` file *is* checked-in to enumerate
+required and optional properties.
+
+Copy `.env.template` to `.env` and fill in the required values for your local database connection.
 
 ## Authentication
 
@@ -88,27 +64,6 @@ When adding a new top-level entry-point for Toolbox (such as a new example appli
 URL must be registered with Auth0 as a valid OAuth callback URL. Either Lee or Anselm can update our
 Auth0 config accordingly.
 
-## Developing with HTTPS on `xh.io` domain
-
-It can be useful to run Toolbox locally with HTTPS enabled and on a sub-domain of `xh.io`, 
-especially when testing OAuth, CORS, or cookie dependent features. 
-Follow these steps to run with HTTPS on the `toolbox-local.xh.io:3000` domain:
-1. add this entry to your dev machine's `hosts` file: `127.0.0.1 toolbox-local.xh.io`
-2. start the Grails server with the following additional VM Options:
-```
--Dserver.ssl.enabled=true
--Dserver.ssl.certificate=classpath:local-dev/toolbox-local.xh.io-self-signed.crt
--Dserver.ssl.certificate-private-key=classpath:local-dev/toolbox-local.xh.io-self-signed.key
--Dserver.ssl.trust-certificate=classpath:local-dev/toolbox-local.xh.io-self-signed.ca.crt
-```
-  The referenced files are self-signed certs commited to the repo for local dev purposes.
-  They expire on Sunday, May 21, 2034 at 6:11:28AM.
-3. Visit `https://toolbox-local.xh.io:8080/ping` in your browser to proceed past the SSL warning 
-  for API calls.
-4. Start the GUI with the `startWithHoistSecure` npm script.  Go to `https://toolbox-local.xh.io:3000/app/home` 
-   in your browser and proceed past the SSL warning.
-
-
 ## Wrapper project for Toolbox + Hoist development
 
 A special project / directory structure can be useful for developing Toolbox alongside the Hoist
@@ -123,17 +78,34 @@ use when setting up Toolbox.
 * Create a new `settings.gradle` file within the top-level directory. The contents of this file will
   be a single line: `include "toolbox", "hoist-core"`. This tells Gradle to reference and combine
   the `build.gradle` targets of those two sub-projects into a single umbrella project.
+* Create a new `build.gradle` file within the top-level directory. This is required to support the
+  `co.uzzu.dotenv` plugin used by Toolbox to read environment variables - the plugin must be applied
+  to the root project, which is the top-level wrapper directory in this mode. The Toolbox
+  `build.gradle` file is where the plugin is actually applied, but for that to work in wrapper mode
+  the plugin must be defined within the root (wrapper) project also, requiring a build file. Its
+  contents should be:
+    ```groovy
+    buildscript {
+        repositories {
+            gradlePluginPortal()
+        }
+        dependencies {
+            classpath "co.uzzu.dotenv:gradle:4.0.0" // keep in sync with version in Toolbox build.gradle
+        }
+    }
+    ```
 * From the checked-out `toolbox` sub-directory, copy the `gradle` directory and the `gradlew` (or
   `gradlew.bat` if on Windows) wrapper script and paste the copies into the top-level wrapper
   directory.
-  * Your top-level directory should now contain four sub-directories, `settings.gradle`, and the
-    `gradlew` script.
+    * Your top-level directory should now contain four sub-directories, `settings.gradle`,
+      `build.gradle`, and the `gradlew` script.
 * From the top-level directory, run `./gradlew` to ensure that Gradle can properly configure the
   unified build.
-* If using IntelliJ, create a newJ project by running through the "New project from existing
+* If using IntelliJ, create a new project by running through the "New project from existing
   sources..." workflow and pointing the IDE at the top-level `settings.gradle` file.
-  * IntelliJ should detect that this is a Gradle/Grails project, download and index the server-side
-    dependencies, and set up an appropriate "Run Configuration" to start the Toolbox server.
+    * IntelliJ should detect that this is a Gradle/Grails project, download and index the
+      server-side dependencies, and set up an appropriate "Run Configuration" to start the Toolbox
+      server.
 
 Having all three repos checked out in a single IntelliJ project can be useful to have the code
 on-hand, but to actually run Toolbox using the local Hoist libraries some additional steps are
@@ -146,8 +118,28 @@ required.
 * To run the client using the local `hoist-react`, start your local webpack-dev-server from the
   `toolbox/client-app` directory by running `yarn startWithHoist`.
 
+## Developing with HTTPS on `xh.io` domain
+
+It can be useful to run Toolbox locally with HTTPS enabled and on a sub-domain of `xh.io`,
+especially when testing OAuth, CORS, or cookie dependent features. Follow these steps to run with
+HTTPS on the `toolbox-local.xh.io:3000` domain:
+
+1. Add this entry to your dev machine's `hosts` file: `127.0.0.1 toolbox-local.xh.io`
+2. Start the Grails server with the additional VM options below. The referenced files are
+   self-signed certs commited to the repo for local dev purposes.
+    ```
+    -Dserver.ssl.enabled=true
+    -Dserver.ssl.certificate=classpath:local-dev/toolbox-local.xh.io-self-signed.crt
+    -Dserver.ssl.certificate-private-key=classpath:local-dev/toolbox-local.xh.io-self-signed.key
+    -Dserver.ssl.trust-certificate=classpath:local-dev/toolbox-local.xh.io-self-signed.ca.crt
+    ```
+3. Visit `https://toolbox-local.xh.io:8080/ping` in your browser to proceed past the SSL warning
+   for API calls.
+4. Start the GUI with the `startWithHoistSecure` npm script. Go to
+   `https://toolbox-local.xh.io:3000/app/` in your browser and proceed past the SSL warning.
+
 ------------------------------------------
 
 📫☎️🌎 [email protected] | <https://xh.io/contact>
 
-Copyright © 2022 Extremely Heavy Industries Inc.
+Copyright © 2024 Extremely Heavy Industries Inc.

Diff for: build.gradle

@@ -3,12 +3,13 @@ import static java.lang.Boolean.parseBoolean
 buildscript {
     repositories {
         mavenCentral()
-        maven {url 'https://plugins.gradle.org/m2/'}
+        gradlePluginPortal()
         maven {url 'https://repo.grails.org/grails/core'}
     }
     dependencies {
         classpath "org.grails:grails-gradle-plugin:$grailsGradlePluginVersion"
         classpath "org.grails.plugins:hibernate5:$grailsHibernatePluginVersion"
+        classpath "co.uzzu.dotenv:gradle:$dotEnvGradlePluginVersion"
     }
 }
 
@@ -19,6 +20,12 @@ apply plugin:'idea'
 apply plugin:'war'
 apply plugin:'org.grails.grails-web'
 
+// Must be applied to root project - matters when running Toolbox in a wrapper project mode w/hoist-core.
+// See README.md for more details on this configuration, including contents of top-level build.gradle file.
+gradle.rootProject {
+    apply plugin:'co.uzzu.dotenv.gradle'
+}
+
 repositories {
     mavenCentral()
     maven {url 'https://repo.grails.org/grails/core'}
@@ -106,6 +113,8 @@ bootRun {
     jvmArgs(allJvmArgs)
     sourceResources sourceSets.main
     systemProperties hoistMetaData
+    // Bring .env sourced environment variables into bootRun JVM process.
+    environment env.allVariables()
 }
 
 tasks.withType(GroovyCompile) {
@@ -133,3 +142,11 @@ tasks.war.doFirst {
     properties.putAll(hoistMetaData)
     infoFile.withOutputStream {properties.store(it, null)}
 }
+
+// Ensure that all variables defined in .env.template are set in local .env
+tasks.bootRun.doFirst {
+    def missingEnvVars = env.allVariablesOrNull().findAll {it.value == null}.collect {it.key}
+    if (missingEnvVars) {
+        throw new GradleException("Environment variables listed in .env.template not set in local .env file as required: ${missingEnvVars}")
+    }
+}

Diff for: gradle.properties

@@ -8,8 +8,9 @@ hoistCoreVersion=21.0-SNAPSHOT
 groovyVersion=3.0.21
 grailsVersion=6.2.0
 grailsGradlePluginVersion=6.2.0
-gormVersion=8.1.0
+dotEnvGradlePluginVersion=4.0.0
 grailsHibernatePluginVersion=8.1.0
+gormVersion=8.1.0
 hazelcast.version=5.3.7
 logback.version=1.2.7