chore: sync upstream (#404)

* chore: sync upstream

* fix: repo references

* fix: fixes with upstream

* chore: continue sync upstream

* fix: lint and snapshots

* fix: deps and lint

Co-authored-by: niftylettuce <[email protected]>

Author: shaunwarman

README.md

@@ -67,7 +67,7 @@ These microservices are preconfigured for security, performance, and graceful re
 
 * Webapp server → [web.js](template/web.js)
 * API server → [api.js](template/api.js)
-* Job scheduler → [bull.js](template/bull.js)
+* Job scheduler → [bree.js](template/bree.js)
 * Proxy server → [proxy.js](template/proxy.js)
 
 ### Front-end
@@ -302,6 +302,181 @@ We strongly recommend using [SemaphoreCI][], [PM2][], and [Digital Ocean][do] fo
 
 5. If you specify an environment variable value for `AWS_CF_DOMAIN` and `NODE_ENV=production` is set then your assets will need to be published to Amazon S3/Cloudfront. To do so run `npm start publish-assets` (or with yarn as `yarn start publish-assets`).  This command automatically sets `NODE_ENV=production` for you as well via `cross-env`.
 
+#### Provisioning
+
+See the [ansible](ansible/) folder for our [Ansible][] configuration and playbooks, which we use to provision servers with.
+
+We recommend you to install [yamllint][] and configure it in your editor while working with [Ansible][] playbooks.
+
+Also note that [ansible-lint][] is a helpful linting tool you can use if you plan on making changes to playbooks.  Note that our current playbooks have several existing lint errors.
+
+First you must provision Ubuntu 18.04 LTS 64-bit server(s) using [Digital Ocean][digital-ocean], [Linode][], [Vultr][], or your host of choice.  These newly provisioned server(s) should have your SSH key automatically added.
+
+Follow the [Deployment](#deployment) guide below for automatic provisioning and deployment instructions.
+
+##### Deployment
+
+1. Set up host configuration by copying the `hosts.yml` file template:
+
+   ```sh
+   cp ansible/playbooks/templates/hosts.yml hosts.yml
+   ```
+
+2. Edit this configuration and update the file with your newly created server aliases and IP addresses.  You can add more than one host to each group if you are setting up load balancing.  Refer to the [Naming Convention](#naming-convention) documentation for our recommended approach to server alias naming.  Note that this file is automatically ignored by git.  If you have a private repository and would like to commit this, then remove `hosts.yml` from the root `.gitignore` file.
+
+   ```sh
+   vim hosts.yml
+   ```
+
+3. Set up environment configuration by copying the `env` file template:
+
+   ```sh
+   cp ansible/playbooks/templates/env .env.production
+   ```
+
+4. Edit this configuration and reference the official [Lad][] documentation for a list of all available environment variables (or see [.env.defaults](.env.defaults)).  **You will need to open this file in your preferred editor** and set the values for any fields containing `TODO`, whereby you replace `TODO` with the appropriate value.  Preserve double quotes where they are already defined.
+
+   ```sh
+   vim .env.production
+   ```
+
+5. Generate [pm2][] [ecosystem files][ecosystem-files] using our automatic template generator. We created an [ansible-playbook.js](ansible-playbook.js) which loads the `.env.production` environment variables rendered with [@ladjs/env][] into `process.env`, which then gets used in the playbooks.  This is a superior, simple, and the only known dotenv approach we know of in Ansible. Newly created `ecosystem-api.json`, `ecosystem-bree.json`, `ecosystem-web.json` files will now be created for you in the root of the repository.  If you ever more add or change IP addresses, you can simply re-run this command.
+
+   ```sh
+   node ansible-playbook ansible/playbooks/ecosystem.yml -l 'localhost'
+   ```
+
+6. Set up the web and API server(s) (see [patterns and ansible-playbook flags docs](https://docs.ansible.com/ansible/latest/user_guide/intro_patterns.html#patterns-and-ansible-playbook-flags) if you need help).  If you completely (or partially) run this playbook (or any others below), then the second time you try to run it may not succeed.  This is because we prevent root user access through security hardening.  To workaround this, run the same command but without `-e 'ansible_user=root'` appended as it will default to the `devops` user created.
+
+   ```sh
+   node ansible-playbook ansible/playbooks/http.yml -e 'ansible_user=root' -l 'http'
+   ```
+
+7. Set up the Bree server(s):
+
+   ```sh
+   node ansible-playbook ansible/playbooks/bree.yml -e 'ansible_user=root' -l 'bree'
+   ```
+
+8. Set up the Redis server:
+
+   ```sh
+   node ansible-playbook ansible/playbooks/redis.yml -e 'ansible_user=root' -l 'redis'
+   ```
+
+9. Set up the Mongo server:
+
+   ```sh
+   node ansible-playbook ansible/playbooks/mongo.yml -e 'ansible_user=root' -l 'mongo'
+   ```
+
+10. Set up GitHub deployment keys for all the servers. Note that the `deployment-keys` directory is ignored from git, so if you have a private repository and wish to commit it, then remove `deployment-keys` from the `.gitignore` file.
+
+    ```sh
+    node ansible-playbook ansible/playbooks/deployment-keys.yml -l 'http:bree'
+    ```
+
+11. Go to your repository "Settings" page on GitHub, click on "Deploy keys", and then add a deployment key for each servers' deployment key copied to the `deployment-keys` directory.  If you're on macOS, you can use the `pbcopy` command to copy each file's contents to your clipboard.  Use tab completion for speed, and replace the server names and paths with yours:
+
+    ```sh
+    cat deployment-keys/api-1-li-dal.forwardemail.net.pub | pbcopy
+
+    #
+    # NOTE: repeat the above command for all servers
+    # and after running the command, it will copy
+    # the key to your clipboard for you to paste as
+    # a new deploy key (make sure to use read-only access)
+    #
+    ```
+
+12. Set up PM2 deployment directories on all the servers:
+
+    ```sh
+    pm2 deploy ecosystem-web.json production setup
+    ```
+
+    ```sh
+    pm2 deploy ecosystem-api.json production setup
+    ```
+
+    ```sh
+    pm2 deploy ecosystem-bree.json production setup
+    ```
+
+13. Create a SSL certificate at [Namecheap][] (we recommend a 5 year wildcard certificate), set up the certificate, and download and extract the ZIP file with the certificate (emailed to you) to your computer. We do not recommend using tools like [LetsEncrypt][] and `certbot` due to complexity when you have (or scale to) a cluster of servers set up behind load balancers.  In other words, we've tried approaches like `lsyncd` in combination with `crontab` for `certbot` renewals and automatic checking.  Furthermore, using this exposes the server(s) to downtime as ports `80` and `443` may need to be shut down so that `certbot` can use them for certificate generation.  This is not a reliable approach, and simply renewing certificates once a year is vastly simpler and also makes using load balancers trivial.  Instead you can use a provider like [Namecheap][] to get a cheap SSL certificate, then run a few commands as we've documented below. This command will prompt you for an absolute file path to the certificates you downloaded. Renewed your certificate after 1 year? Simply follow this step again.  Do not set a password on the certificate files.  When using the `openssl` command (see Namecheap instructions), you need to use `*.example.com` with an asterisk followed by a period if you are registering a wildcard certificate.
+
+    ```sh
+    ansible-playbook ansible/playbooks/certificates.yml
+    ```
+
+    > **Important:** If you renew or change certificates in the future, then after running the previous command, you will subsequently need to reload the processes as such:
+
+    ```sh
+    #
+    # NOTE: See the "Important" note above BEFORE running this command.
+    #       This command ONLY APPLIES for certificate renewals/changes.
+    #
+    pm2 deploy ecosystem-web.json production exec "pm2 reload web"
+    pm2 deploy ecosystem-api.json production exec "pm2 reload api"
+    ```
+
+14. (Optional) Create a Google application credentials profile file and store it locally.  You only need this if you want to support automatic translation.  The following command will prompt you for the absolute file path (e.g. `/path/to/client-profile.json`).  See the [mandarin][] docs for more information.
+
+    ```sh
+    ansible-playbook ansible/playbooks/gapp-creds.yml -l 'http:bree'
+    ```
+
+15. Copy the `.env.production` file and create an AWS config file on the servers:
+
+    ```sh
+    node ansible-playbook ansible/playbooks/env.yml -l 'http:bree'
+    ```
+
+16. Run an initial deploy to all the servers:
+
+    ```sh
+    pm2 deploy ecosystem-web.json production
+    ```
+
+    ```sh
+    pm2 deploy ecosystem-api.json production
+    ```
+
+    ```sh
+    pm2 deploy ecosystem-bree.json production
+    ```
+
+17. Save the process list on the servers so when if the server were to reboot, it will automatically boot back up the processes:
+
+    ```sh
+    pm2 deploy ecosystem-web.json production exec "pm2 save"
+    ```
+
+    ```sh
+    pm2 deploy ecosystem-api.json production exec "pm2 save"
+    ```
+
+    ```sh
+    pm2 deploy ecosystem-bree.json production exec "pm2 save"
+    ```
+
+
+16. Test by visiting your web and API server in your browser (click "proceed to unsafe" site and bypass certificate warning).
+
+17. Configure your DNS records for the web and API server hostnames and respective IP addresses.
+
+18. Test by visiting your web and API server in your browser (in an incognito window).  There should not be any certificate warnings (similar to the one that occurred in step 15).
+
+19. (Optional) Remove the local `.env.production` file for security purposes.  If you do this, then make sure you have a backup, or securely back up off the server in the future before destroying the server.
+
+    ```sh
+    rm .env.production
+    ```
+
+20. (Optional) Remove the local certificate files you downloaded locally and specified in step 11.  If you do this, then make sure you have a backup, or securely back up off the server in the future before destroying the server.
+
+21. Finished. If you need to deploy again, then push your changes to GitHub `master` branch and then follow step 14 again.  We recommend you to read the [Ansible getting started guide][ansible-guide], as it provides you with insight into commands like `ansible all -a "echo hello"` which can be run across all or specific servers.
+
 #### Tests
 
 We use [ava][] and [nyc][] for testing and code coverage.
@@ -706,7 +881,7 @@ template
 │   │   └── uncaught.js
 │   ├── manifest.json
 │   └── robots.txt
-├── bull.js
+├── bree.js
 ├── config
 │   ├── env.js
 │   ├── index.js
@@ -833,7 +1008,7 @@ If you are seeking permission to use these trademarks, then please [contact us](
 [MIT](LICENSE) © [Nick Baugh](http://niftylettuce.com)
 
 
-## 
+##
 
 <a href="#"><img src="media/lad-footer.png" alt="#" /></a>
 
@@ -927,8 +1102,6 @@ If you are seeking permission to use these trademarks, then please [contact us](
 
 [mongoose]: http://mongoosejs.com
 
-[bull]: https://github.com/OptimalBits/bull
-
 [ladjs-auth]: https://github.com/ladjs/auth
 
 [csrf-alternative]: https://scotthelme.co.uk/csrf-is-dead/
@@ -1024,3 +1197,25 @@ If you are seeking permission to use these trademarks, then please [contact us](
 [cabinjs options]: https://github.com/cabinjs/axe#options
 
 [certbot options]: https://certbot.eff.org/docs/using.html#id11
+
+[ansible]: https://github.com/ansible/ansible
+
+[yamllint]: https://github.com/adrienverge/yamllint
+
+[ansible-lint]: https://github.com/ansible/ansible-lint
+
+[digital-ocean]: https://m.do.co/c/2ffb8129b8d6
+
+[linode]: https://www.linode.com/?r=a2840b36770c7020730251a5643428ddbf2e284e
+
+[vultr]: https://www.vultr.com/?ref=7429848
+
+[ecosystem-files]: https://pm2.keymetrics.io/docs/usage/application-declaration/
+
+[@ladjs/env]: https://github.com/ladjs/env
+
+[namecheap]: https://namecheap.com
+
+[letsencrypt]: https://letsencrypt.org/
+
+[ansible-guide]: https://docs.ansible.com/ansible/latest/user_guide/intro_getting_started.html

cli.js

@@ -15,7 +15,7 @@ const cli = cac();
 
 cli
   .command('<name>', 'Generate a new project')
-  .action(name => {
+  .action((name) => {
     const folderName = name;
     const targetPath = path.resolve(folderName);
     console.log(`> Generating project in ${targetPath}`);

package.json

@@ -24,7 +24,8 @@
     "Shaun Warman <[email protected]> (https://shaunwarman.com/)"
   ],
   "dependencies": {
-    "cac": "^6.5.8",
+    "@ladjs/browserslist-config": "^0.0.1",
+    "cac": "^6.6.1",
     "camelcase": "^5.3.1",
     "github-username-regex": "^1.0.0",
     "is-email": "^1.0.0",
@@ -39,21 +40,22 @@
     "uppercamelcase": "^3.0.0"
   },
   "devDependencies": {
-    "@commitlint/cli": "^8.3.5",
-    "@commitlint/config-conventional": "^8.3.4",
-    "ava": "^2.4.0",
-    "codecov": "^3.6.5",
-    "cross-env": "^6.0.3",
-    "eslint": "^6.8.0",
+    "@commitlint/cli": "^9.1.1",
+    "@commitlint/config-conventional": "^9.1.1",
+    "ava": "^3.11.0",
+    "codecov": "^3.7.2",
+    "cross-env": "^7.0.2",
+    "eslint": "^7.5.0",
     "eslint-config-xo-lass": "^1.0.3",
-    "eslint-plugin-compat": "^3.5.1",
-    "husky": "^3.1.0",
-    "lint-staged": "^10.1.4",
-    "nyc": "^15.0.1",
-    "remark-cli": "^8.0.0",
-    "remark-preset-github": "^2.0.0",
+    "eslint-plugin-compat": "^3.8.0",
+    "eslint-plugin-no-smart-quotes": "^1.1.0",
+    "husky": "^4.2.5",
+    "lint-staged": "^10.2.11",
+    "nyc": "^15.1.0",
+    "remark-cli": "^8.0.1",
+    "remark-preset-github": "^2.0.2",
     "supertest": "^4.0.2",
-    "xo": "^0.25.3"
+    "xo": "^0.32.1"
   },
   "engines": {
     "node": ">=10"

sao.js

@@ -23,7 +23,7 @@ module.exports = {
     name: {
       message: 'What is the name of the new project',
       default: ':folderName:',
-      validate: value => {
+      validate: (value) => {
         if (process.env.NODE_ENV === 'test' && value === 'lad') return true;
         return isValidNpmName(value);
       }
@@ -48,19 +48,19 @@ module.exports = {
       message: 'What is your email (the author’s)',
       default: conf.get('init-author-email') || ':gitEmail:',
       store: true,
-      validate: value => (isEmail(value) ? true : 'Invalid email')
+      validate: (value) => (isEmail(value) ? true : 'Invalid email')
     },
     website: {
       message: 'What is your personal website (the author’s)',
       default: conf.get('init-author-url') || '',
       store: true,
-      validate: value => (value === '' || isURL(value) ? true : 'Invalid URL')
+      validate: (value) => (value === '' || isURL(value) ? true : 'Invalid URL')
     },
     username: {
       message: 'What is your GitHub username or organization',
       default: ':gitUser:',
       store: true,
-      validate: value =>
+      validate: (value) =>
         githubUsernameRegex.test(value) ? true : 'Invalid GitHub username'
     },
     repo: {
@@ -70,7 +70,7 @@ module.exports = {
           slug(answers.name)
         )}`;
       },
-      validate: value =>
+      validate: (value) =>
         isURL(value) &&
         value.indexOf('https://github.com/') === 0 &&
         value.lastIndexOf('/') !== value.length - 1
@@ -87,8 +87,8 @@ module.exports = {
       type: 'confirm',
       default: true
     },
-    bull: {
-      message: 'Do you need a job scheduler (@ladjs/bull)',
+    bree: {
+      message: 'Do you need a job scheduler (bree)',
       type: 'confirm',
       default: true
     },
@@ -101,7 +101,7 @@ module.exports = {
       message: 'Do you need automatic multi-lingual support',
       type: 'confirm',
       default: true,
-      when: answers => answers.web || answers.api
+      when: (answers) => answers.web || answers.api
     }
   },
   filters: {
@@ -120,9 +120,9 @@ module.exports = {
 
     'web.js': 'web === true',
     'api.js': 'api === true',
-    'bull.js': 'bull === true',
+    'bree.js': 'bree === true',
     'proxy.js': 'proxy === true',
-    'jobs/**': 'bull === true',
+    'jobs/**': 'bree === true',
 
     'test/config/snapshots/': false,
     'test/config/snapshots/**': false,
@@ -137,10 +137,10 @@ module.exports = {
     README: 'README.md',
     env: '.env'
   },
-  post: async ctx => {
+  post: async (ctx) => {
     ctx.gitInit();
 
-    // TODO: ctx.answers.bull
+    // TODO: ctx.answers.bree
     // - remove from pkg
     // - remove config
     // - remove tests