support for mongodb host config

Author: jimmyjayp

.gitignore

@@ -1 +1,3 @@
 .DS_Store
+.sandbox-config.sh
+log/

.prettierrc

@@ -0,0 +1,6 @@
+{
+	"printWidth": 100,
+	"singleQuote": true,
+	"trailingComma": "es5",
+	"useTabs": true
+}

README.md

@@ -1,71 +1,100 @@
 # CodeStream Backend Services
 
-On the backend, CodeStream requires several services to provide all the
-functionality needed for the clients. Different configurations are supported
-most natably the use of the broadcaster and rabbitMQ for On-Prem vs. PubNub and
-AWS SQS for CodeStream Cloud. This repo contains all the code for these
-services.
+On the backend (aka. the server-side), CodeStream runs a number of services to
+provide all the functionality needed for the clients. The default development
+environment will use the codestream broadcaster and rabbitMQ. Mail services
+are disabled by default.
+## Development Setup
 
-## Installation & Setup Using dev_tools
-
-Installation for development on the CodeStream network.
+_Note: CodeStream employees may prefer to use the dev_tools sandbox as it will
+provide most of the ancillary resources you'll need. [Details
+here](docs/codestream-sandbox-setup.md)._
 
 ### Prerequisites
-1. Install the dev_tools tookkit
-   [here](https://github.com/teamcodestream/dev_tools).
-1. Install mongo - this is available as a dev_tools sandbox or you can install
-   any version. Instructions for the mongo sandbox are
-   [here](https://github.com/teamcodestream/mongodb_tools).
-1. Make sure you can access the CodeStream network via the VPN.
-1. Review how we manage our [server configurations](api_server/README.unified-cfg-file.md).
-   If you have any custom alterations to the standard configuration, you will
-   need to be familiar with the procedures in this document.
+
+1. Mac or Linux computer using zsh or bash.
+
+1. Official CodeStream builds (CI) use Nodejs 12.14.1 with npm 6.13.4
+
+1. [Docker Desktop](https://www.docker.com/products/docker-desktop) which we'll
+   use to provide MongoDB and a pre-configured RabbitMQ.
+
+If you do not wish to use docker, you'll need to provide both of these services:
+
+1. MongoDB 3.4.9 with `mongodb://localhost/codestream` providing full access to
+   create collections and indexes in the `codestream` database. If you're
+   willing to run docker, the instructions below will show you how to install a
+   MongoDB docker container.
+
+1. RabbitMQ 3.7.x with the delayed message exchange plugin. You'll also need to
+   create a codestream user with access. [Notes here](api_server/docs/rabbitmq.md).
 
 ### Installation
-1. If you're using dev_tools on your own computer, bring it up to date
-   (`dt-selfupdate -y`). You don't need to do this if you're using a managed EC2
-   instance.
-1. Update your secrets (`dt-update-secrets -y`).
-1. Select a codestream configuration to use (details documented
-   [here](README.unified-cfg-file.md)). To get up and running quickly, this
-   command will select out-of-the-box 'codestream-cloud' as your configuration.
-	```
-	$ echo codestream-cloud > ~/.codestream/config/codestream-cfg-default.local
-	```
-1. Open a new terminal window
-1. Load your dev_tools mongo sandbox into your shell if you're using one and
-   start the mongo service.
-	```
-	$ dt-load mongo
-	$ mdb-service start
-	```
-   If using your own mongo installation, make sure it's running and accessible
-   without credentials on **localhost** (the default mongo connect url assumes
-   `mongodb://localhost/codestream`).
-1. Install the codestream-server repo (only specify `-I` if you are *not* using
-   or have not loaded a dev_tools mongo sandbox). Select a name for your backend
-   sandbox (we'll use `csbe`):
-	```
-	dt-sb-new-sandbox -yCD -t cs_server -n csbe
-	```
-1. Load your codestream backend sandbox:
-	```
-	$ dt-load csbe
-	```
-1. Create a playground for setting up future terminals with your mongo + csbe
-   sandboxes. This will create a playground with a default name of `csbe` (not
-   to be confused with the **csbe** sandbox).
-	```
-	$ dt-sb-create-playground -t $CSBE_TOP/sandbox/playgrounds/default.template
-	```
-
-You are ready to go.  From this point forward use the following command to setup
-new shells for codestream backend development:
-```
-$ dt-load-playground csbe
-```
-
-## Installation & Setup without dev_tools
-
-To be written
 
+1. Fork the
+   [codestream-server](https://github.com/teamcodestream/codestream-server) repo
+   and clone it.
+
+1. Setup your shell's environment
+   ```
+   cd codestream-server
+   source dev-env.sh     # custom settings go in .sandbox-config.sh
+   ```
+
+1. Install all the node modules
+   ```
+   npm run install:all
+   ```
+
+1. Install the rabbitmq docker container pre-configured for codestream (the
+   container name will be csrabbitmq)
+   ```
+   npm run run:docker:csrabbitmq
+   ```
+
+1. Create a docker volume for mongo and launch the mongodb docker container.
+   The docker volume will ensure the data persists beyond the lifespan of the
+   container.
+   ```
+   npm run run:docker:csmongo
+   ```
+
+1. In a separate shell, source in the `dev-env.sh` environment and start up the
+   api service. It will repeatedly try to connect to the broadcaster. That's ok.
+   Move on once you've started it.
+   ```
+   source dev-env.sh
+   npm run start:api
+   ```
+
+1. In a another separate shell, source in the `dev-env.sh` environment and start
+   up the broadcaster service.
+   ```
+   source dev-env.sh
+   npm run start:broadcaster
+   ```
+
+1. If you want to use or work on the onprem admin UI, that will need two more
+   shells. This first one will run webpack and rebuild the bundle file as your
+   files change.
+   ```
+   source dev-env.sh
+   npm run start:opadm
+   ```
+
+1. The inbound email service is disabled in the default config.
+   ```
+   source dev-env.sh
+   npm run start:mailin
+   ```
+
+1. The outbound email service is also disabled in the default config.
+   ```
+   source dev-env.sh
+   npm run start:mailout
+   ```
+
+Point your CodeStream extension to http://localhost:12000. You should be able to
+register and create codemarks. The onprem admin console is at http://localhost:12002
+
+Develop to your heart's content!!!!  We _love_ pull-requests.

api_server/bin/api_server.js

@@ -6,6 +6,8 @@
 
 'use strict';
 
+const UUID = require('uuid').v4;
+
 // load configurations
 const ApiConfig = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/config/config');
 const ModuleDirectory = process.env.CSSVC_BACKEND_ROOT + '/api_server/modules';
@@ -14,6 +16,7 @@ const ClusterWrapper = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_
 const StringifySortReplacer = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/stringify_sort_replacer');
 const ServerClass = require(process.env.CSSVC_BACKEND_ROOT + '/api_server/lib/api_server/api_server');
 const getOnPremSupportData = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/get_onprem_support_data');
+const customSchemaMigrationMatrix = require(process.env.CSSVC_BACKEND_ROOT + '/shared/server_utils/custom_schema_migration');
 
 // establish our data collections
 const DataCollections = {
@@ -49,15 +52,30 @@ const MongoCollections = Object.keys(DataCollections).concat([
 ]);
 
 (async function() {
+	if (ApiConfig.configIsMongo()) {
+		// set option so structured config will perform a schema version migration if needed upon initial load
+		ApiConfig.performMigrationUsing(customSchemaMigrationMatrix);
+		// set option to create a new config using the supplied default in the event no configs exist (empty database)
+		ApiConfig.loadDefaultConfigIfNoneFrom(
+			process.env.CS_API_DEFAULT_CFG_FILE ||
+				process.env.CSSVC_BACKEND_ROOT + '/api_server/etc/configs/default.json',
+			// generate installation id for first time start-up
+			(cfg) => {
+				if (!cfg.sharedGeneral.installationId) cfg.sharedGeneral.installationId = UUID();
+			}
+		);
+	}
+
 	// changes to Config will be available globally via the /config/writeable.js module
-	const Config = await ApiConfig.loadPreferredConfig();
+	const Config = await ApiConfig.loadPreferredConfig({ wait: true });
 
 	// establish our logger
 	const Logger = new SimpleFileLogger(Config.apiServer.logger);
+	ApiConfig.logger = Logger;
 
 	// onprem support data (service versions, docker registry info, on-prem version)
 	let onPremSupportData;
-	if (!Config.adminServer.adminServerDisabled) {
+	if (Config.isOnPrem) {
 		onPremSupportData = await getOnPremSupportData(Logger);
 		console.info('OnPrem Config:', JSON.stringify(onPremSupportData, StringifySortReplacer, 8));
 	}

api_server/bin/ensure-indexes.js

@@ -98,8 +98,10 @@ function WaitUntilFinished() {
 (async function() {
 	let mongoClient, db;
 	try {
-		const config = await ApiConfig.loadPreferredConfig();
-		mongoClient = await MongoClient.connect(config.storage.mongo.url, { useNewUrlParser: true });
+		const mongoUrl = ApiConfig.configIsMongo()
+			? ApiConfig.options.mongoUrl
+			: await ApiConfig.loadPreferredConfig().storage.mongo.url;
+		mongoClient = await MongoClient.connect(mongoUrl, { useNewUrlParser: true });
 		db = mongoClient.db();
 	}
 	catch (error) {

api_server/config/config.js

@@ -18,10 +18,14 @@ const customConfigFunc = require(process.env.CSSVC_BACKEND_ROOT + '/shared/serve
 // function customRestartFunc(priorConfig, currentConfig) {
 // }
 
-module.exports = StructuredConfigFactory.create({
+var Config;
+
+Config = Config || StructuredConfigFactory.create({
 	configFile: process.env.CS_API_CFG_FILE || process.env.CSSVC_CFG_FILE,
 	mongoUrl: process.env.CSSVC_CFG_URL,
 	showConfigProperty: 'apiServer.showConfig',
 	// customRestartFunc,
 	customConfigFunc
 });
+
+module.exports = Config;