Skip to content

Commit

Permalink
[ENH] Update federation instructions to use JSON local nodes config f…
Browse files Browse the repository at this point in the history
…ile (#143)

* add info on using JSON local nodes config file

* update text formatting
  • Loading branch information
alyssadai authored Dec 4, 2023
1 parent c9edf14 commit d39e7cf
Showing 1 changed file with 46 additions and 26 deletions.
72 changes: 46 additions & 26 deletions docs/federate.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Query federation is not necessary, if you:
## Setting up for local federation
Federated graph queries in neurobagel are provided by the federation API (`f-API`) service.
The neurobagel `f-API` takes a single user query and then sends it to every
neurobagel node API (`n-API`) it is aware of, collects and combinesthe responses,
neurobagel node API (`n-API`) it is aware of, collects and combines the responses,
and sends them back to the user as a single answer.

!!! note
Expand All @@ -40,22 +40,54 @@ and sends them back to the user as a single answer.
before you set up local federation. If you do not have any local
`n-APIs` to federate over, you can just use our public query tool directly at [query.neurobagel.org](https://query.neurobagel.org/).

In your command line, create and navigate to a new directory where you will keep the configuration
files for your new `f-API`. In this directory, create two files:
In your command line, create and navigate to a new directory where you will keep the configuration files for your new `f-API`.
In this directory, create three files:

### `fed.env` environment file

Create a text file called `fed.env` to hold environment variables needed for the `f-API` deployment.
### `local_nb_nodes.json` configuration file
Create a JSON file called `local_nb_nodes.json` which will contain the URLs and (arbitrary) names of the local nodes you wish to federate over.
Each node must be denoted by a dictionary `{}` with two key-value pairs:
`"NodeName"` for the name of the node,
`"ApiURL"` for the URL of the API exposed for that node.
Multiple nodes must be wrapped in a list `[]`.

Let's assume there are two local nodes already running on different servers of your institutional network, and you want to set up federation across both nodes:

- a node named `"node_archive"` running on your local computer on port `8000` and
- a node named `"node_recruitment"` running on a different computer with the local IP `192.168.0.1`, listening on the default http port `80`.
In your `fed.env` file you would configure this as follows:

``` {.bash .annotate title="docker-compose.yml"}
In your `local_nb_nodes.json` file you would configure this as follows:
``` {.json title="local_nb_nodes.json"}
[
{
"NodeName": "node_archive",
"ApiURL": "http://localhost:8000",
},
{
"NodeName": "node_recruitment",
"ApiURL": "http://192.168.0.1"
}
]
```

!!! Info "Nodes that do not need to be manually configured"
We maintain a list of public Neurobagel nodes
[here](https://github.com/neurobagel/menu/blob/main/node_directory/neurobagel_public_nodes.json).
By default every new `f-API` will lookup this list
on startup and include it in the list of nodes to
federate over.
This also means that you do not have to manually
configure public nodes, i.e. you **do not have to explicitly add them** to your `local_nb_nodes.json` file.

To add one or more local nodes to the list of nodes known to your `f-API`, simply add more dictionaries to this file.


### `fed.env` environment file

Create a text file called `fed.env` to hold environment variables needed for the `f-API` deployment.

``` {.bash .annotate title="fed.env"}
# Configuration for f-API
# List of known local node APIs: (node_URL, node_NAME)
LOCAL_NB_NODES=(http://localhost:8000, node_archive) (http://192.168.0.1, node_recruitment)
# Define the port that the f-API will run on INSIDE the docker container (default 8000)
NB_API_PORT=8000
# Define the port that the f-API will be exposed on to the host computer (and likely the outside network)
Expand All @@ -80,10 +112,7 @@ NB_QUERY_PORT_HOST=3000
as it will appear to a user on their own machine
- otherwise the request will fail..

Each node to be federated over is described in the variable `LOCAL_NB_NODES` by a comma-delimited tuple of the form `(node_URL, node_NAME)`.

You can add one or more local nodes to the list of nodes known to your `f-API` in this way.
Just adjust the above code snippet according to your own deployment, and store it in a file called `fed.env`.
Copy and adjust the above code snippet according to your own deployment, and store it in `fed.env`.


### `docker-compose.yml` docker config file
Expand All @@ -100,7 +129,7 @@ to launch the `f-API` together with a connected query tool.

Copy the following snippet into your `docker-compose.yml` file.
You should not have to change anything about this file.
All local configuration changes are done in the `fed.env` file.
All local configuration changes should be made in either the `local_nb_nodes.json` or `fed.env` files.

``` {.yaml .annotate title="docker-compose.yml"}
version: "3.8"
Expand All @@ -110,9 +139,9 @@ services:
image: "neurobagel/federation_api:${NB_API_TAG:-latest}"
ports:
- "${NB_API_PORT_HOST:-8000}:${NB_API_PORT:-8000}"
volumes:
- "${PWD}/local_nb_nodes.json:/usr/src/local_nb_nodes.json:ro"
environment:
- LOCAL_NB_NODES=${LOCAL_NB_NODES} # (1)!
- NB_API_PORT=${NB_API_PORT:-8000}
query:
image: "neurobagel/query_tool:${NB_QUERY_TAG:-latest}"
Expand All @@ -122,18 +151,9 @@ services:
- API_QUERY_URL=${API_QUERY_URL:-http://localhost:8000/}
```

1. We maintain a list of public neurobagel nodes
[here](https://github.com/neurobagel/menu/blob/main/node_directory/neurobagel_public_nodes.json).
By default every new `f-API` will lookup this list
on startup and include it in the list of nodes to
federate over.
This also means that you do not have to manually
configure public nodes, i.e. you **do not have to explicitly add them** to the `LOCAL_NB_NODES` variable) in your `fed.env` file.


## Launch f-API and query tool
Once you have created your `fed.env` and `docker-compose.yml` files
as described above, you can simply launch the services by running
Once you have created your `local_nb_nodes.json`, `fed.env`, and `docker-compose.yml` files as described above, you can simply launch the services by running

`docker compose --env-file fed.env up -d`

Expand Down

0 comments on commit d39e7cf

Please sign in to comment.