Skip to content

1.6.0
Compare
Choose a tag to compare
@awsbuild awsbuild released this 06 Mar 12:39
1.6.0
04982c8

Docker Images

Components

Fixes & Changes

  • #284 - Added support for S3 Object stores
  • #305 - Define groups for models
  • #306 - Fixed logical deletion for AnalysisModels
  • #311 - Improved error logging in worker monitor
  • #313 - Added option to set run dir location in worker
  • #317 - Added Backwards compatibility tests to CI
  • #321 - Updated Django to 3.0.3
  • #323 - Added 'server_info' endpoint for details on the running API
  • #325 - Fixed Database migration issue from version 1.1.2
  • #335 - Accept compressed CSV files as portfolio uploads

New Features

Object file storage

Added support for AWS S3 object storage in both the server and worker containers.

Its enabled by setting the environment variable STORAGE_TYPE=S3 in either the conf.ini file or under the docker-compose environment section.
The default value is STORAGE_TYPE=stored-fs which is the previous shared docker volume between containers. Details in Pull Request #314

Configuration options

Conf.ini Example

Required

  • AWS_ACCESS_KEY_ID Your Amazon Web Services access key, as a string.
  • AWS_SECRET_ACCESS_KEY Your Amazon Web Services secret access key, as a string.
  • AWS_BUCKET_NAME S3 bucket name, as a string.

Optional

  • AWS_LOCATION A path prefix that will be prepended to all uploads
  • AWS_QUERYSTRING_EXPIRE The number of seconds that a generated URL is valid for.
  • AWS_QUERYSTRING_AUTH Setting AWS_QUERYSTRING_AUTH to False to remove query parameter authentication from generated URLs. This can be useful if your S3 buckets are public.
  • AWS_S3_ENDPOINT_URL Override the default AWS service address URL for S3, e.g. AWS_S3_ENDPOINT_URL=http://localhost:4572
  • AWS_S3_REGION_NAME Name of the AWS S3 region to use (eg. eu-west-1)

Local S3 testing

The object store can be tested locally using LocalStack.
Run this by using the docker-compose file s3.docker-compose.yml

export OASIS_MODEL_DATA_DIR=<..path to piwind ..>
docker-compose -f s3.docker-compose.yml up -d

Server version information

Added a new endpoint http://<server-ip>:<server-port>/server_info/ which returns the version tag and non-sensitive configuration information

Example response (Requires authenticated)

{
  "version": "1.6.0",
  "config": {
    "DEBUG": true,
    "LANGUAGE_CODE": "en-us",
    "TIME_ZONE": "UTC",
    "WSGI_APPLICATION": "src.server.oasisapi.wsgi.application",
    "DEFAULT_FILE_STORAGE": "django.core.files.storage.FileSystemStorage",
    "DB_ENGINE": "django.db.backends.sqlite3",
    "STORAGE_TYPE": "shared-fs",
    "MEDIA_ROOT": "/shared-fs/",
    "AWS_STORAGE_BUCKET_NAME": "<server-bucket-name>",
    "AWS_LOCATION": "server",
    "ROTATE_REFRESH_TOKEN": true,
    "ACCESS_TOKEN_LIFETIME": "3600.0",
    "REFRESH_TOKEN_LIFETIME": "172800.0"
  }
}

Grouping Fields

A model can now set a list of OED fields that it uses to define correlation groups when running analyses, these are set as a property of the model.
This is done through the group_fields key in model_settings.json under data_settings.
When generating oasis files or running the model end to end, the argument -M or --model-settings-json should point to the model_settings.json file to use this. For example:

$ oasislmf model generate-oasis-files --model-settings-json meta-data/model_settings.json -C oasislmf.json

Command line arguments for group fields take priority, followed by those defined by the model. If neither exist, loc_id is used as the fall-through option.

model_settings.json example

{
    "model_settings":{
        ...
    },    
    "data_settings": {
        "group_fields": ["PortNumber", "AccNumber", "LocNumber"]
    }
}

Tooltip field for model settings

Added an optional longer descrtioption field to the model settings schema.
OasisLMF/OasisLMF#523

Example:

      "boolean_parameters":[
         {
            "name":"peril_wind",
            "desc":" ..short label.."
            "tooltip": "  ...  Longer descrtiption for this parameter ..."
            "default":false
         },
        ...
      ]

Setting a worker's run directory

Environment variable BASE_RUN_DIR which is used to override the base run directory path.

Example:

 worker:
    ...
    environment:
     - OASIS_BASE_RUN_DIR=/home/worker/run
    ...

The above ill move the temporary run dirs from /tmp/<random_name> to /home/worker/run/<random_name>

Backwards compatibility checks

Added tests which run the current branch (to be published) images against the last release.

  • Worker (prev release) + Server (next release)
  • Server (prev release) + Worker (next release)
Assets 3
Loading