1.7.1

Docker Images

• coreoasis/model_worker:1.7.1
• coreoasis/api_server:1.7.1
• coreoasis/model_worker:1.7.1-slim
• coreoasis/api_server:1.7.1-slim

Components

• oasislmf 1.7.1
• ktools 3.2.6

Oasis Platform

Fixes

• #359 - Model data files not available in worker

Changes

• New worker option KTOOLS_LEGACY_GUL_STREAM

Important note: setting KTOOLS_ALLOC_RULE_GUL = 0 will no longer change the gulcalc stream type, it now switches the back allocation rule.
To use the older stream type pass KTOOLS_LEGACY_GUL_STREAM = True to a worker image via the conf.ini or environment variables

Models running a custom run module will need to add the extra parameter gul_legacy_stream to match runner.py

Oasislmf MDK

For detailed notes please see the Oasislmf 1.7.1 release

Issues Fixed

• #553 - calc rule mapping error
• #550 - genbash fmcalc and full_correlation
• #548 - GUL Alloc Rule 0
• #437 - Invalid coverage file generated

1.7.0

Docker Images

• coreoasis/model_worker:1.7.0
• coreoasis/api_server:1.7.0
• coreoasis/model_worker:1.7.0-slim
• coreoasis/api_server:1.7.0-slim

Components

• oasislmf 1.7.0
• ktools 3.2.5

Fixes & Changes

• #342 - automate milestone creation
• #346 - Fix schema for server_info endpoint
• #339 - Add collapseable option to parameter grouping (model settings schema)
• #345 - Added sets of configuration parameters (model settings schema)

New Features

Schema update - model settings

• Added a placeholder section for storing multiple sets of parameter values, (Issue #345).

  ... 
 "multi_parameter_options":[ 
        {
            "name": "location_dependent_config_options",
            "desc": "Group description",
            "tooltip": "Detailed overview"
            "config": {  ...JSON DATA... }
        }, 
   ] 

• Added collapsible boolean options to parameter_groups

    ...
      "parameter_groups":[
        {
            "name": "boolean_group",
            "desc": "This is the presentation group for bools",
            "priority_id": 1,
            "presentation_order": ["peril_wind", "peril_surge"],
            "collapsible": true,
            "default_collapsed": false
        },

1.6.1

Docker Images

• coreoasis/model_worker:1.6.1
• coreoasis/api_server:1.6.1
• coreoasis/model_worker:1.6.1-slim
• coreoasis/api_server:1.6.1-slim

Components

• oasislmf 1.6.0
• ktools 3.2.2

Fixes

• #343 - Avoid task failure for missing non-essential files on inputs generation

1.6.0

Docker Images

• coreoasis/model_worker:1.6.0
• coreoasis/api_server:1.6.0
• coreoasis/model_worker:1.6.0-slim
• coreoasis/api_server:1.6.0-slim

Components

• oasislmf 1.6.0
• ktools 3.2.2

Fixes & Changes

• #284 - Added support for S3 Object stores
• #305 - Define groups for models
• #306 - Fixed logical deletion for AnalysisModels
• #311 - Improved error logging in worker monitor
• #313 - Added option to set run dir location in worker
• #317 - Added Backwards compatibility tests to CI
• #321 - Updated Django to 3.0.3
• #323 - Added 'server_info' endpoint for details on the running API
• #325 - Fixed Database migration issue from version 1.1.2
• #335 - Accept compressed CSV files as portfolio uploads

New Features

Object file storage

Added support for AWS S3 object storage in both the server and worker containers.

Its enabled by setting the environment variable STORAGE_TYPE=S3 in either the conf.ini file or under the docker-compose environment section.
The default value is STORAGE_TYPE=stored-fs which is the previous shared docker volume between containers. Details in Pull Request #314

Configuration options

Conf.ini Example

Required

• AWS_ACCESS_KEY_ID Your Amazon Web Services access key, as a string.
• AWS_SECRET_ACCESS_KEY Your Amazon Web Services secret access key, as a string.
• AWS_BUCKET_NAME S3 bucket name, as a string.

Optional

• AWS_LOCATION A path prefix that will be prepended to all uploads
• AWS_QUERYSTRING_EXPIRE The number of seconds that a generated URL is valid for.
• AWS_QUERYSTRING_AUTH Setting AWS_QUERYSTRING_AUTH to False to remove query parameter authentication from generated URLs. This can be useful if your S3 buckets are public.
• AWS_S3_ENDPOINT_URL Override the default AWS service address URL for S3, e.g. AWS_S3_ENDPOINT_URL=http://localhost:4572
• AWS_S3_REGION_NAME Name of the AWS S3 region to use (eg. eu-west-1)

Local S3 testing

The object store can be tested locally using LocalStack.
Run this by using the docker-compose file s3.docker-compose.yml

export OASIS_MODEL_DATA_DIR=<..path to piwind ..>
docker-compose -f s3.docker-compose.yml up -d

Server version information

Added a new endpoint http://<server-ip>:<server-port>/server_info/ which returns the version tag and non-sensitive configuration information

Example response (Requires authenticated)

{
  "version": "1.6.0",
  "config": {
    "DEBUG": true,
    "LANGUAGE_CODE": "en-us",
    "TIME_ZONE": "UTC",
    "WSGI_APPLICATION": "src.server.oasisapi.wsgi.application",
    "DEFAULT_FILE_STORAGE": "django.core.files.storage.FileSystemStorage",
    "DB_ENGINE": "django.db.backends.sqlite3",
    "STORAGE_TYPE": "shared-fs",
    "MEDIA_ROOT": "/shared-fs/",
    "AWS_STORAGE_BUCKET_NAME": "<server-bucket-name>",
    "AWS_LOCATION": "server",
    "ROTATE_REFRESH_TOKEN": true,
    "ACCESS_TOKEN_LIFETIME": "3600.0",
    "REFRESH_TOKEN_LIFETIME": "172800.0"
  }
}

Grouping Fields

A model can now set a list of OED fields that it uses to define correlation groups when running analyses, these are set as a property of the model.
This is done through the group_fields key in model_settings.json under data_settings.
When generating oasis files or running the model end to end, the argument -M or --model-settings-json should point to the model_settings.json file to use this. For example:

$ oasislmf model generate-oasis-files --model-settings-json meta-data/model_settings.json -C oasislmf.json

Command line arguments for group fields take priority, followed by those defined by the model. If neither exist, loc_id is used as the fall-through option.

model_settings.json example

{
    "model_settings":{
        ...
    },    
    "data_settings": {
        "group_fields": ["PortNumber", "AccNumber", "LocNumber"]
    }
}

Tooltip field for model settings

Added an optional longer descrtioption field to the model settings schema.
OasisLMF/OasisLMF#523

Example:

      "boolean_parameters":[
         {
            "name":"peril_wind",
            "desc":" ..short label.."
            "tooltip": "  ...  Longer descrtiption for this parameter ..."
            "default":false
         },
        ...
      ] 

Setting a worker's run directory

Environment variable BASE_RUN_DIR which is used to override the base run directory path.

Example:

 worker:
    ...
    environment:
     - OASIS_BASE_RUN_DIR=/home/worker/run
    ...

The above ill move the temporary run dirs from /tmp/<random_name> to /home/worker/run/<random_name>

Backwards compatibility checks

Added tests which run the current branch (to be published) images against the last release.

• Worker (prev release) + Server (next release)
• Server (prev release) + Worker (next release)

1.5.1

Docker Images

• coreoasis/model_worker:1.5.1
• coreoasis/api_server:1.5.1
• coreoasis/model_worker:1.5.1-slim
• coreoasis/api_server:1.5.1-slim

Components

• oasislmf 1.5.1
• ktools 3.2.2

Hot Fixed

• #309 - worker: Missing error logs from input generation
• #307 - worker: Show subprocess output in worker logs
• Fixed IL file generation issue from oasislmf==1.5.0

1.5.0

Docker Images

coreoasis/model_worker:1.5.0
coreoasis/api_server:1.5.0
coreoasis/model_worker:1.5.0-slim
coreoasis/api_server:1.5.0-slim

Components

oasislmf 1.5.0
ktools 3.2.2

Update to the Model Settings schema

For more details see the pull request notes

Under model_settings

• numeric_parameters, for unbound floats and integers
• dropdown_parameters, same format as 'events' or 'occurrence' but under a generic 'name'
• parameter_groups, for grouping related parameters of different types

New section data_settings

Added for datafile_selectors which is used to connect an uploaded file under /v1/data_files/ to an analyses

{
   "model_settings":{ 
        ...
   },
   "lookup_settings": {
       ...
   }
   "data_settings":{
      "datafile_selectors":[
          { ... }
      ]
   }
}

Changelog

• #304 - Slim image builds (optional)
• #303 - model settings schema Update
• #302 - Add Maven Swagger API build test
• #297 - Remove previous output results on run error
• #297 - Fix Log and traceback storage
• #222 - Update to model settings schema
• #275 - Fixed delete operations with multi-part as content type in swagger
• #274 - Fixed reverting behaviour for complex models (custom gulcalc)
• #281 - Added Task Queued state
• #268 - Store run trace on success
• #287 - Store ktools log directory in an Analyses
• #283 - Fixed Worker not releasing memory from Python process

New Features

1.4.2 (Release Candidate 2)

Docker Images

coreoasis/model_worker:1.4.2-rc2
coreoasis/api_server:1.4.2-rc2

Components

oasislmf 1.4.7rc1
ktools 3.2.1

Changelog

• Hotfix (rc2) - Remove previous output results on run error
• Hotfix (rc2) - Fix Log and traceback storage
• #222 - Update to model settings schema
• #275 - Fixed delete operations with multipart as content type in swagger
• #274 - Fixed reverting behaviour for complex models (custom gulcalc)
• #281 - Added Task Queued state
• #268 - Store run trace on success
• #287 - Store ktools log directory in an Analyses
• #283 - Fixed Worker not releasing memory from Python process

Feature Notes

Accessing Ktools logs

On an analyses run (pass or fail) if a ktools log directory is created the API will tar up <run-dir>/log and store as output under /v1/analyses/{id}/run_log_file/ (GET, DELETE)

Worker Job Status Update

Added two extra analyses.status states INPUTS_GENERATION_QUEUED and RUN_QUEUED

Analyses states overview:

• NEW - Newly created analyses Object, portfolio successfully linked to a model.
• INPUTS_GENERATION_QUEUED - Generate input files job sent to Celery queue.
• INPUTS_GENERATION_STARTED - Generate input Job accepted by a Celery worker.
• INPUTS_GENERATION_CANCELLED - Cancel signal sent, halting input files generation.
• INPUTS_GENERATION_ERROR - Error while processing Input files, see /v1/analyses/{id}/input_generation_traceback_file/ for
• READY - Input files generated, analyses ready for execution.
• RUN_QUEUED - Analyses run sent Celery queue.
• RUN_STARTED - Analyses run accepted by a Celery worker.
• RUN_CANCELLED - Cancel signal sent, halting an Analysis's execution
• RUN_ERROR - An error occurred during execution see /v1/analyses/{id}/run_traceback_file/ for details
• RUN_COMPLETED - Run complete and results available at /v1/analyses/{id}/output_file/

Explicitly setting a custom gulcalc

From Oasislmf==1.4.7rc1 and onwards, custom models can explicitly set the drop-in component name.
The if the script/binary is not found on the current $PATH the MDK will raise an error.

CLI Example

$ oasislmf model run --model-custom-gulcalc MY_GULCALC

    ... 

COMPLETED: oasislmf.model_execution.bin.csv_to_bin in 0.02s
STARTED: oasislmf.model_execution.bin.csv_to_bin
COMPLETED: oasislmf.model_execution.bin.csv_to_bin in 0.03s
STARTED: oasislmf.model_execution.bin.prepare_run_inputs
COMPLETED: oasislmf.model_execution.bin.prepare_run_inputs in 0.0s
STARTED: oasislmf.model_execution.runner.run       
Run error: Custom Gulcalc command "MY_GULCALC" explicitly set but not found in path.

For worker deployment add the following to an oasislmf.json file

{
    ... 
    "model_custom_gulcalc": "<MY_GULCALC>"
    ... 
}

1.4.2 (Release Candidate 1)

-- See Release Candidate 2 --

1.4.1

Docker Images

coreoasis/model_worker:1.4.1
coreoasis/api_server:1.4.1

Components

oasislmf 1.4.6
ktools 3.2.1

Changelog

• Fixes #280 - JSON file schemas compatibility with swagger
• Added models/{id}/versions endpoint
• Improved worker environment variables logging and defaults

Worker Environment variable changes

KTOOLS_BATCH_COUNT

KTOOLS_BATCH_COUNT has changed to KTOOLS_NUM_PROCESSES so it aligns with the MDK flag --ktools-num-processes if not set the worker will now default to the value ktools_num_processes set in oasislmf.conf with a fallback of -1 (if not set anywhere) which means 'Use all available threads'

OASIS_MODEL_SETTINGS_FILE

Workers will auto-update their model_settings file when connecting to celery. If a valid file exists in /var/oasis/meta-data/model_settings.json its posted to the API under models/{id}/settings the file path can be edited by setting OASIS_MODEL_SETTINGS_FILE

Compose file Example:

- OASIS_MODEL_SETTINGS_FILE = /var/oasis/meta-data/<MODEL_NAME>_model_settings.json

New Features

models/{id}/resource_file --> models/{id}/settings

The previous endpoints point for model specific parameters is changing to the following.
These will only accept JSON data validated using the schema model_settings.json.

A validation error on POST will return 400: Bad request response.

For an example of the settings file using the update schema see:
OasisPiWind/meta-data/model_settings.json

analyses/{id}/settings_file --> analyses/{id}/settings

The analysis settings endpoint is also moving from a file upload to JSON data which is validated using
analysis_settings.json.

worker versions

Workers will also post its internal component versions to the API under:

Example response

{
  "ver_ktools": "fmcalc : version: 3.2.1 - git update: Fri Dec 20 10:50:16 2019 +0000t",
  "ver_oasislmf": "1.4.6",
  "ver_platform": "1.4.1 - b53e9a4, Fri Dec 20 12:37:09 UTC 2019"
}

1.4.0

Docker Images

coreoasis/model_worker:1.4.0
coreoasis/api_server:1.4.0

Components

oasislmf 1.4.5
ktools 3.2.0

Changes

• Added /v1/models/{id}/settings to replace /v1/models/{id}/resource_file/
• Added /v1/analyses/{id}/settings to replace /v1/analyses/{id}/settings_file/
• New worker environment variable MODEL_SETTINGS_FILE if set workers auto-update model settings on connection
• Removed environment variable WRITE_EXPOSURE_SUMMARY in favour of DISABLE_EXPOSURE_SUMMARY
• Workers log internal versions to the API and docker logs
• Schemas updated for analysis_settings and model_settings