Oasis Release v2.4.0

Docker Images (Platform)

Docker Images (User Interface)

Components

Changelogs

OasisPlatform Changelog - 2.4.0

#1097 - Extend sub-tasks to track and log any retries attempts
#1158 - Worker controller detects pending V1 task and tries to scale V2 workers
#1160 - Fix/testing oed4 update
#1123 - Generate oasis files is not using the pre-analysis adjusted location file for V2 runs
#1162 - Fix/v2 location loading
#1163 - Workaround for ods tools issue #174
#1035 - Remove debug mounts and refs in default compose file
#1047 - Tidy up docker compose examples
#1052 - Lot3 - worker monitor compatibility fix needed
#1070 - Use version 2 of the API on api ls commands
#904 - Lot3 - Load model data from object storage
#1089 - Fixed CVEs from fiona package
#1094 - Updated Package Requirements: twisted==24.7.0rc1
#1101 - Only run attach project issues workflow from Our repo - not forks
#1128 - Update python 3.12
#1142 - Project PR action - incorrectly running on external PRs
#689 - Add mechanism to list, and return individual files from tar

OasisLMF Changelog - 2.4.0

#1394 - Net RI losses do not use -z in summarycalc
#1607 - Fix/lookup sort keys
#1591 - Dynamic footprint has incorrect type 1 losses
#1611 - fix loss_out len in account level back allocation
#1615 - Non-useful error message for missing PolInceptionDate with RA Basis reinsurance
#1552 - Make analysis and model setting able to modify any computation step parameters (MDK parameters)
#1618 - Fix testing failures (API Client)
#1619 - Fix/ci pre analysis testing
#1624 - Feature/oed v4
#1625 - feat: security enhancements

ODS_Tools Changelog - 4.0.0

#146 - OED Settings merger
#167 - Fix default for do_disaggregation in schema
#170 - Update OED schema to release v4
#172 - CI Fix - branches renamed in OED repo
#173 - Feature/v4 integration

Release Notes

OasisPlatform Notes

Update subtasks when a retry is attempted - (PR #1155)

Added two new fields to analyses sub-task object

retry_count number of failed retry attempts
retry_log error trace for each failed retry attempt (appended to on each retry)

Worker controller detects pending V1 task and tries to scale V2 workers - (PR #1158)

Fixed issue where a analysis run requests is on a V1 queue will be viewed as a waiting V2 analyses.
This causes the auto-scaling to spin workers up and down.

Fixed platform Unit tests for OED4 update - (PR #1160)

Fix pre analysis hook loading - (PR #1161)

The main branch uses the Oasis-Data-Manager to fetch and store files. There a bug in ffspec that dosn't create local dirs when mkdir is called. Added a work around to make sure the local storage dir exists.

Update the location file loading in platform V2 runs - (PR #1162)

Needed to support the changes in OasisLMF/OasisLMF#1624 ~ Note cyber models will probably not with V2 runs, but should be fine with V1

Workaround for ods tools validation issue #174 - (PR #1163)

See: OasisLMF/ODS_Tools#174

Docker compose, removed debugging options from default - (PR #1036)

The default file docker-compose.yml has all debugging settings removed, instead these are now in compose/debug.docker-compose.yml

Tidy up docker compose examples - (PR #1047)

Removed all outdated examples from compose/

Lot3 work - (PR #958)

Added SQL endpoints for exposure data queries (Currently disabled)
Replaced Remote file storage manager with OasisDataManager package
Added the option to read model_data from a remote object store (S3, Azure BlobStorage)

There are two sets of 'remote storage' configuration options.

General Storage, for files like results and exposure in an ObjectStore - Env var prefix = OASIS_{option}
Model Data Storage, for reading 'model_data' during an analysis execution - Env var prefix = OASIS_WORKER_MODEL_DATA_{option}

For Valid options see src/common/filestore/filestore.py

https://github.com/OasisLMF/OasisPlatform/blob/cc6907ada4b9d67a6f02372572fb81f067f04f21/src/common/filestore/filestore.py#L21-L59

https://github.com/OasisLMF/OasisPlatform/blob/cc6907ada4b9d67a6f02372572fb81f067f04f21/src/common/filestore/filestore.py#L63-L83

Example

OASIS_WORKER_MODEL_DATA_STORAGE_TYPE: S3
OASIS_WORKER_MODEL_DATA_AWS_BUCKET_NAME: oasislmf-model-library-oasis-piwind
OASIS_WORKER_MODEL_DATA_AWS_ACCESS_KEY_ID:  {_ID_}
OASIS_WORKER_MODEL_DATA_AWS_SECRET_ACCESS_KEY:  {_KEY_}
OASIS_WORKER_MODEL_DATA_ROOT_DIR: model_data/

When set on a worker container, these are used to create a model_storage.json file which is then passed into the execution command so model data is pulled directly from OASIS_WORKER_MODEL_DATA_AWS_BUCKET_NAME

Fixed CVEs from fiona package - (PR #1089)

Updated Package Requirements: fiona==1.10b1 for CVE-2023-45853 & CVE-2020-14152

Updated Package Requirements: twisted==24.7.0rc1 - (PR #1094)

Only run attach project issues workflow from Our repo - not forks - (PR #1101)

CI fix for external PRs

Update main platform branch to latest Oasislmf package - (PR #1102)

Update docker base images and python packages - (PR #1128)

Update docker base images to ubuntu LTS 24.04
regenerate python package requirements files
Load CVE scanning DB from Github actions cache (Trivy)

Project PR action - incorrectly running on external PRs - (PR #1142)

External PR fail on the actions Job 'Project PR' - disable this action if the PR is from a Fork

Add endpoints to list and return individual files from tar - (PR #1151)

The following endpoints were added:

/analyses/{id}/{tar_file}_tar_list/: List the contents of {tar_file}
/analyses/{id}/{tar_file}_tar_extract/?filename={filename}: Extract {filename} from {tar_file}

These endpoints can be used on the following tar_files: input_files, output_files.
Issue: #689

OasisLMF Notes

Calculate summarycalc reinsurance without all zeros flag - (PR #1601)

Running summarycalc with the -z flag for Net RI was running into significant performance issues.

To resolve this, this PR:

removes the output all zeros flag when running summarycalc or summarypy
adds an additional check on the the mean_idx to catch non-zeros that were previously being filtered out

Issue: #1394

Fix lookup `sort_values` of keys - (PR #1607)

The ordering of pandas df.sort_values was inconsistent during testing. Passing the kind='stable' argument resolves this issue.

Fix fmpy back allocation for account level term - (PR #1611)

Trim loss_out to the correct length before account level term back allocation

fixes error message for missing PolInceptionDate - (PR #1614)

For RA based reinsurance introduced in 2.3.11 OasisLMF/OasisLMF#1576
Solution is to look for both non '_x' and '_x' variants of these column numbers in the row if they exist and output them.

acc_info = {
    field: row[f'{field}_x'] if f'{field}_x' in row else row[f'{field}']
    for field in RISK_LEVEL_FIELD_MAP[oed.REINS_RISK_LEVEL_ACCOUNT]
    if f'{field}_x' in row or f'{field}' in row
}

Should output the correct error message as shown in the example

Error: PolInceptionDate missing for {'PortNumber': '1', 'AccNumber': 'A11111'}, cannot use AttachmentBasis [RA]. Please check the account file

Make analysis and model setting able to modify any computation step parameters (MDK parameters) - (PR #1552)

To be accessible to the user and modeler via analysis and model settings, MDK parameters use to need a specific treatment in the code to allow the overwrite.
This PR makes it automatic by using the ods_tools settings object that help merge the different settings and parameters in a coherent way.

The package schema can be created on CLI using: oasislmf model generate-computation-settings-json-schema

Fix testing failures (API Client) - (PR #1618)

Objected returned from a mocked API post has changed between versions of the responses
Pinning to responses<=0.25.3 for now, but test should be updated later

Fix/ci pre analysis testing - (PR #1619)

Fix CI testing with ODS-tools main (using OEDv4)

Integrate new OED v4 coverages and perils in OasisLMF - (PR #1624)

ODS_Tools Notes

Add capacity to merge multiple settings files. - (PR #146)

OED settings can contain several layer of settings coming from different sources (user, modeler, platform host)
This feature allow to merge those setting into one comprehensive settings file.
supported feature:
default : setting can specify a default value that can be overwritten by other setting
restricted: a key can be restricted to disallow other source to overwrite it
min, max: check that value is within min and max boundaries
option: to limit the possible value to only a specific list

Fix default for do_disaggregation to true - (PR #169)

Match schema default to the oasislmf package

Update OED schema to release v4 - (PR #170)

The validation now uses the v4.0.0 version of OED.
For more details on OED v4 see the release notes

Release 2.4.0