Merge pull request #25 from OasisLMF/add/transformdocs

Add documentation on the odtf tool

Author: benhayes21

requirements.txt

@@ -20,3 +20,4 @@ sphinx-jsonschema==1.19.1
 requests
 ods-tools
 sphinxcontrib-redoc
+lxml_html_clean

src/index.rst

@@ -109,6 +109,7 @@ It allows regression of the models after updates to the Oasis Platform code, and
     sections/OasisLMF-package.rst
     sections/ODS-tools.rst
     sections/ODS.rst
+    sections/ODTF.rst
     sections/OED.rst
     sections/ORD.rst
     sections/platform_1

src/sections/ODS-tools.rst

@@ -113,12 +113,25 @@ Example :
     ods_tools convert --location path_to_location_file --path output folder
 |
 
+ODS Tools can also be used to transform location and account data from other formats to OED and vice versa,
+depenting on the mapping files provided. For more information on this, see here: :doc:`../sections/ODTF`.
+
+Example :
+
+|
+.. code-block:: python 
+
+    ods_tools transform --config-file configuration.yaml
+|
+
+
 .. note::
-    See ``ods_tools convert --help`` for more options.
+    See ``ods_tools convert --help`` and ``ods_tools transform --help`` for more options.
 |
 
 
 
+
 .. _links_ODS_Tools:
 
 Links for further information

src/sections/ODTF.rst

@@ -0,0 +1,138 @@
+ODTF
+=========
+Data transformation tool in ODS-tools
+
+On this page:
+-------------
+
+* :ref:`intro_ODTF`
+* :ref:`inputs`
+* :ref:`use`
+* :ref:`validation`
+
+----
+
+.. _intro_ODTF:
+
+Introduction
+************
+
+
+The `transform` command can be used to convert data from one format to another (e.g., from the AIR Cede format to OED). It will convert from a specific version of the source format to a specific version of the target format, for example from AIR Cede 10.0.0 to OED 3.0.2.
+
+As of :doc:`../sections/ODS-tools` 3.2.3, we support conversions between AIR Cede and OED formats.
+
+
+----
+
+.. _inputs:
+
+Inputs
+************
+
+
+
+For a transformation to run, the necessary input files should be present.
+The input files you will need to run a transformation are:
+
+
+
+**Configuration file**
+
+See, e.g., this `example configuration file <https://github.com/OasisLMF/ODS_Tools/blob/main/ods_tools/odtf/examples/example_config.yaml>`_:
+
+|
+.. code-block:: yaml
+
+    transformations:
+      loc: # Transformation name
+        input_format:
+          name: Cede_Location
+          version: 10.0.0
+        output_format:
+          name: OED_Location
+          version: 3.0.2
+        runner:
+          batch_size: 150000 # Number of rows to process in a single batch
+        extractor:
+          options:
+            path: ./cede_location_1000.csv # Path to the input file
+            quoting: minimal
+        loader:
+          options:
+            path: ./oed_location_1000.csv # Path to the output file
+            quoting: minimal
+|
+ 
+The configuration file contains a list of transformations to run (currently loc for location and acc for account data).
+Each transformation includes name and version of the input and output formats, the (optional) batch size, and the paths to the input (extractor) and output (loader) files.
+
+
+
+
+**Input data**
+
+The input data should be in the format that you want to transform from. For example, if you want to transform data from AIR Cede to OED, the input data should be in the AIR Cede format.
+File types supported:
+.csv
+
+
+
+
+**Mapping file**
+
+A mapping file is a file in .yaml format that describes how to run a conversion between the source and target formats and vice versa.
+Multiple mapping files can be used together to define a mapping between a source and destination format that do not appear in the same mapping file. I.e., A mapping file for model A to B and and a mapping file for Model B to C, can be used to transform data directly from A to C.
+
+Transformations can copy one field into another, substitute field values using a replace function, or include conditional transformation using a where clause. For columns that can contain multiple values (the LocPerils column in AIR Cede which could contain, for example "CF, CH, EQ"), the replace_multiple allows to input a separator used in the cell to split the values.
+Only the transformations involving columns present in the input file will be run.
+
+
+For example, see the `Cede-OED mapping file <https://github.com/OasisLMF/ODS_Tools/blob/main/ods_tools/odtf/data/mappings/mapping_loc_Cede-OED.yaml>`_
+
+
+
+----
+
+.. _use:
+
+Usage
+************
+
+
+Command line usage..
+
+    ods_tools transform [-h] --config-file CONFIG_FILE [-v LOGGING_LEVEL] [--nocheck NOCHECK]
+
+Transform data format to/from OED.
+
+options:
+  -h, --help            show this help message and exit
+
+  --config-file CONFIG_FILE
+                        Path to the config file
+
+  -v LOGGING_LEVEL, --logging-level LOGGING_LEVEL
+                        logging level (debug:10, info:20, warning:30, error:40, critical:50)
+
+  --nocheck NOCHECK     if True, OED file will not be checked after transformation
+
+
+----
+
+.. _validation:
+
+Validation
+************
+
+
+Validation is performed after a conversion to make sure that the output file is valid with respect to specific rules.
+A data validation file contains comparisons of various metrics in both the input file and output file.
+For example, the sum of Total Insured Value grouped by Occupancy Type and Currency. The fields and operations are defined by the user in the validation definition files.
+The validation definition file is in .yaml format.
+See, for example, this `example validation file <https://github.com/OasisLMF/ODS_Tools/blob/main/ods_tools/odtf/data/validators/validation_OED_Location_loc.yaml>`_.
+
+
+.. note::
+  The ODTF and the transform command are adapted from the `Open Data Transformation Framework <https://oasislmf.github.io/OpenDataTransform/>`_.
+|