Ray config files over ibm-gen2 VPC

Author: Cohen-J-Omer

.gitignore

@@ -1,2 +1,3 @@
 *.pyc
 __pycache__
+*.egg-info

README.md

@@ -1,28 +1,28 @@
 # Tool to generate Lithops configuration file
 
-Lithopscloud is a CLI tool that greatly simplifies user experience by generating Lithops and Ray configuration files.
+ibm-ray-config is a CLI tool that greatly simplifies user experience by generating Ray configuration files for IBM services.
 
 ## Setup
 
-The tool been mostly tested with Ubuntu 18.04/20.04, but should work with most Linux systems
+The tool been mostly tested with Ubuntu 18.04/20.04 and Fedora 35, but should work with most Linux systems
 Requirements: `ssh-keygen` utility installed:
 ```
 sudo apt install openssh-client
 ```
 
-Install `lithopscloud` from pip repository
+Install `ibm-ray-config` from pip repository
 
 ```
-pip install lithopscloud
+pip install ibm-ray-config
 ```
 
 ## Usage
 Use the configuration tool as follows
 
 ```
-lithopscloud [--iam-api-key IAM_API_KEY] [-i INPUT_FILE] [-o OUTPUT_PATH] [--verify_config CONFIG_FILE_PATH] [--version] [--backend backend] [--defaults] 
+ibm-ray-config [--iam-api-key IAM_API_KEY] [-i INPUT_FILE] [-o OUTPUT_PATH] [--version] [--defaults] 
 ```
-Get a short description of the available flags via ```lithopscloud --help```
+Get a short description of the available flags via ```ibm-ray-config --help```
 
 <br/>
 
@@ -34,135 +34,40 @@ Get a short description of the available flags via ```lithopscloud --help```
  | iam-api-key   | |yes|IBM Cloud API key. To generate a new API Key, adhere to the following [guide](https://www.ibm.com/docs/en/spectrumvirtualizecl/8.1.3?topic=installing-creating-api-key)
  | input-file    |<compute_backend>/defaults.py| no | Existing config file to be used as a template in the configuration process |
  | output-path   |A randomly generated path to a randomly named yaml file | no |A custom location the config file will be written to |
- | verify-config <img width=125/>| | no |Verifies the integrity of an existing config file and outputs a usable config file based on it. Currently doesn't support gen2 backends. 
- | version       | | no |Returns lithopscloud's package version|
+ | version       | | no |Returns ibm-ray-config's package version|
+
 
-<br/>
 
-#### Verify Additional ways of configuration
-Using the ```verify-config``` option enables verification of additional valid ways of configuration, that otherwise 
-left unchecked. This mode scans for any possible subset of valid parameters and extract them to output a new 
-lithops config file (e.g., verify cos configured by HMAC credentials).
-To utilize simply run ```lithopscloud --verify-config CONFIG_FILE_PATH -o OUTPUT_FILE_PATH```
-<br/> Please note that this feature doesn't currently support the verification of gen2 backends. 
-
-## Supported backends:
-<table>
-<tr>
-<th align="center">
-<p>
-<span style="color:orange">Standalone Compute Backends</span> 
-</p>
-</th>
-<th align="center">
-
-<p>
-<span style="color:orange">Serverless Compute Backends</span> 
-</p>
-</th>
-<th align="center">
-<p>
-<span style="color:orange">Storage Backends</span> 
-</p>
-</th>
-</tr>
-<tr>
-<td>
-
-- Gen2/Lithops
-- Gen2/Ray
-- Local Host
-
-</td>
-<td>
-
-- IBM Cloud Functions
-- IBM Code Engine
-</td>
-<td>
-
-- IBM Cloud Object Storage
-
-</td>
-</tr>
-</table>
-
-### Using lithopscloud to generate config file without user interaction
-In order to let lithopscloud generate config file based on some defaults and create vpc and all its peripherial assets automatically, please run:
+### Using ibm-ray-config to generate config file without user interaction
+In order to let ibm-ray-config generate config file based on some defaults and create vpc and all its peripheral assets automatically, please run:
 
 ```
-lithopscloud -a <API_KEY> -b <BACKEND> --defaults
+ibm-ray-config -a <API_KEY>  --defaults
 ```
 
-* currently, lithops `gen2` is the only supported backend
-
-### Using lithopscloud config tool programmatically
+### Using ibm-ray-config config tool programmatically
 Notice, not all fields are mandatory. Unspecified resources will be created automatically on the backend.
 
 E.g.
-If existing vpc id not provided - vpc will be created automatically with all required peripherial resources like security groups, gateway.. etc following minimal default requierments
+If existing vpc id not provided - vpc will be created automatically with all required peripheral resources like security groups, gateway.. etc following minimal default requirements
 If ssh key details not provided - new ssh key pair will be generated and registered in ibm cloud
 
-
-##### Lithops Gen2
-```
-from lithopscloud import generate_config
-from lithopscloud import LITHOPS_GEN2, LITHOPS_CF, LITHOPS_CE, RAY_GEN2, LOCAL_HOST
-
-api_key = '<IAM_API_KEY>'
-region = 'eu-de'
-generate_config(LITHOPS_GEN2, api_key, region, cos_bucket_name='kpavel-bucket', image_id='r010-5a674db7-95aa-45c5-a2f1-a6aa9d7e93ad', key_id='r010-fe6cb103-60e6-46bc-9cb5-14e415990849', ssh_key_filename='/home/kpavel/.ssh/id_rsa', profile_name='bx2-2x8', vpc_id='r010-af1adda4-e4e5-4060-9aa2-7a0c981aff8e')
-
-```
-
-Mandatory fields are: backend_type (LITHOPS_GEN2), api_key, region and cos_bucket.
-Minimal example:
-
-```
-from lithopscloud import generate_config
-from lithopscloud import LITHOPS_GEN2, RAY_GEN2
-
-api_key = <IAM_API_KEY>
-region = 'ca-tor'
-cos_bucket_name='kpavel-bucket'
-config_file = generate_config(LITHOPS_GEN2, api_key, region, cos_bucket_name=cos_bucket_name)
-```
-
 ###### Ray Gen2
 ```
-from lithopscloud import generate_config
-from lithopscloud import RAY_GEN2
+from ibm_ray_config import generate_config
 
 api_key = '<IAM_API_KEY>'
 region = 'eu-de'
-generate_config(RAY_GEN2, api_key, region, image_id='r010-5a674db7-95aa-45c5-a2f1-a6aa9d7e93ad', key_id='r010-fe6cb103-60e6-46bc-9cb5-14e415990849', ssh_key_filename='/home/kpavel/.ssh/id_rsa', profile_name='bx2-2x8', vpc_id='r010-af1adda4-e4e5-4060-9aa2-7a0c981aff8e', min_workers=1, max_workers=1)
+generate_config(iam_api_key=api_key, region=region, image_id='r010-5a674db7-95aa-45c5-a2f1-a6aa9d7e93ad', key_id='r010-fe6cb103-60e6-46bc-9cb5-14e415990849', ssh_key_filename='/home/kpavel/.ssh/id_rsa', profile_name='bx2-2x8', vpc_id='r010-af1adda4-e4e5-4060-9aa2-7a0c981aff8e', min_workers=1, max_workers=1)
 ```
 
-Mandatory fields are: backend_type (LITHOPS_GEN2), api_key, region and cos_bucket.
+Mandatory fields are:  api_key, region.
 Minimal example:
 
 ```
-from lithopscloud import generate_config
-from lithopscloud import RAY_GEN2
+from ibm_ray_config import generate_config
 
 api_key = <IAM_API_KEY>
 region = 'eu-de'
-config_file = generate_config(RAY_GEN2, api_key, region)
+config_file = generate_config(iam_api_key=api_key, region=region)
 ```
-
-## For Contributors
-
-### Add new unsupported sections to config file
-
-#### If the new configuration is provider specific
-
-1. implement [__config_builder.py__](src/lithopscloud/modules/config_builder.py) interface
-2. add your implementation under __provider__ package
-3. add reference to your implementation in the list of exported modules, e.g. [lithops modules](src/lithopscloud/modules/lithops/__init__.py__)
-
-
-#### If the new configuration is common for multiple providers
-
-1. implement [__config_builder.py__](src/lithopscloud/modules/config_builder.py) interface to hold common logic
-2. add your implementation to __modules__ package
-3. extend your common implementation under each provider package, e.g. [lithops image.py](src/lithopscloud/modules/lithops/image.py) and [ray image.py](src/lithopscloud/modules/ray/image.py) to have config file specific logic

bin/lithopscloud renamed to bin/ibm-ray-config

@@ -2,7 +2,7 @@
 # -*- coding: utf-8 -*-
 import re
 import sys
-from lithopscloud.main import builder
+from ibm_ray_config.main import builder
 
 if __name__ == '__main__':
     sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])

setup.cfg

@@ -1,12 +1,12 @@
 [metadata]
-name = lithopscloud
-version = 1.0.0b96
-author = kpavel
-author_email = kpavel@il.ibm.com
-description = Interactive tool for generation of lithops configuration
+name = ibm-ray-config
+version = 1.0.0b1
+author = Omer J Cohen
+author_email = omer.cohen.ibm.com
+description = Interactive tool generating ray configuration for IBM platforms 
 long_description = file: README.md
 long_description_content_type = text/markdown
-url = https://github.com/lithops-cloud/lithopscloud
+url = https://github.com/<INSERT_HERE>/ibm-ray-config
 classifiers =
     Programming Language :: Python :: 3
 
@@ -16,18 +16,15 @@ package_dir =
 packages = find:
 python_requires = >=3.6
 scripts =
-    bin/lithopscloud
+    bin/ibm-ray-config
 install_requires =
     click >=8.0.1
     inquirer >=2.7.0
     ibm-cloud-sdk-core
     ibm-platform-services
-    ibm-cos-sdk
     PyYAML >=5.4.1
-    ibm-code-engine-sdk
     ibm_watson
     docker >=5.0.3
-    lithops>=2.5.4
     ibm-vpc
 
 [options.packages.find]

src/ibm_ray_config/__init__.py

@@ -0,0 +1 @@
+from ibm_ray_config.main import generate_config, delete_config

src/ibm_ray_config/main.py

@@ -0,0 +1,147 @@
+import importlib
+import pkg_resources
+import os
+import click
+import yaml
+
+from ibm_ray_config.modules.utils import color_msg, Color, verify_paths
+
+# currently supporting configuration for ray above IBM Gen2 VPC. 
+IBM_GEN2 = 'IBM Gen2'
+backends =[{'name': IBM_GEN2, 'path': 'gen2.ray'}]
+
+def select_backend(input_file, default_config_suffix=''):
+    """returns a base config file for the backend and its package """
+    backend = backends[0] # currently supporting a single backend
+   
+    base_config = {}
+
+    if input_file:
+        with open(input_file) as f:
+            base_config = yaml.safe_load(f)
+    else:
+        base_config = load_base_config(backend, default_config_suffix)
+
+    # find the right modules
+    backend_pkg = importlib.import_module(f"ibm_ray_config.modules.{backend['path']}")
+
+    return base_config, backend_pkg
+
+def load_base_config(backend, default_config_suffix=''):
+    """returns a default base configuration dict for the chosen backend """
+    backend_path = backend['path'].replace('.', '/')
+    dir_path = os.path.dirname(os.path.realpath(__file__))
+    input_file = f"{dir_path}/modules/{backend_path}/defaults{default_config_suffix}.yaml"
+
+    base_config = None
+    with open(input_file) as f:
+        base_config = yaml.safe_load(f)
+
+    return base_config
+
+def validate_api_keys(base_config, modules, iam_api_key, compute_iam_endpoint):
+    """validates the api key specified.
+    returns a base config dict, updated with the api-key field populated and the post api-key module removal modules list  """
+    # The API_KEY module is invoked and popped from the list. 
+    api_key_module = modules[0]
+    base_config = api_key_module(base_config).run(api_key=iam_api_key,
+                                                  compute_iam_endpoint=compute_iam_endpoint)
+
+    modules = modules[1:]
+    return base_config, modules
+    
+@click.command()
+@click.option('--output-file', '-o', help='Output filename to save configurations')
+@click.option('--input-file', '-i', help=f'Template for the new configuration')
+@click.option('--iam-api-key', '-a', help='IAM_API_KEY')
+@click.option('--version', '-v', help=f'Get package version', is_flag=True)
+@click.option('--compute-iam-endpoint', help='IAM endpoint url used for compute instead of default https://iam.cloud.ibm.com')
+@click.option('--endpoint', help='IBM Cloud API endpoint')
+@click.option('--defaults', help=f'Create defaults if not exist and generate default config', is_flag=True)
+@click.option('--pr', '-g', help=f'Temporary workaround for ray gen2 only. If specified, use provider setup from PR github', is_flag=True, default=False)
+def builder(iam_api_key, output_file, input_file, version, compute_iam_endpoint, endpoint, defaults, pr):
+
+    if version:
+        print(f"{pkg_resources.get_distribution('ibm-ray-config').project_name} "
+              f"{pkg_resources.get_distribution('ibm-ray-config').version}")
+        exit(0)
+
+    print(color_msg("\nWelcome to ibm_ray_config export helper\n", color=Color.YELLOW))
+
+    input_file, output_file = verify_paths(input_file, output_file)
+
+    default_config_suffix = ''
+    if pr:
+        default_config_suffix = '_pr'
+
+    base_config, backend_pkg = select_backend(input_file, default_config_suffix)
+    
+    modules = backend_pkg.MODULES
+    base_config['create_defaults'] = defaults
+    base_config, modules = validate_api_keys(base_config, modules, iam_api_key, compute_iam_endpoint)
+
+    if endpoint and 'ibm_vpc' in base_config:
+        base_config['ibm_vpc']['endpoint'] = endpoint
+    elif endpoint and 'provider' in base_config:
+        base_config['provider']['endpoint'] = endpoint
+
+    for module in modules:
+        next_module = module(base_config)
+        
+        if defaults:
+            base_config = next_module.create_default()
+        else:
+            base_config = next_module.run()
+
+    with open(output_file, 'w') as outfile:
+        del base_config['create_defaults']
+        yaml.dump(base_config, outfile, default_flow_style=False)
+
+    if hasattr(backend_pkg, 'finish_message'):
+        print(backend_pkg.finish_message(output_file))
+    else:
+        print("\n\n=================================================")
+        print(color_msg(f"Cluster config file: {output_file}", color=Color.LIGHTGREEN))
+        print("=================================================")
+
+def error(msg):
+    print(msg)
+    raise Exception(msg)
+
+def generate_config(*args, **kwargs):
+    """A programmatic way to create configuration files, to be used externally by user."""
+    backend = backends[0] # currently supporting a single backend
+    _, output_file = verify_paths(None, None)
+    
+    # now update base config with backend specific params
+    base_config = importlib.import_module(f"ibm_ray_config.modules.{backend['path']}").load_config(backend, *args, **kwargs)
+    
+    # now find the right modules
+    modules = importlib.import_module(f"ibm_ray_config.modules.{backend['path']}").MODULES
+    
+    for module in modules:
+        base_config = module(base_config).verify(base_config)
+
+    with open(output_file, 'w') as outfile:
+        yaml.dump(base_config, outfile, default_flow_style=False)
+
+    print("\n\n=================================================")
+    print(color_msg(f"Cluster config file: {output_file}", color=Color.LIGHTGREEN))
+    print("=================================================")
+
+    return output_file
+
+def delete_config(config_file_path):
+    config = None
+    with open(config_file_path) as f:
+        config = yaml.safe_load(f)
+        
+    from ibm_ray_config.modules.gen2 import delete_config
+    delete_config(config)      
+    
+if __name__ == '__main__':
+    try:
+        builder()
+    except KeyboardInterrupt:
+        # User interrupt the program
+        exit()