Refresh README, examples and changelog with recent changes (#78)

This PR refreshes README, examples of our and changelog to better
emphasise recent changes in the validation/open API of ours
clarifying read-only properties in our API.

Specifically DAGRun dag_id has not been marked as read-only so far
which was a bug, and some of our "dev" examples set the read only
properties, which could mislead our users.

We've recently clarified that those properties are read-only and
we are now updating the API documentation and examples to show that
the good way of using related APIs.

Also the examples have been refrehsed and modernized a bit as well
as the dev example has been synchronized automatically (via
pre-commit) to make sure it is the same in the README and in dev
where we are using it to run the tests with the API.

Minimum version of Python have been set to 3.7 and CHANGELOG was
updated to reflect all the changes, as well as it has been
exposed in Project_URL of the package in PyPI.

Link in README have been changed to URL so that they are properly
rendered in README in `pypi` package documentation (currently
following the links to other filess in the repo there fails.

Author: potiuk

.pre-commit-config.yaml

@@ -21,8 +21,14 @@ default_language_version:
   python: python3
 minimum_pre_commit_version: "1.20.0"
 repos:
+  - repo: meta
+    hooks:
+      - id: identity
+        name: Print input to the static check hooks for troubleshooting
+      - id: check-hooks-apply
+        name: Check if all hooks apply to the repository
   - repo: https://github.com/Lucas-C/pre-commit-hooks
-    rev: v1.1.9
+    rev: v1.5.1
     hooks:
       - id: insert-license
         name: Add license for all md files
@@ -66,4 +72,13 @@ repos:
           - license-templates/LICENSE.txt
           - --fuzzy-match-generates-todo
         files: >
-          \.properties$|\.cfg$|\.conf$|\.ini$|\.txt$$
+          \.properties$|\.cfg$|\.conf$|\.ini$|\.txt$$
+  - repo: local
+    hooks:
+      - id: update-example-in-readme
+        name: Update example python script in README.md
+        entry: ./dev/insert_readme_example.py
+        language: python
+        files: ^dev/test_python_client.py$|^README.md$
+        pass_filenames: false
+        require_serial: true

CHANGELOG.md

@@ -18,11 +18,14 @@
  -->
 
  # v2.6.0
- Apache Airflow API version: 2.6.0
+ 
+Apache Airflow API version: 2.6.0
 
  ### Major changes:
 
- - NA
+ - Minimum Python version is 3.7
+ - DAGRun dag_id parameter is properly validated as read-only and setting it might result in an error:
+   "`dag_id` is a read-only attribute" This might break some workflows that used examples from the documentation. 
 
  ### Major Fixes
 

README.md

@@ -19,16 +19,12 @@
 
 # Apache Airflow Python Client
 
-> **_NOTE:_**  The Apache Airflow Client is still under active development and some methods
-> or APIs might be broken. Please raise an issue in github if you encounter any such issues.
-
-
-
 ## Requirements.
 
-Python >= 3.6
+Python >= 3.7
 
 ## Installation & Usage
+
 ### pip install
 
 You can install directly using pip:
@@ -41,7 +37,7 @@ pip install apache-airflow-client
 
 Or install via [Setuptools](http://pypi.python.org/pypi/setuptools).
 
-```sh
+```shell
 git clone [email protected]:apache/airflow-client-python.git
 cd airflow-client-python
 python setup.py install --user
@@ -53,40 +49,116 @@ Then import the package:
 import airflow_client.client
 ```
 
+## Changelog
+
+See [CHANGELOG.md](https://github.com/apache/airflow-client-python/blob/main/CHANGELOG.md) for keeping
+track on what has changed in the client.
+
+
 ## Getting Started
 
-Please follow the [installation procedure](#installation--usage) and then run the following:
+Please follow the [installation procedure](#installation--usage) and then run the following
+example python script:
 
 ```python
-import airflow_client.client
-from pprint import pprint
-from airflow_client.client.api import config_api
+import uuid
 
+import airflow_client.client
+try:
+    # If you have rich installed, you will have nice colored output of the API responses
+    from rich import print
+except ImportError:
+    print("Output will not be colored. Please install rich to get colored output: `pip install rich`")
+    pass
+from airflow_client.client.api import config_api, dag_api, dag_run_api
+from airflow_client.client.model.dag_run import DAGRun
+
+# The client must use the authentication and authorization parameters
+# in accordance with the API server security policy.
+# Examples for each auth method are provided below, use the example that
+# satisfies your auth use case.
 #
-# In case of the basic authentication below. Make sure:
-#  - Airflow is configured with the basic_auth as backend:
-#     auth_backend = airflow.api.auth.backend.basic_auth
-#  - Make sure that the client has been generated with securitySchema Basic.
+# In case of the basic authentication below, make sure that Airflow is
+# configured also with the basic_auth as backend additionally to regular session backend needed
+# by the UI. In the `[api]` section of your `airflow.cfg` set:
+#
+# auth_backend = airflow.api.auth.backend.session,airflow.api.auth.backend.basic_auth
+#
+# Make sure that your user/name are configured properly - using the user/password that has admin
+# privileges in Airflow
 
 # Configure HTTP basic authorization: Basic
 configuration = airflow_client.client.Configuration(
-    host="http://localhost/api/v1",
+    host="http://localhost:8080/api/v1",
     username='admin',
     password='admin'
 )
 
+# Make sure in the [core] section, the  `load_examples` config is set to True in your airflow.cfg
+# or AIRFLOW__CORE__LOAD_EXAMPLES environment variable set to True
+DAG_ID = "example_bash_operator"
 
 # Enter a context with an instance of the API client
 with airflow_client.client.ApiClient(configuration) as api_client:
-    # Create an instance of the API class
-    api_instance = config_api.ConfigApi(api_client)
 
+    errors = False
+
+    print('[blue]Getting DAG list')
+    dag_api_instance = dag_api.DAGApi(api_client)
+    try:
+        api_response = dag_api_instance.get_dags()
+        print(api_response)
+    except airflow_client.client.OpenApiException as e:
+        print("[red]Exception when calling DagAPI->get_dags: %s\n" % e)
+        errors = True
+    else:
+        print('[green]Getting DAG list successful')
+
+
+    print('[blue]Getting Tasks for a DAG')
+    try:
+        api_response = dag_api_instance.get_tasks(DAG_ID)
+        print(api_response)
+    except airflow_client.client.exceptions.OpenApiException as e:
+        print("[red]Exception when calling DagAPI->get_tasks: %s\n" % e)
+        errors = True
+    else:
+        print('[green]Getting Tasks successful')
+
+
+    print('[blue]Triggering a DAG run')
+    dag_run_api_instance = dag_run_api.DAGRunApi(api_client)
+    try:
+        # Create a DAGRun object (no dag_id should be specified because it is read-only property of DAGRun)
+        # dag_run id is generated randomly to allow multiple executions of the script
+        dag_run = DAGRun(
+            dag_run_id='some_test_run_' + uuid.uuid4().hex,
+        )
+        api_response = dag_run_api_instance.post_dag_run(DAG_ID, dag_run)
+        print(api_response)
+    except airflow_client.client.exceptions.OpenApiException as e:
+        print("[red]Exception when calling DAGRunAPI->post_dag_run: %s\n" % e)
+        errors = True
+    else:
+        print('[green]Posting DAG Run successful')
+
+    # Get current configuration. Note, this is disabled by default with most installation.
+    # You need to set `expose_config = True` in Airflow configuration in order to retrieve configuration.
+    conf_api_instance = config_api.ConfigApi(api_client)
     try:
-        # Get current configuration
-        api_response = api_instance.get_config()
-        pprint(api_response)
-    except airflow_client.client.ApiException as e:
-        print("Exception when calling ConfigApi->get_config: %s\n" % e)
+        api_response = conf_api_instance.get_config()
+        print(api_response)
+    except airflow_client.client.OpenApiException as e:
+        print("[red]Exception when calling ConfigApi->get_config: %s\n" % e)
+        errors = True
+    else:
+        print('[green]Config retrieved successfully')
+
+    if errors:
+        print ('\n[red]There were errors while running the script - see above for details')
+    else:
+        print ('\n[green]Everything went well')
 ```
 
-See [README](./airflow_client/README.md#documentation-for-api-endpoints) for full client API documentation.
+See [README](https://github.com/apache/airflow-client-python/blob/main/README.md#documentation-for-api-endpoints) 
+for full client API documentation.

dev/insert_readme_example.py

@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from pathlib import Path
+
+BASE_DIR = Path(__file__).resolve().parent.parent
+
+if __name__ == '__main__':
+    example = (BASE_DIR / "dev" / "test_python_client.py").read_text().splitlines()
+    example = example[example.index("# under the License.")+1:]   # Remove license header
+    readme_lines = (BASE_DIR / "README.md").read_text().splitlines()
+    result_lines = []
+    skip_lines = False
+    changes_made = False
+    for line in readme_lines:
+        if line.strip().startswith("example python script:"):
+            result_lines.append(line)
+            result_lines.append("")
+            result_lines.append("```python")
+            result_lines.extend(example)
+            result_lines.append("```")
+            skip_lines = True
+            changes_made = True
+        else:
+            if not skip_lines:
+                result_lines.append(line)
+            if line.strip() == "```":
+                skip_lines = False
+    (BASE_DIR / "README.md").write_text("\n".join(result_lines))
+    if not changes_made:
+        raise Exception("Could not find example python script in README.md to replace. "
+                        "Please make sure `example python script:` line is in README.md")

dev/test_python_client.py

@@ -14,8 +14,15 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+import uuid
+
 import airflow_client.client
-from pprint import pprint
+try:
+    # If you have rich installed, you will have nice colored output of the API responses
+    from rich import print
+except ImportError:
+    print("Output will not be colored. Please install rich to get colored output: `pip install rich`")
+    pass
 from airflow_client.client.api import config_api, dag_api, dag_run_api
 from airflow_client.client.model.dag_run import DAGRun
 
@@ -25,11 +32,13 @@
 # satisfies your auth use case.
 #
 # In case of the basic authentication below, make sure that Airflow is
-# configured with the basic_auth as backend:
+# configured also with the basic_auth as backend additionally to regular session backend needed
+# by the UI. In the `[api]` section of your `airflow.cfg` set:
 #
-# auth_backend = airflow.api.auth.backend.basic_auth
+# auth_backend = airflow.api.auth.backend.session,airflow.api.auth.backend.basic_auth
 #
-# Make sure that your user/name are configured properly
+# Make sure that your user/name are configured properly - using the user/password that has admin
+# privileges in Airflow
 
 # Configure HTTP basic authorization: Basic
 configuration = airflow_client.client.Configuration(
@@ -38,46 +47,67 @@
     password='admin'
 )
 
-dag_id = "example_bash_operator"
+# Make sure in the [core] section, the  `load_examples` config is set to True in your airflow.cfg
+# or AIRFLOW__CORE__LOAD_EXAMPLES environment variable set to True
+DAG_ID = "example_bash_operator"
 
 # Enter a context with an instance of the API client
 with airflow_client.client.ApiClient(configuration) as api_client:
-    # Get current configuration
-    conf_api_instance = config_api.ConfigApi(api_client)
-    try:
-        api_response = conf_api_instance.get_config()
-        pprint(api_response)
-    except airflow_client.client.OpenApiException as e:
-        print("Exception when calling ConfigApi->get_config: %s\n" % e)
 
+    errors = False
 
-    # Get dag list
+    print('[blue]Getting DAG list')
     dag_api_instance = dag_api.DAGApi(api_client)
     try:
         api_response = dag_api_instance.get_dags()
-        pprint(api_response)
+        print(api_response)
     except airflow_client.client.OpenApiException as e:
-        print("Exception when calling DagAPI->get_dags: %s\n" % e)
+        print("[red]Exception when calling DagAPI->get_dags: %s\n" % e)
+        errors = True
+    else:
+        print('[green]Getting DAG list successful')
 
 
-    # Get tasks for a DAG (TODO: issue#20)
+    print('[blue]Getting Tasks for a DAG')
     try:
-        api_response = dag_api_instance.get_tasks(dag_id)
-        pprint(api_response)
+        api_response = dag_api_instance.get_tasks(DAG_ID)
+        print(api_response)
     except airflow_client.client.exceptions.OpenApiException as e:
-        print("Exception when calling DagAPI->get_tasks: %s\n" % e)
+        print("[red]Exception when calling DagAPI->get_tasks: %s\n" % e)
+        errors = True
+    else:
+        print('[green]Getting Tasks successful')
 
 
-    # Trigger a dag run (TODO: issue#21)
+    print('[blue]Triggering a DAG run')
     dag_run_api_instance = dag_run_api.DAGRunApi(api_client)
     try:
-        # Create a DAGRun object
+        # Create a DAGRun object (no dag_id should be specified because it is read-only property of DAGRun)
+        # dag_run id is generated randomly to allow multiple executions of the script
         dag_run = DAGRun(
-            dag_run_id='some_test_run',
-            dag_id=dag_id,
-            external_trigger=True,
+            dag_run_id='some_test_run_' + uuid.uuid4().hex,
         )
-        api_response = dag_run_api_instance.post_dag_run(dag_id, dag_run)
-        pprint(api_response)
+        api_response = dag_run_api_instance.post_dag_run(DAG_ID, dag_run)
+        print(api_response)
     except airflow_client.client.exceptions.OpenApiException as e:
-        print("Exception when calling DAGRunAPI->post_dag_run: %s\n" % e)
+        print("[red]Exception when calling DAGRunAPI->post_dag_run: %s\n" % e)
+        errors = True
+    else:
+        print('[green]Posting DAG Run successful')
+
+    # Get current configuration. Note, this is disabled by default with most installation.
+    # You need to set `expose_config = True` in Airflow configuration in order to retrieve configuration.
+    conf_api_instance = config_api.ConfigApi(api_client)
+    try:
+        api_response = conf_api_instance.get_config()
+        print(api_response)
+    except airflow_client.client.OpenApiException as e:
+        print("[red]Exception when calling ConfigApi->get_config: %s\n" % e)
+        errors = True
+    else:
+        print('[green]Config retrieved successfully')
+
+    if errors:
+        print ('\n[red]There were errors while running the script - see above for details')
+    else:
+        print ('\n[green]Everything went well')

setup.cfg

@@ -32,10 +32,11 @@ license_files =
    NOTICE
 project_urls =
     Documentation=https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html
+    Changelog=https://github.com/apache/airflow-client-python/CHANGELOG.md
     Bug Tracker=https://github.com/apache/airflow-client-python/issues
     Source Code=https://github.com/apache/airflow-client-python
 
 [options]
 zip_safe = False
 include_package_data = True
-python_requires = ~=3.6
+python_requires = ~=3.7