From a518e01fcc71c0a80ba6da5c07d038502a57b7f4 Mon Sep 17 00:00:00 2001 From: Bas Harenslak Date: Tue, 19 Dec 2023 12:22:20 +0100 Subject: [PATCH] Update --- README.md | 144 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 75 insertions(+), 69 deletions(-) diff --git a/README.md b/README.md index 8b9405e..80224ed 100644 --- a/README.md +++ b/README.md @@ -4,28 +4,27 @@ > This repository serves as a demo for how to structure a custom package.\ > Feel free to copy and adjust to your own needs. -This Python library hosts custom Airflow Operators, Sensors, Notifiers, etc. that are common across **[insert company name]** data teams. This guide will walk you through how this library works, how to use it in your project, making changes, running tests, and contributing your changes. +This Python library hosts custom Airflow Operators, Sensors, Notifiers, etc. that are common across **[insert company name]** data teams. This guide will walk you through how this library works, how to use it in your project, making changes, running tests, and contributing changes. ## Table of Contents - [Installing the library](#installing-the-library) - [Using the library](#using-the-library) -- [Versioning](#versioning) -- [Continuous Integration/Continuous Deployment (CI/CD)](#continuous-integrationcontinuous-deployment-cicd) -- [Contributing](#contributing) - * [Getting started](#getting-started) - * [Making changes](#making-changes) - * [Testing](#testing) - * [Pull requests](#pull-requests) - * [Update your project dependency](#update-your-project-dependency) - [Library structure](#library-structure) -- [How do I ...?](#how-do-i-) +- [Contributing](#contributing) + * [Getting started](#getting-started) + * [Making changes](#making-changes) + * [Testing](#testing) + * [Versioning](#versioning) + * [Continuous Integration/Continuous Deployment (CI/CD)](#continuous-integrationcontinuous-deployment-cicd) + * [Pull requests](#pull-requests) + * [Update your project dependency](#update-your-project-dependency) Table of contents generated using https://derlin.github.io/bitdowntoc. ## Installing the library -Since this repository is built only for example purposes, we publish a Python package only to [TestPyPI](https://test.pypi.org/project/democorp-airflow). Therefore, you'll need to add TestPyPI as an (extra) index URL. You can install the package using pip: +Since this repository is built only for example purposes, we name it `democorp-airflow` and publish a Python package only to [TestPyPI](https://test.pypi.org/project/democorp-airflow). Therefore, you'll need to add TestPyPI as an (extra) index URL. You can install the package using pip: ```bash pip install -i https://test.pypi.org/simple/ democorp-airflow @@ -45,18 +44,58 @@ Import example: from democorp_airflow.operators.example import ExampleOperator ``` -## Versioning - -We use `setuptools-scm` to automatically manage versioning based on git tags. When you create a new tag, the library version is updated accordingly, and a release for the given tag is made. See the files in `.github/workflows` for inspiration. - -See [1](https://github.com/pypa/setuptools_scm/), [2](https://www.moritzkoerber.com/posts/versioning-with-setuptools_scm/) for more details. +## Library structure -## Continuous Integration/Continuous Deployment (CI/CD) +The library is organized as follows: -Our CI/CD pipeline is managed using GitHub Actions: +``` +. +├── .github +│ └── workflows +│ ├── ci.yaml +│ └── publish.yaml +├── democorp_airflow +│ ├── __init__.py +│ ├── hooks/ +│ │ ├── __init__.py +│ │ ├── example.py +│ │ └── ... +│ ├── notifiers/ +│ │ ├── __init__.py +│ │ └── ... +│ ├── operators/ +│ │ ├── __init__.py +│ │ └── ... +│ └── sensors/ +│ ├── __init__.py +│ └── ... +├── tests/ +│ ├── __init__.py +│ ├── conftest.py +│ └── democorp_airflow +│ ├── hooks/ +│ │ ├── __init__.py +│ │ ├── test_example.py +│ │ └── ... +│ ├── notifiers/ +│ │ ├── __init__.py +│ │ └── ... +│ ├── operators/ +│ │ ├── __init__.py +│ │ └── ... +│ └─ sensors/ +│ ├── __init__.py +│ └── ... +├── README.md +├── pyproject.toml +└── setup.py +``` -- On every commit, syntax checks and unit tests are run to ensure code quality. See `.github/workflows/ci.yaml`. -- When a new tag is pushed, the library is built, published to TestPyPI, and a GitHub release is created. See `.github/workflows/publish.yaml`. +- CI/CD code is stored in `.github/workflows` +- Modules are organized in `democorp_airflow/` according to Airflow components names (`hooks`, `operators`, etc.) but you're free to name modules fitting your use case +- The `tests/` directory contains unit tests for each component, this folder structure matches the package structure (pytests calls this structure [tests outside application code](https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#tests-outside-application-code)) +- `pyproject.toml` is used for package configuration and versioning +- `setup.py` is used for backwards compatibility of newer `pyproject.toml` syntax ## Contributing @@ -80,63 +119,30 @@ Before submitting your changes, ensure that all tests pass: pytest tests/ ``` +### Versioning + +We use `setuptools-scm` to automatically manage versioning based on git tags. When you push a new tag, the library version is updated accordingly, and a release for the given tag is made. See [.github/workflows/publish.yaml](.github/workflows/publish.yaml). + +See [1](https://github.com/pypa/setuptools_scm/), [2](https://www.moritzkoerber.com/posts/versioning-with-setuptools_scm/) for more details. + +### Continuous Integration/Continuous Deployment (CI/CD) + +The CI/CD pipeline is managed using GitHub Actions: + +- On every commit, syntax checks and unit tests are run to ensure code quality. See [.github/workflows/ci.yaml](.github/workflows/ci.yaml). +- When a new tag is pushed, the library is built, published to TestPyPI, and a GitHub release is created. See [.github/workflows/publish.yaml](.github/workflows/publish.yaml). + ### Pull requests 1. Commit your changes and push them to your branch. 1. Create a pull request from your branch to the `main` branch. 1. The CI/CD pipeline will automatically run tests on your pull request. -1. Once the tests pass and your code is reviewed, your contribution will be merged into the main repository. +1. Once the tests pass and your code is reviewed/accepted, your contribution will be merged into the main repository. ### Update your project dependency -Once a new version of the `custom-package-demo` library is released, you need to update your Airflow project to use it. +Once a new version of the `democorp-airflow` library is released, you need to update your Airflow project to use it. 1. Update the version constraint in your `requirements.txt` file. 1. After updating the package, it's crucial to test your project to ensure that the new version works as expected and doesn't introduce any compatibility issues or bugs. Run `astro dev start` to test changes locally. -1. Commit your changes to the requirements.txt file to Git so that other developers can easily reproduce your project environment. - -## Library structure - -The library is organized as follows: - -``` -custom_package_demo/ -├── hooks/ -│ ├── __init__.py -│ └── ... -├── notifiers/ -│ ├── __init__.py -│ └── ... -├── operators/ -│ ├── __init__.py -│ └── ... -├── sensors/ -│ ├── __init__.py -│ └── ... -├── tests/ -│ ├── hooks/ -│ │ ├── __init__.py -│ │ └── ... -│ ├── notifiers/ -│ │ ├── __init__.py -│ │ └── ... -│ ├── operators/ -│ │ ├── __init__.py -│ │ └── ... -│ ├── sensors/ -│ │ ├── __init__.py -│ │ └── ... -├── README.md -├── pyproject.toml -└── setup.py -``` - -- `tests/` directory contains unit tests for each component -- `pyproject.toml` is used for package configuration and versioning -- `setup.py` is used for backwards compatibility of newer `pyproject.toml` syntax - -Happy coding! - -## How do I ...? - -Explain usage of custom components here... +1. Commit your changes to the `requirements.txt` file to Git so that other developers can easily reproduce your project environment.