Add pre-commit hook for common misspelling check in files (apache#18964)

This PR adds codespell to the pre-commit hooks. This will specifically help
us a bit in resolving sphinx errors.

From the project page:
It does not check for word membership in a complete dictionary, but instead looks for a set of common misspellings.
Therefore it should catch errors like "adn", but it will not catch "adnasdfasdf".
This also means it shouldn't generate false-positives when you use a niche term it doesn't know about.

This means the sphinx errors are not solved completely.

.github/workflows/build-images.yml

@@ -307,12 +307,12 @@ jobs:
       - name: "Build PROD images ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
         run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
         env:
-          # GITHUB_REGISTRY_PULL_IMAGE_TAG is overriden to latest in order to build PROD image using "latest"
+          # GITHUB_REGISTRY_PULL_IMAGE_TAG is overridden to latest in order to build PROD image using "latest"
           GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
       - name: "Push PROD images ${{ matrix.python-version }}:${{ env.GITHUB_REGISTRY_PUSH_IMAGE_TAG }}"
         run: ./scripts/ci/images/ci_push_production_images.sh
         env:
-          # GITHUB_REGISTRY_PULL_IMAGE_TAG is overriden to latest in order to build PROD image using "latest"
+          # GITHUB_REGISTRY_PULL_IMAGE_TAG is overridden to latest in order to build PROD image using "latest"
           GITHUB_REGISTRY_PULL_IMAGE_TAG: "latest"
 
   cancel-on-ci-build:

.pre-commit-config.yaml

@@ -258,6 +258,18 @@ repos:
         exclude: |
           (?x)
           ^airflow/_vendor/
+  - repo: https://github.com/codespell-project/codespell
+    rev: v2.1.0
+    hooks:
+      - id: codespell
+        name: Run codespell to check for common misspellings in files
+        entry: codespell
+        language: python
+        types: [text]
+        exclude: ^airflow/_vendor/|^CHANGELOG.txt|^airflow/www/static/css/material-icons.css
+        args:
+          - --ignore-words=docs/spelling_wordlist.txt
+          - --skip=docs/*/commits.rst,airflow/providers/*/*.rst,*.lock,INTHEWILD.md,*.min.js
   - repo: local
     hooks:
       - id: lint-openapi

BREEZE.rst

@@ -2191,7 +2191,7 @@ This is the current syntax for  `./breeze <./breeze>`_:
                  check-executables-have-shebangs check-extras-order check-hooks-apply
                  check-integrations check-merge-conflict check-xml daysago-import-check
                  debug-statements detect-private-key doctoc dont-use-safe-filter end-of-file-fixer
-                 fix-encoding-pragma flake8 flynt forbid-tabs helm-lint identity
+                 fix-encoding-pragma flake8 flynt codespell forbid-tabs helm-lint identity
                  incorrect-use-of-LoggingMixin insert-license isort json-schema language-matters
                  lint-dockerfile lint-openapi markdownlint mermaid mixed-line-ending mypy mypy-helm
                  no-providers-in-core-examples no-relative-imports pre-commit-descriptions

STATIC_CODE_CHECKS.rst

@@ -188,6 +188,8 @@ require Breeze Docker images to be installed locally.
 ------------------------------------ ---------------------------------------------------------------- ------------
 ``flynt``                              Runs flynt
 ------------------------------------ ---------------------------------------------------------------- ------------
+``codespell``                          Checks for common misspellings in files.
+------------------------------------ ---------------------------------------------------------------- ------------
 ``forbid-tabs``                        Fails if tabs are used in the project
 ------------------------------------ ---------------------------------------------------------------- ------------
 ``helm-lint``                          Verifies if helm lint passes for the chart

UPDATING.md

@@ -283,7 +283,7 @@ No breaking changes.
 
 ### `activate_dag_runs` argument of the function `clear_task_instances` is replaced with `dag_run_state`
 
-To achieve the previous default behaviour of `clear_task_instances` with `activate_dag_runs=True`, no change is needed. To achieve the previous behaviour of `activate_dag_runs=False`, pass `dag_run_state=False` instead. (The previous paramater is still accepted, but is deprecated)
+To achieve the previous default behaviour of `clear_task_instances` with `activate_dag_runs=True`, no change is needed. To achieve the previous behaviour of `activate_dag_runs=False`, pass `dag_run_state=False` instead. (The previous parameter is still accepted, but is deprecated)
 
 ### `dag.set_dag_runs_state` is deprecated
 

airflow/_vendor/connexion/__init__.py

@@ -6,7 +6,7 @@
 from .apps import AbstractApp  # NOQA
 from .decorators.produces import NoContent  # NOQA
 from .exceptions import ProblemException  # NOQA
-# add operation for backwards compatability
+# add operation for backwards compatibility
 from .operations import compat
 from .problem import problem  # NOQA
 from .resolver import Resolution, Resolver, RestyResolver  # NOQA

airflow/example_dags/plugins/workday.py

@@ -16,7 +16,7 @@
 # specific language governing permissions and limitations
 # under the License.
 
-"""Plugin to demostrate timetable registration and accomdate example DAGs."""
+"""Plugin to demonstrate timetable registration and accommodate example DAGs."""
 
 # [START howto_timetable]
 from datetime import timedelta

airflow/providers/cncf/kubernetes/utils/pod_launcher.py

@@ -263,7 +263,7 @@ def read_pod_logs(
             )
         except BaseHTTPError:
             self.log.exception('There was an error reading the kubernetes API.')
-            # Reraise to be catched by self.monitor_pod.
+            # Reraise to be caught by self.monitor_pod.
             raise
 
     @tenacity.retry(stop=tenacity.stop_after_attempt(3), wait=tenacity.wait_exponential(), reraise=True)

airflow/providers/google/cloud/hooks/bigquery.py

@@ -1422,7 +1422,7 @@ def _build_new_schema(
             # Turn schema_field_updates into a dict keyed on field names
             schema_fields_updates = {field["name"]: field for field in deepcopy(schema_fields_updates)}
 
-            # Create a new dict for storing the new schema, initated based on the current_schema
+            # Create a new dict for storing the new schema, initiated based on the current_schema
             # as of Python 3.6, dicts retain order.
             new_schema = {field["name"]: field for field in deepcopy(current_schema)}
 

airflow/providers/microsoft/azure/operators/wasb_delete_blob.py

@@ -65,7 +65,7 @@ def __init__(
         self.ignore_if_missing = ignore_if_missing
 
     def execute(self, context: dict) -> None:
-        self.log.info('Deleting blob: %s\nin wasb://%s', self.blob_name, self.container_name)
+        self.log.info('Deleting blob: %s\n in wasb://%s', self.blob_name, self.container_name)
         hook = WasbHook(wasb_conn_id=self.wasb_conn_id)
 
         hook.delete_file(

airflow/providers/microsoft/azure/sensors/wasb.py

@@ -57,7 +57,7 @@ def __init__(
         self.check_options = check_options
 
     def poke(self, context: dict):
-        self.log.info('Poking for blob: %s\nin wasb://%s', self.blob_name, self.container_name)
+        self.log.info('Poking for blob: %s\n in wasb://%s', self.blob_name, self.container_name)
         hook = WasbHook(wasb_conn_id=self.wasb_conn_id)
         return hook.check_for_blob(self.container_name, self.blob_name, **self.check_options)
 

airflow/providers/ssh/hooks/ssh.py

@@ -261,13 +261,13 @@ def get_conn(self) -> paramiko.SSHClient:
 
         if not self.allow_host_key_change:
             self.log.warning(
-                'Remote Identification Change is not verified. '
-                'This wont protect against Man-In-The-Middle attacks'
+                "Remote Identification Change is not verified. "
+                "This won't protect against Man-In-The-Middle attacks"
             )
             client.load_system_host_keys()
 
         if self.no_host_key_check:
-            self.log.warning('No Host Key Verification. This wont protect against Man-In-The-Middle attacks')
+            self.log.warning("No Host Key Verification. This won't protect against Man-In-The-Middle attacks")
             # Default is RejectPolicy
             client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
         else:
@@ -400,7 +400,7 @@ def _pkey_from_private_key(self, private_key: str, passphrase: Optional[str] = N
         for pkey_class in self._pkey_loaders:
             try:
                 key = pkey_class.from_private_key(StringIO(private_key), password=passphrase)
-                # Test it acutally works. If Paramiko loads an openssh generated key, sometimes it will
+                # Test it actually works. If Paramiko loads an openssh generated key, sometimes it will
                 # happily load it as the wrong type, only to fail when actually used.
                 key.sign_ssh_data(b'')
                 return key

airflow/settings.py

@@ -550,7 +550,7 @@ def initialize():
 
 # Display alerts on the dashboard
 # Useful for warning about setup issues or announcing changes to end users
-# List of UIAlerts, which allows for specifiying the message, category, and roles the
+# List of UIAlerts, which allows for specifying the message, category, and roles the
 # message should be shown to. For example:
 #   from airflow.www.utils import UIAlert
 #

airflow/templates.py

@@ -32,7 +32,7 @@ def is_safe_attribute(self, obj, attr, value):
         Allow access to ``_`` prefix vars (but not ``__``).
 
         Unlike the stock SandboxedEnvironment, we allow access to "private" attributes (ones starting with
-        ``_``) whilst still blocking internal or truely private attributes (``__`` prefixed ones).
+        ``_``) whilst still blocking internal or truly private attributes (``__`` prefixed ones).
         """
         return not jinja2.sandbox.is_internal_attribute(obj, attr)
 

airflow/utils/db.py

@@ -771,7 +771,7 @@ def check_task_tables_without_matching_dagruns(session) -> Iterable[str]:
         try:
             metadata.reflect(only=[model.__tablename__])
         except exc.InvalidRequestError:
-            # Table doesn't exist, but try the other ones incase the user is upgrading from an _old_ DB
+            # Table doesn't exist, but try the other ones in case the user is upgrading from an _old_ DB
             # version
             pass
 

airflow/www/fab_security/manager.py

@@ -857,7 +857,7 @@ def _search_ldap(self, ldap, con, username):
         if len(self.auth_roles_mapping) > 0:
             request_fields.append(self.auth_ldap_group_field)
 
-        # preform the LDAP search
+        # perform the LDAP search
         log.debug(
             "LDAP search for '%s' with fields %s in scope '%s'"
             % (filter_str, request_fields, self.auth_ldap_search)
@@ -1017,7 +1017,7 @@ def auth_user_ldap(self, username, password):
             user_attributes = {}
 
             # Flow 1 - (Indirect Search Bind):
-            #  - in this flow, special bind credentials are used to preform the
+            #  - in this flow, special bind credentials are used to perform the
             #    LDAP search
             #  - in this flow, AUTH_LDAP_SEARCH must be set
             if self.auth_ldap_bind_user:
@@ -1051,7 +1051,7 @@ def auth_user_ldap(self, username, password):
 
             # Flow 2 - (Direct Search Bind):
             #  - in this flow, the credentials provided by the end-user are used
-            #    to preform the LDAP search
+            #    to perform the LDAP search
             #  - in this flow, we only search LDAP if AUTH_LDAP_SEARCH is set
             #     - features like AUTH_USER_REGISTRATION & AUTH_ROLES_SYNC_AT_LOGIN
             #       will only work if AUTH_LDAP_SEARCH is set

airflow/www/static/js/graph.js

@@ -400,7 +400,7 @@ function startOrStopRefresh() {
 
 $('#auto_refresh').change(() => {
   if ($('#auto_refresh').is(':checked')) {
-    // Run an initial refesh before starting interval if manually turned on
+    // Run an initial refresh before starting interval if manually turned on
     handleRefresh();
     localStorage.removeItem('disableAutoRefresh');
   } else {

airflow/www/static/js/tree.js

@@ -473,7 +473,7 @@ document.addEventListener('DOMContentLoaded', () => {
 
   $('#auto_refresh').change(() => {
     if ($('#auto_refresh').is(':checked')) {
-      // Run an initial refesh before starting interval if manually turned on
+      // Run an initial refresh before starting interval if manually turned on
 
       handleRefresh();
       localStorage.removeItem('disableAutoRefresh');

breeze

@@ -149,7 +149,7 @@ function breeze::setup_default_breeze_constants() {
     AIRFLOW_SOURCES_TO=${AIRFLOW_SOURCES_TO:="/opt/airflow"}
     export AIRFLOW_SOURCES_TO
 
-    # Unlike in CI scripts, in breeze by default production image ist installed from sources
+    # Unlike in CI scripts, in breeze by default production image is installed from sources
     export AIRFLOW_INSTALLATION_METHOD="."
 
     # If it set is set to specified version, then the source version of Airflow

breeze-complete

@@ -102,6 +102,7 @@ end-of-file-fixer
 fix-encoding-pragma
 flake8
 flynt
+codespell
 forbid-tabs
 helm-lint
 identity

chart/README.md

@@ -40,7 +40,7 @@ cluster using the [Helm](https://helm.sh) package manager.
 * Supported Airflow version: ``1.10+``, ``2.0+``
 * Supported database backend: ``PostgresSQL``, ``MySQL``
 * Autoscaling for ``CeleryExecutor`` provided by KEDA
-* PostgresSQL and PgBouncer with a battle-tested configuration
+* PostgreSQL and PgBouncer with a battle-tested configuration
 * Monitoring:
    * StatsD/Prometheus metrics for Airflow
    * Prometheus metrics for PgBouncer

chart/tests/helm_template_generator.py

@@ -80,7 +80,7 @@ def create_validator(api_version, kind, kubernetes_version):
 
 
 def validate_k8s_object(instance, kubernetes_version):
-    # Skip PostgresSQL chart
+    # Skip PostgreSQL chart
     labels = jmespath.search("metadata.labels", instance)
     if "helm.sh/chart" in labels:
         chart = labels["helm.sh/chart"]

dev/README_RELEASE_PROVIDER_PACKAGES.md

@@ -212,7 +212,7 @@ rm -rf ${AIRFLOW_REPO_ROOT}/dist/*
 ./breeze prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both
 ```
 
-if you ony build few packages, run:
+if you only build few packages, run:
 
 ```shell script
 ./breeze prepare-provider-packages --version-suffix-for-pypi rc1 --package-format both \

dev/import_all_classes.py

@@ -45,7 +45,7 @@ def import_all_classes(
     :param provider_ids - provider ids that should be loaded.
     :param print_imports - if imported class should also be printed in output
     :param print_skips - if skipped classes should also be printed in output
-    :return: tupple of list of all imported classes and all warnings generated
+    :return: tuple of list of all imported classes and all warnings generated
     """
     imported_classes = []
     tracebacks = []

docs/apache-airflow-providers-amazon/secrets-backends/aws-secrets-manager.rst

@@ -104,7 +104,7 @@ For example, if you want to set parameter ``connections_prefix`` to ``"airflow/c
     backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
     backend_kwargs = {"connections_prefix": "airflow/connections", "variables_prefix": null, "profile_name": "default"}
 
-Example of storing Google Secrets in AWS Secrets Manger
+Example of storing Google Secrets in AWS Secrets Manager
 """"""""""""""""""""""""""""""""""""""""""""""""""""""""
 For connecting to a google cloud conn, all the fields must be in the extra field, and their names follow the pattern
 ``extra_google_cloud_platform__value``. For example:

docs/apache-airflow-providers-ssh/connections/ssh.rst

@@ -51,7 +51,7 @@ Extra (optional)
     * ``timeout`` - Deprecated - use conn_timeout instead.
     * ``compress`` - ``true`` to ask the remote client/server to compress traffic; ``false`` to refuse compression. Default is ``true``.
     * ``no_host_key_check`` - Set to ``false`` to restrict connecting to hosts with no entries in ``~/.ssh/known_hosts`` (Hosts file). This provides maximum protection against trojan horse attacks, but can be troublesome when the ``/etc/ssh/ssh_known_hosts`` file is poorly maintained or connections to new hosts are frequently made. This option forces the user to manually add all new hosts. Default is ``true``, ssh will automatically add new host keys to the user known hosts files.
-    * ``allow_host_key_change`` - Set to ``true`` if you want to allow connecting to hosts that has host key changed or when you get 'REMOTE HOST IDENTIFICATION HAS CHANGED' error.  This wont protect against Man-In-The-Middle attacks. Other possible solution is to remove the host entry from ``~/.ssh/known_hosts`` file. Default is ``false``.
+    * ``allow_host_key_change`` - Set to ``true`` if you want to allow connecting to hosts that has host key changed or when you get 'REMOTE HOST IDENTIFICATION HAS CHANGED' error.  This won't protect against Man-In-The-Middle attacks. Other possible solution is to remove the host entry from ``~/.ssh/known_hosts`` file. Default is ``false``.
     * ``look_for_keys`` - Set to ``false`` if you want to disable searching for discoverable private key files in ``~/.ssh/``
     * ``host_key`` - The base64 encoded ssh-rsa public key of the host or "ssh-<key type> <key data>" (as you would find in the ``known_hosts`` file). Specifying this allows making the connection if and only if the public key of the endpoint matches this value.
 

docs/apache-airflow/howto/set-up-database.rst

@@ -27,7 +27,7 @@ The document below describes the database engine configurations, the necessary c
 Choosing database backend
 -------------------------
 
-If you want to take a real test drive of Airflow, you should consider setting up a database backend to **MySQL**, **PostgresSQL** , **MsSQL**.
+If you want to take a real test drive of Airflow, you should consider setting up a database backend to **MySQL**, **PostgreSQL** , **MsSQL**.
 By default, Airflow uses **SQLite**, which is intended for development purposes only.
 
 Airflow supports the following database engine versions, so make sure which version you have. Old versions may not support all SQL statements.
@@ -230,7 +230,7 @@ If you use a current Postgres user with custom search_path, search_path can be c
 
    ALTER USER airflow_user SET search_path = public;
 
-For more information regarding setup of the PostgresSQL connection, see `PostgreSQL dialect <https://docs.sqlalchemy.org/en/13/dialects/postgresql.html>`__ in SQLAlchemy documentation.
+For more information regarding setup of the PostgreSQL connection, see `PostgreSQL dialect <https://docs.sqlalchemy.org/en/13/dialects/postgresql.html>`__ in SQLAlchemy documentation.
 
 .. note::
 

docs/apache-airflow/modules_management.rst

@@ -112,7 +112,7 @@ In the case above, there are the ways you could import the python files:
 .. code-block:: python
 
    from my_company.common_package.common_module import SomeClass
-   from my_company.common_package.subpackge.subpackaged_util_module import AnotherClass
+   from my_company.common_package.subpackage.subpackaged_util_module import AnotherClass
    from my_company.my_custom_dags.base_dag import BaseDag
 
 You can see the ``.airflowignore`` file at the root of your folder. This is a file that you can put in your

docs/apache-airflow/pipeline_example.csv

@@ -68,7 +68,7 @@ Serial Number,Company Name,Employee Markme,Description,Leave
 9.78174E+12,COLLABORATION IN LEARNING,MAL LEE AND LORRAE WARD,ACER PRESS,0
 9.78086E+12,RE-IMAGINING EDUCATIMarkL LEADERSHIP,BRIAN J.CALDWELL,ACER PRESS,0
 9.78086E+12,TOWARDS A MOVING SCHOOL,FLEMING & KLEINHENZ,ACER PRESS,0
-9.78086E+12,DESINGNING A THINKING A CURRICULAM,SUSAN WILKS,ACER PRESS,0
+9.78086E+12,DESIGNING A THINKING A CURRICULAM,SUSAN WILKS,ACER PRESS,0
 9.78086E+12,LEADING A DIGITAL SCHOOL,MAL LEE AND MICHEAL GAFFNEY,ACER PRESS,0
 9.78086E+12,NUMERACY,WESTWOOD,ACER PRESS,0
 9.78086E+12,TEACHING ORAL LANGUAGE,JOHN MUNRO,ACER PRESS,0
@@ -87,7 +87,7 @@ Serial Number,Company Name,Employee Markme,Description,Leave
 9.78818E+12,TULSIDAS ' RAMAYAMark,Mark,ACK,0
 9.78818E+12,TALES OF HANUMAN,-,ACK,0
 9.78818E+12,VALMIKI'S RAMAYAMark,A C K,ACK,1
-9.78818E+12,THE BEST OF INIDAN WIT AND WISDOM,Mark,ACK,0
+9.78818E+12,THE BEST OF INIDAN WITH AND WISDOM,Mark,ACK,0
 9.78818E+12,MORE TALES FROM THE PANCHTANTRA,AMarkNT PAL,ACK,0
 9.78818E+12,THE GREAT MUGHALS {5-IN-1},AMarkNT.,ACK,0
 9.78818E+12,FAMOUS SCIENTISTS,Mark,ACK,0

docs/apache-airflow/security/kerberos.rst

@@ -78,7 +78,7 @@ If you need more granular options for your kerberos ticket the following options
     forwardable = True
 
     # Allow to include or remove local IP from kerberos token.
-    # This is particulary useful if you use Airflow inside a VM NATted behind host system IP.
+    # This is particularly useful if you use Airflow inside a VM NATted behind host system IP.
     include_ip = True
 
 Keep in mind that Kerberos ticket are generated via ``kinit`` and will your use your local ``krb5.conf`` by default.

docs/apache-airflow/security/webserver.rst

@@ -220,7 +220,7 @@ webserver_config.py itself if you wish.
         ) -> Dict[str, Union[str, List[str]]]:
 
             # Creates the user info payload from Github.
-            # The user previously allowed your app to act on thier behalf,
+            # The user previously allowed your app to act on their behalf,
             #   so now we can query the user and teams endpoints for their data.
             # Username and team membership are added to the payload and returned to FAB.