Skip to content

Commit

Permalink
[13-1, 14] Change definition of scaling factor alpha , add center a…
Browse files Browse the repository at this point in the history 
…nd approximations for Hermite functions, add Fourier transform utility functions (#15)

* BLD:
- first test of a GitHub CI for testing

* FIX:
- fixed wrong pip install and python versions in CI ?

* FIX:
- fixed loading of test files that do not contain test but dependencies that cannot be loaded ?

* BUG:
- escaped imports of unaccessible dependencies for test-file generation in GitHub CI tests by making the `if __name__ == "__main__"` include these imports and function definitions

* TST:
- test whether Cython build can be removed from GitHub actions for testing ?

* TST:
- reverted removal of Cython for testing

* PKG:
- moved the full package back into an `src`-folder

BLD:
- updated `pyproject.toml` and `setup.py` to account for the movement to the `src`-folder
- removed dedicated Cython build from CI test pipeline

* BLD:
- changed CI pipeline target branches

* BLD:
- augmented CI checks with format, type, and lint checks

* BUG:
- fixed CI pipeline wrong folder include ?

BLD:
- added isort check

* BUG:
- fixed broken `isort` usage in CI?

* FIX:
- fixed missing `isort` dependency in CI?
- fixed wrong import sort order in Cython Hermite functions

* FIX:
- fixed missing `colorama` dependency for `isort` in CI ?

* FIX:
- type-ignored Cython import that was not properly resolved by `pyright` ?

* FIX:
- fixed missing import of Cython module for `pyright`  in CI ?

* FIX:
- again trying to resolve the wrong import error of Cython module by `pyright` in CI ?

* FIX:
- fixed wrong `pyright` Cython import error of Cython import in CI

* BLD:
- removed pushes to `develop` from the GitHub CI actions

* BLD:
- added missing comma to the name of the GitHub CI action

* [10 develop] Add coverage to CI pipeline (#11)

* DOC:
- added Python versions and `black` code style to `README`

* DOC:
- added `isort` badge to `README`

* TST:
- added `--no-jit`-flag to `pytest` to enable proper coverage of Numba functions

* tmp:
- first test of CI with coverage report ?

* BUG:
- fixed failure of `pytest-xdist` and `pytest-cov` in GitHub CI (works locally) ?

* BUG:
- fixed accidentally placed `\` for `./tests` in coverage CI action ?

* BUG:
- added codecov to CI pipeline ?

* TST:
- tried to readd `pytest-xdist` for coverage reports ?

* wip:
- reset example Jupyter notebook number 3

* MAINT:
- made `_get_num_workers` a function of the `_utils`-model

TST:
- increased coverage to 100% by testing `np.float32` x-values for the Hermite functions was well as super negative numbers of requested workers

* DOC:
- added setup, installation, and development instructions to `README`

* DOC:
- switched back from `README.rst` to `README.md`

BLD:
- made CI pipeline push actions apply to the main branch only

* BLD:
- added `mypy` to CI pipeline

BUG:
- fixed type error that `mypy` uncovered for the `filename_parameters_mapping` for the Hermite function reference generation

* wip:
- test if CI can still pass ?

* wip:
- CI apparently still runs; removed check

* BLD:
- added `pycodestyle` as a part of the CI checks

BUG:
- fixed a line too long error revealed by `pycodestyle`

* DOC:
- added CI test status to `README` ?

* ENH:
- added functions that help for computing discrete/continuous Fourier transforms (`fourier_transform._fft_utils`)

TST:
- implemented a test suite for `fourier_transform._fft_utils`

* Issue 14 #14
DOC:
- updated equations with new definitions of $\alpha$
- updated `README` with new equations and added respective reference that these definitions are based on

* Issue 14 #14
MAINT:
- removed redundant Numba version of the legacy Hermite functions and `jit`-compiled the NumPy-version instead
- fixed minor documentation issues

* Issue 14 #14
MAINT:
- adapted interface to Hermite functions to the new definition of $\alpha$ in terms of both implementation and docs

* Issue 14 #14
DOC:
- updated example scripts and resulting plots with new definition of $\alpha$

* Issue 14 #14
TST:
- generated new references files for testing the new definition of $\alpha$ for the Hermite functions
- adapted Hermite function tests to properly run for the new definition of $\alpha$
- configured Hermite function tests via global constants and kept comments in them more vague
- made test for Cramér's inequality of the Hermite functions ensure that the boundary value is not only an upper bound but actually reached
- changed `requirements` for GitHub CI because with a re-arrangement of the `if __name__ == "__main__"` in the Hermite function reference generation file, `sympy` and `tqdm` became CI requirements as well

* BUG:
- fixed failing `ruff` for too long line in docstring of new `_fft_utils`

BLD:
- temporarily enabled CI on push to current branch to check if it still passes after an adaption in `pyright`

* Issue 14 #14
BLD:
- disabled GitHub CI on push to the current branch again after verifying that it still runs

* Issue 13 #13 (3)
ENH:
- added `x_center` to the Hermite function basis for shifting the Hermite functions in x-direction
- added input validation for `x_center`

MAINT:
- renamed `hermite_functions._interface` to `hermite_functions._func_interface` to prepare for the class interface
- made type handling of Hermite function input validation smarter and safer by first converting to the target type and then testing only for this one
- made argument types for `x` of the Hermite functions broader by also including lists and tuples

TST:
- made input validation tests for Hermite functions be executed before the Hermite function reference tests
- made input validation tests for Hermite functions not only check failure but also success cases
- added tests for the new `x_center` and ensured that it does not modify the original `x`-values for the Hermite functions (after this causes some problems)

DOC:
- updated all equations and the `README` to clarify the new `x_center` for the Hermite functions
- updated installation commands (also mentioned venv activation) and test/check commands in `README`

* DOC:
- added y-axis to Hermite-function-scale-effect-plot
- fixed first vertical x-axis for order `n=0` not showing in the plot at all

* Issue #13 (3)
MAINT:
- moved Hermite function input validation to dedicated file

ENH:
- pushed forgotten files that add `x_center` to Hermite functions

TST:
- pushed forgotten files that test the new `x_center` of the Hermite functions and improved the testing of the Hermite function input validation

* TST:
- split up tests for Hermite functions into dedicated test files to be more flexible

* Issue #13 (3)
DOC:
- added common matplotlib style to unify all plots
- updated example scripts with new matplotlib style
- added shifted center to Hermite function example 01

* DOC:
- made example plots 01 and 02 for Hermite functions more visually appealing/less overwhelming

* Issue #13 (6.1)
ENH:
- added auxiliary script that auto-generates a Python file with spline coefficients for almost exactly evaluating the largest zero (= outermost roots) of the Hermite functions up to order ~100_000

* Issue #13 (6.1)
MAINT:
- refactored the reference creation for the largest roots of the Hermite functions to not minimize the relative but the absolute error
- relaxed permitted absolute error in reference creation for the largest roots of the Hermite functions because the SciPy-reference is not that accurate

DOC:
- mentioned that the SciPy-reference for the largest roots of the Hermite functions is not perfectly accurate

* Issue #13 (6)
ENH:
- added script to create reference data for the largest extrema of the Hermite functions
- added new `_approximations`-module for Hermite functions that offers approximations for their largest roots, largest extrema, and fadeout points

TST:
- added tests for the new `_approximations`-module

DOC:
- added legend to first plot in script that creates reference data for the largest roots of the Hermite functions

* MAINT:
- refactored the reference data generation for the largest roots and extrema of the Hermite functions to fit the splines based on relative errors
- updated reference data and plots created by these scripts
- made this reference data generation a developer-only script

TST:
- made test for largest root of Hermite functions more meaningful by adapting the tolerance for checking to the actual magnitude of the root position
- made test for largest extremum of Hermite functions more meaningful by checking the y-values (and not the x-values) of the extremum against a numerical optimisation (also with magnitude-adaptive tolerances)

* DOC:
- made all plot saves in examples available for developers only
- aligned plots in terms of visuals (especially titles and sizes)
- updated `README` with new plots and developer-only environment variable

* BLD:
- added additional scripts checks to GitHub CI
- made all tools cause actual failures rather than just printing test output ?
- broke CI on purpose, by badly formatting `auxiliary_scripts\02_hermite_functions_largest_extremum.py` and `src\robust_hermite_ft\hermite_functions\_c_hermite.pyx`

* BLD:
- CI tests fixed `black` formatting after confirming that it works
- `isort` is the next to check

* BLD:
- fixed `isort` CI failure after confirming that it works
- `pyright` is the next to check

* BLD:
- temporarily broke `isort` CI check again to check its output

* BLD:
- fixed `isort` problem again
- checking for reasons why `pyright`-check fails by running it separately without output ?

* BLD:
- fixed `pyright` error handling in CI
- `mypy` is the next one to check, but for now it's checked whether it just runs through even though there is an error

* BLD:
- pushing updated CI workflow file for properly testing that `mypy` just runs through
- added `matplotlib` to CI requirements to avoid `pyright` from throwing an error ?

* BLD:
- going back to the start for tests after `mypy` also stopped CI correctly
- starting with `black` test

* BLD:
- `black` successfully stopped the CI without customised handling
- `isort` is the next to check without customised handling

* BLD:
- also `isort` stops the CI on failure
- next to check is `pyright`

* BLD:
- reverted GitHub CI to its original working state

* DOC:
- updated `README` with new CI commands

* ENH:
- Hermite functions now support all kinds of real numeric scalars/Arrays as `x`-values, integer scalars as `n`-values, and real numeric scalars for `alpha` and `x_center` after making the input validation way more relaxed and smarter

TST:
- test for Hermite function input validation now covers Python and Numpy scalars individually and on top of that also includes Pandas series and Python Arrays
- test for data link to original `x` in Hermite functions now evaluates dynamically whether memory is shared by checking if changes in the copy/view are reflected in the original
- added a completely new test set whether the Hermite functions yield the same results for all different kinds of `x`-values, `n`-values, `alpha`- and `x-center` values
  • Loading branch information
@MothNik
MothNik authored Jul 30, 2024
1 parent 75808fa commit 3c663ba
Show file tree
Hide file tree
Showing 53 changed files with 6,417 additions and 863 deletions.
22 changes: 15 additions & 7 deletions .github/workflows/python-package.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,19 +30,27 @@ jobs:
- name: Run format check
run: |
black --check --diff --color ./examples ./src ./tests
black --check --diff --color ./auxiliary_scripts ./examples ./src ./tests
- name: Run Import sort check
- name: Run import sort check
run: |
isort --check --diff --color ./examples ./src ./tests
isort --check --diff --color ./auxiliary_scripts ./examples ./src ./tests
- name: Run static type check
- name: Run pyright static type check
run: |
pyright
pyright ./auxiliary_scripts ./examples ./src ./tests
- name: Run Python lint check
- name: Run mypy static type check
run: |
ruff check ./examples ./src ./tests
mypy ./auxiliary_scripts ./examples ./src ./tests
- name: Run ruff lint check
run: |
ruff check ./auxiliary_scripts ./examples ./src ./tests
- name: Run pycodestyle check
run: |
pycodestyle ./auxiliary_scripts ./examples ./src ./tests --max-line-length=88 --ignore=E203,W503
- name: Run Cython check
run: |
Expand Down
73 changes: 51 additions & 22 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,9 @@
[![python-3.12](https://img.shields.io/badge/python-3.12-blue.svg)](https://www.python.org/downloads/release/python-3120/)
[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![code style: isort](https://img.shields.io/badge/code%20style-isort-000000.svg)](https://pycqa.github.io/isort/)
[![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)
[![codecov](https://codecov.io/gh/MothNik/robust_hermite_ft/branch/10-improve-and-add-coverage-to-CI/graph/badge.svg)](https://codecov.io/gh/MothNik/robust_hermite_ft/branch/10-improve-and-add-coverage-to-CI)
![tests](https://github.com/MothNik/robust_hermite_ft/actions/workflows/python-package.yml/badge.svg)
<br><br>

You want to compute the Fourier transform of a signal, but your signal can be corrupted by outliers? If so, this package is for you even though you will have to say goodbye to the _"fast"_ in _Fast Fourier Transform_ 🏃🙅‍♀️
Expand All @@ -25,33 +27,42 @@ Currently, the package is not yet available on PyPI. To install it, you can clon
git clone https://github.com/MothNik/robust_hermite_ft.git
```

and from within the repositories root directory, install it with
and from within the repositories root directory, install it for normal use

```bash
pip install -e .
# activate your virtual environment, e.g., source venv/bin/activate
pip install .
```

for normal use or
or for development with all the development dependencies

```bash
pip install -e .["dev"]
# activate your virtual environment, e.g., source venv/bin/activate
pip install .["dev"]
```

for development which will also install the development dependencies.

⚠️ **Warning**: This will require a C-compiler to be installed on your system to
compile the Cython code.

When working in developer mode, an environment variable has to be added to run certain
scripts.

```
ROBHERMFT_DEVELOPER = true
```

### 🔎 Code quality

The following checks for `black`, `isort`, `pyright`, `ruff`, and
`cython-lint` - that are also part of the CI pipeline - can be run with

```bash
black --check --diff --color ./examples ./src ./tests
isort --check --diff --color ./examples ./src ./tests
pyright
ruff check ./examples ./src ./tests
black --check --diff --color ./auxiliary_scripts ./examples ./src ./tests
isort --check --diff --color ./auxiliary_scripts ./examples ./src ./tests
pyright ./auxiliary_scripts ./examples ./src ./tests
mypy ./auxiliary_scripts ./examples ./src ./tests
ruff check ./auxiliary_scripts ./examples ./src ./tests
pycodestyle ./auxiliary_scripts ./examples ./src ./tests --max-line-length=88 --ignore=E203,W503
cython-lint src/robust_hermite_ft/hermite_functions/_c_hermite.pyx
```

Expand All @@ -61,6 +72,8 @@ To run the tests - almost like in the CI pipeline - you can use

```bash
pytest --cov=robust_hermite_ft ./tests -n="auto" --cov-report=xml -x --no-jit
# or for a nice HTML coverage report
# pytest --cov=robust_hermite_ft ./tests -n="auto" --cov-report=html -x --no-jit
```

for parallelized testing whose coverage report will be stored in the folder
Expand All @@ -74,22 +87,23 @@ transform. However, their evaluation can be a bit tricky.

The module `hermite_functions` offers a numerically stable way to evaluate Hermite
functions or arbitrary order $n$ and argument - that can be scaled with a factor
$\alpha$:
$\alpha$ and shifted by a constant $\mu$:

<p align="center">
<img src="docs/hermite_functions/DilatedHermiteFunctions_DifferentScales.png" width="1000px" />
<img src="docs/hermite_functions/01-DilatedHermiteFunctions_DifferentScales.png" width="1000px" />
</p>

The Hermite functions are defined as
After a slight modification of the definitions in [[1]](#references), the Hermite
functions can be written as

<p align="left">
<img src="docs/hermite_functions/equations/DilatedHermiteFunctions.png" width="500px" />
<img src="docs/hermite_functions/equations/Dilated_Hermite_Functions_Of_Generic_X.png" width="500px", height="91px" />
</p>

with the Hermite polynomials

<p align="left">
<img src="docs/hermite_functions/equations/DilatedHermitePolynomials.png" width="681px" />
<img src="docs/hermite_functions/equations/Dilated_Hermite_Polynomials_Of_Generic_X.png" width="764px", height="65px" />
</p>

By making use of logarithm tricks, the evaluation that might involve infinitely high
Expand All @@ -100,19 +114,19 @@ results.
For doing so, the relation between the dilated and the non-dilated Hermite functions

<p align="left">
<img src="docs/hermite_functions/equations/HermiteFunctions_UndilatedToDilated.png" width="321px" />
<img src="docs/hermite_functions/equations/HermiteFunctions_UndilatedToDilated.png" width="366px", height="32px" />
</p>

and the recurrence relation for the Hermite functions

<p align="left">
<img src="docs/hermite_functions/equations/HermiteFunctions_RecurrenceRelation.png" width="699px" />
<img src="docs/hermite_functions/equations/HermiteFunctions_RecurrenceRelation.png" width="576px", height="68px" />
</p>

are used, but not directly. Instead, the latest evaluated Hermite function is kept at a
value of either -1, 0, or +1 during the recursion and the logarithm of a correction
factor is tracked and applied when the respective Hermite function is finally evaluated
and stored. This approach is based on [[1]](#references).
and stored. This approach is based on [[2]](#references).

The implementation is tested against a symbolic evaluation with `sympy` that uses 200
digits of precision and it can be shown that even orders as high as 2,000 can still be
Expand All @@ -121,14 +135,29 @@ evaluated for this anymore. The factorial for example would already have overflo
orders of 170 in `float64`-precision.

<p align="center">
<img src="docs/hermite_functions/DilatedHermiteFunctions_Stability.png" width="1000px" />
<img src="docs/hermite_functions/02-DilatedHermiteFunctions_Stability.png" width="1000px" />
</p>

As a sanity check, their orthogonality is part of the tests together with a test for
the fact that the absolute values of the Hermite functions for real input cannot exceed
the value $\frac{1}{\pi^{-\frac{1}{4}}\cdot\sqrt{\alpha}}$.
the value $\frac{\sqrt{\alpha}}{\pi^{-\frac{1}{4}}}$.

On top of that `robust_hermite_ft` comes with utility functions to approximate some
special points of the Hermite functions, namely the x-positions of their

- largest root (= outermost zero),
- largest maximum in the outermost oscillation, and
- the point where they numerically fade to zero.

<p align="center">
<img src="docs/hermite_functions/04-HermiteFunctions_SpecialPoints.png" width="1000px" />
</p>

## References

- [1] Bunck B. F., A fast algorithm for evaluation of normalized Hermite
functions, BIT Numer Math (2009), 49, pp. 281–295, DOI: [https://doi.org/10.1007/s10543-009-0216-1](https://doi.org/10.1007/s10543-009-0216-1)
- [1] Dobróka M., Szegedi H., and Vass P., Inversion-Based Fourier Transform as a New
Tool for Noise Rejection, _Fourier Transforms - High-tech Application and Current Trends_
(2017), DOI: [http://dx.doi.org/10.5772/66338](http://dx.doi.org/10.5772/66338)
- [2] Bunck B. F., A fast algorithm for evaluation of normalized Hermite functions,
_BIT Numer Math_ (2009), 49, pp. 281–295, DOI:
[https://doi.org/10.1007/s10543-009-0216-1](https://doi.org/10.1007/s10543-009-0216-1)
Loading

0 comments on commit 3c663ba

Please sign in to comment.