Major Refactoring into API and Subpackages (SciML#371)

* Start to work on DataDrivenBase

* Restructure a bit

* Make the solution  a statsbase type

* Add implicit indicator and dispatch

* exclude tests for now

* Clean up deps and generalize Solution

* Move utils

* More tests

* Fix typo

* Remove old tests

* Adapt further

* Adapt testset

* Add collocation test

* Add some interpolation methods

* Adapt SciML Code style

* Adapt common options slightly

* Adapt docs

* Adapt the docs

* Adapt docs

* Lighten up documentation deps

* Prune deps

* Update the basis methods

* Update make.jl

* Format code

* Adapt for subtyping of basis

* Rmv deprecations

* Slight documentation adapts

* adapt more common options

* Adapt solve dispatch

* Setup subpackages and start working on DMD

* Rmv old test for now

* Adapt Readme

* Format

* Add StaticArrays

* Make basis immutable

* Make abstract types more concrete

* Safety check eqs

* Fixup docs

* Format

* Restructure tests

* Implicit deps on Symbolics and SymbolicUtils

* Fix wrong import

* Working status commit

* More tests

* DataDrivenFunction

* Structure tests

* Remove revise again

* Fix implicits

* Fix Dataset and Solution

* Fix tests

* Empty Matrix backwards

* Change empty matrix

* Better normalization, preprocessing,add internal

* Prune old code

* Add problem to the internal prob

* Fit InternalProb for all

* Add comments and tests

* Further adapt for use within DMD

* Forgot retcode

* Fix up tests

* Store default values

* Adapt test

* Update docs

* Format

* Update docs and format

* Export returncodes

* Updata DMD

* WIP for DMD

* Update for remake and solution parameters

* Update options, fix problem

* Right eval_expression

* Fix solution tests for prob remake

* Fixup small issues and make StaticArrays keyworded

* Fix Complex results and jacobian behaviour

* Formatting

* Fix small issues

* Allow for non-parametric outputs

* Add getters and fix errors

* Add exports, testdeps and some tests

* Remove StaticArrays and make eval concrete

* Change sigdigits and use it to derive the basis

* Adapt rounding

* Adapt test

* Add tests and fix gEDMD

* Adapt the building of the basis

* Add controlled tests

* Adapt algorithms to old structure

* Finalize tests

* Info to test

* Adapt tests for subpackages

* Initial draft for Symbolic Regression jl

* Adapt tests

* try to up CI

* Adapt for right useage

* Fix typo

* Intial draft for SparseRegression

* ADMM and fixes

* Further fixes

* SR3

* Add simple testset

* Add Tests

* Format

* Implicit Sparse Regression

* Remove old stuff

* Format

* Debug the tests

* Try to get tests green

* Less randomness

* Too much synt sugar!

* Diversify the basis

* Relax tests a little

* Relax due to failing of 1.0

* Update Candidate Matrix

* Update problems and build basis

* Whitespace

* Adapt utils

* Update dependencies

* Fix candidate matrix

* Update tests

* Fix pkg imports

* Remove prints

* Check for consistency on GH

* Add necessary idx for implicits

* Try to adapt coverage

* Format

* Try to fix

* Adapt for all files

* Revert CI

* Adapt DMD tests

* Add eval expression tests

* Format

* Adapt CompatHelper to include subdirs

* Try to up Coverage

* Add coverage to sublib

* Fix coverage

* Fix some exports and annotations

* Update docs

* Add more docs

* Missing exports

* Remove direct deps right now

* Remove docs

* Add more examples and adapt docs

* Remove example files

* Format

* Bump version

* Add quickstart to the homepage

* Add rounding

* Remove tests and make SR examples faster

* Remove folders

* Adapt README

* Format

* Limit DataInterpolations

Author: AlCap23

.JuliaFormatter.toml

@@ -0,0 +1 @@
+style = "sciml"

.github/workflows/CI.yml

@@ -13,6 +13,9 @@ jobs:
       matrix:
         group:
           - Core
+          - DataDrivenDMD
+          - DataDrivenSR
+          - DataDrivenSparse
         version:
           - '1'
           - '1.6'
@@ -33,7 +36,12 @@ jobs:
             ${{ runner.os }}-
       - uses: julia-actions/julia-buildpkg@v1
       - uses: julia-actions/julia-runtest@v1
+        env:
+          GROUP: ${{ matrix.group }}
       - uses: julia-actions/julia-processcoverage@v1
-      - uses: codecov/codecov-action@v1
         with:
-          file: lcov.info
+          directories: src,lib/DataDrivenDMD/src,lib/DataDrivenSparse/src,lib/DataDrivenSR/src
+      - uses: codecov/codecov-action@v3
+        with:
+          files: lcov.info
+          flags: ${{ matrix.group }}

.github/workflows/CompatHelper.yml

@@ -23,4 +23,4 @@ jobs:
       - name: CompatHelper.main()
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        run: julia -e 'using CompatHelper; CompatHelper.main()'
+        run: julia -e 'using CompatHelper; CompatHelper.main(;subdirs=["", "docs", "lib/DataDrivenDMD", "lib/DataDrivenSparse", "lib/DataDrivenSR"])'

.github/workflows/FormatCheck.yml

@@ -0,0 +1,42 @@
+name: format-check
+
+on:
+  push:
+    branches:
+      - 'master'
+      - 'release-'
+    tags: '*'
+  pull_request:
+
+jobs:
+  build:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        julia-version: [1]
+        julia-arch: [x86]
+        os: [ubuntu-latest]
+    steps:
+      - uses: julia-actions/setup-julia@latest
+        with:
+          version: ${{ matrix.julia-version }}
+
+      - uses: actions/checkout@v1
+      - name: Install JuliaFormatter and format
+        # This will use the latest version by default but you can set the version like so:
+        #
+        # julia  -e 'using Pkg; Pkg.add(PackageSpec(name="JuliaFormatter", version="0.13.0"))'
+        run: |
+          julia  -e 'using Pkg; Pkg.add(PackageSpec(name="JuliaFormatter"))'
+          julia  -e 'using JuliaFormatter; format(".", verbose=true)'
+      - name: Format check
+        run: |
+          julia -e '
+          out = Cmd(`git diff --name-only`) |> read |> String
+          if out == ""
+              exit(0)
+          else
+              @error "Some files have not been formatted !!!"
+              write(stdout, out)
+              exit(1)
+          end'

.gitignore

@@ -4,4 +4,6 @@
 Manifest.toml
 .DS_Store
 /docs/build/*
-hall_of_fame.*
+/docs/src/libs/*/examples
+hall_of_fame.*
+/tmp/*

Project.toml

@@ -1,16 +1,15 @@
 name = "DataDrivenDiffEq"
 uuid = "2445eb08-9709-466a-b3fc-47e12bd697a2"
 authors = ["Julius Martensen <[email protected]>"]
-version = "0.8.6"
+version = "1.0.0"
 
 [deps]
 CommonSolve = "38540f10-b2f7-11e9-35d8-d573e4eb0ff2"
 DataInterpolations = "82cc6244-b520-54b8-b5a6-8a565e85f1d0"
 DiffEqBase = "2b5f629d-d688-5b77-993f-72d75c75574e"
-Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
 DocStringExtensions = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
 LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
-Literate = "98b081ad-f1c9-55d3-8b20-4c87d4299306"
+MLUtils = "f1d291b0-491e-4a28-83b9-f70985020b54"
 Measurements = "eff96d63-e80a-5855-80a2-b1b0885c5ab7"
 ModelingToolkit = "961ee093-0014-501f-94e3-6117800e7a78"
 Parameters = "d96e819e-fc66-5662-9728-84c9c7592b0a"
@@ -19,43 +18,31 @@ QuadGK = "1fd47b50-473d-5c70-9696-f719f8f3bcdc"
 Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
 RecipesBase = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
 Reexport = "189a3867-3050-52da-a836-e630ba90ab69"
-Requires = "ae029012-a4dd-5104-9daa-d747884805df"
+Setfield = "efcf1570-3423-57d1-acb7-fd33fddbac46"
 Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
 StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
-SymbolicUtils = "d1185830-fcd6-423d-90d6-eec64667417b"
-Symbolics = "0c5d862f-8b57-4792-8d23-62f2024744c7"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
 [compat]
 CommonSolve = "0.2"
-DataInterpolations = "3"
+DataInterpolations = "3.7"
 DiffEqBase = "6"
-Distributions = "^0.25.9, 0.25"
 DocStringExtensions = "0.7, 0.8, 0.9"
-Flux = "^0.12.4"
-Literate = "2"
-Measurements = "2.7"
 ModelingToolkit = "7, 8"
 Parameters = "0.12"
 ProgressMeter = "1.6"
 QuadGK = "2.4"
 RecipesBase = "1"
 Reexport = "1.0"
-Requires = "1"
+Setfield = "1"
 StatsBase = "0.32.0, 0.33"
-SymbolicRegression = "0.9"
-SymbolicUtils = "0.19"
-Symbolics = "4"
 julia = "1.6"
 
 [extras]
-DiffEqOperators = "9fdde737-9c7f-55bf-ade8-46b3f136cc48"
-Flux = "587475ba-b771-5e3f-ad9e-33799f191a9c"
-OrdinaryDiffEq = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed"
+Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
 Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
 SafeTestsets = "1bc83da4-3b8d-516f-aca4-4fe02f6d838f"
-SymbolicRegression = "8254be44-1295-4e6a-a16d-46603ac705cb"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 
 [targets]
-test = ["DiffEqOperators", "Flux", "SymbolicRegression", "OrdinaryDiffEq", "Test", "Random", "SafeTestsets"]
+test = ["Test", "Random", "SafeTestsets", "Pkg"]

README.md

@@ -27,7 +27,7 @@ the documentation which contains the un-released features.
 using DataDrivenDiffEq
 using ModelingToolkit
 using OrdinaryDiffEq
-
+using DataDrivenSparse
 using LinearAlgebra
 
 # Create a test problem
@@ -44,18 +44,18 @@ u0 = [1.0;0.0;0.0]
 tspan = (0.0,100.0)
 dt = 0.1
 prob = ODEProblem(lorenz,u0,tspan)
-sol = solve(prob, Tsit5(), saveat = dt, progress = true)
+sol = solve(prob, Tsit5(), saveat = dt)
 
 
 ## Start the automatic discovery
-ddprob = ContinuousDataDrivenProblem(sol)
+ddprob = DataDrivenProblem(sol)
 
 @variables t x(t) y(t) z(t)
 u = [x;y;z]
 basis = Basis(polynomial_basis(u, 5), u, iv = t)
 opt = STLSQ(exp10.(-5:0.1:-1))
-ddsol = solve(ddprob, basis, opt, normalize = true)
-print(ddsol, Val{true})
+ddsol = solve(ddprob, basis, opt, options = DataDrivenCommonOptions(digits = 1))
+println(get_basis(ddsol))
 ```
 
 ```

docs/Project.toml

@@ -1,16 +1,13 @@
 [deps]
 DataDrivenDiffEq = "2445eb08-9709-466a-b3fc-47e12bd697a2"
-DiffEqOperators = "9fdde737-9c7f-55bf-ade8-46b3f136cc48"
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
-Flux = "587475ba-b771-5e3f-ad9e-33799f191a9c"
 LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
 Literate = "98b081ad-f1c9-55d3-8b20-4c87d4299306"
 ModelingToolkit = "961ee093-0014-501f-94e3-6117800e7a78"
 OrdinaryDiffEq = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed"
 Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
-SymbolicRegression = "8254be44-1295-4e6a-a16d-46603ac705cb"
-Symbolics = "0c5d862f-8b57-4792-8d23-62f2024744c7"
+StableRNGs = "860ef19b-820b-49d6-a774-d7a799459cd3"
+StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
 
 [compat]
 Documenter = "0.27"
-SymbolicRegression = "0.9"