Simplify library installation (#909)

* Add gate module in src/ansys/dpf/

* remove ansys-dpf-gate from direct dependencies

* Add grpc/dpf module to src/ansys/

* remove ansys-grpc-dpf from requirements_dev.txt

* Fix assert in test_data_tree.py/test_runtime_core_config

* Temporary fix to dpf_mesh_to_vtk_py

* By default ignore changes to src/ansys/dpf/gate, ansys/dpf/gatebin and src/ansys/grpc/. This should be overriden when updating using "git add --force my/ignore/file.foo"

* Remove requirements_dev.txt with gatebin

* Add gatebin for windows directly

* Revert "Remove requirements_dev.txt with gatebin"

This reverts commit 8b0137d.

* Remove gatebin from requirements_dev.txt

* Refactor package for single wheel

* Add manylinux_1 versions of gatebin binaries

* Add grpcio as dependency (was working due to something in requirements_test.txt having it as dependency)

* Add google-api-python-client as dependency (was working due to something in requirements_test.txt having it as dependency)

* Exclude gate, gatebin and grpc from api-doc generation

* Fix title underlines in examples

* Update .gitignore

* Update .pre-commit-config.yaml

* Move everything into src/ansys/dpf/core, also move grpc to src/ansys/dpf/core/grpc/dpf/

* Fix imports due to refactor

* Fix imports due to refactor

* Add missing refactored files

* Fix gate/load_pi.py/_try_use_gatebin()

* Fix gate/load_pi.py/_find_outdated_ansys_version()

* Add .so gatebin binaries

* Fix expected path to gatebin module in test_service.py

* Update codacy.yml and .gitignore

* Try and fix 02-volume_averaged_stress.py for Docker

* Fix 04-extrapolation_stress_3d.py for Docker

* Switch to setuptools as backend building tool to allow for several packages in the library

* Fix ansys/grpc/dpf instead of ansys/dpf/grpc in ignores

* Revert operators/math changes due to testing

* Include ansys.dpf.gate.generated in packaged packages

* Include all in setup("packages")

* Revert changes to 04-extrapolation_stress_3d.py

* Add a 10mn timeout to Test API Entry on Docker

* Try fix elemental_nodal_to_nodal_fc.outputs[1].type_names

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/ as well as ansys_dpf_gate and ansys_dpf_gatebin

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/ as well as ansys_dpf_gate and ansys_dpf_gatebin

* update_operators.yml now unzips ansys_grpc_dpf in src/ansys/ as well as ansys_dpf_gate and ansys_dpf_gatebin

* Improve debugging for operators/build;py

* Temp hack to bypass broken operators sweeping_phase specs

* Revert "Temp hack to bypass broken operators sweeping_phase specs"

This reverts commit bed5415.

* Temporarily bypass the mesh_to_vtk operator

* Fix use of input standalone-suffix

* update operators (#916)

Co-authored-by: PProfizi <[email protected]>

* Remove previous grpc and gate files and force git add

* Still bypass mesh_to_pyvista_op

* Bypass bug on plot for 00-compute_and_average.py

* Move all metadata to pyproject.toml, keep setup.py for retro-compatibility

* Add update_dpf_dependencies.py script

* Add update_dpf_dependencies.py script

* Remove support for 221 (incompatible with this refactoring of the package)

* Updated dpf dependencies

* Updated dpf dependencies

* Updated dpf dependencies

* Updated dpf dependencies

* Updated dpf dependencies

* Updated dpf dependencies

* Add missing generic_data_container_pb2.py and generic_data_container_pb2_grpc.py

* Update with latest dist from dpf-standalone

* Update with latest dist from dpf-standalone

* Remove custom requirements and custom wheels inputs and parameters from CI

* Rename ANSYS_DPF_GATE_ROOT -> ANSYSDPFPYGATE_ROOT, ANSYS_DPF_GRPC_ROOT -> DPFDV_ROOT

* update operators (#1109)

Co-authored-by: PProfizi <[email protected]>

* Fix gatebin integration and core.helpers integration

* Use branches _integrate_gate for dpf-standalone

* Fix test_load_api_without_awp_root_no_gatebin

* Fix gate.load_api._try_use_gatebin

* Set branch for update_operators.yml to _test_pygate

* Update update_dpf_dependencies.py for debug

* Update update_dpf_dependencies.py for improved path treatment

* Update Operators for DPF 241_test_pygate on maint/integrate_pygate (#1132)

* update operators

* Update ci.yml

---------

Co-authored-by: PProfizi <[email protected]>
Co-authored-by: Paul Profizi <[email protected]>

* Fix doc step

* Make package discovery dynamic

* Fix package_data

* Fix build_doc.bat fluid images copy

* update operators (#1136)

Co-authored-by: PProfizi <[email protected]>

* Improve update PR text

* Set CI back to main dpf-standalone branches

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: PProfizi <[email protected]>

Author: PProfizi

.ci/build_doc.bat

@@ -4,7 +4,8 @@ call sphinx-apidoc -o ../docs/source/api ../src/ansys ../src/ansys/dpf/core/log.
 ../src/ansys/dpf/core/field_base.py ../src/ansys/dpf/core/cache.py ../src/ansys/dpf/core/misc.py ^
 ../src/ansys/dpf/core/check_version.py ../src/ansys/dpf/core/operators/build.py ../src/ansys/dpf/core/operators/specification.py ^
 ../src/ansys/dpf/core/vtk_helper.py ../src/ansys/dpf/core/label_space.py ../src/ansys/dpf/core/examples/python_plugins/* ^
-../src/ansys/dpf/core/examples/examples.py ../src/ansys/dpf/core/property_fields_container.py ^
+../src/ansys/dpf/core/examples/examples.py ../src/ansys/dpf/gate/* ../src/ansys/dpf/gatebin/* ../src/ansys/grpc/dpf/* ^
+ ../src/ansys/dpf/core/property_fields_container.py ^
  -f --implicit-namespaces --separate  --no-headings
 pushd .
 cd ../docs/
@@ -15,14 +16,14 @@ dir
 
 rem Patch pyVista issue with elemental plots
 
-xcopy  source\examples\04-advanced\02-volume_averaged_stress\sphx_glr_02-volume_averaged_stress_001.png   build\html\_images\sphx_glr_02-volume_averaged_stress_001.png  /y
-xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_001.png                     build\html\_images\sphx_glr_02-fluids_results_001.png          /y
-xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_002.png                     build\html\_images\sphx_glr_02-fluids_results_002.png          /y
-xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_003.png                     build\html\_images\sphx_glr_02-fluids_results_003.png          /y
-xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_004.png                     build\html\_images\sphx_glr_02-fluids_results_004.png          /y
-xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_005.png                     build\html\_images\sphx_glr_02-fluids_results_005.png          /y
-xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_006.png                     build\html\_images\sphx_glr_02-fluids_results_006.png          /y
-xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_007.png                     build\html\_images\sphx_glr_02-fluids_results_007.png          /y
-xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_thumb.png                   build\html\_images\sphx_glr_02-fluids_results_thumb.png        /y
+xcopy  source\examples\04-advanced\02-volume_averaged_stress\sphx_glr_02-volume_averaged_stress_001.png   build\html\_images  /y /f /i
+xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_001.png                     build\html\_images  /y /f /i
+xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_002.png                     build\html\_images  /y /f /i
+xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_003.png                     build\html\_images  /y /f /i
+xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_004.png                     build\html\_images  /y /f /i
+xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_005.png                     build\html\_images  /y /f /i
+xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_006.png                     build\html\_images  /y /f /i
+xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_007.png                     build\html\_images  /y /f /i
+xcopy  source\examples\12-fluids\02-fluids_results\sphx_glr_02-fluids_results_thumb.png                   build\html\_images  /y /f /i
 
 popd

.ci/update_dpf_dependencies.py

@@ -0,0 +1,82 @@
+"""Script to update ansys-dpf-gate, ansys-dpf-gatebin and ansys-grpc-dpf based on repositories
+
+This script should only be used to quickly test changes to any of these dependencies.
+Actual commit of updated code should not occur.
+The GitHub pipelines take care of the actual update in ansys-dpf-core.
+
+Define environment variables to know where to get the code from:
+- "DPFDV_ROOT" defines the DPF repo where ansys-grpc-dpf resides.
+  Will unzip the latest wheel built in DPF/proto/dist/.
+- "ANSYSDPFPYGATE_ROOT" defines where the ansys-dpf-pygate repository resides.
+
+It will update the current repo
+or the repo defined by the environment variable "ANSYSDPFCORE_ROOT" if it exists.
+"""
+import os
+import glob
+import pathlib
+import platform
+import shutil
+import zipfile
+
+
+grpc_path_key = "DPFDV_ROOT"
+gate_path_key = "ANSYSDPFPYGATE_ROOT"
+core_path = pathlib.Path(__file__).parent.parent.resolve()
+if "ANSYSDPFCORE_ROOT" in os.environ:
+    core_path = os.environ["ANSYSDPFCORE_ROOT"]
+
+grpc_path = os.getenv(grpc_path_key, None)
+gate_path = os.getenv(gate_path_key, None)
+
+if grpc_path is not None:
+    # Update ansys-grpc-dpf with latest in proto/dist
+    print("Updating ansys.grpc.dpf")
+    dist_path = os.path.join(grpc_path, "proto", "dist", "*")
+    print(f"from {dist_path}")
+    destination = os.path.join(core_path, "src")
+    print(f"into {destination}")
+    latest_wheel = max(glob.glob(dist_path), key=os.path.getctime)
+    with zipfile.ZipFile(latest_wheel, 'r') as wheel:
+        for file in wheel.namelist():
+            # print(file)
+            if file.startswith('ansys/'):
+                wheel.extract(
+                    file,
+                    path=destination,
+                )
+    print("Done updating ansys-grpc-dpf")
+else:
+    print(f"{grpc_path_key} environment variable is not defined. "
+          "Cannot update ansys-grpc-dpf.")
+
+if gate_path is not None:
+    # Update ansys-dpf-gate
+    print("Updating ansys.dpf.gate")
+    dist_path = os.path.join(gate_path, "ansys-dpf-gate", "ansys")
+    print(f"from {dist_path}")
+    destination = os.path.join(core_path, "src", "ansys")
+    print(f"into {destination}")
+    shutil.copytree(
+        src=dist_path,
+        dst=destination,
+        dirs_exist_ok=True,
+        ignore=lambda directory, contents: ["__pycache__"] if directory[-5:] == "gate" else [],
+    )
+    print("Done updating ansys-dpf-gate")
+
+    # Update ansys-dpf-gatebin
+    print("Updating ansys.dpf.gatebin")
+    dist_path = os.path.join(gate_path, "ansys-dpf-gatebin", "ansys")
+    print(f"from {dist_path}")
+    destination = os.path.join(core_path, "src", "ansys")
+    print(f"into {destination}")
+    shutil.copytree(
+        src=dist_path,
+        dst=destination,
+        dirs_exist_ok=True,
+    )
+    print(f"Done updating ansys-dpf-gatebin for {platform.system()}")
+else:
+    print(f"{gate_path_key} environment variable is not defined. "
+          "Cannot update ansys-dpf-gate or ansys-dpf-gatebin.")

.flake8

@@ -1,7 +1,8 @@
 [flake8]
-exclude = venv, src/ansys/dpf/core/operators, __init__.py, docs/build, .venv
+exclude = venv, src/ansys/dpf/core/operators, __init__.py, docs/build, .venv, src/ansys/dpf/gate, src/ansys/dpf/gatebin, src/ansys/grpc/dpf
 select = W191, W391, E115, E117, E122, E124, E125, E301, W291, W293, E225, E231, E303, E501, F401, F403
-per-file-ignores = __init__.py:F401,F403
+per-file-ignores =
+    __init__.py:F401,F403
 count = True
 max-complexity = 10
 max-line-length = 100

.github/workflows/ci.yml

@@ -14,21 +14,6 @@ on:
         description: 'Suffix of the branch on standalone'
         required: false
         default: ''
-      custom-requirements:
-        description: "Path to requirements.txt file to install"
-        required: false
-        type: string
-        default: 'requirements/requirements_dev.txt'
-      custom-wheels:
-        description: "extra pip --find-links argument to find custom dpf wheels"
-        required: false
-        type: string
-        default: './dpf-standalone/v241/dist'
-      custom-wheels-docker:
-        description: "extra pip --find-links argument to find custom dpf wheels"
-        required: false
-        type: string
-        default: './dpf-standalone/dist'
 
 
 concurrency:
@@ -73,8 +58,6 @@ jobs:
       wheel: true
       wheelhouse: false
       standalone_suffix: ${{ github.event.inputs.standalone_branch_suffix || '' }}
-      custom-requirements: ${{ github.event.inputs.custom-requirements || 'requirements/requirements_dev.txt' }}
-      custom-wheels: ${{ github.event.inputs.custom-wheels || './dpf-standalone/v241/dist' }}
     secrets: inherit
 
   docker_tests:
@@ -83,8 +66,6 @@ jobs:
     with:
       ANSYS_VERSION: "241"
       standalone_suffix: ${{ github.event.inputs.standalone_branch_suffix || '' }}
-      custom-requirements: ${{ github.event.inputs.custom-requirements || 'requirements/requirements_dev.txt' }}
-      custom-wheels: ${{ github.event.inputs.custom-wheels-docker || './dpf-standalone/dist' }}
     secrets: inherit
 
   docker_examples:
@@ -94,8 +75,6 @@ jobs:
       ANSYS_VERSION: "241"
       python_versions: '["3.8"]'
       standalone_suffix: ${{ github.event.inputs.standalone_branch_suffix || '' }}
-      custom-requirements: ${{ github.event.inputs.custom-requirements || 'requirements/requirements_dev.txt' }}
-      custom-wheels: ${{ github.event.inputs.custom-wheels-docker || './dpf-standalone/dist' }}
     secrets: inherit
 
   docs:
@@ -104,8 +83,6 @@ jobs:
     with:
       ANSYS_VERSION: "241"
       standalone_suffix: ${{ github.event.inputs.standalone_branch_suffix || '' }}
-      custom-requirements: ${{ github.event.inputs.custom-requirements || 'requirements/requirements_dev.txt' }}
-      custom-wheels: ${{ github.event.inputs.custom-wheels || './dpf-standalone/v241/dist' }}
       event_name: ${{ github.event_name }}
     secrets: inherit
 
@@ -129,8 +106,6 @@ jobs:
       ANSYS_VERSION: "241"
       python_versions: '["3.8"]'
       standalone_suffix: ${{ github.event.inputs.standalone_branch_suffix || '' }}
-      custom-requirements: ${{ github.event.inputs.custom-requirements || 'requirements/requirements_dev.txt' }}
-      custom-wheels: ${{ github.event.inputs.custom-wheels || './dpf-standalone/v241/dist' }}
     secrets: inherit
 
   retro_232:
@@ -142,7 +117,6 @@ jobs:
       python_versions: '["3.8"]'
       DOCSTRING: false
       standalone_suffix: ''
-      custom-wheels: './dpf-standalone/v232/dist --pre'
     secrets: inherit
 
   retro_231:
@@ -153,7 +127,6 @@ jobs:
       ANSYS_VERSION: "231"
       python_versions: '["3.8"]'
       DOCSTRING: false
-      custom-wheels: './.github'
     secrets: inherit
 
   retro_222:
@@ -164,18 +137,6 @@ jobs:
       ANSYS_VERSION: "222"
       python_versions: '["3.8"]'
       DOCSTRING: false
-      custom-wheels: './.github'
-    secrets: inherit
-
-  retro_221:
-    name: "retro 221"
-    if: startsWith(github.head_ref, 'master') || github.event.action == 'ready_for_review' || !github.event.pull_request.draft
-    uses: ./.github/workflows/tests.yml
-    with:
-      ANSYS_VERSION: "221"
-      python_versions: '["3.8"]'
-      DOCSTRING: false
-      custom-wheels: './.github'
     secrets: inherit
 
   pydpf-post:
@@ -186,7 +147,5 @@ jobs:
       ANSYS_VERSION: "241"
       post_branch: "master"
       standalone_suffix: ${{ github.event.inputs.standalone_branch_suffix || '' }}
-      custom-requirements: ${{ github.event.inputs.custom-requirements || 'requirements/requirements_dev.txt' }}
-      custom-wheels: ${{ github.event.inputs.custom-wheels || './dpf-standalone/v241/dist' }}
       test_docstrings: "true"
     secrets: inherit

.github/workflows/ci_release.yml

@@ -107,15 +107,6 @@ jobs:
       DOCSTRING: false
     secrets: inherit
 
-  retro_221:
-    name: "retro 221"
-    uses: ./.github/workflows/tests.yml
-    with:
-      ANSYS_VERSION: "221"
-      python_versions: '["3.8"]'
-      DOCSTRING: false
-    secrets: inherit
-
   pydpf-post_241:
     name: "PyDPF-Post with 241"
     uses: ./.github/workflows/pydpf-post.yml
@@ -146,13 +137,6 @@ jobs:
       ANSYS_VERSION: "222"
     secrets: inherit
 
-  pydpf-post_221:
-    name: "PyDPF-Post with 221"
-    uses: ./.github/workflows/pydpf-post.yml
-    with:
-      ANSYS_VERSION: "221"
-    secrets: inherit
-
   gate:
     name: "gate"
     uses: ./.github/workflows/gate.yml
@@ -182,7 +166,7 @@ jobs:
   draft_release:
     name: "Draft Release"
     if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
-    needs: [style, tests, docs, examples, retro_232, retro_231, retro_222, retro_221, gate, docker_tests]
+    needs: [style, tests, docs, examples, retro_232, retro_231, retro_222, gate, docker_tests]
     runs-on: ubuntu-latest
     steps:
       - name: "Download artifacts"

.github/workflows/docs.yml

@@ -17,16 +17,6 @@ on:
         required: false
         type: string
         default: ''
-      custom-requirements:
-        description: "Path to requirements.txt file to install"
-        required: false
-        type: string
-        default: ''
-      custom-wheels:
-        description: "extra pip '--find-links XXX' argument to find custom dpf wheels"
-        required: false
-        type: string
-        default: ''
       event_name:
         description: "Name of event calling"
         required: true
@@ -50,16 +40,6 @@ on:
         required: false
         type: string
         default: ''
-      custom-requirements:
-        description: "Path to requirements.txt file to install"
-        required: false
-        type: string
-        default: ''
-      custom-wheels:
-        description: "extra pip --find-links argument to find custom dpf wheels"
-        required: false
-        type: string
-        default: ''
 
 env:
   PACKAGE_NAME: ansys-dpf-core
@@ -103,9 +83,7 @@ jobs:
           install_extras: plotting
           wheel: false
           wheelhouse: false
-          extra-pip-args: ${{ inputs.custom-wheels && format('--find-links {0}', inputs.custom-wheels) || '' }}
           standalone_suffix: ${{ inputs.standalone_suffix }}
-          custom-requirements: ${{ inputs.custom-requirements }}
 
       - name: "Setup headless display"
         uses: pyvista/setup-headless-display-action@v1

.github/workflows/examples.yml

@@ -17,16 +17,6 @@ on:
         required: false
         type: string
         default: ''
-      custom-requirements:
-        description: "Path to requirements.txt file to install"
-        required: false
-        type: string
-        default: ''
-      custom-wheels:
-        description: "extra pip --find-links argument to find custom dpf wheels"
-        required: false
-        type: string
-        default: ''
 # Can be called manually
   workflow_dispatch:
     inputs:
@@ -45,16 +35,6 @@ on:
         required: false
         type: string
         default: ''
-      custom-requirements:
-        description: "Path to requirements.txt file to install"
-        required: false
-        type: string
-        default: ''
-      custom-wheels:
-        description: "extra pip --find-links argument to find custom dpf wheels"
-        required: false
-        type: string
-        default: ''
 
 env:
   PACKAGE_NAME: ansys-dpf-core
@@ -107,9 +87,7 @@ jobs:
           install_extras: plotting
           wheelhouse: false
           wheel: false
-          extra-pip-args: ${{ inputs.custom-wheels && format('--find-links {0}', inputs.custom-wheels) || '' }}
           standalone_suffix: ${{ inputs.standalone_suffix }}
-          custom-requirements: ${{ inputs.custom-requirements }}
 
       - name: "Prepare Testing Environment"
         uses: ansys/pydpf-actions/[email protected]