[MLIR] Existing Python bindings issues

[makslevental]

I plan to propose a partial rewrite/refactor of the Python bindings later this year. This github issue collects up some of the problems I've noticed/identified and some associated PRs/issues

1. Decoupled/divorced lifetimes between C++/C/Python; succinctly this revolves around liveOperations
   • [mlir][py] invalidate nested operations when parent is deleted #93339
   • [mlir][python] Tracking of live operations fails. #69730
   • [mlir][python] Tracking of PyOperations references in liveOperations map breaks down in reasonable scenarios #92344
   • Python bidings don't know that modules are operations #93337
   • [mlir][python] Running passes can invalidate objects and lead to crashes #63916
   • [MLIR:Python] Fix race on PyOperations. #139721
   • half-hearted attempt fix via removal: [mlir][python][wip] remove liveOperations #92631
   • unfinished weak-ref system: [mlir][operation] weak refs #97340
2. The CMake build system is "mysterious and important", which is to say it's abstruse and inflexible; succinctly this affects modularity vis-a-vis piecemeal dialect registration/distribution
   • [mlir:python] Add a mechanism to register upstream dialects from Python #74245

   • llvm-project/mlir/examples/standalone/python/CMakeLists.txt

     Line 43 in 5e83a5b

     # TODO: Remove this in favor of showing fine grained registration once

   • [mlir][python] allow upstream dialect registration #74252
   • [mlir][python] add pyproject.toml #124878
   • [MLIR] Enabling external dialects to be shared libs for C API users #108253
   • [mlir][python] set the registry free #72477
   • [mlir] Add option to disable MLIR Python dev package configuration. #117934
3. Upstream and downstream bindings constitute a two-tier system, with upstream being first class and downstream being second class; specifically I'm referring to the difference between upstream types/attrs and downstream types/attrs
   • which gets at a broader problem/issue/limitation: none of the upstream impl code is reusable because none of the headers are in mlir/include. Note, this is not malicious/accidental, but is connected to the aforementioned build system issues
4. Minor/ungrouped outstanding issues:
   • [mlir-python] Emit only dialect EnumAttr registrations. #117918
   • [mlir][python] C++ API demo #71133, which simply alludes to a plausible alternative structure/implementation

[llvmbot]

@llvm/issue-subscribers-mlir

Author: Maksim Levental (makslevental)

I plan to propose a partial rewrite/refactor of the Python bindings later this year. This github issue collects up some of the problems I've noticed/identified and some associated PRs/issues

1. Decoupled/divorced lifetimes between C++/C/Python; succinctly this revolves around liveOperations
   • [mlir][py] invalidate nested operations when parent is deleted #93339
   • [mlir][python] Tracking of live operations fails. #69730
   • [mlir][python] Tracking of PyOperations references in liveOperations map breaks down in reasonable scenarios #92344
   • Python bidings don't know that modules are operations #93337
   • [mlir][python] Running passes can invalidate objects and lead to crashes #63916
   • half-hearted attempt fix via removal: [mlir][python][wip] remove liveOperations #92631
   • unfinished weak-ref system: [mlir][operation] weak refs #97340
2. The CMake build system is "mysterious and important", which is to say it's abstruse and inflexible; succinctly this affects modularity vis-a-vis piecemeal dialect registration/distribution
   • [mlir:python] Add a mechanism to register upstream dialects from Python #74245

   • llvm-project/mlir/examples/standalone/python/CMakeLists.txt

     Line 43 in 5e83a5b

     # TODO: Remove this in favor of showing fine grained registration once

   • [mlir][python] allow upstream dialect registration #74252
   • [mlir][python] add pyproject.toml #124878
   • [MLIR] Enabling external dialects to be shared libs for C API users #108253

[llvmbot]

@llvm/issue-subscribers-mlir-python

Author: Maksim Levental (makslevental)

I plan to propose a partial rewrite/refactor of the Python bindings later this year. This github issue collects up some of the problems I've noticed/identified and some associated PRs/issues

1. Decoupled/divorced lifetimes between C++/C/Python; succinctly this revolves around liveOperations
   • [mlir][py] invalidate nested operations when parent is deleted #93339
   • [mlir][python] Tracking of live operations fails. #69730
   • [mlir][python] Tracking of PyOperations references in liveOperations map breaks down in reasonable scenarios #92344
   • Python bidings don't know that modules are operations #93337
   • [mlir][python] Running passes can invalidate objects and lead to crashes #63916
   • half-hearted attempt fix via removal: [mlir][python][wip] remove liveOperations #92631
   • unfinished weak-ref system: [mlir][operation] weak refs #97340
2. The CMake build system is "mysterious and important", which is to say it's abstruse and inflexible; succinctly this affects modularity vis-a-vis piecemeal dialect registration/distribution
   • [mlir:python] Add a mechanism to register upstream dialects from Python #74245

   • llvm-project/mlir/examples/standalone/python/CMakeLists.txt

     Line 43 in 5e83a5b

     # TODO: Remove this in favor of showing fine grained registration once

   • [mlir][python] allow upstream dialect registration #74252
   • [mlir][python] add pyproject.toml #124878
   • [MLIR] Enabling external dialects to be shared libs for C API users #108253
   • https://github.com/llvm/llvm-project/pull/72477/files
3. Upstream and downstream bindings constitute a two-tier system, with upstream being first class and downstream being second class; specifically I'm referring to the difference between upstream types/attrs and downstream types/attrs
   • which gets at a broader problem/issue/limitation: none of the upstream impl code is reusable because none of the headers are in mlir/include. Note, this is not malicious/accidental, but is connected to the aforementioned build system issues
4. Minor/ungrouped outstanding issues:
   • [mlir-python] Emit only dialect EnumAttr registrations. #117918
   • [mlir][python] C++ API demo #71133, which simply alludes to a plausible alternative structure/implementation