GH-45129: [Python][C++] Fix usage of deprecated C++ functionality on pyarrow by raulcd · Pull Request #45189 · apache/arrow

raulcd · 2025-01-07T12:32:07Z

Rationale for this change

We are using two C++ deprecated APIs:

we are using decimal instead of smallest_decimal
we are using Arrow:Status on GetRecordBatchReader instead of Arrow::Result

What changes are included in this PR?

Update code to use non deprecated functions.

Are these changes tested?

Yes via CI with existing tests.

Are there any user-facing changes?

No

GitHub Issue: [Python][C++] Fix deprecations when building pyarrow #45129

github-actions · 2025-01-07T12:32:37Z

⚠️ GitHub issue #45129 has been automatically assigned in GitHub to PR creator.

raulcd · 2025-01-07T12:40:31Z

@github-actions crossbow submit -g python

github-actions · 2025-01-07T12:43:15Z

Revision: c1deb19cdb624e1f5e8363fc73c6afe28529270f

Submitted crossbow builds: ursacomputing/crossbow @ actions-d40040ac1a

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

raulcd · 2025-01-07T14:32:14Z

The cython2 failure is related:

  In file included from /build/python/pyarrow/src/arrow/python/async.h:22,
                   from /build/python/build/temp.linux-x86_64-cpython-310/_parquet.cpp:830:
  /build/python/pyarrow/src/arrow/python/common.h: In instantiation of 'arrow::py::SmartPtrNoGIL<SmartPtr, Ts>& arrow::py::SmartPtrNoGIL<SmartPtr, Ts>::operator=(V&&) [with V = std::unique_ptr<arrow::RecordBatchReader>&; SmartPtr = std::unique_ptr; Ts = {arrow::RecordBatchReader}]':
  /build/python/build/temp.linux-x86_64-cpython-310/_parquet.cpp:25556:56:   required from here
  /build/python/pyarrow/src/arrow/python/common.h:263:20: error: use of deleted function 'std::unique_ptr<_Tp, _Dp>& std::unique_ptr<_Tp, _Dp>::operator=(const std::unique_ptr<_Tp, _Dp>&) [with _Tp = arrow::RecordBatchReader; _Dp = std::default_delete<arrow::RecordBatchReader>]'
    263 |     Base::operator=(std::forward<V>(v));
        |     ~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
  In file included from /opt/conda/envs/arrow/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/memory:78,
                   from /build/python/build/temp.linux-x86_64-cpython-310/_parquet.cpp:770:
  /opt/conda/envs/arrow/lib/gcc/x86_64-conda-linux-gnu/13.3.0/include/c++/bits/unique_ptr.h:523:19: note: declared here
    523 |       unique_ptr& operator=(const unique_ptr&) = delete;
        |                   ^~~~~~~~

the following patch would work but I don't think this should be the fix, as I am pretty sure std::forward should be used here for other operators:

diff --git a/python/pyarrow/src/arrow/python/common.h b/python/pyarrow/src/arrow/python/common.h
index 4a7886695e..30acf0d6ed 100644
--- a/python/pyarrow/src/arrow/python/common.h
+++ b/python/pyarrow/src/arrow/python/common.h
@@ -260,7 +260,7 @@ class SmartPtrNoGIL : public SmartPtr<Ts...> {
   template <typename V>
   SmartPtrNoGIL& operator=(V&& v) {
     auto release_guard = optional_gil_release();
-    Base::operator=(std::forward<V>(v));
+    Base::operator=(std::move(v));
     return *this;
   }

raulcd · 2025-01-07T14:42:10Z

@github-actions crossbow submit test-conda-python-3.10-cython2

github-actions · 2025-01-07T14:44:23Z

Revision: 07f7acdb3c41b8299ebe2903c39ab6c3c562faed

Submitted crossbow builds: ursacomputing/crossbow @ actions-3db166d478

Task	Status
test-conda-python-3.10-cython2

raulcd · 2025-01-07T15:22:16Z

@github-actions crossbow submit -g python

raulcd · 2025-01-07T15:29:16Z

@github-actions crossbow submit -g python

github-actions · 2025-01-07T15:31:58Z

Revision: babbe9fa049f5fcc1144f45150cae737aa8425bf

Submitted crossbow builds: ursacomputing/crossbow @ actions-f876b9ef9d

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

python/pyarrow/_parquet.pxd

raulcd · 2025-01-08T09:47:01Z

@github-actions crossbow submit test-conda-python-3.10-cython2

github-actions · 2025-01-08T09:49:11Z

Revision: f292cc101be901b98b0d9357db5b564efee76de7

Submitted crossbow builds: ursacomputing/crossbow @ actions-58dca82b3c

Task	Status
test-conda-python-3.10-cython2

pitrou · 2025-01-13T10:39:28Z

Ok, so the shared_ptr version works but not the unique_ptr one? In this case, we can keep using the shared_ptr approach.

pitrou · 2025-01-13T10:39:55Z

By the way, why do we want to support Cython 2? @jorisvandenbossche

jorisvandenbossche · 2025-01-13T10:44:57Z

I would say we can drop the cython 2 support.

raulcd · 2025-01-13T11:13:11Z

I would say we can drop the cython 2 support.

I created an issue to track that as it is not the first time we say the same. I'll do and then I'll rebase this one. I'll ping to merge once CI is green again.

…ty on pyarrow

…mentation of SmartPtrNoGIL operator

raulcd · 2025-01-17T12:02:19Z

@github-actions crossbow submit -g python

github-actions · 2025-01-17T12:04:52Z

Revision: 0a1d214

Submitted crossbow builds: ursacomputing/crossbow @ actions-74244302df

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

raulcd · 2025-01-17T12:44:34Z

CI failures are unrelated, they are failing on main and I've opened individual issues to track them.

@pitrou I've rebased after the required cython bump. This should be ready to review/merge now that there's no CI failure.

conbench-apache-arrow · 2025-01-22T00:11:57Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 984519d.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.

github-actions bot added Component: Python awaiting committer review Awaiting committer review labels Jan 7, 2025

raulcd force-pushed the GH-45129 branch from 2fe23a2 to c1deb19 Compare January 7, 2025 14:28

raulcd marked this pull request as ready for review January 7, 2025 15:22

This comment was marked as outdated.

Sign in to view

raulcd requested a review from pitrou January 7, 2025 16:38

pitrou reviewed Jan 8, 2025

View reviewed changes

python/pyarrow/_parquet.pxd Outdated Show resolved Hide resolved

raulcd mentioned this pull request Jan 13, 2025

[Python] Raise minimum supported cython to >=3 #45237

Closed

raulcd added 4 commits January 17, 2025 12:11

apacheGH-45129: [Python][C++] Fix usage of deprecated C++ functionali…

736829d

…ty on pyarrow

We seem to require a shared_ptr insted of unique_ptr due to our imple…

a38d756

…mentation of SmartPtrNoGIL operator

This sould be SharedPtrNoGIL as we are usng no_gil

4b2e3d4

Revert to unique_ptr as returned from C++

0a1d214

raulcd force-pushed the GH-45129 branch from f292cc1 to 0a1d214 Compare January 17, 2025 11:11

raulcd requested a review from pitrou January 17, 2025 12:44

pitrou approved these changes Jan 21, 2025

View reviewed changes

raulcd merged commit 984519d into apache:main Jan 21, 2025
14 checks passed

raulcd removed the awaiting committer review Awaiting committer review label Jan 21, 2025

raulcd mentioned this pull request Jan 21, 2025

[Python][C++] Fix deprecations when building pyarrow #45129

Closed

raulcd deleted the GH-45129 branch January 21, 2025 17:26

Conversation

raulcd commented Jan 7, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Jan 7, 2025

Uh oh!

raulcd commented Jan 7, 2025

Uh oh!

github-actions bot commented Jan 7, 2025

Uh oh!

raulcd commented Jan 7, 2025

Uh oh!

raulcd commented Jan 7, 2025

Uh oh!

github-actions bot commented Jan 7, 2025

Uh oh!

raulcd commented Jan 7, 2025

Uh oh!

This comment was marked as outdated.

raulcd commented Jan 7, 2025

Uh oh!

github-actions bot commented Jan 7, 2025

Uh oh!

Uh oh!

raulcd commented Jan 8, 2025

Uh oh!

github-actions bot commented Jan 8, 2025

Uh oh!

pitrou commented Jan 13, 2025

Uh oh!

pitrou commented Jan 13, 2025

Uh oh!

jorisvandenbossche commented Jan 13, 2025

Uh oh!

raulcd commented Jan 13, 2025

Uh oh!

raulcd commented Jan 17, 2025

Uh oh!

github-actions bot commented Jan 17, 2025

Uh oh!

raulcd commented Jan 17, 2025

Uh oh!

Uh oh!

conbench-apache-arrow bot commented Jan 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

raulcd commented Jan 7, 2025 •

edited by github-actions bot

Loading