[SPARK-32435][PYTHON] Remove heapq3 port from Python 3

### What changes were proposed in this pull request? This PR removes the manual port of `heapq3.py` introduced from SPARK-3073. The main reason of this was to support Python 2.6 and 2.7 because Python 2's `heapq.merge()` doesn't not support `key` and `reverse`. See - https://docs.python.org/2/library/heapq.html#heapq.merge in Python 2 - https://docs.python.org/3.8/library/heapq.html#heapq.merge in Python 3 Since we dropped the Python 2 at SPARK-32138, we can remove this away. ### Why are the changes needed? To remove unnecessary codes. Also, we can leverage bug fixes made in Python 3.x at `heapq`. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Existing tests should cover. I locally ran and verified: ```bash ./python/run-tests --python-executable=python3 --testname="pyspark.tests.test_shuffle" ./python/run-tests --python-executable=python3 --testname="pyspark.shuffle ExternalSorter" ./python/run-tests --python-executable=python3 --testname="pyspark.tests.test_rdd RDDTests.test_external_group_by_key" ``` Closes apache#29229 from HyukjinKwon/SPARK-32435. Authored-by: HyukjinKwon <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>
lyft · Jul 27, 2020 · a82aee0 · a82aee0
1 parent 6ab29b3
commit a82aee0
Show file tree

Hide file tree

Showing 9 changed files with 5 additions and 1,232 deletions.
diff --git a/LICENSE b/LICENSE
@@ -222,7 +222,6 @@ external/spark-ganglia-lgpl/src/main/java/com/codahale/metrics/ganglia/GangliaRe
 Python Software Foundation License
 ----------------------------------
 
-pyspark/heapq3.py
 python/docs/source/_static/copybutton.js
 
 BSD 3-Clause

diff --git a/LICENSE-binary b/LICENSE-binary
@@ -557,12 +557,6 @@ jakarta.ws.rs:jakarta.ws.rs-api https://github.com/eclipse-ee4j/jaxrs-api
 org.glassfish.hk2.external:jakarta.inject
 
 
-Python Software Foundation License
-----------------------------------
-
-pyspark/heapq3.py
-
-
 Public Domain
 -------------
 

diff --git a/dev/.rat-excludes b/dev/.rat-excludes
@@ -49,7 +49,6 @@ jsonFormatter.min.js
 .*log
 pyspark-coverage-site/*
 cloudpickle/*
-heapq3.py
 join.py
 SparkExprTyper.scala
 SparkILoop.scala

diff --git a/dev/tox.ini b/dev/tox.ini
@@ -16,4 +16,4 @@
 [pycodestyle]
 ignore=E226,E241,E305,E402,E722,E731,E741,W503,W504
 max-line-length=100
-exclude=python/pyspark/cloudpickle/*.py,heapq3.py,shared.py,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*
+exclude=python/pyspark/cloudpickle/*.py,shared.py,python/docs/source/conf.py,work/*/*.py,python/.eggs/*,dist/*,.git/*
diff --git a/licenses-binary/LICENSE-heapq.txt b/licenses-binary/LICENSE-heapq.txt
diff --git a/licenses/LICENSE-heapq.txt b/licenses/LICENSE-heapq.txt
diff --git a/python/pylintrc b/python/pylintrc
@@ -29,7 +29,7 @@ profile=no
 
 # Add files or directories to the ignoreList. They should be base names, not
 # paths.
-ignore=pyspark.heapq3
+#ignore=
 
 # Pickle collected data for later comparisons.
 persistent=yes