Skip to content

Commit 1e1f647

Browse files
committed
Launcher, reproduced error in test
This is the error: Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set
1 parent 7680a4b commit 1e1f647

File tree

1 file changed

+13
-0
lines changed

1 file changed

+13
-0
lines changed

tests/test_spark_launcher.py

+13
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,16 @@ def spark():
1111

1212
def test_spark_session_is_ok(spark):
1313
assert isinstance(spark.session, SparkSession)
14+
15+
16+
def test_bug_different_versions_error(spark):
17+
"""
18+
Test intended to reproduce an error
19+
20+
Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with different
21+
minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set
22+
"""
23+
example_cols = ["column1"]
24+
example_data = [["value1"], ["value2"]]
25+
df = spark.session.createDataFrame(example_data, example_cols)
26+
df.show()

0 commit comments

Comments
 (0)