Skip to content

Commit c7b46d4

Browse files
minixalphaHyukjinKwon
authored andcommitted
[SPARK-21877][DEPLOY, WINDOWS] Handle quotes in Windows command scripts
## What changes were proposed in this pull request? All the windows command scripts can not handle quotes in parameter. Run a windows command shell with parameter which has quotes can reproduce the bug: ``` C:\Users\meng\software\spark-2.2.0-bin-hadoop2.7> bin\spark-shell --driver-java-options " -Dfile.encoding=utf-8 " 'C:\Users\meng\software\spark-2.2.0-bin-hadoop2.7\bin\spark-shell2.cmd" --driver-java-options "' is not recognized as an internal or external command, operable program or batch file. ``` Windows recognize "--driver-java-options" as part of the command. All the Windows command script has the following code have the bug. ``` cmd /V /E /C "<other command>" %* ``` We should quote command and parameters like ``` cmd /V /E /C ""<other command>" %*" ``` ## How was this patch tested? Test manually on Windows 10 and Windows 7 We can verify it by the following demo: ``` C:\Users\meng\program\demo>cat a.cmd echo off cmd /V /E /C "b.cmd" %* C:\Users\meng\program\demo>cat b.cmd echo off echo %* C:\Users\meng\program\demo>cat c.cmd echo off cmd /V /E /C ""b.cmd" %*" C:\Users\meng\program\demo>a.cmd "123" 'b.cmd" "123' is not recognized as an internal or external command, operable program or batch file. C:\Users\meng\program\demo>c.cmd "123" "123" ``` With the spark-shell.cmd example, change it to the following code will make the command execute succeed. ``` cmd /V /E /C ""%~dp0spark-shell2.cmd" %*" ``` ``` C:\Users\meng\software\spark-2.2.0-bin-hadoop2.7> bin\spark-shell --driver-java-options " -Dfile.encoding=utf-8 " Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). ... ``` Author: minixalpha <[email protected]> Closes apache#19090 from minixalpha/master.
1 parent 0c03297 commit c7b46d4

7 files changed

+22
-7
lines changed

bin/beeline.cmd

+3-1
Original file line numberDiff line numberDiff line change
@@ -17,4 +17,6 @@ rem See the License for the specific language governing permissions and
1717
rem limitations under the License.
1818
rem
1919

20-
cmd /V /E /C "%~dp0spark-class.cmd" org.apache.hive.beeline.BeeLine %*
20+
rem The outermost quotes are used to prevent Windows command line parse error
21+
rem when there are some quotes in parameters, see SPARK-21877.
22+
cmd /V /E /C ""%~dp0spark-class.cmd" org.apache.hive.beeline.BeeLine %*"

bin/pyspark.cmd

+3-1
Original file line numberDiff line numberDiff line change
@@ -20,4 +20,6 @@ rem
2020
rem This is the entry point for running PySpark. To avoid polluting the
2121
rem environment, it just launches a new cmd to do the real work.
2222

23-
cmd /V /E /C "%~dp0pyspark2.cmd" %*
23+
rem The outermost quotes are used to prevent Windows command line parse error
24+
rem when there are some quotes in parameters, see SPARK-21877.
25+
cmd /V /E /C ""%~dp0pyspark2.cmd" %*"

bin/run-example.cmd

+4-1
Original file line numberDiff line numberDiff line change
@@ -19,4 +19,7 @@ rem
1919

2020
set SPARK_HOME=%~dp0..
2121
set _SPARK_CMD_USAGE=Usage: ./bin/run-example [options] example-class [example args]
22-
cmd /V /E /C "%~dp0spark-submit.cmd" run-example %*
22+
23+
rem The outermost quotes are used to prevent Windows command line parse error
24+
rem when there are some quotes in parameters, see SPARK-21877.
25+
cmd /V /E /C ""%~dp0spark-submit.cmd" run-example %*"

bin/spark-class.cmd

+3-1
Original file line numberDiff line numberDiff line change
@@ -20,4 +20,6 @@ rem
2020
rem This is the entry point for running a Spark class. To avoid polluting
2121
rem the environment, it just launches a new cmd to do the real work.
2222

23-
cmd /V /E /C "%~dp0spark-class2.cmd" %*
23+
rem The outermost quotes are used to prevent Windows command line parse error
24+
rem when there are some quotes in parameters, see SPARK-21877.
25+
cmd /V /E /C ""%~dp0spark-class2.cmd" %*"

bin/spark-shell.cmd

+3-1
Original file line numberDiff line numberDiff line change
@@ -20,4 +20,6 @@ rem
2020
rem This is the entry point for running Spark shell. To avoid polluting the
2121
rem environment, it just launches a new cmd to do the real work.
2222

23-
cmd /V /E /C "%~dp0spark-shell2.cmd" %*
23+
rem The outermost quotes are used to prevent Windows command line parse error
24+
rem when there are some quotes in parameters, see SPARK-21877.
25+
cmd /V /E /C ""%~dp0spark-shell2.cmd" %*"

bin/spark-submit.cmd

+3-1
Original file line numberDiff line numberDiff line change
@@ -20,4 +20,6 @@ rem
2020
rem This is the entry point for running Spark submit. To avoid polluting the
2121
rem environment, it just launches a new cmd to do the real work.
2222

23-
cmd /V /E /C "%~dp0spark-submit2.cmd" %*
23+
rem The outermost quotes are used to prevent Windows command line parse error
24+
rem when there are some quotes in parameters, see SPARK-21877.
25+
cmd /V /E /C ""%~dp0spark-submit2.cmd" %*"

bin/sparkR.cmd

+3-1
Original file line numberDiff line numberDiff line change
@@ -20,4 +20,6 @@ rem
2020
rem This is the entry point for running SparkR. To avoid polluting the
2121
rem environment, it just launches a new cmd to do the real work.
2222

23-
cmd /V /E /C "%~dp0sparkR2.cmd" %*
23+
rem The outermost quotes are used to prevent Windows command line parse error
24+
rem when there are some quotes in parameters, see SPARK-21877.
25+
cmd /V /E /C ""%~dp0sparkR2.cmd" %*"

0 commit comments

Comments
 (0)