[QST]The GPU does not seem to be called when submitting a Spark application【DataFrameExample,LR,GBDT etc】

[YeahNew]

What is your question?
Hello, when I use the IntelliJIdea to package LogisticRegressionSummaryExample.scala (in $SPARK_HOME/exampls/src/main/scala/org/apache/spark/examples/ml, and the DataFrameExample and GradientBoostedTreeRegressorExample have also been tried) and submit it to spark standalone mode, and add the configuration parameters for calling the GPU. In the Spark://master:8080 webUI interface, I can see that the GPU is used, and I can also find that the GPU resource is called by the application from the historyServer webUI. But it is strange that if I use nvidia-smi to check the GPU on the node terminal, I find that the GPU does not seem to be called. Then I removed the parameters for configuring the GPU to be called, and only used the CPU, and found that the running time of the application did not differ much

I changed the input data file, and the data size is about 1G.
the cudf-0.14-cuda10-2.jar and rapids-4-spark_2.12-0.1.0.jar are located at /opt/sparkRapidsPlugin/

in /etc/profile:
#Spark3.0-GPU Accelerator
export SPARK_RAPIDS_DIR=/opt/sparkRapidsPlugin
export SPARK_CUDF_JAR=${SPARK_RAPIDS_DIR}/cudf-0.14-cuda10-2.jar
export SPARK_RAPIDS_PLUGIN_JAR=${SPARK_RAPIDS_DIR}/rapids-4-spark_2.12-0.1.0.jar

The following is my submission case:
./bin/spark-submit --classs org.hik.LogisticRegressionSummaryExampl
--master spark://master:7077
--executor-cores 2
--conf spark.task.cpus 1
--conf spark.executor.resource.gpu.amount=4
--conf spark.executor.resource.gpu.discoveryScript=./examples/src/main/scripts/getGpusResources.sh
--conf spark.task.resource.gpu.amount=1
--conf spark.plugins=com.nvidia.spark.SQLPlugin
--jars ${SPARK_CUDF_JAR}, ${SPARK_RAPIDS_PLUGIN_JAR}, LR.jar

we can see from the webUI that the GPU is called [http://master:8080]

We can also see that the GPU is called from the historyServer webUI [http://master:18080]

However, through the nvidia-smi we can see that the GPU is not being used.

Actually I suspect that the GPU is indeed not called.
I spent 2 days and still did not solve this problem, which made me very confused.
Would you help me solve it? Thank you very much.

[jlowe]

nvidia-smi can be used to see if the GPU is being utilized at a snapshot in time and if an active process has an active device context for a GPU.

A couple of things you can quickly check to verify the RAPIDS Accelerator plugin is active. In the driver startup logs, you should see the following log:

SQLExecPlugin: Installing extensions to enable rapids GPU SQL support. To disable GPU support set `spark.rapids.sql.enabled` to false

You can also visit the SQL tab in the Spark UI to look at the visual description of the query plan. Search for nodes that have Gpu in their name which will indicate its an operation destined for the GPU when the plan is executed.

You can also set the config spark.rapids.sql.explain to NOT_ON_GPU to have the plugin log any operations in a query that are not going to be performed on the GPU. Setting that same config to ALL will log the entire query plan showing what will and will not run on the GPU.

[revans2]

Also be aware that most of the ML operations that you are looking at use RDD to do the processing and that is something we don't currently accelerate. The only thing we might accelerate in the examples you pointed out is reading the input data. And that is only if it is in a type we support, like parquet, orc, or CSV

[YeahNew]

Also be aware that most of the ML operations that you are looking at use RDD to do the processing and that is something we don't currently accelerate. The only thing we might accelerate in the examples you pointed out is reading the input data. And that is only if it is in a type we support, like parquet, orc, or CSV

Yes, my input data file is CSV, which is about 1G. And I change the submission case:
./bin/spark-submit --classs org.hik.LogisticRegressionSummaryExampl
--master spark://master:7077
--executor-cores 2
--conf spark.task.cpus 1
--conf spark.executor.resource.gpu.amount=4
--conf spark.executor.resource.gpu.discoveryScript=./examples/src/main/scripts/getGpusResources.sh
--conf spark.task.resource.gpu.amount=1
--conf spark.plugins=com.nvidia.spark.SQLPlugin
--conf spark.rapids.sql.enabled=true
--jars ${SPARK_CUDF_JAR}, ${SPARK_RAPIDS_PLUGIN_JAR}, LR.jar

But it doesn't seem to work，I found that there are no Gpu nodes from the SQL tab in the Spark UI.
Of course, you can see the used GPU in the resource column of Executors as shown in fig.3 above.
It's weird.

[jlowe]

Do you see the startup log message showing the plugin is initializing? Also run with --conf spark.rapids.sql.explain=ALL and examine the logs to see the explain output from the plugin for the query. That will be useful in debugging why parts are not being applied to the GPU.

[YeahNew]

Do you see the startup log message showing the plugin is initializing? Also run with --conf spark.rapids.sql.explain=ALL and examine the logs to see the explain output from the plugin for the query. That will be useful in debugging why parts are not being applied to the GPU.

No, I can not find the plugin is initializing in the startup log message even if I run with --conf spark.rapids.sql.explain=ALL and --conf spark.rapids.sql.enabled='true'. I should have configured this plugin successfully, so the WebUI interface of Spark and HistoryServer can see the GPU.
I'm very confused as to what went wrong? Would you like to show a copy of your configuration information and submitted cases on GPU?

[revans2]

GPU scheduling is built into spark. You don't need this plugin to be able to schedule a job with a GPU, so if you see GPU resources on the UI it does not indicate if you successfully installed the accelerator or not.

[YeahNew]

GPU scheduling is built into spark. You don't need this plugin to be able to schedule a job with a GPU, so if you see GPU resources on the UI it does not indicate if you successfully installed the accelerator or not.

Thank you, I have solved the question successfully.