Skip to content

Commit 7c7745f

Browse files
yaooqinndongjoon-hyun
authored andcommitted
[SPARK-50917][EXAMPLES] Add Pi Scala example to work both for Connect and Classic
### What changes were proposed in this pull request? This PR adds a SparkDataFramePi Scala example to work both for Connect and Classic ### Why are the changes needed? The SparkPi example, mostly as the first step for users to get to know Spark, should be able to run on Spark Connect mode. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? Manually build and test ```log bin/spark-submit --remote 'sc://localhost' --class org.apache.spark.examples.sql.SparkDataFramePi examples/jars/spark-examples_2.13-4.1.0-SNAPSHOT.jar WARNING: Using incubator modules: jdk.incubator.vector 25/01/23 15:00:03 INFO BaseAllocator: Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true. 25/01/23 15:00:03 INFO DefaultAllocationManagerOption: allocation manager type not specified, using netty as the default type 25/01/23 15:00:03 INFO CheckAllocator: Using DefaultAllocationManager at memory/netty/DefaultAllocationManagerFactory.class Pi is roughly 3.1388756943784717 25/01/23 15:00:04 INFO ShutdownHookManager: Shutdown hook called 25/01/23 15:00:04 INFO ShutdownHookManager: Deleting directory /private/var/folders/84/dgr9ykwn6yndcmq1kjxqvk200000gn/T/spark-25ed842e-5888-47ce-bb0b-442385d643cb ``` ### Was this patch authored or co-authored using generative AI tooling? no Closes #49617 from yaooqinn/SPARK-50917. Authored-by: Kent Yao <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit e823afa) Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 2de9be2 commit 7c7745f

File tree

1 file changed

+42
-0
lines changed

1 file changed

+42
-0
lines changed
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
// scalastyle:off println
19+
package org.apache.spark.examples.sql
20+
21+
import org.apache.spark.sql.SparkSession
22+
import org.apache.spark.sql.functions._
23+
24+
/** Computes an approximation to pi with SparkSession/DataFrame APIs */
25+
object SparkDataFramePi {
26+
def main(args: Array[String]): Unit = {
27+
val spark = SparkSession
28+
.builder()
29+
.appName("Spark DataFrame Pi")
30+
.getOrCreate()
31+
import spark.implicits._
32+
val slices = if (args.length > 0) args(0).toInt else 2
33+
val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
34+
val count = spark.range(0, n, 1, slices)
35+
.select((pow(rand() * 2 - 1, lit(2)) + pow(rand() * 2 - 1, lit(2))).as("v"))
36+
.where($"v" <= 1)
37+
.count()
38+
println(s"Pi is roughly ${4.0 * count / (n - 1)}")
39+
spark.stop()
40+
}
41+
}
42+
// scalastyle:on println

0 commit comments

Comments
 (0)