Skip to content
This repository was archived by the owner on Jun 8, 2021. It is now read-only.

Commit

Permalink
test fix
Browse files Browse the repository at this point in the history
  • Loading branch information
seraogianluca committed Jul 10, 2020
1 parent 22b3827 commit 63d6fb6
Show file tree
Hide file tree
Showing 9 changed files with 29 additions and 56 deletions.
2 changes: 1 addition & 1 deletion doc/hadoop.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
1) [Point](#1-point)
2) [Main](#2-main)
3) [Mapper](#3-mapper)
4) [Combine](#4-combiner)
4) [Combiner](#4-combiner)
5) [Reducer](#4-reducer)

## 1. Point
Expand Down
7 changes: 3 additions & 4 deletions scripts/run_spark.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,8 @@
for i in $(seq $2)
do
echo "Run: "$i >> run.txt
spark-submit --master yarn spark.py $1 "out/spark/$3k/$4_$5/centroids_$i" >> run.txt
spark-submit --master yarn spark.py $1 "output.txt" >> run.txt
echo "centroids:" >> run.txt
hadoop fs -get "out/spark/$3k/$4_$5/centroids_$i/part-00000" ./centroids$i.txt
cat ./centroids$i.txt >> run.txt
rm ./centroids$i.txt
cat "output.txt" >> run.txt
echo "" >> run.txt
done
6 changes: 3 additions & 3 deletions scripts/stats.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import glob
import os

path = "../tests/hadoop/no_opt_output_3_7_2.txt"
path = "../tests/spark/100k/output_3_13.txt"

times = []
iteration_time = []
Expand All @@ -13,10 +13,10 @@

for line in file:
if line.startswith("execution time:"):
et = round(float(line.split()[2])/1000, 4)
et = round(float(line.split()[2]), 4)
times.append(et)
if line.startswith("init centroid execution:"):
ic = round(float(line.split()[3])/1000, 4)
ic = round(float(line.split()[5]), 4)
init_centroids.append(ic)
if line.startswith("n_iter"):
i = int(line.split()[1])
Expand Down
11 changes: 11 additions & 0 deletions tests/spark/100k/output_3_13.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -199,3 +199,14 @@ centroids:
8.72294 8.17847 -3.8905
9.81878 -2.61919 5.39548


execution time: 415.1549 s
execution time confidence interval: [299.1694,531.1405]
execution time variance: 23659.4466
init centroids time: 5.0730 s
init centroids confidence interval: [3.7430,6.4030]
init centroids variance: 3.1110
average iterations: 41.90
iteration execution time: 9.9684 s
iteration execution time confidence interval: [8.0919,11.8448]
iteration execution time variance: 6.1925
11 changes: 11 additions & 0 deletions tests/spark/100k/output_3_7.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -137,3 +137,14 @@ centroids:
5.51647 8.24958 -8.63566
6.87136 7.25463 -9.27385


execution time: 417.4038 s
execution time confidence interval: [334.7866,500.0209]
execution time variance: 12004.3157
init centroids time: 5.0867 s
init centroids confidence interval: [4.1647,6.0086]
init centroids variance: 1.4948
average iterations: 50.00
iteration execution time: 8.2463 s
iteration execution time confidence interval: [6.5952,9.8975]
iteration execution time variance: 4.7950
12 changes: 0 additions & 12 deletions tests/spark/1k/output_3_13.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,6 @@ centroids:
8.2191 9.29974 2.31736
9.10867 9.44385 -5.93477


execution time: 40.9729 s
execution time confidence interval: [35.3375,46.6083]
execution time variance: 55.8525
Expand All @@ -209,14 +208,3 @@ average iterations: 11.80
iteration execution time: 3.5078 s
iteration execution time confidence interval: [2.7095,4.3060]
iteration execution time variance: 1.1207

execution time: 40.9729 s
execution time confidence interval: [35.9522,45.9937]
execution time variance: 50.7750
init centroids time: 4.2371 s
init centroids confidence interval: [3.1257,5.3485]
init centroids variance: 2.1725
average iterations: 11.80
iteration execution time: 3.5078 s
iteration execution time confidence interval: [2.7095,4.3060]
iteration execution time variance: 1.1207
12 changes: 0 additions & 12 deletions tests/spark/1k/output_3_7.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,6 @@ centroids:
4.5943 -6.04007 -0.03004
9.67375 4.73747 -0.66995


execution time: 34.2601 s
execution time confidence interval: [29.0245,39.4958]
execution time variance: 48.2103
Expand All @@ -151,14 +150,3 @@ average iterations: 8.00
iteration execution time: 5.0593 s
iteration execution time confidence interval: [3.1232,6.9953]
iteration execution time variance: 6.5924

execution time: 34.2601 s
execution time confidence interval: [29.5955,38.9247]
execution time variance: 43.8276
init centroids time: 3.8501 s
init centroids confidence interval: [2.8745,4.8257]
init centroids variance: 1.6739
average iterations: 8.00
iteration execution time: 5.0593 s
iteration execution time confidence interval: [3.1232,6.9953]
iteration execution time variance: 6.5924
12 changes: 0 additions & 12 deletions tests/spark/1k/output_7_13.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,6 @@ centroids:
5.87694 -6.64158 -6.58053 0.4321 1.43218 -6.82584 -4.78189
6.59617 -9.69672 -0.90496 3.70853 -1.05576 -5.25313 9.2416


execution time: 36.6952 s
execution time confidence interval: [33.6611,39.7292]
execution time variance: 16.1898
Expand All @@ -208,14 +207,3 @@ average iterations: 9.60
iteration execution time: 3.5809 s
iteration execution time confidence interval: [2.9711,4.1907]
iteration execution time variance: 0.6540

execution time: 36.6952 s
execution time confidence interval: [33.9920,39.3983]
execution time variance: 14.7180
init centroids time: 4.0291 s
init centroids confidence interval: [2.9415,5.1167]
init centroids variance: 2.0805
average iterations: 9.60
iteration execution time: 3.5809 s
iteration execution time confidence interval: [2.9711,4.1907]
iteration execution time variance: 0.6540
12 changes: 0 additions & 12 deletions tests/spark/1k/output_7_7.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,6 @@ centroids:
2.28782 5.11654 1.32413 -7.93625 3.1831 3.28441 -8.32611
9.22185 6.12421 6.95811 -8.28856 -0.93157 -3.23746 5.70776


execution time: 39.5095 s
execution time confidence interval: [31.8482,47.1709]
execution time variance: 103.2311
Expand All @@ -149,14 +148,3 @@ average iterations: 10.20
iteration execution time: 3.9319 s
iteration execution time confidence interval: [2.6541,5.2097]
iteration execution time variance: 2.8716

execution time: 39.5095 s
execution time confidence interval: [32.6838,46.3353]
execution time variance: 93.8465
init centroids time: 4.4854 s
init centroids confidence interval: [3.1927,5.7781]
init centroids variance: 2.9390
average iterations: 10.20
iteration execution time: 3.9319 s
iteration execution time confidence interval: [2.6541,5.2097]
iteration execution time variance: 2.8716

0 comments on commit 63d6fb6

Please sign in to comment.