Skip to content

Commit 8e31554

Browse files
LuciferYangHyukjinKwon
authored andcommitted
[SPARK-40742][CORE][SQL] Fix Java compilation warnings related to generic type
### What changes were proposed in this pull request? This pr aims to fix following Java compilation warnings related to generic type: ``` 2022-10-08T01:43:33.6487078Z /home/runner/work/spark/spark/core/src/main/java/org/apache/spark/SparkThrowable.java:54: warning: [rawtypes] found raw type: HashMap 2022-10-08T01:43:33.6487456Z return new HashMap(); 2022-10-08T01:43:33.6487682Z ^ 2022-10-08T01:43:33.6487957Z missing type arguments for generic class HashMap<K,V> 2022-10-08T01:43:33.6488617Z where K,V are type-variables: 2022-10-08T01:43:33.6488911Z K extends Object declared in class HashMap 2022-10-08T01:43:33.6489211Z V extends Object declared in class HashMap 2022-10-08T01:50:21.5951932Z /home/runner/work/spark/spark/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsAtomicPartitionManagement.java:55: warning: [rawtypes] found raw type: Map 2022-10-08T01:50:21.5999993Z createPartitions(new InternalRow[]{ident}, new Map[]{properties}); 2022-10-08T01:50:21.6000343Z ^ 2022-10-08T01:50:21.6000642Z missing type arguments for generic class Map<K,V> 2022-10-08T01:50:21.6001272Z where K,V are type-variables: 2022-10-08T01:50:21.6001569Z K extends Object declared in interface Map 2022-10-08T01:50:21.6002109Z V extends Object declared in interface Map 2022-10-08T01:50:21.6006655Z /home/runner/work/spark/spark/sql/catalyst/src/main/java/org/apache/spark/sql/connector/util/V2ExpressionSQLBuilder.java:216: warning: [rawtypes] found raw type: Literal 2022-10-08T01:50:21.6007121Z protected String visitLiteral(Literal literal) { 2022-10-08T01:50:21.6007395Z ^ 2022-10-08T01:50:21.6007673Z missing type arguments for generic class Literal<T> 2022-10-08T01:50:21.6008032Z where T is a type-variable: 2022-10-08T01:50:21.6008324Z T extends Object declared in interface Literal 2022-10-08T01:50:21.6008785Z /home/runner/work/spark/spark/sql/catalyst/src/main/java/org/apache/spark/sql/util/NumericHistogram.java:56: warning: [rawtypes] found raw type: Comparable 2022-10-08T01:50:21.6009223Z public static class Coord implements Comparable { 2022-10-08T01:50:21.6009503Z ^ 2022-10-08T01:50:21.6009791Z missing type arguments for generic class Comparable<T> 2022-10-08T01:50:21.6010137Z where T is a type-variable: 2022-10-08T01:50:21.6010433Z T extends Object declared in interface Comparable 2022-10-08T01:50:21.6010976Z /home/runner/work/spark/spark/sql/catalyst/src/main/java/org/apache/spark/sql/util/NumericHistogram.java:191: warning: [unchecked] unchecked method invocation: method sort in class Collections is applied to given types 2022-10-08T01:50:21.6011474Z Collections.sort(tmp_bins); 2022-10-08T01:50:21.6011714Z ^ 2022-10-08T01:50:21.6012050Z required: List<T> 2022-10-08T01:50:21.6012296Z found: ArrayList<Coord> 2022-10-08T01:50:21.6012604Z where T is a type-variable: 2022-10-08T01:50:21.6012926Z T extends Comparable<? super T> declared in method <T>sort(List<T>) 2022-10-08T02:13:38.0769617Z /home/runner/work/spark/spark/sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/OperationManager.java:85: warning: [rawtypes] found raw type: AbstractWriterAppender 2022-10-08T02:13:38.0770287Z AbstractWriterAppender ap = new LogDivertAppender(this, OperationLog.getLoggingLevel(loggingMode)); 2022-10-08T02:13:38.0770645Z ^ 2022-10-08T02:13:38.0770947Z missing type arguments for generic class AbstractWriterAppender<M> 2022-10-08T02:13:38.0771330Z where M is a type-variable: 2022-10-08T02:13:38.0771665Z M extends WriterManager declared in class AbstractWriterAppender 2022-10-08T02:13:38.0774487Z /home/runner/work/spark/spark/sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/LogDivertAppender.java:268: warning: [rawtypes] found raw type: Layout 2022-10-08T02:13:38.0774940Z Layout l = ap.getLayout(); 2022-10-08T02:13:38.0775173Z ^ 2022-10-08T02:13:38.0775441Z missing type arguments for generic class Layout<T> 2022-10-08T02:13:38.0775849Z where T is a type-variable: 2022-10-08T02:13:38.0776359Z T extends Serializable declared in interface Layout 2022-10-08T02:19:55.0035795Z [WARNING] /home/runner/work/spark/spark/connector/avro/src/main/java/org/apache/spark/sql/avro/SparkAvroKeyOutputFormat.java:56:17: [rawtypes] found raw type: SparkAvroKeyRecordWriter 2022-10-08T02:19:55.0037287Z [WARNING] /home/runner/work/spark/spark/connector/avro/src/main/java/org/apache/spark/sql/avro/SparkAvroKeyOutputFormat.java:56:13: [unchecked] unchecked call to SparkAvroKeyRecordWriter(Schema,GenericData,CodecFactory,OutputStream,int,Map<String,String>) as a member of the raw type SparkAvroKeyRecordWriter 2022-10-08T02:19:55.0038442Z [WARNING] /home/runner/work/spark/spark/connector/avro/src/main/java/org/apache/spark/sql/avro/SparkAvroKeyOutputFormat.java:75:31: [rawtypes] found raw type: DataFileWriter 2022-10-08T02:19:55.0039370Z [WARNING] /home/runner/work/spark/spark/connector/avro/src/main/java/org/apache/spark/sql/avro/SparkAvroKeyOutputFormat.java:75:27: [unchecked] unchecked call to DataFileWriter(DatumWriter<D>) as a member of the raw type DataFileWriter ``` ### Why are the changes needed? Fix Java compilation warnings. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions. Closes apache#38198 from LuciferYang/fix-java-warn. Lead-authored-by: yangjie01 <[email protected]> Co-authored-by: YangJie <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
1 parent 6c182da commit 8e31554

File tree

8 files changed

+20
-16
lines changed

8 files changed

+20
-16
lines changed

connector/avro/src/main/java/org/apache/spark/sql/avro/SparkAvroKeyOutputFormat.java

+3-2
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
import org.apache.avro.file.CodecFactory;
2626
import org.apache.avro.file.DataFileWriter;
2727
import org.apache.avro.generic.GenericData;
28+
import org.apache.avro.generic.GenericDatumWriter;
2829
import org.apache.avro.generic.GenericRecord;
2930
import org.apache.avro.mapred.AvroKey;
3031
import org.apache.avro.mapreduce.AvroKeyOutputFormat;
@@ -53,7 +54,7 @@ protected RecordWriter<AvroKey<GenericRecord>, NullWritable> create(
5354
CodecFactory compressionCodec,
5455
OutputStream outputStream,
5556
int syncInterval) throws IOException {
56-
return new SparkAvroKeyRecordWriter(
57+
return new SparkAvroKeyRecordWriter<>(
5758
writerSchema, dataModel, compressionCodec, outputStream, syncInterval, metadata);
5859
}
5960
}
@@ -72,7 +73,7 @@ class SparkAvroKeyRecordWriter<T> extends RecordWriter<AvroKey<T>, NullWritable>
7273
OutputStream outputStream,
7374
int syncInterval,
7475
Map<String, String> metadata) throws IOException {
75-
this.mAvroFileWriter = new DataFileWriter(dataModel.createDatumWriter(writerSchema));
76+
this.mAvroFileWriter = new DataFileWriter<>(new GenericDatumWriter<>(writerSchema, dataModel));
7677
for (Map.Entry<String, String> entry : metadata.entrySet()) {
7778
this.mAvroFileWriter.setMeta(entry.getKey(), entry.getValue());
7879
}

connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala

+1-1
Original file line numberDiff line numberDiff line change
@@ -1075,7 +1075,7 @@ abstract class AvroSuite
10751075
.save(s"$tempDir/${UUID.randomUUID()}")
10761076
}.getMessage
10771077
assert(message.contains("Caused by: java.lang.NullPointerException: "))
1078-
assert(message.contains("null in string in field Name"))
1078+
assert(message.contains("null value for (non-nullable) string at test_schema.Name"))
10791079
}
10801080
}
10811081

core/src/main/java/org/apache/spark/SparkThrowable.java

+1-1
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ default boolean isInternalError() {
5151
}
5252

5353
default Map<String, String> getMessageParameters() {
54-
return new HashMap();
54+
return new HashMap<>();
5555
}
5656

5757
default QueryContext[] getQueryContext() { return new QueryContext[0]; }

sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsAtomicPartitionManagement.java

+1
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@
4545
@Experimental
4646
public interface SupportsAtomicPartitionManagement extends SupportsPartitionManagement {
4747

48+
@SuppressWarnings("unchecked")
4849
@Override
4950
default void createPartition(
5051
InternalRow ident,

sql/catalyst/src/main/java/org/apache/spark/sql/connector/util/V2ExpressionSQLBuilder.java

+2-2
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ public class V2ExpressionSQLBuilder {
4747

4848
public String build(Expression expr) {
4949
if (expr instanceof Literal) {
50-
return visitLiteral((Literal) expr);
50+
return visitLiteral((Literal<?>) expr);
5151
} else if (expr instanceof NamedReference) {
5252
return visitNamedReference((NamedReference) expr);
5353
} else if (expr instanceof Cast) {
@@ -213,7 +213,7 @@ public String build(Expression expr) {
213213
}
214214
}
215215

216-
protected String visitLiteral(Literal literal) {
216+
protected String visitLiteral(Literal<?> literal) {
217217
return literal.toString();
218218
}
219219

sql/catalyst/src/main/java/org/apache/spark/sql/util/NumericHistogram.java

+8-7
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919

2020
import java.util.ArrayList;
2121
import java.util.Collections;
22+
import java.util.List;
2223
import java.util.Random;
2324

2425

@@ -53,20 +54,20 @@ public class NumericHistogram {
5354
*
5455
* @since 3.3.0
5556
*/
56-
public static class Coord implements Comparable {
57+
public static class Coord implements Comparable<Coord> {
5758
public double x;
5859
public double y;
5960

6061
@Override
61-
public int compareTo(Object other) {
62-
return Double.compare(x, ((Coord) other).x);
62+
public int compareTo(Coord other) {
63+
return Double.compare(x, other.x);
6364
}
6465
}
6566

6667
// Class variables
6768
private int nbins;
6869
private int nusedbins;
69-
private ArrayList<Coord> bins;
70+
private List<Coord> bins;
7071
private Random prng;
7172

7273
/**
@@ -146,7 +147,7 @@ public void addBin(double x, double y, int b) {
146147
*/
147148
public void allocate(int num_bins) {
148149
nbins = num_bins;
149-
bins = new ArrayList<Coord>();
150+
bins = new ArrayList<>();
150151
nusedbins = 0;
151152
}
152153

@@ -163,7 +164,7 @@ public void merge(NumericHistogram other) {
163164
// by deserializing the ArrayList of (x,y) pairs into an array of Coord objects
164165
nbins = other.nbins;
165166
nusedbins = other.nusedbins;
166-
bins = new ArrayList<Coord>(nusedbins);
167+
bins = new ArrayList<>(nusedbins);
167168
for (int i = 0; i < other.nusedbins; i += 1) {
168169
Coord bin = new Coord();
169170
bin.x = other.getBin(i).x;
@@ -174,7 +175,7 @@ public void merge(NumericHistogram other) {
174175
// The aggregation buffer already contains a partial histogram. Therefore, we need
175176
// to merge histograms using Algorithm #2 from the Ben-Haim and Tom-Tov paper.
176177

177-
ArrayList<Coord> tmp_bins = new ArrayList<Coord>(nusedbins + other.nusedbins);
178+
List<Coord> tmp_bins = new ArrayList<>(nusedbins + other.nusedbins);
178179
// Copy all the histogram bins from us and 'other' into an overstuffed histogram
179180
for (int i = 0; i < nusedbins; i++) {
180181
Coord bin = new Coord();

sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/LogDivertAppender.java

+2-1
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
package org.apache.hive.service.cli.operation;
1919
import java.io.CharArrayWriter;
20+
import java.io.Serializable;
2021
import java.util.Map;
2122
import java.util.regex.Pattern;
2223

@@ -265,7 +266,7 @@ private static StringLayout initLayout(OperationLog.LoggingLevel loggingMode) {
265266
Map<String, Appender> appenders = root.getAppenders();
266267
for (Appender ap : appenders.values()) {
267268
if (ap.getClass().equals(ConsoleAppender.class)) {
268-
Layout l = ap.getLayout();
269+
Layout<? extends Serializable> l = ap.getLayout();
269270
if (l instanceof StringLayout) {
270271
layout = (StringLayout) l;
271272
break;

sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/OperationManager.java

+2-2
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@
3939
import org.apache.hive.service.cli.session.HiveSession;
4040
import org.apache.hive.service.rpc.thrift.TRowSet;
4141
import org.apache.hive.service.rpc.thrift.TTableSchema;
42-
import org.apache.logging.log4j.core.appender.AbstractWriterAppender;
42+
import org.apache.logging.log4j.core.Appender;
4343
import org.slf4j.Logger;
4444
import org.slf4j.LoggerFactory;
4545

@@ -82,7 +82,7 @@ public synchronized void stop() {
8282

8383
private void initOperationLogCapture(String loggingMode) {
8484
// Register another Appender (with the same layout) that talks to us.
85-
AbstractWriterAppender ap = new LogDivertAppender(this, OperationLog.getLoggingLevel(loggingMode));
85+
Appender ap = new LogDivertAppender(this, OperationLog.getLoggingLevel(loggingMode));
8686
((org.apache.logging.log4j.core.Logger)org.apache.logging.log4j.LogManager.getRootLogger()).addAppender(ap);
8787
ap.start();
8888
}

0 commit comments

Comments
 (0)