Why would Jackson be omitting type information for serialized polymorphic types when using ProGuard?

[eric-creekside]

[Also asked on SO here]

In a project that's being obfuscated with ProGuard (using the Gradle plugin), I have lots of serialization/deserialization using Jackson. I've found that in the obfuscated builds, certain kinds of serialization of polymorphic types is omitting the type information (specified with @JsonTypeInfo(use = Id.NAME, include = As.PROPERTY, property = "type")), which causes some very strange behavior during deserialization.

For example, here's the (correct) serialized form when run with unobfuscated code:

{
  "owner" : "Jay Leno",
  "cars" : [ {
    "type" : "Corvette",
    "name" : "Corvette",
    "year" : 1963
  }, {
    "type" : "Aztek",
    "name" : "Ugly",
    "year" : 2003
  } ]
}

Here's the output of the exact same code after running through ProGuard:

{
  "owner" : "Jay Leno",
  "cars" : [ {
    "a" : 1963,
    "name" : "Corvette"
  }, {
    "a" : 2003,
    "name" : "Ugly"
  } ]
}

Note the "type" property is missing from the array elements in the obfuscated form. This causes Jackson to deserialize those objects incorrectly.

Here is an MRE of the code; sorry for including so much code in a question, but this is as minimal as I could get it and still demonstrate the context and problem.

Also viewable as a gist.

@JsonAutoDetect(fieldVisibility = Visibility.ANY)
@JsonTypeInfo(use = Id.NAME, include = As.PROPERTY, property = "type")
@JsonSubTypes({
		@JsonSubTypes.Type(value = Corvette.class),
		@JsonSubTypes.Type(value = Aztek.class)
})
public abstract class CarModel {

	public abstract String getName();
	public abstract int getYear();
}

@Getter
@AllArgsConstructor
@ToString
public class Corvette extends CarModel {

	private int year;

	@JsonCreator
	public Corvette(@JsonProperty("name") String name, @JsonProperty("year") int year) {
		this.year = year;
	}

	@Override
	@ToString.Include
	public String getName() {
		return "Corvette";
	}
}

@Getter
@Setter
@AllArgsConstructor
@ToString
public class Aztek extends CarModel {

	private int year;

	@JsonCreator
	public Aztek(@JsonProperty("name") String name, @JsonProperty("year") int year) {
		this.year = year;
	}

	@Override
	@ToString.Include
	public String getName() {
		return "Ugly";
	}
}

@Getter
public class Inventory {

	private String owner;
	private List<CarModel> cars;

	@JsonCreator
	public Inventory(@JsonProperty("owner") String owner, @JsonProperty("cars") List<CarModel> cars) {
		this.owner = owner;
		this.cars = cars;
	}

	@Override
	public String toString() {
		StringBuilder builder = new StringBuilder("Inventory [");
		builder.append("owner=")
			.append(owner)
			.append(", cars=<");

		System.out.println("First cars element is of type " + cars.get(0).getClass());

		cars.forEach(car -> builder.append(car).append(", "));
		builder.append(">");

		builder.append("]");
		return builder.toString();
	}
}

public class JacksonPolymorphicSerialization {

	private final static ObjectMapper JSONMapper =
			JsonMapper.builder()
			.configure(DEFAULT_VIEW_INCLUSION, false)
			.configure(FAIL_ON_UNKNOWN_PROPERTIES, false)
			.serializationInclusion(Include.NON_ABSENT)
			.build();
	private static final ObjectReader JSONReader = JSONMapper.readerFor(Inventory.class);
	private static final ObjectWriter JSONWriter = JSONMapper.writerFor(Inventory.class).withDefaultPrettyPrinter();


	public static void main(String[] args) throws JacksonException {
		Corvette corvette = new Corvette(1963);
		Aztek aztek = new Aztek(2003);
		Inventory inventory = new Inventory("Jay Leno", List.of(corvette, aztek));

		String json = JSONWriter.writeValueAsString(inventory);
		System.out.println("Serialized form:\n" + json);

		System.out.println();

		inventory = JSONReader.readValue(json);
		System.out.println("Deserialized object: " + inventory);
	}
}

Here's the Gradle build file with ProGuard configuration:

buildscript {
	repositories {
		mavenCentral()
	}
	dependencies {
		classpath 'com.guardsquare:proguard-gradle:7.6.1'
	}
}

plugins {
	id 'java'
	id "io.freefair.lombok" version "8.12.1"
}

repositories {
	mavenCentral()
}

def appMainClass = 'rizzo.test.JacksonPolymorphicSerialization'

java {
	toolchain {
		languageVersion = JavaLanguageVersion.of(17)
	}
}

jar {
	duplicatesStrategy = 'exclude'
	
	manifest {
		attributes 'Main-Class': "${appMainClass}"
	}
	
	from {
		configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
	}
}

dependencies {
	implementation group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.18.2'
	implementation group: 'com.fasterxml.jackson.core', name: 'jackson-annotations', version: '2.18.2'
}

task proguard(type: proguard.gradle.ProGuardTask) {
	dependsOn classes
	
	verbose
	printmapping "${buildDir}/proguard-mapping.txt"

	injars "${buildDir}/classes/java/main"
	outjars "${buildDir}/classes/obfuscated/"

	libraryjars "${System.getProperty('java.home')}/jmods/java.base.jmod", jarfilter: '!**.jar', filter: '!module-info.class'
	libraryjars "${System.getProperty('java.home')}/jmods/java.logging.jmod", jarfilter: '!**.jar', filter: '!module-info.class'
	libraryjars "${System.getProperty('java.home')}/jmods/java.desktop.jmod", jarfilter: '!**.jar', filter: '!module-info.class'
	libraryjars "${System.getProperty('java.home')}/jmods/java.xml.jmod", jarfilter: '!**.jar', filter: '!module-info.class'
	libraryjars "${System.getProperty('java.home')}/jmods/java.sql.jmod", jarfilter: '!**.jar', filter: '!module-info.class'

	// This will contain the app dependencies.
	libraryjars sourceSets.main.compileClasspath

	dontoptimize
	keepdirectories

	// Preserve getters and setters
	keepclassmembers 'class * { \
		*** get*(); \
		void set*(***); \
	}'

	// Keep Jackson-annotated methods and fields
	keepclassmembers "class * { \
			@com.fasterxml.jackson.annotation.Json* \
			<fields>; \
			@com.fasterxml.jackson.annotation.Json* \
			<methods>; \
		}"

	// Keep the main class entry point.
	keep "public class ${appMainClass} { \
			public static void main(java.lang.String[]); \
		 }"

	// This helps produce useful stack traces (see https://www.guardsquare.com/manual/configuration/examples#stacktrace)
	renamesourcefileattribute 'SourceFile'
	keepattributes '*Annotation*,EnclosingMethod,SourceFile,LineNumberTable'

	doLast {
		delete "${buildDir}/classes/java/main"
		copy {
			from "${buildDir}/classes/obfuscated/"
			into "${buildDir}/classes/java/main"
		}
		
	}
}

Run ./gradlew jar to build, you'll get a normal (unobfuscated) JAR. Run ./gradlew proguard jar and the resulting JAR will be obfuscated. Either way, you can then run with java -jar ...

Here's what happens with each.

Unobfuscated:

> java -jar build/libs/ProguardJacksonTest.jar           
Serialized form:
{
  "owner" : "Jay Leno",
  "cars" : [ {
    "type" : "Corvette",
    "name" : "Corvette",
    "year" : 1963
  }, {
    "type" : "Aztek",
    "name" : "Ugly",
    "year" : 2003
  } ]
}

First cars element is of type class rizzo.test.Corvette
Deserialized object: Inventory [owner=Jay Leno, cars=<Corvette(year=1963, getName=Corvette), Aztek(year=2003, getName=Ugly), >]

Output using the obfuscated build:

> java -jar build/libs/ProguardJacksonTest.jar
Serialized form:
{
  "owner" : "Jay Leno",
  "cars" : [ {
    "a" : 1963,
    "name" : "Corvette"
  }, {
    "a" : 2003,
    "name" : "Ugly"
  } ]
}

Exception in thread "main" java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class rizzo.test.b (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; rizzo.test.b is in unnamed module of loader 'app')
        at rizzo.test.d.toString(SourceFile:29)
        at java.base/java.lang.StringConcatHelper.stringOf(StringConcatHelper.java:453)
        at java.base/java.lang.StringConcatHelper.simpleConcat(StringConcatHelper.java:408)
        at rizzo.test.JacksonPolymorphicSerialization.main(SourceFile:38)

The ClassCastException indicates that the cars List is actually populated with LinkedHashMap instances rather than CarModel instances as it should. But I guess that makes sense given the serialized form that's missing type information - I suppose Jackson is just deserializing that into maps since it doesn't have the actual expected type at runtime (thanks, stoopid type erasure!).

If I change Inventory.cars to be a CarModel[] instead of List<CarModel> then it works even when obfuscated. However, that's not feasible in the real application project and the number of places this problem occurs.

The real question is, why is Jackson omitting the type info from the serialized form when the code has been obfuscated? What could obfuscation be doing that causes Jackson to lose the type info, and what can I do about it?

[pjfanning]

We could flip this on its head, why is proguard stopping jackson from working? Can you not go to proguard experts for this? There is no proguard expertise in the Jackson team.

[pjfanning]

So how is going to work if you obfuscate the class names of Corvette and Aztek as c and a? That will affect the serialization and the derived type field which in your config is based on the class name.

[pjfanning]

https://www.baeldung.com/java-jackson-polymorphic-deserialization has its first example where it supplies name values in the JsonSubTypes.Type annotations. This might help because the name values should be used in serialization instead or relying on the class names.

[eric-creekside]

I'm not exactly sure what you're saying or asking, but I'll try to answer by explaining generally...
The obfuscator basically alters the bytecode, renaming packages, classes, fields, and methods. You point it to a bag of classes (or JARs containing classes) and whenever it renames something, it also updates all references to that thing that exist in the bag of input classes.
To preserve JSON structure, I've configured it to keep (not rename) all getters and setters as well as any fields or methods that are annotated with any Jackson annotation (for example, my @JsonCreator annotations and associated methods are not altered). This generally works; where it's running into trouble is with generics like my List<CarModel>. The mystery is, what in the obfuscated classes is triggering Jackson to ignore the @JsonTypeInfo annotation during serialization, when it's serializing the concrete instances in the list?
Hope this helps.

[pjfanning]

I don't understand what is not clear. You are creating a setup that relies on Java Reflection and class names but you are trying to complicate by obfuscating the class names. Why not use other options to lessen the reliance on Reflection. I provided a Baeldung link. Why not try different annotation setups? If you only want to tweak proguard settings and are not willing to tweak the Jackson annotations then best of luck.

[eric-creekside]

I didn't see the Baeldung pointer, must have come in as I was writing my last response.
I did already try using @JsonTypeName annotations and it didn't seem to make a difference. I'll resurrect that again and post the code changes and exact results.

I appreciate any help and interaction in trying to solve this thorny problem. Having said that, the tone of the last sentence, "If you only want to tweak proguard settings and are not willing to tweak the Jackson annotations then best of luck." suggests that you're missing my context: I'm trying to apply obfuscation to a large, complex code base that already works, with minimal changes to that code. Changing Jackson annotations and serialization formats is, in that context, a last resort. Jackson works quite well, even with obfuscated code (when leaving the right things unobfuscated); as already stated, it's the generic collection that's a sticking point.

[yawkat]

It is probably easiest to simply disable obfuscation for your model classes. It wouldn't hide much anyway given that the relevant names still have to be there for the serialization to work, be it as actual element names or in a json string somewhere.

That said, this is not really a jackson issue. Jackson just uses reflection to look at type and field names. If you use proguard to rename those types and fields, of course the serialized representation is also going to change, there is nothing jackson can do about it.

[eric-creekside]

First, in this simple example that's easy but in the real application we user Jackson to serialize and deserialize lots of things, not just simple DTOs or anemic models.
Secondly, I respectfully disagree with the assertion that obfuscation and Jackson are fundamentally difficult to be friends. You just have to figure out the smallest set of things to leave unobfuscated that results in the same JSON outputs. I know it's doable because I've seen plenty of examples and have even achieved it myself - but WITHOUT any fields of type List<T>.

Maybe I need to restate the question: where in the Jackson code does it determine whether or not to include the type info defined by @JsonTypeInfo and @JsonSubTypes during serialization? Given a map to those locations, I can probably figure out my way home from there.

[yawkat]

The best way to marry the two is to just leave the model classes unobfuscated. It will not change much for a reverse engineer and save you a lot of trouble.

I say this as someone who implemented an obfuscator as his thesis, worked on reverse engineered code bases for years, and was a contributor to jadx. But if you want to try anyway, good luck.

[pjfanning]

jackson-databind/src/main/java/com/fasterxml/jackson/databind/introspect/AnnotationIntrospectorPair.java

Line 210 in d98215c

JsonTypeInfo.Value v = _primary.findPolymorphicTypeInfo(config, ann);

Generally, search for where the annotations like JsonTypeInfo and JsonSubTypes are referenced in the code. GitHub has good search features if your IDE does not.

It is hard enough to test Jackson in the unit tests without having to worry about people using tools to obfuscate and generally mess around without the class structure. There are other JSON libs out there that are not as reliant on Reflection as Jackson is. Could you consider alternate JSON libs? I program mainly in Scala and generally use the JSON libs that rely on working on the deser logic at compile time instead of at runtime (like Jackson).

[eric-creekside]

The best way to marry the two is to just leave the model classes unobfuscated. It will not change much for a reverse engineer and save you a lot of trouble.

I say this as someone who implemented an obfuscator as his thesis, worked on reverse engineered code bases for years, and was a contributor to jadx. But if you want to try anyway, good luck.

Just for kicks I added this to my ProGuard configuration to preserve basically everything in my little example:
keep 'class rizzo.test.* { *; }'
That results in the JAR preserving all class, method, and field names:

The results are the same, no "type": in the serialized output:

 4:13PM:~/dev/ProguardJacksonTest > java -jar build/libs/ProguardJacksonTest.jar
Serialized form:
{
  "owner" : "Jay Leno",
  "cars" : [ {
    "name" : "Corvette",
    "year" : 1963
  }, {
    "name" : "Ugly",
    "year" : 2003
  } ]
}

Exception in thread "main" java.lang.ClassCastException: class java.util.LinkedHashMap cannot be cast to class rizzo.test.CarModel (java.util.LinkedHashMap is in module java.base of loader 'bootstrap'; rizzo.test.CarModel is in unnamed module of loader 'app')
	at rizzo.test.Inventory.toString(SourceFile:29)
	at java.base/java.lang.StringConcatHelper.stringOf(StringConcatHelper.java:453)
	at java.base/java.lang.StringConcatHelper.simpleConcat(StringConcatHelper.java:408)
	at rizzo.test.JacksonPolymorphicSerialization.main(SourceFile:38)

Compared to the unobfuscated JAR:

Serialized form:
{
  "owner" : "Jay Leno",
  "cars" : [ {
    "type" : "Vette",
    "name" : "Corvette",
    "year" : 1963
  }, {
    "type" : "Tek",
    "name" : "Ugly",
    "year" : 2003
  } ]
}

Deserialized object: Inventory [owner=Jay Leno, cars=<Corvette(year=1963, getName=Corvette), Aztek(year=2003, getName=Ugly), >]

The only difference is the absence of "type" : on the list/array elements.

[eric-creekside]

There are other JSON libs out there that are not as reliant on Reflection as Jackson is. Could you consider alternate JSON libs?

Not in my context - existing large, complex code base that makes heavy use of Jackson to serialize varying types of objects. Changing that now would be cost-prohibitive.

[eric-creekside]

The solution was small but subtle. To make code like this example work correctly after obfuscation you must instruct ProGuard to keep the "Signature" data of the original bytecode.

Changing this line from build.gradle:

	keepattributes '*Annotation*,EnclosingMethod,SourceFile,LineNumberTable'

to this:

	keepattributes '*Annotation*,EnclosingMethod,Signature,SourceFile,LineNumberTable'

allows the example to run successfully even after obfuscation.

The description of keepattributes values is here. TL;DR it tells ProGuard to keep the generic signature of the class or field, which is all that remains of the type info (due to type erasure), and Jackson needs that type info when it determines how to serialize the List.

It's not clear to me what the "obfuscation theory" or tradeoffs involved in keeping that data nor why ProGuard's default configuration does not keep it.

[cowtowncoder]

Thank you for sharing @eric-creekside ! I can't think of anything specific to signature (as long as types and annotations are retained) either. But it is good to know a work-around at least.