Null inclusion bug in Jackson 2.9.10

[bovenson]

Facing the same problem with #2092 . Agree to JsonNode is taken to be literal expression of intent, so JsonNode.toString should keep null field (wrappered by NullNode). But, object mapper which specified JsonInclude.Include.NON_NULL option should skip NullNode when serialize JsonNode, because null value in json string means literal Java (JVM) null values for Java.

In contrast, gson's behavior is more reasonable

// json string
{
  "did": "did",
  "type": "type",
  "data": {
    "value": "valule",
    "d": null,
    "list": ["A", null, "B"]
  }
}

// java code
String s = "{\n" +
                "  \"did\": \"did\",\n" +
                "  \"type\": \"type\",\n" +
                "  \"data\": {\n" +
                "    \"value\": \"valule\",\n" +
                "    \"d\": null,\n" +
                "    \"list\": [\"A\", null, \"B\"]\n" +
                "  }\n" +
                "}";
Gson gson = new Gson();
JsonObject jo = gson.fromJson(s, JsonObject.class);
System.out.println(gson.toJson(jo));
System.out.println("-");
System.out.println(jo.toString());

// output
{"did":"did","type":"type","data":{"value":"valule","list":["A",null,"B"]}}
-
{"did":"did","type":"type","data":{"value":"valule","d":null,"list":["A",null,"B"]}}

By the way, when I try to skip NullNode through configuration, new SimpleModule().addSerializer(NullNode.class, new NullNodeAdapter()) seems doesn't work, filtering doesn't work with JsonNode?

[cowtowncoder]

Two questions/comments

1. Does this still occur with more recent versions (2.11.2)?
2. Can you show how Jackson works (failing unit test)? Knowing how Gson works may be interesting but it is most important for me to know the exact issue -- textual descriptions are often easy to misunderstand, esp. since there is difference between Java nulls (which stand for both omission and explicit null values) and JsonNode representation within document (where absence and explicit nulls are expressed differently)
3. JsonNode.toString() is NOT guaranteed to produce legit JSON (until 2.10.x, see Make JsonNode.toString() use shared ObjectMapper to produce valid json #2187) -- it was only ever meant to be diagnostic tool -- always use ObjectMapper.writeValue() for serializing all values, including JsonNode (this recommendation remains for 2.10.x although in that version you can actually rely on well-formed valid JSON)

[bovenson]

Two questions/comments

1. Does this still occur with more recent versions (2.11.2)?
2. Can you show how Jackson works (failing unit test)? Knowing how Gson works may be interesting but it is most important for me to know the exact issue -- textual descriptions are often easy to misunderstand, esp. since there is difference between Java nulls (which stand for both omission and explicit null values) and JsonNode representation within document (where absence and explicit nulls are expressed differently)
3. JsonNode.toString() is NOT guaranteed to produce legit JSON (until 2.10.x, see Make JsonNode.toString() use shared ObjectMapper to produce valid json #2187) -- it was only ever meant to be diagnostic tool -- always use ObjectMapper.writeValue() for serializing all values, including JsonNode (this recommendation remains for 2.10.x although in that version you can actually rely on well-formed valid JSON)

1. It still occur with 2.11.2, as shown in the appended code.
2. Some of jackson’s behavior makes me confused, such as, even i specified JsonInclude.Include.NON_NULL for object mapper, but it still produce null field when getting json string, as an explanation in issue Null inclusion bug in Jackson 2.9.6 #2092 , that filter not work with JsonNode, obviously SimpleModule doesn't work too. It makes me desperate that i cann't remove null field completely when getting json string, and this may produce NPE problems, of cause, the NPE is due to unrobust code.
3. Using toString() to get json string is a bad idea, my fault, forget that.

// code; jackson-core & jackson-databind with version 2.11.2
public class TestJson {
    static class NullNodeAdapter extends JsonSerializer<NullNode> {

        @Override
        public void serialize(NullNode jsonNodes,
                              JsonGenerator jg,
                              SerializerProvider serializerProvider) throws IOException {
            jg.writeString("NONE");
        }
    }

    public static void main(String[] args) throws JsonProcessingException {
        String s = "{\n" +
                "  \"did\": \"did\",\n" +
                "  \"type\": \"type\",\n" +
                "  \"data\": {\n" +
                "    \"value\": \"valule\",\n" +
                "    \"d\": null,\n" +
                "    \"list\": [\"A\", null, \"B\"]\n" +
                "  }\n" +
                "}";
        SimpleModule simpleModule = new SimpleModule()
                .addSerializer(NullNode.class, new NullNodeAdapter());
        ObjectMapper mapper = new ObjectMapper()
                .registerModule(simpleModule)
                .setSerializationInclusion(JsonInclude.Include.NON_NULL);

        JsonNode jn = mapper.readValue(s, JsonNode.class);
        System.out.println(mapper.writeValueAsString(jn));

        Gson gson = new Gson();
        JsonObject jo = gson.fromJson(s, JsonObject.class);
        System.out.println(gson.toJson(jo));
    }
}

// output; jackson's & gson's output
{"did":"did","type":"type","data":{"value":"valule","d":null,"list":["A",null,"B"]}}
{"did":"did","type":"type","data":{"value":"valule","list":["A",null,"B"]}}

[cowtowncoder]

Ok, here's a quick answer: JsonNode is handled rather different from POJOs, and is expected to be exact representation of JSON read and is generally not filtered at all. As such, serialization inclusion settings do not affect contents of JsonNode (*).
There have been requests to allow filtering, and plans to do so:

https://github.com/FasterXML/jackson-future-ideas/wiki/JSTEP-3

but this would probably use different settings, for backwards-compatibility reasons.

So, when using JsonNode, any filtering would need to be implemented by user: it will be serialized exactly as-is, with minimal modifications.

(*) there may be an edge case for possible usage of JsonNode properties in POJO -- in those cases I think root-level JsonNode is handled according to settings, given that it is a POJO property. But within node itself filtering is not applied.

[bovenson]

Glad to know the plan, and thanks.