Skip to content

Reduce/eliminate introspection on java.* classes #4907

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
gsmet opened this issue Jan 15, 2025 · 12 comments
Open

Reduce/eliminate introspection on java.* classes #4907

gsmet opened this issue Jan 15, 2025 · 12 comments
Labels
3.0 Issue planned for initial 3.0 release to-evaluate Issue that has been received but not yet evaluated
Milestone
3.0.0-rc5

Comments

@gsmet
Copy link

gsmet commented Jan 15, 2025

Is your feature request related to a problem? Please describe.

This is a follow-up to this discussion in the Quarkus tracker: quarkusio/quarkus#45537 (comment) .

In the Quarkus project, @zakkak is leading an effort to allow throwing exceptions when GraalVM/Mandrel tries to do reflection calls on elements that were not registered for reflection. This effort leads to all sort of discoveries and in the issue above we discovered that Jackson tries to collect information on java.* classes, which should probably be avoided: these classes can't be annotated with Jackson classes and they are well known so we could probably hardcode what's needed.

We are having this issue with at least AnnotatedFieldCollector/AnnotatedMethodCollector/AnnotatedCreatorCollector. There might be others.

Examples of this issue with stacktraces:

org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.util.Date.getDeclaredFields()

without it being registered for runtime reflection. Add java.util.Date.getDeclaredFields() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  [email protected]/java.lang.Class.getDeclaredFields(DynamicHub.java:1251)
  com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector._findFields(AnnotatedFieldCollector.java:73)
  com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector._findFields(AnnotatedFieldCollector.java:71)
  com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector.collect(AnnotatedFieldCollector.java:48)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.sql.Date.getDeclaredFields()

without it being registered for runtime reflection. Add java.sql.Date.getDeclaredFields() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  [email protected]/java.lang.Class.getDeclaredFields(DynamicHub.java:1251)
  com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector._findFields(AnnotatedFieldCollector.java:73)
  com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector.collect(AnnotatedFieldCollector.java:48)
  com.fasterxml.jackson.databind.introspect.AnnotatedFieldCollector.collectFields(AnnotatedFieldCollector.java:43)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.sql.Date.getDeclaredMethods()

without it being registered for runtime reflection. Add java.sql.Date.getDeclaredMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  [email protected]/java.lang.Class.getDeclaredMethods(DynamicHub.java:1258)
  com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1300)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:115)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:49)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.util.Date.getDeclaredMethods()

without it being registered for runtime reflection. Add java.util.Date.getDeclaredMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  [email protected]/java.lang.Class.getDeclaredMethods(DynamicHub.java:1258)
  com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1300)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:115)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:54)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.io.Serializable.getDeclaredMethods()

without it being registered for runtime reflection. Add java.io.Serializable.getDeclaredMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  [email protected]/java.lang.Class.getDeclaredMethods(DynamicHub.java:1258)
  com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1300)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:115)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:54)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.lang.Cloneable.getDeclaredMethods()

without it being registered for runtime reflection. Add java.lang.Cloneable.getDeclaredMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  [email protected]/java.lang.Class.getDeclaredMethods(DynamicHub.java:1258)
  com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1300)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:115)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:54)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.lang.Comparable.getDeclaredMethods()

without it being registered for runtime reflection. Add java.lang.Comparable.getDeclaredMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  [email protected]/java.lang.Class.getDeclaredMethods(DynamicHub.java:1258)
  com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1300)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:115)
  com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:54)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.sql.Date.getDeclaredConstructors()

without it being registered for runtime reflection. Add java.sql.Date.getDeclaredConstructors() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  [email protected]/java.lang.Class.getDeclaredConstructors(DynamicHub.java:1264)
  com.fasterxml.jackson.databind.util.ClassUtil.getConstructors(ClassUtil.java:1345)
  com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector._findPotentialConstructors(AnnotatedCreatorCollector.java:115)
  com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector.collect(AnnotatedCreatorCollector.java:70)
org.graalvm.nativeimage.MissingReflectionRegistrationError: The program tried to reflectively access

   java.sql.Date.getDeclaredMethods()

without it being registered for runtime reflection. Add java.sql.Date.getDeclaredMethods() to the reflection metadata to solve this problem. See https://www.graalvm.org/latest/reference-manual/native-image/metadata/#reflection for help.
  [email protected]/java.lang.Class.getDeclaredMethods(DynamicHub.java:1258)
  com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1300)
  com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector._findPotentialFactories(AnnotatedCreatorCollector.java:197)
  com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector.collect(AnnotatedCreatorCollector.java:71)

Describe the solution you'd like

Ideally, we should avoid dynamically resolving this information for java.* classes.

Usage example

No response

Additional context

No response
@gsmet gsmet added the to-evaluate Issue that has been received but not yet evaluated label Jan 15, 2025
@gsmet gsmet mentioned this issue Jan 15, 2025
Open
@cowtowncoder cowtowncoder added the 2.19 Issues planned at 2.19 or later label Jan 26, 2025
@cowtowncoder
Copy link
Member

cowtowncoder commented Jan 26, 2025 •
edited
Loading

Hmmmh. Ok, now I remember the challenge: although there is existing mechanism to avoid annotation introspection for JDK classes (although only applied to JDK collection types), actual introspection of declared fields etc is still performed. So that'd need to be changed and it's bigger change.

Preventing this traversal would also prevent use of mix-in annotations on JDK types.
I'll probably make this a MapperFeature but one that is opt-out (i.e. default being to skip introspection).

cowtowncoder added a commit that referenced this issue Jan 26, 2025
@cowtowncoder
Minor work for #4907: reduce _annotation_ introspection of JDK types …
b4719f6 
…further
@cowtowncoder
Copy link
Member

Another question: once I get to implement something, is there some tool I could use to see if access is violated? It'd be good for me to verify changes locally before asking others to check.

cowtowncoder added a commit that referenced this issue Jan 26, 2025
@cowtowncoder
Minor work for #4907: reduce _annotation_ introspection of JDK types …
3f6fbb4 
…further (#4927)
@cowtowncoder
Copy link
Member

Ok so just removing --add-opens from pom.xml will not expose these (although it does trigger 4 failures, one for EnumSet another for EnumMap type detection, 2 others for trying to force access to JDK internal collections -- I fixed 2 others that happened).

Is there maybe another compiler option to get warnings? Or is this with Graal processing?

@gsmet I'd be happy to work a bit on this now but need to see warnings to verify calls are no longer made.
(ideally there'd be tests too of course but first things first).

@gsmet
Copy link
Author

gsmet commented Feb 9, 2025

@cowtowncoder sorry, things have been quite chaotic lately. Let me try to assemble a simple reproducer for you.

@gsmet
Copy link
Author

gsmet commented Feb 9, 2025

@cowtowncoder the feedback loop is not exactly fast as it requires building a native executable but the following reproducer reproduces the issue:

jackson-reproducer.zip

See the README in it for detailed instructions. Let me know if you encounter any issue.

You should be able to override Jackson in the POM to point to a SNAPSHOT.

@cowtowncoder
Copy link
Member

Thank you @gsmet !

@cowtowncoder
Copy link
Member

cowtowncoder commented Feb 11, 2025 •
edited
Loading

Ok after couple of small tweaks (figuring out dir ref to GraalVM etc), got build going. So I think this is fine.

EDIT: and after building, running service with given settings does indeed report expected warnings.

@cowtowncoder cowtowncoder mentioned this issue Apr 13, 2025
Closed
@cowtowncoder cowtowncoder added the 3.0 Issue planned for initial 3.0 release label Apr 26, 2025
@cowtowncoder
Copy link
Member

@gsmet Trying to work on this again, now that 2.19.0 is released. Noticing that I can only build against 2.18.*; attempts to building against 2.19(.0) fail due to one thing added to jackson-annotations in 2.19.

Would it be easy to indicate if I can make local modifications to reproducer (aside from jackson-databind version in pom.xml) -- or if need be, create new reproducer zip?
Apologies for extra work, but I'd like to verify if my changes help -- now that I can see what specifically is printing warnings.

cowtowncoder added a commit that referenced this issue May 3, 2025
@cowtowncoder
Work towards #4907: reduce introspection for "java.*" classes (and ma…
0b769e2 
…ybe others too)
cowtowncoder added a commit that referenced this issue May 7, 2025
@cowtowncoder
Prepping for #4907, make introspection more granular/dynamic
5a273bd
cowtowncoder added a commit that referenced this issue May 7, 2025
@cowtowncoder
Prepping for #4907, make introspection more granular/dynamic (#5146)
707d8c6
@cowtowncoder
Copy link
Member

Quick note: been investigating this, learning much more about the challenges.

There are multiple ways to (try to) achieve this, f.ex:

  1. Reduce introspection of accessors at low-level, for JDK types. Could exclude Fields without problems, but for Methods not doable without allow-listing some types (getters relied on for some serialization of StackTraceElement, ThreadGroup, as-POJO-serialization for Optional -- see (WIP) Reflection testing on 2.x wrt #4907 #5119 )
  2. Try to do (de)serializer lookup by providers/defaults before (most?) introspection, to prevent introspection for "known" JDK types (ones with handlers)

Of these, both seem difficult to safely do for 2.x, so I will focus more on 3.0; but if it happened that a safe set of changes were found, would backport into 2.x (2.20).

FWTW, I am thinking (2) would be most beneficial route but we'll see.

@cowtowncoder cowtowncoder mentioned this issue May 9, 2025
Open
cowtowncoder added a commit that referenced this issue May 20, 2025
@cowtowncoder
Start working on #4907
9b121c2
cowtowncoder added a commit that referenced this issue May 20, 2025
@cowtowncoder
Start working on #4907 (#5167)
9aa9fd0
cowtowncoder added a commit that referenced this issue May 20, 2025
@cowtowncoder
Part 2 of changes for #4907 (converter access)
05f1365
cowtowncoder added a commit that referenced this issue May 21, 2025
@cowtowncoder
Part 2 of changes for #4907 (converter access) (#5168)
ab25a78
@cowtowncoder
Copy link
Member

Ok some incremental progress, for 3.0 (.0-rc5): now for deserialization side, full introspection is avoided for deserialization of "well-known" types (ones for which Jackson or modules has explicitly registered deserializer) such as java.sql.Date used in reproduction.

For serialization calls are deferred as well, with the unfortunate exception for @JsonValue introspection which must be done for accessors (Fields, Getters).
To avoid that, different approach is needed; will need to think about that.

Note, though, that there is parallel effort via #5149 which would:

  1. Allow for explicit disabling of Field/Method/Constructor introspection for specific categories of classes (like Core JDK) AND
  2. Would be implemented for 2.x (2.20)

and which I hope to work on. But haven't had much progress yet (at idea/design stage).

cowtowncoder added a commit that referenced this issue May 21, 2025
@cowtowncoder
Third part of #4907: change ordering of serializer lookups
bed68c5
cowtowncoder added a commit that referenced this issue May 21, 2025
@cowtowncoder
Third part of #4907: change ordering of serializer lookups to defer i…
ff42202 
…ntrospection further (#5172)
@cowtowncoder
Copy link
Member

Phew! I figured out how to order things to avoid Method introspection for serialization of explicitly supported types: as I mentioned earlier this is for 3.0.0.
Will mark this as closed.

cowtowncoder added a commit that referenced this issue May 21, 2025
@cowtowncoder
Update release notes wrt #4907
649744e
@cowtowncoder cowtowncoder removed the 2.19 Issues planned at 2.19 or later label May 21, 2025
@cowtowncoder cowtowncoder added this to the 3.0.0-rc5 milestone May 21, 2025
@cowtowncoder
Copy link
Member

@gsmet Given that work to re-order things is based on working around specific type (java.sql.Date in this case), there are likely other paths that would still trigger same issue. So I would be interesting in finding more if you happen to have time. These can be in 2.x (whichever version); specifically thinking of Lists and Maps -- scalar types are probably covered.

cowtowncoder added a commit that referenced this issue May 21, 2025
@cowtowncoder
More follow-up work wrt #4907
80155ba
cowtowncoder added a commit that referenced this issue May 21, 2025
@cowtowncoder
More follow-up work wrt #4907 (#5173)
2b1a8fd
cowtowncoder added a commit that referenced this issue May 21, 2025
@cowtowncoder
Yet more tweaking wrt #4907
631b916
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.0 Issue planned for initial 3.0 release to-evaluate Issue that has been received but not yet evaluated
Projects
None yet
Milestone
3.0.0-rc5
Development

No branches or pull requests
2 participants
@cowtowncoder @gsmet