Side effects from `serde_json/preserve_order` feature impact user applications

[sergiimk]

Describe the bug

In v55.0.0 the PR #21767 added the preserve_order feature to serde_json dependency in datafusion-physical-plan.

This feature replaces BTreeMap-backed serde_json::Value with an IndexMap globally.

This is a major side-effect for applications embedding datafusion:

• It can affect performance
• It makes REST API return keys in different order
• It impacts many tests that may have depended on sorted keys
• It affects our cryptography that relies on stable sorted JSON representation.

Enabling this feature just doesn't seem like a choice that a library should make as its effects are disruptive and non-local.

To Reproduce

Expected behavior

I suspect preserve_order is only needed for testing and could be enabled in dev-dependencies without affecting applications that link datafusion. CC @adriangb

Additional context

No response

[adriangb]

Thanks for reporting @sergiimk and sorry for the issue, we should have caught this. There' a fix PR open and it's being tracked for the next patch release.

[sergiimk]

Really appreciate a quick turnaround @adriangb!

I see the issue is linked in 56.0.0 release. Any chance it can be included into the planned 55.2.0 instead? Would love to be able to finish upgrade to 55 instead of skipping one major release.

[adriangb]

I did link it there too! #25758 (comment)