Console command for users to merge field definitions

[vnali]

In Craft 5 Plugin Upgrade Guide in notion.site, there is an announcement:

We will provide a console command for users to merge field definitions that are made redundant by multi-instance support.

and I couldn't find further information about it in repos, roadmap, ...

I have two questions about it:

• After Craft 5 migration, there are so many similar global fields in different field layout. especially fields that were in Matrix context that now are available in global context. Can we merge these fields and their values easily via console? -or it is something more related to multi instance support as announcement said that implies this is only related to fields in same layout-
• Is it going to be released in 5.0 or next minor releases?

I appreciate more details if details are planned yet.

[brandonkelly]

We’re planning on adding a fields/merge command which will aid in combining similarly-configured fields. The hope was that it would be done by the time 5.0 GA is released, but it’s more likely it will come later since bug fixes and porting plugins need to come first.

[MoritzLost]

@brandonkelly Any chance this will make it into 5.3, or is there any ETA for this?

We're holding off on our Craft 5 upgrades mostly due to the unwieldy amount of work required to consolidate all the duplicated fields left by the migration. Having a console command that allows us to selectively merge duplicated fields and that also deals with merging existing content in the production environment (similar to the the entrify commands) would be a godsend.

[brandonkelly]

It’s on the list for 5.3

[MoritzLost]

@brandonkelly That's great to hear! 🚀

[mikolamin]

Thanks, look forward to it!

[reilnuud]

@brandonkelly please forgive me if I missed something elsewhere, but has there been any movement on this?

[brandonkelly]

Just tagged Craft 5.3.0-beta.1 with new fields/merge and entry-types/merge commands 🎉

See the PRs for a full description of how they work.

When it comes to cleaning up your entry types after upgrading to Craft 5, I’d recommend taking the following approach:

1. Create a database backup.
2. For any redundant fields, first check their settings to be sure there aren’t any subtle differences that warrant two separate fields. Then run fields/merge for each one.
3. For any redundant entry types, first compare their field layouts. If there are any field instances in the outgoing entry type whose values should get merged into an existing instance in the persisting entry type, make sure that:
   • they both reference the same custom field; and
   • they both have the same field handle. (Doesn’t matter if the handles are overridden, as long as they’re the same.)
4. Create a new database backup.
5. Run the entry-types/merge command for each of the redundant entry types.

Both fields/merge and entry-types/merge will end up creating a content migration. When propagating the changes to other environments, it’s critical that this migration gets run after project config changes have been applied. (craft up will handle it, but you’ll need to enter yes at the prompt about running content migrations if you’re running that in an interactive shell.)

We’re targeting Craft 5.3 for August 6. If you want to help test in the meantime, change your craftcms/cms requirement in composer.json to:

"craftcms/cms": "^5.3.0-beta.1",

Then run composer update.

[jannisborgers]

I’m just thinking about our matrix fields and nearly all of them have at least one of the following fields - mainTitle, introText, mainText, cta - more often than not all these fields are created as plain text default settings field. Having a command that can find identical fields and merge them would be a big help.

@mofman While I agree that this could be useful and save some time, I’m testing this right now on a multi-site installation where I ended up with 199 fields (not 500, but more than I expected) due to matrix and super-table nesting… and it’s a manageable process once you get the hang of it.

Additionally, while I agree most of the duplicates are supposed to be the same fields, a typo in any property of any field would result in a cancelled automatic merge. I believe doing it semi-automatic like with this 5.3 command is not a lot more time-consuming.

@brandonkelly Thanks a lot! This comes at the right time for me, just as I finished a Craft3→4→5 update. I already reduced the fields from 199 to 140 within 20 minutes (when migrating the content by hand, that would have been half a day).

One thing that bugs me is that the handle overrides stay in place even after merging fields. Can you provide an additional command to automatically remove all handle overrides from field layouts, if the original handle is the same as the override? Something like entry-types/clear-handle-overrides? (not sure if entry-types or fields is the right namespace)

This is kind of the reverse of the what the Craft 5 entrification process does. It would make it a lot easier to rename fields in the future, and have the handle change propagate to all entry types that already have the field in use.

[brandonkelly]

@jannisborgers Good point. Just updated the command so overridden labels, handles, and instructions will be cleared out if they now match the persisting fields’ values. (e121d28)

[brandonkelly]

Just added a new fields/auto-merge command for the next release, which should speed the process up quite a bit if there are many identically-configured fields. (#15472)

[mofman]

Found 31 Plain Text fields with identical settings:

- altText (1 usage)
- date (1 usage)
- date2 (1 usage)
- itemText (1 usage)
- itemTitle (1 usage)
- mainName (1 usage)
- mainName2 (1 usage)
- mainText10 (1 usage)
- mainText13 (1 usage)
- mainText16 (1 usage)
- mainText22 (1 usage)
- mainText3 (1 usage)
- mainText9 (1 usage)
- mainTitle10 (1 usage)
- mainTitle12 (1 usage)
- mainTitle15 (1 usage)
- mainTitle17 (1 usage)
- mainTitle20 (1 usage)
- mainTitle21 (1 usage)
- mainTitle27 (1 usage)
- mainTitle3 (1 usage)
- mainTitle7 (1 usage)
- position (1 usage)
- position2 (1 usage)
- quote (1 usage)
- quote2 (1 usage)
- quote3 (1 usage)
- quote4 (1 usage)
- quoteBy (1 usage)
- quoteBy2 (1 usage)
- statsTitle (1 usage)

@brandonkelly just getting arround to testing this, the auto-merge command still isnt't working quite how I imagined it would. Given the output above as an example I would have thought it would have taken into account the original name and only merged fields with the same name? So for example; all the mainTitleXX fields would be merged into a single mainTitle field etc.

Flattening all these fields down to a single 'Text' field is going too far for us...

[brandonkelly]

@mofman Field names, handles, and instructions can be overridden within field layouts, so those aren’t taken into account when looking for duplicates.

The command will automatically set the field layout overrides for names, handles, and instructions when needed, to ensure they don’t change as a result of the command.

[jannisborgers]

@brandonkelly Will CKEditor fields gain MergeableFieldInterface and be able to merge together in a future release? Apart from plain text fields, they are the most common duplicates I find (after converting all Redactor fields to CKEditor fields).

[jannisborgers]

Also, the ability to sort the fields table by handle would ease finding duplicates. The search filter is so fuzzy that it finds a lot of fields when searching handles like "text", "copy" or "heading". Sorting by handle would also group all duplicates with incremented handles together.

[brandonkelly]

Yep, it’s on the list for CKEditor 4.2, which we’ll start working on once Craft 5.3 is out.

Just made all columns sortable on the Entry Types and Fields index pages. (7b5b664)

[reilnuud]

@brandonkelly Are there any plans to support usage of the command like with more than two fields at once? Would really speed up the process for us. fields/auto-merge is great, but tends to grab one or two fields sometimes that we want to actually keep separate, so it'd be something like:

php craft fields/auto-merge field field1 field2 field3

Currently, all the fields after the first two are ignored.

[brandonkelly]

Yii doesn’t currently support variadic CLI actions (actions with an unknown number of arguments), so I just PR’d support for it (yiisoft/yii2#20247). Can consider making this possible if/when that’s merged and Yii is updated with it.

[reilnuud]

@brandonkelly appreciate the quick response on this one!

[brandonkelly]

Added support for merging 3+ fields at a time, for Craft 5.7 🎉 (#16719)

[mikolamin]

@brandonkelly I know that Redactor type is deprecated, but I cannot change all fields faster. And I got the message "Redactor fields don’t support merging." while I run the command fields/merge. Is it possible to fix it?

[brandonkelly]

Just tagged Redactor 4.2.0 with support for field merging.

[mikolamin]

@brandonkelly Thanks a lot for the swift update!

[mikolamin]

@brandonkelly I also noticed that the craft\fields\Table field do not implement the interface MergeableFieldInterface, is it possible to fix this? Thanks!

[brandonkelly]

Unfortunately it’s not possible to merge Table fields because there’s no way to reconcile the data from two different fields. Cell data is indexed based on column IDs that are unique to the field.