Allow arbitrary "dedup unit" sizes in RGBGFX

[ISSOtm]

This would be useful for "8×16 OAM mode", mainly, but also metatile setups that require four unique tile IDs (usually those that don't go through an extra indirection layer).

[CharlesVanEeckhout]

Image from #1412 that shows what order to read the tiles

[Rangi42]

Given the 2bpp two-plane encoding format, I assume we would limit the dedup unit widths to multiples of 8px, even if the heights can be anything.

Also, how should {tile,attr,pal}map data be handled? For the example use cases, I expect that the "dedup unit" would also be the "map unit" -- e.g. if OAM objects are 8x16px, or metatiles are 32x32px, then you'd want your tilemaps to have one byte/ID per unit. Likewise for attrmaps, at least for OAM objects, since those support X/Y flip; but in the general non-8x8 case, attributes might not even make sense.

[ISSOtm]

The idea I had in mind was that -U would affect the way data is generated, but not how it's emitted beyond ordering.

[Rangi42]

Something to note with a general "dedup unit size" setting, as opposed to an 8x16px OAM mode toggle switch: the 8x8px tiles within the unit could still matter.

If you say -U 4,4 to mean "dedup 4x4-tile (32x32-px) metatiles", you'll still care about whether the 8x8px tiles within it are traversed in row-major or column-major order. (That's different from whether -Z traverses the 32x32px units themselves in column-major order!)

You would also want the ability to output a metatilemap (and likewise for attrmap and palmap), not just a per-8x8px-tile map. For example, here's an input image:

Without -U (or with -U 1,1), you'd get this 2bpp output (skipping the blue tiles):

With -U 4,4, you'd get this 2bpp output:

But unlike with OAM graphics, you probably don't care about the 2bpp output of metatiled input. You more likely want two kinds of map data: the tilemaps of each metatile (4 x 4 = 16 bytes per metatile, 10 metatiles), and the metatile map of the whole input image:

0 1 2 3
4 1 5 6
7 8 8 8
4 9 4 0

On the other hand, with OAM graphics you don't care that each 8x16px unit is made of of two 8x8px tiles -- you basically want the tiles to be 8x16px each. You're also much more likely to want X/Y flip support, whereas I'm not aware of metatile engines that support flipping that way (you would need to have flip attributes for the individual tiles with each metatile, and mirror the rows/columns themselves).

TL;DR I would consider implementing a --oam flag for the common case of 8x16 tile support, and separately considering how best to support metatiles.