Character variant sequences

[jsbien]

Please see http://unicode.org/faq/vs.html for basing information about variant sequences.
I'm curious how it looks from a point view of a font designer. Just found https://docs.microsoft.com/en-us/typography/opentype/spec/cmap#format-14-unicode-variation-sequences. However the practical consequences of it are not clear for me. In particular the difference between Default and Non-Default UVS table is not clear for me.

[psb1558]

There was recently a thread on the Glyphs forum that I found clarifying. The Variation Sequence is a mechanism for specifying variant shapes of Unicode-encoded characters, similar in function to OpenType Stylistic Sets and Character Variants (though these offer a good bit more flexibility). It is supported by Glyphs, the font editor I use.

The Microsoft document describes the technical details of how Variation Sequences must be implemented in a font. There are two methods, Default and Non-Default, which appear to me to be functionally the same. I suppose Default simply means that that is the method that font editors like Glyphs are expected to use unless there's some reason to do it differently (but I can't guess what that reason would be).

More interesting, from my point of view, are the FAQ and the associated lists of variants. From these I take two points relevant to Junicode:

• You can't define your own Variation Sequences; you can only use those defined by Unicode. (From the MIcrosoft doc it looks as if there are no technical reasons for this; it's just Unicode policy.)
• There are no Latin characters in the lists of Standardized Variants supplied by Unicode. They seem to think of it as a support mainly for Asian languages and Emoji.

So there can be no variations of this kind in Junicode unless they are defined by Unicode at some point; and they seem parsimonious about handing them out for the Latin script.

I have no idea whether applications support Variation Sequences, but support has to be supplied at that level too for them to work.

[jsbien]

I'm glad it would not be a technical problems.
As for the Unicode Consortium policy, I consider it self-contradictory. In the FAQ referenced above it recommends using variants for historical scripts. On the other hand the proposals http://unicode.org/L2/L2011/11059-latin-cyr-var.pdf and https://www.unicode.org/L2/L2013/13153-variants.pdf has been rejected without a publicly available explanation (at least I was unable to find it).

[jsbien]

I just learned from Karl Pentzlin that his proposal is just stalled: after the long discussion no action was taken.

[jsbien]

If I understand correctly, there is no limitations concerning the use of variant sequences for private characters. What about making some MUFI characters accessible also with variant sequences? It would allow to test how the software handle this (in particular Emacs and XeLaTeX).

[psb1558]

These days I like to stick very close to the specification—the reason being that, even if all applications handle a non-standard feature correctly now (and I haven't tested Variant Sequences to determine if this is the case), there is no guarantee that some major app won't come along that refuses to recognize them. Right now, some Adobe apps disable features they judge are not needed for a particular language: will they support VSs for languages that use the Latin script? I don't know: maybe I'll test.

As I understand the matter, the VS is similar in function to Character Variants (which I use liberally). The one thing the VS gets you that the CV doesn't is the ability to indicate "in plain text" (rather than in the markup) that a particular variant is needed.

I wonder why it would be worthwhile implementing VSs when CVs seem to be functioning well?

[jsbien]

Exactly for the reason you mentioned: CV are for rendering, VS are for the input. But I can agree it is perhaps premature to start experimenting with VS.

[TheKnightWho]

Just to chime in on this - I think the reason that I would (ultimately) prefer VS over CV is for the reason that it's theoretically possible to get new VS added to the Unicode standard. It is frustrating that they don't seem to be willing to actually do it, though - I have no idea what the reasoning could be.

[jsbien]

Karl Pentzlin's proposal has been discussed on the Unicode mailing list and the archive should be available. I think it should be found and read. I intend to do it in some future, but I don't know yet when.
Pentzlin's proposal had several independent aspects. I understand the main objections has been to giving names to variant sequences, which for me is quite OK.

[TheKnightWho]

I am also fine with the lack of names for individual sequences. So long as it's possible to specify some kind of identifier for the style, which the StandardizedVariants doc does for many of them, then that works for me.

[psb1558]

I think the prospect of getting VS characters added to Unicode is faint and distant, given that there are currently precisely zero Latin-script characters there. I don't know exactly what the thinking is among the Unicoders, but from the outside it looks as if they're thinking that it is simply not a feature for Latin script. I personally think that's a mistake, probably based on a misconception about Latin script (that it is always and everywhere a "simple" script), but I doubt that anyone is going to budge them without first opening a wider discussion about the character of Latin script and the purposes of VS.

I understand the concern about font-specific features. For what it's worth, very much in the front of my mind as I've been working on Junicode 2 is to come up with an OpenType feature scheme that is rational enough to be standardized, and (once the Junicode feature set is in a stable place) to come up with a feature file that can be easily applied to any MUFI font. I've had a few preliminary words with Tarrin on this subject, but I haven't had time (so far) to write up my thoughts about this in detail.

Janusz--your reply came in as I was writing this. I'll try to find Karl Pentzlin's proposal and have a look.

[marrus-sh]

Just going to chime in and say that Unicode tag characters (any number of U+E0020..U+E007E, followed by U+E007F) might be one path for a sort of “private‐use variation selector”. The use of tag characters isn’t clearly‐defined in Unicode, aside from language tags (deprecated, starting with U+E0001) and flag emoji (like 🏴󠁧󠁢󠁳󠁣󠁴󠁿, where the tag characters are effectively used as variation selectors on U+1F3F4 🏴 WAVING BLACK FLAG).

Like variation selectors, tag characters are “default ignorable”, and they are unlikely to be used for anything outside of emoji anytime soon. So it seems to me like it might be preferable to an unsanctioned use of a variation selector, which would introduce conflicts if Unicode ever changes their mind and starts designating official variation sequences for Latin characters.

[jsbien]

Sounds interesting.

[psb1558]

Yes, it does sound interesting. I'm going to look into it.

[jsbien]

like 🏴󠁧󠁢󠁳󠁣󠁴󠁿

FYI, my Emacs (27.1, Debian stable) has problems with this emoji. I will probably come back to this topic later in the "Emacs" thread. However it works out of the box in Emacs 28 compiled locally.

[psb1558]

I've just been looking at the Noto Color Emoji and BabelStone Flags fonts, which use the tag characters. These are presently used in only one way, for variants on U+1F3F4 WAVING BLACK FLAG. The tags correspond to the ASCII character set, so you can use them to spell out region tags. So (using friendly naming instead of Unicodes), the sequence blackflag g.tag b.tag e.tag n.tag g.tag cancel.tag will yield the U.K. English flag.

Whereas the Variation Selectors depend on a specialized lookup type in the font, Noto Color Emoji and BabelStyle Flags implement tags as ligature lookups. They put them in ccmp (so they're always on), but it seems to me they might (in theory anyway) work in any feature, say a Stylistic Set, so they could be switched off if they caused a performance hit. Further, since the tags are used to spell out "words" of arbitrary length, the mechanism would seem to be almost infinitely flexible.

The tag characters are ignorable, so they shouldn't trip up search engines—but who knows?

Things to think about.

• Since only the flag sequences are defined by Unicode, use of these tags for MUFI characters would be about as font-specific as OpenType features. If one can be standardized, maybe by a new MUFI initiative, then so can the other.
• In Noto Color Emoji, the tag characters are zero-width and empty; so in an editor they don't give visual feedback. Testing with the BabelStone Flags font, things seem to work about the same way, though there are differences that I'm not sure I understand completely.
• You've got to enter a five-digit code-point for each tag character, and this is tedious. I suspect that these would be at least as difficult to use as OpenType feature tags, which are more widely used and (I hope) more familiar.

So—a mixed reading from my point of view. Plusses and minuses to such an approach.

[jsbien]

The visual feedback in Emacs seems satisfactory: there is no glyph but you can place the cursor on a tag and delete it or e.g. check its properties. This is the situation when Emacs doesn't use a font with tags. I have Noto Color Emoji installed, but looks like Emacs has to be configured to use it.

[jsbien]

use of these tags for MUFI characters would be about as font-specific as OpenType features

Yes, but as I already said earlier, tags/VS are for input, OT features are for rendering. So it's OK for me.
I don't think we need a new MUFI inititiative, a de facto standard introduced by Junicode can get formal support later from MUFI or some other body.

[TheKnightWho]

Given that (from a user perspective) it works in the same way that a hypothetical VS implementation would, this should give us an idea of how such a system could work in practice, too.

I may be being naive, but I feel a proposal for VS in the Latin script can only be strengthened by having a real-world implementation to point at as evidence that it not only could work, but that there is demand for it as well.

Assuming, of course, that this all works as expected.

[jsbien]

The proposals of new characters can be submitted only if there is already a font supporting them. I think the same requirement is natural for VS proposals.

[psb1558]

The VS FAQ says quite specifically not to roll your own. I could try it out experimentally, but I'm not likely to publish a big run of MUFI VS's contrary to the specification.

[jsbien]

An advantage of tags over VS is that you can use them mnemotechnically, see below.

BTW, even Emacs 28 has problem with generating the correct Postscript for emojis, I had to make just a screenshot.

[psb1558]

I looked at the StackExchange thread. Very complicated, but mostly that was just Lisp scripting in Emacs, which is always a pain. I tested in FontGoggles, where it was pretty straightforward.

I'd be interested in trying out some characters exprerimentally. Any input as to which characters to experiment with and what the sequences of tags might look like?

[jsbien]

As for Emacs, in Emacs 28 it is also straightforward.

As for characters, my candidate is A749 and its variant now at F000F. As for the specific sequence of tags, this is a more difficult question. Perhaps you can, at least for testing purposes, keep it synchronized with OT features using TAG LATIN SMALL LETTER C, TAG LATIN SMALL LETTER V, TAG DIGIT ONE, TAG DIGIT SEVEN.

[jsbien]

As for F000F (A749), what about making it/them just a variants of "l"? Just an idea :-)

[psb1558]

I want to keep the cvNNs and tags in sync, so I'll make them variants of l everywhere. That also frees up cv64 for future use.

[psb1558]

I'd like to keep tag sequences as short as possible for reasons of performance and file size. It occurs to me now that one place to look for inspiration might be the MUFI entity references, which are built out of a standardized collection of abbreviations, e.g. "ins" for insular, "lig" for ligature, placed in a particular order. The tag doesn't need to specify the base character, because that's there in the sequence.

So if we think of U+F000F as U+A749 with flourish (not sure this is right, but just illustrating), the tag sequence would be

U+A749 f.tag l.tag o.tag u.tag r.tag cancel.tag

Which is a long sequence for a ligature-type lookup, but I don't think there would be much of a performance hit.

[jsbien]

It's OK for me.

[jsbien]

I don't see it in tag_key.pdf

[psb1558]

Oops. I'll add it.

[psb1558]

Fixed now.

[jsbien]

Thanks!

[psb1558]

An interim report. Inputting tags in a word processor is very tricky because they are "default ignorable" and invisible, and somehow you don't know if they're going to land in the right sequence. So I used an (optional) scheme that Junicode uses elsewhere: character entity references. In LibreOffice a sequence like this:

ooꝉ&__f;&__l;&__o;&__u;&__r;&__ca;oo

comes out looking like this:

Apply ss10 (which enables entities and tags) to it and you get this:

Success!
In HTML it's more straightforward. Here's a screenshot of a test page, which tests both standard HTML entity references (like ꝉ) and the Junicode entity references I tried out in LibreOffice:

Success again! That's Firefox, but it also works in Safari and Chrome (on the Mac). I haven't been able to test on a Windows computer (Edge) yet.
No luck in Adobe InDesign (which uses a proprietary layout engine):

But that one probably doesn't matter so much.

Next I'll test with XeTeX, LuaTeX, MS Word, and Affinity Publisher, then various Linux apps, and will try to test on a Windows machine.

[jsbien]

Great! Looking forward to XeTeX test results.

[TheKnightWho]

It looks good. Is the cancel tag always necessary at the end of any sequence?

[psb1558]

@jsbien: I'll probably be looking at XeTeX and LuaTeX today. I expect it to go well, since XeTeX uses Harfbuzz for a layout engine, like LibreOffice and Firefox.

@TheKnightWho: I wondered that too. It's probably there because the tag sequence is indeterminate in length (the two-character Regional Indicator sequence, which functions similarly, doesn't need a terminating character), but it's not strictly necessary in OpenType because the compiler reorders ligature rules so that longer ones come first. Still, it's there in the Unicode docs (and you know I like adhering to the spec), and it may provide some clarity for users, like the semicolon that ends a character entity reference.

[psb1558]

Thanks for this. The section on the cancel tag is also interesting.

We'll see how it works without. Every added tag is both an input headache and a performance hit.

[TheKnightWho]

It makes for confusing reading, because it appears that the syntax for tags as intended (i.e. with language tags) is quite different to the syntax of tags as actually used (dummy characters to force ligatures). I get the impression that in another universe, this could have been a way to introduce CSS-like style features into the Unicode standard, which is an interesting concept.

It also explains why Unicode has both Tags and VS, as I was struggling to understand the point of having both before.

[marrus-sh]

Yes, tags were originally added for the purpose of essentially marking up spans of text with additional information (language information to start), but in practice it turned out to be incredibly fragile (copying and pasting part of a page might, or might not, mess with the language information depending on which part you copied!). So they were deprecated, and Unicode decided not to pursue anything else in that direction. Then many years later, the emoji folks needed a way of specifying arbitrary strings for flag identification (to start; there was once a very controversial proposal to use Wikidata QNames to specify arbitrary emoji), so they undeprecated and repurposed the tag characters for that, since they were already in Unicode and didn’t have a clear purpose anymore.

[marrus-sh]

My understanding is that the reason for the cancel tag at the end of the sequence is essentially to create the equivalent of an empty element, so in the flag of Scotland example, <🏴GBSCT> is the bit before the cancel tag, and the cancel tag gives a </🏴GBSCT>. Since I don’t think any software actually processes these tag sequences, I’m not sure this actually matters. From a font perspective, I think it would be fine to have the ligature exclude the cancel tag, and then make the cancel tag character invisible for those who wish to include it.

[psb1558]

Yes, I've made the cancel tag zero-width and empty. It won't do any harm if people include it. With any luck, it won't even interfere with kerning.

[psb1558]

BTW, the performance hit from these complex lookups is pretty obvious in LibreOffice. A page with lots of them could be painful. I'm thinking of making all those MUFI entity elements (which can be up to seven letters long) into two-letter sequences. That should be mnemonic enough, and it will be rare to need a sequence of more than two tags (three with the cancel at the end).

[TheKnightWho]

Do you have any idea of why performance is degrading so much?

[psb1558]

No. I've noticed it before in LibreOffice with very complex OpenType operations (e.g. long ligature substitutions). I don't think it's Harfbuzz, because I'm not seeing it in Firefox.

[psb1558]

In both XeLaTeX and LuaLaTeX, a minimal file (pasted below) yields this:

Success again! (The tag sequence shortened to "fl" and the cancel tag omitted.)

\documentclass[12pt,letterpaper,openany]{book}
\usepackage[quiet]{fontspec}
\setmainfont{Junicode Two Beta}[StylisticSet=10]
\newcommand{\lhighstrokeflourish}{ꝉ\char"E0066\char"E006C}
\begin{document}
  oo{\lhighstrokeflourish}oo
\end{document}

The same minimal test in MS Word for Mac v. 16.58, using Junicode's character entities (I think Word uses its own proprietary layout engine):

Success again—the only notable failure so far is Adobe InDesign.

[kenmcd]

Success again—the only notable failure so far is Adobe InDesign.

Did you try the InDesign World-Ready Paragraph Composer?
It uses Harfbuzz (like LibreOffice) instead of the normal ID shaper.
So my guess is it should work.

[psb1558]

Thanks for the tip! Yes, it works perfectly (in this limited test, anyway):

[psb1558]

Here's a list of proposed tag sequences. They correspond to the (usually longer) MUFI abbreviations used to build character entity references. Most of the time only one two-letter sequence will be needed, but they can be concatenated when necessary. These are not for users building composite characters (I don't think that will be possible), but for me to use making tag sequences. I hope they're mnemonic enough.

A number of these will I think not be needed (and I've left off a couple that I'm sure won't be needed).

Comments?

[TheKnightWho]

Two that might be useful are "uc" (uppercase) and "lc" (lowercase), to allow the display of all caps/no caps text without it being encoded that way. This comes up on Wikisource from time to time, when the opening sentence is written in capitals, or whatever. I appreciate that that would be faffy to implement, though.

[TheKnightWho]

A (very) minor request is that it would be good for the Q long form to refer to the number of following characters (or the overall length - I don't mind).

[psb1558]

Good idea. make it lo01 and lo02.

[TheKnightWho]

Slightly more detailed suggestions/comments:

• Antiphon (A + __a + __0) ➜ A + __s + __l, as this is a diagonal stroke.

• q + __s + __l may cause confusion with U+A759. MUFI specify that it's a central diagonal stroke.

• Sanctus (unsure of current assignment) ➜ S + __s + __l (I think, though it's a little different).

• "Long s with slanted descending stroke" (s + __l + __o + __a + __1) ➜ s + __l + __o + __s + __l. MUFI have caused confusion here, because what they refer to as a diagonal stroke, as on these glyphs, differs from U+1E9C "long s with diagonal stroke", which has what they call a bar slash.

• Hymnus (Y + __a + __0) ➜ Y + __s + __l.

• Is d + __i + __n + __a + __0 perhaps more suited as d + __u + __n? It feels closer to an uncial form, but you know better than I do.

• h + __a + __0 ➜ h + __d + __e (same logic as uncial M)

• i + __a + __0 ➜ i + __s + __h + __2 (two horizontal strokes)

• j + __a + __0 ➜ j + __s + __h + __2

• Assuming i + __a + __1 and __2 refer to Roman numerals from my recent request, maybe some kind of "rm" tag makes sense?

• v + __s + __l ➜ v + __s + __t (as distinct from the diagonal stroke, which is longer).

• v + __s + __t + __r + __u I think should be v + __s + __h + __r + __u.

• v + __s + __t + __r + __u + __0 + __2 likewise. Also, is there a reason for the leading 0?

• Same question with x + __s + __l + __0 + __2 re the leading 0.

• x + __s + __l [+ etc] ➜ x + __s + __t [+ etc] (same logic as with v).

[psb1558]

Thanks for this. It'll help as I revise the list and make up the tag sequences.

[TheKnightWho]

One concern that I've had is the application of multiple tags to the same character, and how any implementation would handle it. Please do let me know if I have the wrong idea (because I hope that I do), but the only solution seems to be to encode compound ligatures. However:

a) Even if you limit combinations to a maximum of two, you're looking at 1,431 compound tags. Let's say 200 realistic possibilities. There are certainly situations where you might want three, though (e.g. small caps uncial m with macron).

b) Even that assumes that m + __s + __c + __u + __n would be automatically processed identically to m + __u + __n + __s + __c. I don't think it will be, because that would cause issues for flag country codes (and, incidentally, any combinations of tag letters that are anagrams). That doubles the number. If we wanted combinations like m + __s + __c + __u + __n + __m + __a (i.e. with the macron), it becomes completely out of control (nearly 25,000 theoretical combinations, each with 6 ways of being entered).

I would be surprised if the same issues applied to VS, because they're designed with this sort of usage in mind, whereas tags are more of a happy accident for certain use cases that seem to lack scalability.

[psb1558]

I explained myself badly. When I had made up my list of tag-pairs, it struck me that it looked like a set of building blocks, but it's not: it's just meant to describe what the font contains. For example, MUFI has an x with slash across the right lower leg. That is produced by the sequence x s.tag l.tag r.tag l.tag for "x + slash + right lower". There's also a "left lower" tag-pair, but you can't enter x s.tag l.tag l.tag l.tag and get an x with a slash across the lower left leg because there's no such thing in the font. Those tags will be silently ignored (but you'll still get the x).

Similarly, although there are tag-pairs describing diacritics in the list, that doesn't mean you can use them to build character + diacritic combinations. Unicode provides a much better way to accomplish that--plus, as @TheKnightWho points out, that kind of thing would lead to an impossible number of combinations to cover with tags.

You can just use the sequences that are defined in the font.

There's a tentative list of tag sequences (covering only the same ground as cv01-cv52, but that's just a start) in the document tag_key.pdf, and I will shortly have a font for people to test with.

[psb1558]

My plan is to cover the same territory as the cvNN features, and then see what else (if anything) has to be done. So far I've done cv01-cv52 (A-Z, a-z), that is 160 tag sequences. The number of sequences might double by the time I get to cv99, but I wouldn't expect it to get much bigger than that. That would be roughly comparable to the number of rules in the smcp (small cap) feature.

As the only purpose of the tag-pairs is to be mnemonic, it would be useful to make them occur in a predicable order. But so far I've just used a common-sense order.

Early days.

[TheKnightWho]

Sorry - I may be explaining myself badly. Correct me if you aren't creating tags for these, but I don't think it makes a difference to the point: if I wanted to, say, apply 'onum', 'pnum' and 'sups' to a sequence of numbers, would you specify an order in which the relevant tags need to be added, or will you be creating multiple lookups to cover each possible input?

I may be simply misunderstanding something here, so apologies if that's all it is.

(To be clear - I don't mind which, but it helps in understanding the limitations inherent to any implementation!)

[psb1558]

Okay, I see what you mean. This kind of tag is like a modifier attached to a single character. It can't be applied to a run of characters. So this makes most sense as an alternative way of doing the work of the cvNN features.

[psb1558]

But this system is quite compatible with other OpenType features. If you use the sequence d i.tag n.tag to make an insular d, smcp can then be applied to that to get a small cap insular d.

[TheKnightWho]

Thanks - that makes sense. I suppose the examples I gave have the advantage of being standardised anyway, and it would be a bit of a regressive step to (more than) quadruple the size of any numerical text input. I can certainly see why tags were introduced in their initial form, even if it didn’t really work.

As you say - early days.

[psb1558]

I ran into difficulties with the scheme I devised before. Without getting into details about it, the problem turned out to be having tag sequences of of varying lengths. I had to either (1) make them all the same length or (2) terminate each one with cancel.tag. I chose (1) and decided that each sequence would contain a base character and two tags. That meant that the scheme of tags would be less expressive than before--a bit less mnemonic. it is laid out here. (The good thing about revising the scheme was that I could eliminate tag sequences that weren't needed.)

A list of characters covered by the new tag scheme is here. It contains two or three errors that will come right when the font is rebuilt, probably some time in the next few days. Until then, the fonts currently posted use the old scheme and aren't usable.

[jsbien]

Just curious: how do you create the tag symbols in tag_key.pdf? The document properties say it was created with LibreOffice, but tag_key.odt represents the tag characters differently.

[psb1558]

You can't do it until you've installed version 1.047, which I am building right now.

[psb1558]

I've just put up the variable and *.otf versions; the ttf will take a little while longer.

[psb1558]

There's the *.ttf. Be sure to delete old version before installing.

[jsbien]

Still curious how you create the tag symbols :-)

[jsbien]

Your XeTeX/LuaTex minimal example doesn't work for me. I slightly changed it to account in particular for the changed tags.
JSBtest_lua.txt
JSBtest_lua.pdf
JSBtest_lua.log

[jsbien]

I tested also some other tags and their work OK.

[jsbien]

As for F0011, F0012, F0013, F0014 and F0021, I would like to remind that we have now in Unicode 'LATIN SMALL LETTER OLD POLISH O' (U+A7C1). So you can use it as the base character of tag sequences instead of or additionally to just "o".

[psb1558]

Would you then count the things from old Polish printing that I've been counting as variants of o+slash as variants of U+A7C0/1?

Because there are so few free cvNN variants, I don't want to do them twice, as variants of both oslash and U+A7C0/1.

[jsbien]

Yes. Actually I don't need cv access at all, the tags are sufficient for me.

[psb1558]

I'll fix this. I will revise the cvNN too, because I want to keep the two systems in sync. They are good for different situations: if you want a variant to appear throughout the text, a cvNN feature is still the best option for most users. The tags can be used to override it, since they are resolved first.

[psb1558]

The \lhighstrokeflourish command should start with plain l (U+006C), then \char"E0073\char"E0066.

[jsbien]

Thanks! Sorry for bothering you with a stupid question :-(

[psb1558]

But happy to help out! These tags are new to most of us, including me.

[psb1558]

I have posted a new version of tag_key.pdf with instructions and descriptions.