Publish and maintain a validation suite along with the spec

[belluzj]

What I'm calling a validation suite would be a collection of small UFO files that each demonstrate an aspect of the specification, a bit like test cases for software.

What would the suite look like?

Small UFO files

Each aspect of the UFO format would be represented in at least one matching UFO file (granularity to be defined, for example 1 big UFO that covers the whole specification is not very useful; 1 UFO per field of the fontinfo.plist is probably too many).

Each UFO of the suite would have a purpose, something to show.

Evolving alongside the specification

Existing aspects of the specification would get matching UFOs in the suite. New specification proposals should come accompanied with UFOs ready for inclusion in the suite.

This proposal goes hand-in-hand with another proposal to encourage custom lib keys to be accompanied by a small specification in the same format as the official UFO specification. Having a UFO that demonstrates the use of each custom lib key would help understand the various features bolted on the UFO format through lib keys, check which implementations cover them, and facilitate their inclusion into the official specification.

With assertions about semantics of their contents in terms of compiled fonts

The suite would be independent from a reference implementation, it would stand on its own and specify the expected results of each UFO file in terms of compiled fonts. For example, each UFO could come with a Python script that opens a compiled font file with fontTools.ttLib and checks that certain assertions about the file are true, meaning that the UFO data was interpreted correctly while compiling the font.

Who would benefit from the suite?

As a test suite for UFO implementers

Taking into account 2 main categories of tools that implement the UFO format, we have "editors" and "compilers".

• editors can read the UFO, change stuff, and write it back. Examples: with a user interface: Robofont, FontLab 7, Glyphs.app (natively? or through glyphsLib) and any other font editor with UFO support; from the command line or scripts: defcon, fontParts etc.
• compilers can read the UFO and make a TTF, OTF etc out of it. Example: all font editors that include a menu item to "generate fonts" or equivalent; from the command-line: fontmake (and others?)

The validation suite would be useful for both kinds of implementers:

• any editor should have a way to read a UFO and write it back, and using the validation suite they could check easily how many features of the UFO specification they can interpret well enough to write them back to the disk in exactly the same way. If the editor exposes an API method called readAndWriteBack(sourceUFO, targetUFO), that validation suite can be scripted as a suite of automated round-trip tests (as what we have in glyphsLib)

• compilers could, in a similar way, compile each UFO of the suite and run the provided semantic assertions about the compiled fonts. Given an exposed API called compile(sourceUFO, targetTTF), an automated test suite can compile each UFO file into a font and check that the font has the expected features that are expressed in the UFO.

As a software implementer, having such a test suite is always a huge help.

As a discussion tool between implementers and their users

Now that implementers know exactly which parts of the spec their tools cover, they can advertise that fact to their users and clients; in the same way, users who know which features of the UFO format they're using can procure software that supports their use-cases. The validation suite would serve as a kind of "certification of compliance with the UFO specification".

To give an idea of how valuable that information would be, specification compliance could be shown as a table like the ones you can find on MDN (imagine font tools instead of browsers and UFO features instead of units):

Users of UFO-based tools could also use the validation suite as a bug report tool/repository of use-cases, to ensure that everything they do with UFOs is actually documented in the suite, and that all implementers are made aware of these possibilities when they run the suite.

As a documentation and discussion tool for the format

Everyone who wants to understand the UFO specification would benefit from having examples attached to the specification, be it the existing specification or new proposals that are under review.

To sum, such a validation suite would be widely useful to about anyone who is involved with the UFO format. Parts of this suite probably already exist as test cases for various libraries such as ufoLib.

[alerque]

In software unit testing, these sort of data files with an input and expected output are usually called fixtures.

And YES, this is a great idea!

[benkiel]

FYI, this is on the agenda. The validators in ufoLib can be part of a start to this.

[typesupply]

I agree 100% with this. I started a comprehensive UFO3 test suite a long time ago based on the WOFF test suite. It probably stalled because I was trying to do too many things at that time. I just dug up the files from the ancient UFO SVN and put them here. It's been ages, but if this is a good starting point feel free to use it.

[madig]

Something like https://github.com/BurntSushi/toml-test could serve as an example for a test suite.