Charles,

the main point is that Unicode, just like any international standard, is not perfect. Character variants are not meant to provide a way for font developers to deal with situations where Unicode is completely wrong (i.e. where the user hijacks some Unicode codepoint in order to encode some information that actually has nothing to do what that codepoint). It has more to do with situations where Unicode is not entirely right, or where there are several alternative skeletons for a given letter which can be used interchangably.

Hi,

Which thread? Idea sounds a lot like Unicode Variation Selectors...?

The thread was

http://typophile.com/node/55586

Adam,

Surely it is Unicode's business to straighten out. For example, in old English, the thorn and eth were two symbols with a common meaning -- from The Cambridge Encyclopedia of the English Language, page 17: This symbol [thorn] and [eth] (see below) were in fact interchangeable: a scribe might use first one, then the other, in the same manuscript . . . The thorn was taken from Runic writings, the origin of the eth is obscure, though probably it was take from an early Irish letter. The point being that English borrowed characters from several written traditions, and to some extent modified them (e.g., paet as an abbreviation, etc.).

Both thorn and eth are encoded; granted eth has a use beyond Old English, but the thorn so encoded is not a pure runic character -- that is elsewhere.

If two languages share basically the same characters, while having a few differences, it would seem to be the Unicode consortium's business to to include the variants. If left up to type designers, we are back to the characters in a language being font dependent. And that is a real nightmare.

Charles, eth and thorn are used contrastively in Icelandic; this is why both are included not only in Unicode but also the Latin 1 charset for Western European languages. It sometimes happens that, over time, forms that are variants of a single letter in one orthography become contrastive letters in another orthography. But this doesn't imply that all variants of all letters should be treated as separate characters in an encoding standard: it means that some variants will need to be at some time and for some purposes.

Let's consider one of the specific cases that the new character variant features are designed to address. Unicode encodes a character that is the uppercase Eng, and of course a corresponding lowercase eng: Ŋ and ŋ. The lowercase eng has a single form: it always has this form regardless of the language or regional typography. The uppercase Eng has four different forms, each of which is preferred by some linguistic or regional groups. But all four forms are understood to be the uppercase form of the single lowercase letter. Ergo, they are a single character, with a single case mapping between uppercase and lowercase. One could individually encode the four uppercase forms as separate characters without also encoding four identical lowercase characters. So what we have is a single uppercase character with four glyph variants, each of which is desired by someone as a representation of that character. The first and best method to ensure that the user gets the form he or she wants is to link it to a language system tag. So, for example, the N-like form is linked to the Saami language, and in a font primarily targeting European languages this typically be the default form. But that leaves three forms, all of which are used in Africa, and for which reliable language association information is not available. We know some of the languages that use each form, but not all the languages. So the new character variant features provide a user-controlled mechanism to display the desired form of the character.

Yes, there are some characters in Unicode that are over-unified, i.e. characters that should be separately encoded but are encoded under a single identity. One that comes to mind is ƒ which is both the orthographic f-hook found in many African alphabets and also the old Dutch florin currency symbol. But there are also numerous characters whose locally preferred forms are, indeed, multiple glyph variants of a single letter, often identifiable as such through their relationships to other characters, e.g. via case-mapping.

Further, the process of disunifying existing characters, when it has been done, has been fraught with major legacy document problems. Romanian computing, for example, is still struggling with the disunification of ŞşŢţ and ȘșȚț, as are many font developers.

Josh, the difference between Unicode Variation Selectors and the character variant features that Peter Constable has proposed with SIL is that the former is a character-level mechanism to preserve the identity of variants in plain text and the latter is a glyph-level mechanism to display preferred forms in a user-controlled way (in contrast to the [locl] feature).

I think Variation Selectors could be used for commonly recognised variants of some letters, especially when those variants are known to be preferred by user communities even if not all those communities are identified. But of course that would require defining those variants and the Variation Selector sequences to encode them, getting that approved and published in the Unicode Standard, and going through that same process every time one wanted to add support for more variants.

Then there is the case of purely stylistic variants, which we have not discussed here yet but which can be expected to be mapped with the character variant features. Let's say a user is setting text in a fancy display font and wants all occurences of a particular letter in a paragraph to have a specific form, but for all other letters to have either default forms or the forms of some stylistic set to which the particular letter does not belong. The character variant features provide a mechanism to make such discreet stylistic decisions, in a way that the existing [salt] and [ssXX] features do not. Obviously Unicode Variation Selector sequences are not nearly flexible enough for this purpose.

Any knowledge (or even guesses) as to which application vendors are likely to be supporting this feature (and when)?

The features are in the pipeline for OT 1.6. Apparently SIL are already implementing some form of them, but in terms of major app developers I have no knowledge and wouldn't like to guess. Since Peter Constable has been instrumental in getting these features defined, I'd like to think that this indicates at least some possibility of support in MS apps.

There are two aspects to the support in terms of UI: firstly coming up with a useable interface for selecting among enumerated results of one-to-one-of-many substitutions (something like how Apple present 'Zapf' table substitutions in AAT fonts), and support for user-friendly feature name strings. The latter will also be an issue for Stylistic Set features as of OT 1.6. Actually, it is already an issue because FDK 2.5 supports name strings for Stylistic Set features, but apparently there are issues with such names breaking Stylistic Set support in CS3 (and perhaps other apps), as reported HERE

It's possible that it's just an AFDKO 2.5 bug rather than a general problem with the new format that permits descriptive names. It's being investigated...