0,0 → 1,532 |
gxvalid: TrueType GX validator |
============================== |
|
|
1. What is this |
--------------- |
|
`gxvalid' is a module to validate TrueType GX tables: a collection of |
additional tables in TrueType font which are used by `QuickDraw GX |
Text', Apple Advanced Typography (AAT). In addition, gxvalid can |
validates `kern' tables which have been extended for AAT. Like the |
otvalid module, gxvalid uses Freetype 2's validator framework |
(ftvalid). |
|
You can link gxvalid with your program; before running your own layout |
engine, gxvalid validates a font file. As the result, you can remove |
error-checking code from the layout engine. It is also possible to |
use gxvalid as a stand-alone font validator; the `ftvalid' test |
program included in the ft2demo bundle calls gxvalid internally. |
A stand-alone font validator may be useful for font developers. |
|
This documents documents the following issues. |
|
- supported TrueType GX tables |
- fundamental validation limitations |
- permissive error handling of broken GX tables |
- `kern' table issue. |
|
|
2. Supported tables |
------------------- |
|
The following GX tables are currently supported. |
|
bsln |
feat |
just |
kern(*) |
lcar |
mort |
morx |
opbd |
prop |
trak |
|
The following GX tables are currently unsupported. |
|
cvar |
fdsc |
fmtx |
fvar |
gvar |
Zapf |
|
The following GX tables won't be supported. |
|
acnt(**) |
hsty(***) |
|
The following undocumented tables in TrueType fonts designed for Apple |
platform aren't handled either. |
|
addg |
CVTM |
TPNM |
umif |
|
|
*) The `kern' validator handles both the classic and the new kern |
formats; the former is supported on both Microsoft and Apple |
platforms, while the latter is supported on Apple platforms. |
|
**) `acnt' tables are not supported by currently available Apple font |
tools. |
|
***) There is one more Apple extension, `hsty', but it is for |
Newton-OS, not GX (Newton-OS is a platform by Apple, but it can |
use sfnt- housed bitmap fonts only). Therefore, it should be |
excluded from `Apple platform' in the context of TrueType. |
gxvalid ignores it as Apple font tools do so. |
|
|
We have checked 183 fonts bundled with MacOS 9.1, MacOS 9.2, MacOS |
10.0, MacOS X 10.1, MSIE for MacOS, and AppleWorks 6.0. In addition, |
we have checked 67 Dynalab fonts (designed for MacOS) and 189 Ricoh |
fonts (designed for Windows and MacOS dual platforms). The number of |
fonts including TrueType GX tables are as follows. |
|
bsln: 76 |
feat: 191 |
just: 84 |
kern: 59 |
lcar: 4 |
mort: 326 |
morx: 19 |
opbd: 4 |
prop: 114 |
trak: 16 |
|
Dynalab and Ricoh fonts don't have GX tables except of `feat' and |
`mort'. |
|
|
3. Fundamental validation limitations |
------------------------------------- |
|
TrueType GX provides layout information to libraries for font |
rasterizers and text layout. gxvalid can check whether the layout |
data in a font is conformant to the TrueType GX format specified by |
Apple. But gxvalid cannot check a how QuickDraw GX/AAT renderer uses |
the stored information. |
|
3-1. Validation of State Machine activity |
----------------------------------------- |
|
QuickDraw GX/AAT uses a `State Machine' to provide `stateful' layout |
features, and TrueType GX stores the state transition diagram of |
this `State Machine' in a `StateTable' data structure. While the |
State Machine receives a series of glyph IDs, the State Machine |
starts with `start of text' state, walks around various states and |
generates various layout information to the renderer, and finally |
reaches the `end of text' state. |
|
gxvalid can check essential errors like: |
|
- possibility of state transitions to undefined states |
- existence of glyph IDs that the State Machine doesn't know how |
to handle |
- the State Machine cannot compute the layout information from |
given diagram |
|
These errors can be checked within finite steps, and without the |
State Machine itself, because these are `expression' errors of state |
transition diagram. |
|
There is no limitation about how long the State Machine walks |
around, so validation of the algorithm in the state transition |
diagram requires infinite steps, even if we had a State Machine in |
gxvalid. Therefore, the following errors and problems cannot be |
checked. |
|
- existence of states which the State Machine never transits to |
- the possibility that the State Machine never reaches `end of |
text' |
- the possibility of stack underflow/overflow in the State Machine |
(in ligature and contextual glyph substitutions, the State |
Machine can store 16 glyphs onto its stack) |
|
In addition, gxvalid doesn't check `temporary glyph IDs' used in the |
chained State Machines (in `mort' and `morx' tables). If a layout |
feature is implemented by a single State Machine, a glyph ID |
converted by the State Machine is passed to the glyph renderer, thus |
it should not point to an undefined glyph ID. But if a layout |
feature is implemented by chained State Machines, a component State |
Machine (if it is not the final one) is permitted to generate |
undefined glyph IDs for temporary use, because it is handled by next |
component State Machine and not by the glyph renderer. To validate |
such temporary glyph IDs, gxvalid must stack all undefined glyph IDs |
which can occur in the output of the previous State Machine and |
search them in the `ClassTable' structure of the current State |
Machine. It is too complex to list all possible glyph IDs from the |
StateTable, especially from a ligature substitution table. |
|
3-2. Validation of relationship between multiple layout features |
---------------------------------------------------------------- |
|
gxvalid does not validate the relationship between multiple layout |
features at all. |
|
If multiple layout features are defined in TrueType GX tables, |
possible interactions, overrides, and conflicts between layout |
features are implicitly given in the font too. For example, there |
are several predefined spacing control features: |
|
- Text Spacing (Proportional/Monospace/Half-width/Normal) |
- Number Spacing (Monospaced-numbers/Proportional-numbers) |
- Kana Spacing (Full-width/Proportional) |
- Ideographic Spacing (Full-width/Proportional) |
- CJK Roman Spacing (Half-width/Proportional/Default-roman |
/Full-width-roman/Proportional) |
|
If all layout features are independently managed, we can activate |
inconsistent typographic rules like `Text Spacing=Monospace' and |
`Ideographic Spacing=Proportional' at the same time. |
|
The combinations of layout features is managed by a 32bit integer |
(one bit each for selector setting), so we can define relationships |
between up to 32 features, theoretically. But if one feature |
setting affects another feature setting, we need typographic |
priority rules to validate the relationship. Unfortunately, the |
TrueType GX format specification does not give such information even |
for predefined features. |
|
|
4. Permissive error handling of broken GX tables |
------------------------------------------------ |
|
When Apple's font rendering system finds an inconsistency, like a |
specification violation or an unspecified value in a TrueType GX |
table, it does not always return error. In most cases, the rendering |
engine silently ignores such wrong values or even whole tables. In |
fact, MacOS is shipped with fonts including broken GX/AAT tables, but |
no harmful effects due to `officially broken' fonts are observed by |
end-users. |
|
gxvalid is designed to continue the validation process as long as |
possible. When gxvalid find wrong values, gxvalid warns it at least, |
and takes a fallback procedure if possible. The fallback procedure |
depends on the debug level. |
|
We used the following three tools to investigate Apple's error handling. |
|
- FontValidator (for MacOS 8.5 - 9.2) resource fork font |
- ftxvalidator (for MacOS X 10.1 -) dfont or naked-sfnt |
- ftxdumperfuser (for MacOS X 10.1 -) dfont or naked-sfnt |
|
However, all tests were done on a PowerPC based Macintosh; at present, |
we have not checked those tools on a m68k-based Macintosh. |
|
In total, we checked 183 fonts bundled to MacOS 9.1, MacOS 9.2, MacOS |
10.0, MacOS X 10.1, MSIE for MacOS, and AppleWorks 6.0. These fonts |
are distributed officially, but many broken GX/AAT tables were found |
by Apple's font tools. In the following, we list typical violation of |
the GX specification, in fonts officially distributed with those Apple |
systems. |
|
4-1. broken BinSrchHeader (19/183) |
---------------------------------- |
|
`BinSrchHeader' is a header of a data array for m68k platforms to |
access memory efficiently. Although there are only two independent |
parameters for real (`unitSize' and `nUnits'), BinSrchHeader has |
three additional parameters which can be calculated from `unitSize' |
and `nUnits', for fast setup. Apple font tools ignore them |
silently, so gxvalid warns if it finds and inconsistency, and always |
continues validation. The additional parameters are ignored |
regardless of the consistency. |
|
19 fonts include such inconsistencies; all breaks are in the |
BinSrchHeader structure of the `kern' table. |
|
4-2. too-short LookupTable (5/183) |
---------------------------------- |
|
LookupTable format 0 is a simple array to get a value from a given |
GID (glyph ID); the index of this array is a GID too. Therefore, |
the length of the array is expected to be same as the maximum GID |
value defined in the `maxp' table, but there are some fonts whose |
LookupTable format 0 is too short to cover all GIDs. FontValidator |
ignores this error silently, ftxvalidator and ftxdumperfuser both |
warn and continue. Similar problems are found in format 3 subtables |
of `kern'. gxvalid warns always and abort if the validation level |
is set to FT_VALIDATE_PARANOID. |
|
5 fonts include too-short kern format 0 subtables. |
1 font includes too-short kern format 3 subtable. |
|
4-3. broken LookupTable format 2 (1/183) |
---------------------------------------- |
|
LookupTable format 2, subformat 4 covers the GID space by a |
collection of segments which are specified by `firstGlyph' and |
`lastGlyph'. Some fonts store `firstGlyph' and `lastGlyph' in |
reverse order, so the segment specification is broken. Apple font |
tools ignore this error silently; a broken segment is ignored as if |
it did not exist. gxvalid warns and normalize the segment at |
FT_VALIDATE_DEFAULT, or ignore the segment at FT_VALIDATE_TIGHT, or |
abort at FT_VALIDATE_PARANOID. |
|
1 font includes broken LookupTable format 2, in the `just' table. |
|
*) It seems that all fonts manufactured by ITC for AppleWorks have |
this error. |
|
4-4. bad bracketing in glyph property (14/183) |
---------------------------------------------- |
|
GX/AAT defines a `bracketing' property of the glyphs in the `prop' |
table, to control layout features of strings enclosed inside and |
outside of brackets. Some fonts give inappropriate bracket |
properties to glyphs. Apple font tools warn about this error; |
gxvalid warns too and aborts at FT_VALIDATE_PARANOID. |
|
14 fonts include wrong bracket properties. |
|
|
4-5. invalid feature number (117/183) |
------------------------------------- |
|
The GX/AAT extension can include 255 different layout features, but |
popular layout features are predefined (see |
http://developer.apple.com/fonts/Registry/index.html). Some fonts |
include feature numbers which are incompatible with the predefined |
feature registry. |
|
In our survey, there are 140 fonts including `feat' table. |
|
a) 67 fonts use a feature number which should not be used. |
b) 117 fonts set the wrong feature range (nSetting). This is mostly |
found in the `mort' and `morx' tables. |
|
Apple font tools give no warning, although they cannot recognize |
what the feature is. At FT_VALIDATE_DEFAULT, gxvalid warns but |
continues in both cases (a, b). At FT_VALIDATE_TIGHT, gxvalid warns |
and aborts for (a), but continues for (b). At FT_VALIDATE_PARANOID, |
gxvalid warns and aborts in both cases (a, b). |
|
4-6. invalid prop version (10/183) |
---------------------------------- |
|
As most TrueType GX tables, the `prop' table must start with a 32bit |
version identifier: 0x00010000, 0x00020000 or 0x00030000. But some |
fonts store nonsense binary data instead. When Apple font tools |
find them, they abort the processing immediately, and the data which |
follows is unhandled. gxvalid does the same. |
|
10 fonts include broken `prop' version. |
|
All of these fonts are classic TrueType fonts for the Japanese |
script, manufactured by Apple. |
|
4-7. unknown resource name (2/183) |
------------------------------------ |
|
NOTE: THIS IS NOT A TRUETYPE GX ERROR. |
|
If a TrueType font is stored in the resource fork or in dfont |
format, the data must be tagged as `sfnt' in the resource fork index |
to invoke TrueType font handler for the data. But the TrueType font |
data in `Keyboard.dfont' is tagged as `kbd', and that in |
`LastResort.dfont' is tagged as `lst'. Apple font tools can detect |
that the data is in TrueType format and successfully validate them. |
Maybe this is possible because they are known to be dfont. The |
current implementation of the resource fork driver of FreeType |
cannot do that, thus gxvalid cannot validate them. |
|
2 fonts use an unknown tag for the TrueType font resource. |
|
5. `kern' table issues |
---------------------- |
|
In common terminology of TrueType, `kern' is classified as a basic and |
platform-independent table. But there are Apple extensions of `kern', |
and there is an extension which requires a GX state machine for |
contextual kerning. Therefore, gxvalid includes a special validator |
for `kern' tables. Unfortunately, there is no exact algorithm to |
check Apple's extension, so gxvalid includes a heuristic algorithm to |
find the proper validation routines for all possible data formats, |
including the data format for Microsoft. By calling |
classic_kern_validate() instead of gxv_validate(), you can specify the |
`kern' format explicitly. However, current FreeType2 uses Microsoft |
`kern' format only, others are ignored (and should be handled in a |
library one level higher than FreeType). |
|
5-1. History |
------------ |
|
The original 16bit version of `kern' was designed by Apple in the |
pre-GX era, and it was also approved by Microsoft. Afterwards, |
Apple designed a new 32bit version of the `kern' table. According |
to the documentation, the difference between the 16bit and 32bit |
version is only the size of variables in the `kern' header. In the |
following, we call the original 16bit version as `classic', and |
32bit version as `new'. |
|
5-2. Versions and dialects which should be differentiated |
--------------------------------------------------------- |
|
The `kern' table consists of a table header and several subtables. |
The version number which identifies a `classic' or a `new' version |
is explicitly written in the table header, but there are |
undocumented differences between Microsoft's and Apple's formats. |
It is called a `dialect' in the following. There are three cases |
which should be handled: the new Apple-dialect, the classic |
Apple-dialect, and the classic Microsoft-dialect. An analysis of |
the formats and the auto detection algorithm of gxvalid is described |
in the following. |
|
5-2-1. Version detection: classic and new kern |
---------------------------------------------- |
|
According to Apple TrueType specification, there are only two |
differences between the classic and the new: |
|
- The `kern' table header starts with the version number. |
The classic version starts with 0x0000 (16bit), |
the new version starts with 0x00010000 (32bit). |
|
- In the `kern' table header, the number of subtables follows |
the version number. |
In the classic version, it is stored as a 16bit value. |
In the new version, it is stored as a 32bit value. |
|
From Apple font tool's output (DumpKERN is also tested in addition |
to the three Apple font tools in above), there is another |
undocumented difference. In the new version, the subtable header |
includes a 16bit variable named `tupleIndex' which does not exist |
in the classic version. |
|
The new version can store all subtable formats (0, 1, 2, and 3), |
but the Apple TrueType specification does not mention the subtable |
formats available in the classic version. |
|
5-2-2. Available subtable formats in classic version |
---------------------------------------------------- |
|
Although the Apple TrueType specification recommends to use the |
classic version in the case if the font is designed for both the |
Apple and Microsoft platforms, it does not document the available |
subtable formats in the classic version. |
|
According to the Microsoft TrueType specification, the subtable |
format assured for Windows and OS/2 support is only subtable |
format 0. The Microsoft TrueType specification also describes |
subtable format 2, but does not mention which platforms support |
it. Aubtable formats 1, 3, and higher are documented as reserved |
for future use. Therefore, the classic version can store subtable |
formats 0 and 2, at least. `ttfdump.exe', a font tool provided by |
Microsoft, ignores the subtable format written in the subtable |
header, and parses the table as if all subtables are in format 0. |
|
`kern' subtable format 1 uses a StateTable, so it cannot be |
utilized without a GX State Machine. Therefore, it is reasonable |
to assume that format 1 (and 3) were introduced after Apple had |
introduced GX and moved to the new 32bit version. |
|
5-2-3. Apple and Microsoft dialects |
----------------------------------- |
|
The `kern' subtable has a 16bit `coverage' field to describe |
kerning attributes, but bit interpretations by Apple and Microsoft |
are different: For example, Apple uses bits 0-7 to identify the |
subtable, while Microsoft uses bits 8-15. |
|
In addition, due to the output of DumpKERN and FontValidator, |
Apple's bit interpretations of coverage in classic and new version |
are incompatible also. In summary, there are three dialects: |
classic Apple dialect, classic Microsoft dialect, and new Apple |
dialect. The classic Microsoft dialect and the new Apple dialect |
are documented by each vendors' TrueType font specification, but |
the documentation for classic Apple dialect is not available. |
|
For example, in the new Apple dialect, bit 15 is documented as |
`set to 1 if the kerning is vertical'. On the other hand, in |
classic Microsoft dialect, bit 1 is documented as `set to 1 if the |
kerning is horizontal'. From the outputs of DumpKERN and |
FontValidator, classic Apple dialect recognizes 15 as `set to 1 |
when the kerning is horizontal'. From the results of similar |
experiments, classic Apple dialect seems to be the Endian reverse |
of the classic Microsoft dialect. |
|
As a conclusion it must be noted that no font tool can identify |
classic Apple dialect or classic Microsoft dialect automatically. |
|
5-2-4. gxvalid auto dialect detection algorithm |
----------------------------------------------- |
|
The first 16 bits of the `kern' table are enough to identify the |
version: |
|
- if the first 16 bits are 0x0000, the `kern' table is in |
classic Apple dialect or classic Microsoft dialect |
- if the first 16 bits are 0x0001, and next 16 bits are 0x0000, |
the kern table is in new Apple dialect. |
|
If the `kern' table is a classic one, the 16bit `coverage' field |
is checked next. Firstly, the coverage bits are decoded for the |
classic Apple dialect using the following bit masks (this is based |
on DumpKERN output): |
|
0x8000: 1=horizontal, 0=vertical |
0x4000: not used |
0x2000: 1=cross-stream, 0=normal |
0x1FF0: reserved |
0x000F: subtable format |
|
If any of reserved bits are set or the subtable bits is |
interpreted as format 1 or 3, we take it as `impossible in classic |
Apple dialect' and retry, using the classic Microsoft dialect. |
|
The most popular coverage in new Apple-dialect: 0x8000, |
The most popular coverage in classic Apple-dialect: 0x0000, |
The most popular coverage in classic Microsoft dialect: 0x0001. |
|
5-3. Tested fonts |
----------------- |
|
We checked 59 fonts bundled with MacOS and 38 fonts bundled with |
Windows, where all font include a `kern' table. |
|
- fonts bundled with MacOS |
* new Apple dialect |
format 0: 18 |
format 2: 1 |
format 3: 1 |
* classic Apple dialect |
format 0: 14 |
* classic Microsoft dialect |
format 0: 15 |
|
- fonts bundled with Windows |
* classic Microsoft dialect |
format 0: 38 |
|
It looks strange that classic Microsoft-dialect fonts are bundled to |
MacOS: they come from MSIE for MacOS, except of MarkerFelt.dfont. |
|
|
ACKNOWLEDGEMENT |
--------------- |
|
Some parts of gxvalid are derived from both the `gxlayout' module and |
the `otvalid' module. Development of gxlayout was supported by the |
Information-technology Promotion Agency(IPA), Japan. |
|
The detailed analysis of undefined glyph ID utilization in `mort' and |
`morx' tables is provided by George Williams. |
|
------------------------------------------------------------------------ |
|
Copyright 2004, 2005, 2007 by |
suzuki toshiya, Masatake YAMATO, Red hat K.K., |
David Turner, Robert Wilhelm, and Werner Lemberg. |
|
This file is part of the FreeType project, and may only be used, |
modified, and distributed under the terms of the FreeType project |
license, LICENSE.TXT. By continuing to use, modify, or distribute this |
file you indicate that you have read the license and understand and |
accept it fully. |
|
|
--- end of README --- |