Subversion Repositories Kolibri OS

gxvalid: TrueType GX validator

==============================

1. What is this

---------------

  `gxvalid' is a module to  validate TrueType GX tables: a collection of

  additional tables  in TrueType  font which are  used by  `QuickDraw GX

  Text',  Apple Advanced  Typography  (AAT).  In  addition, gxvalid  can

  validates `kern'  tables which have  been extended for AAT.   Like the

  otvalid  module,   gxvalid  uses  Freetype   2's  validator  framework

  (ftvalid).

  You can link gxvalid with your program; before running your own layout

  engine, gxvalid validates a font  file.  As the result, you can remove

  error-checking code  from the layout  engine.  It is also  possible to

  use  gxvalid  as a  stand-alone  font  validator;  the `ftvalid'  test

  program  included  in the  ft2demo  bundle  calls gxvalid  internally.

  A stand-alone font validator may be useful for font developers.

  This documents documents the following issues.

  - supported TrueType GX tables

  - fundamental validation limitations

  - permissive error handling of broken GX tables

  - `kern' table issue.

2. Supported tables

-------------------

  The following GX tables are currently supported.

    bsln

    feat

    just

    kern(*)

    lcar

    mort

    morx

    opbd

    prop

    trak

  The following GX tables are currently unsupported.

    cvar

    fdsc

    fmtx

    fvar

    gvar

    Zapf

  The following GX tables won't be supported.

    acnt(**)

    hsty(***)

  The following undocumented tables in TrueType fonts designed for Apple

  platform aren't handled either.

    addg

    CVTM

    TPNM

    umif

  *)   The `kern'  validator handles both  the classic and the  new kern

       formats;  the former  is supported  on both  Microsoft  and Apple

       platforms, while the latter is supported on Apple platforms.

  **)  `acnt' tables are not supported by currently available Apple font

       tools.

  ***) There  is  one more  Apple  extension,  `hsty',  but  it  is  for

       Newton-OS, not GX  (Newton-OS is a platform by  Apple, but it can

       use  sfnt- housed bitmap  fonts only).   Therefore, it  should be

       excluded  from  `Apple  platform'  in the  context  of  TrueType.

       gxvalid ignores it as Apple font tools do so.

  We have  checked 183  fonts bundled with  MacOS 9.1, MacOS  9.2, MacOS

  10.0, MacOS X 10.1, MSIE  for MacOS, and AppleWorks 6.0.  In addition,

  we have  checked 67 Dynalab fonts  (designed for MacOS)  and 189 Ricoh

  fonts (designed for Windows and  MacOS dual platforms).  The number of

  fonts including TrueType GX tables are as follows.

    bsln:  76

    feat: 191

    just:  84

    kern:  59

    lcar:   4

    mort: 326

    morx:  19

    opbd:   4

    prop: 114

    trak:  16

  Dynalab  and Ricoh fonts  don't have  GX tables  except of  `feat' and

  `mort'.

3. Fundamental validation limitations

-------------------------------------

  TrueType  GX  provides  layout   information  to  libraries  for  font

  rasterizers  and text layout.   gxvalid can  check whether  the layout

  data in  a font is conformant  to the TrueType GX  format specified by

  Apple.  But gxvalid cannot check  a how QuickDraw GX/AAT renderer uses

  the stored information.

  3-1. Validation of State Machine activity

  -----------------------------------------

    QuickDraw GX/AAT uses a `State Machine' to provide `stateful' layout

    features,  and TrueType GX  stores the  state transition  diagram of

    this `State  Machine' in a  `StateTable' data structure.   While the

    State  Machine receives  a series  of glyph  IDs, the  State Machine

    starts with `start  of text' state, walks around  various states and

    generates various  layout information  to the  renderer, and finally

    reaches the `end of text' state.

    gxvalid can check essential errors like:

      - possibility of state transitions to undefined states

      - existence of glyph  IDs that the State Machine  doesn't know how

        to handle

      - the  State Machine  cannot compute  the layout  information from

        given diagram

    These errors  can be  checked within finite  steps, and  without the

    State Machine itself, because these are `expression' errors of state

    transition diagram.

    There  is no  limitation  about  how long  the  State Machine  walks

    around,  so validation  of  the algorithm  in  the state  transition

    diagram requires infinite  steps, even if we had  a State Machine in

    gxvalid.   Therefore, the  following errors  and problems  cannot be

    checked.

      - existence of states which the State Machine never transits to

      - the  possibility that the  State Machine  never reaches  `end of

        text'

      - the possibility of stack underflow/overflow in the State Machine

        (in  ligature  and  contextual  glyph substitutions,  the  State

        Machine can store 16 glyphs onto its stack)

    In addition, gxvalid doesn't check `temporary glyph IDs' used in the

    chained State Machines  (in `mort' and `morx' tables).   If a layout

    feature  is  implemented by  a  single  State  Machine, a  glyph  ID

    converted by the State Machine is passed to the glyph renderer, thus

    it  should not  point to  an undefined  glyph ID.   But if  a layout

    feature is implemented by  chained State Machines, a component State

    Machine  (if it  is  not the  final  one) is  permitted to  generate

    undefined glyph IDs for temporary use, because it is handled by next

    component State Machine and not  by the glyph renderer.  To validate

    such temporary glyph IDs, gxvalid must stack all undefined glyph IDs

    which  can occur in  the output  of the  previous State  Machine and

    search  them in  the  `ClassTable' structure  of  the current  State

    Machine.  It is too complex to  list all possible glyph IDs from the

    StateTable, especially from a ligature substitution table.

  3-2. Validation of relationship between multiple layout features

  ----------------------------------------------------------------

    gxvalid does  not validate the relationship  between multiple layout

    features at all.

    If  multiple layout  features  are defined  in  TrueType GX  tables,

    possible  interactions,  overrides,  and  conflicts  between  layout

    features are implicitly  given in the font too.   For example, there

    are several predefined spacing control features:

      - Text Spacing          (Proportional/Monospace/Half-width/Normal)

      - Number Spacing        (Monospaced-numbers/Proportional-numbers)

      - Kana Spacing          (Full-width/Proportional)

      - Ideographic Spacing   (Full-width/Proportional)

      - CJK Roman Spacing     (Half-width/Proportional/Default-roman

                               /Full-width-roman/Proportional)

    If all  layout features are  independently managed, we  can activate

    inconsistent  typographic rules  like  `Text Spacing=Monospace'  and

    `Ideographic Spacing=Proportional' at the same time.

    The combinations  of layout features  is managed by a  32bit integer

    (one bit each for selector  setting), so we can define relationships

    between  up  to 32  features,  theoretically.   But  if one  feature

    setting  affects  another   feature  setting,  we  need  typographic

    priority  rules to  validate the  relationship.   Unfortunately, the

    TrueType GX format specification does not give such information even

    for predefined features.

4. Permissive error handling of broken GX tables

------------------------------------------------

  When  Apple's font  rendering system  finds an  inconsistency,  like a

  specification  violation or  an  unspecified value  in  a TrueType  GX

  table, it does not always  return error.  In most cases, the rendering

  engine silently  ignores such wrong  values or even whole  tables.  In

  fact, MacOS is shipped with  fonts including broken GX/AAT tables, but

  no harmful  effects due to  `officially broken' fonts are  observed by

  end-users.

  gxvalid  is designed  to continue  the validation  process as  long as

  possible.  When gxvalid find wrong  values, gxvalid warns it at least,

  and takes  a fallback procedure  if possible.  The  fallback procedure

  depends on the debug level.

  We used the following three tools to investigate Apple's error handling.

    - FontValidator  (for MacOS 8.5 - 9.2)  resource fork font

    - ftxvalidator   (for MacOS X 10.1 -)   dfont or naked-sfnt

    - ftxdumperfuser (for MacOS X 10.1 -)   dfont or naked-sfnt

  However, all tests were done on a PowerPC based Macintosh; at present,

  we have not checked those tools on a m68k-based Macintosh.

  In total, we checked 183 fonts  bundled to MacOS 9.1, MacOS 9.2, MacOS

  10.0, MacOS X  10.1, MSIE for MacOS, and  AppleWorks 6.0.  These fonts

  are distributed  officially, but many broken GX/AAT  tables were found

  by Apple's font tools.  In the following, we list typical violation of

  the GX specification, in fonts officially distributed with those Apple

  systems.

  4-1. broken BinSrchHeader (19/183)

  ----------------------------------

    `BinSrchHeader' is  a header of a  data array for  m68k platforms to

    access memory efficiently.  Although  there are only two independent

    parameters  for real  (`unitSize' and  `nUnits'),  BinSrchHeader has

    three additional parameters which  can be calculated from `unitSize'

    and  `nUnits',  for  fast  setup.   Apple  font  tools  ignore  them

    silently, so gxvalid warns if it finds and inconsistency, and always

    continues  validation.    The  additional  parameters   are  ignored

    regardless of the consistency.

      19  fonts include  such  inconsistencies; all  breaks  are in  the

      BinSrchHeader structure of the `kern' table.

  4-2. too-short LookupTable (5/183)

  ----------------------------------

    LookupTable format 0  is a simple array to get a  value from a given

    GID (glyph  ID); the index of  this array is a  GID too.  Therefore,

    the length  of the array is expected  to be same as  the maximum GID

    value defined  in the `maxp' table,  but there are  some fonts whose

    LookupTable format 0 is too  short to cover all GIDs.  FontValidator

    ignores  this error silently,  ftxvalidator and  ftxdumperfuser both

    warn and continue.  Similar problems are found in format 3 subtables

    of `kern'.  gxvalid  warns always and abort if  the validation level

    is set to FT_VALIDATE_PARANOID.

      5 fonts include too-short kern format 0 subtables.

      1 font includes too-short kern format 3 subtable.

  4-3. broken LookupTable format 2 (1/183)

  ----------------------------------------

    LookupTable  format  2,  subformat  4  covers the  GID  space  by  a

    collection  of  segments which  are  specified  by `firstGlyph'  and

    `lastGlyph'.   Some  fonts  store  `firstGlyph' and  `lastGlyph'  in

    reverse order,  so the segment specification is  broken.  Apple font

    tools ignore this error silently;  a broken segment is ignored as if

    it  did not  exist.   gxvalid  warns and  normalize  the segment  at

    FT_VALIDATE_DEFAULT, or ignore  the segment at FT_VALIDATE_TIGHT, or

    abort at FT_VALIDATE_PARANOID.

      1 font includes broken LookupTable format 2, in the `just' table.

    *) It seems  that all fonts manufactured by  ITC for AppleWorks have

       this error.

  4-4. bad bracketing in glyph property (14/183)

  ----------------------------------------------

    GX/AAT defines a  `bracketing' property of the glyphs  in the `prop'

    table,  to control layout  features of  strings enclosed  inside and

    outside  of   brackets.   Some  fonts   give  inappropriate  bracket

    properties  to glyphs.   Apple  font tools  warn  about this  error;

    gxvalid warns too and aborts at FT_VALIDATE_PARANOID.

      14 fonts include wrong bracket properties.

  4-5. invalid feature number (117/183)

  -------------------------------------

    The GX/AAT extension can  include 255 different layout features, but

    popular      layout      features      are      predefined      (see

    http://developer.apple.com/fonts/Registry/index.html).   Some  fonts

    include feature  numbers which are incompatible  with the predefined

    feature registry.

    In our survey, there are 140 fonts including `feat' table.

    a) 67 fonts use a feature number which should not be used.

    b) 117 fonts set the wrong feature range (nSetting).  This is mostly

       found in the `mort' and `morx' tables.

    Apple  font tools give  no warning,  although they  cannot recognize

    what  the feature  is.   At FT_VALIDATE_DEFAULT,  gxvalid warns  but

    continues in both cases (a, b).  At FT_VALIDATE_TIGHT, gxvalid warns

    and aborts for (a), but continues for (b).  At FT_VALIDATE_PARANOID,

    gxvalid warns and aborts in both cases (a, b).

  4-6. invalid prop version (10/183)

  ----------------------------------

    As most TrueType GX tables, the `prop' table must start with a 32bit

    version identifier: 0x00010000,  0x00020000 or 0x00030000.  But some

    fonts  store nonsense binary  data instead.   When Apple  font tools

    find them, they abort the processing immediately, and the data which

    follows is unhandled.  gxvalid does the same.

      10 fonts include broken `prop' version.

    All  of these  fonts are  classic  TrueType fonts  for the  Japanese

    script, manufactured by Apple.

  4-7. unknown resource name (2/183)

  ------------------------------------

    NOTE: THIS IS NOT A TRUETYPE GX ERROR.

    If  a TrueType  font is  stored  in the  resource fork  or in  dfont

    format, the data must be tagged as `sfnt' in the resource fork index

    to invoke TrueType font handler for the data.  But the TrueType font

    data  in   `Keyboard.dfont'  is  tagged   as  `kbd',  and   that  in

    `LastResort.dfont' is tagged as  `lst'.  Apple font tools can detect

    that the data is in  TrueType format and successfully validate them.

    Maybe  this is possible  because they  are known  to be  dfont.  The

    current  implementation  of the  resource  fork  driver of  FreeType

    cannot do that, thus gxvalid cannot validate them.

      2 fonts use an unknown tag for the TrueType font resource.

5. `kern' table issues

----------------------

  In common terminology of TrueType, `kern' is classified as a basic and

  platform-independent table.  But there are Apple extensions of `kern',

  and  there is  an  extension which  requires  a GX  state machine  for

  contextual kerning.   Therefore, gxvalid includes  a special validator

  for  `kern' tables.   Unfortunately, there  is no  exact  algorithm to

  check Apple's extension, so  gxvalid includes a heuristic algorithm to

  find  the proper validation  routines for  all possible  data formats,

  including    the   data    format   for    Microsoft.     By   calling

  classic_kern_validate() instead of gxv_validate(), you can specify the

  `kern' format  explicitly.  However, current  FreeType2 uses Microsoft

  `kern' format  only, others  are ignored (and  should be handled  in a

  library one level higher than FreeType).

  5-1. History

  ------------

    The original  16bit version of `kern'  was designed by  Apple in the

    pre-GX  era, and  it was  also approved  by  Microsoft.  Afterwards,

    Apple designed a  new 32bit version of the  `kern' table.  According

    to  the documentation, the  difference between  the 16bit  and 32bit

    version is only the size of  variables in the `kern' header.  In the

    following,  we call  the original  16bit version  as  `classic', and

    32bit version as `new'.

  5-2. Versions and dialects which should be differentiated

  ---------------------------------------------------------

    The `kern' table  consists of a table header  and several subtables.

    The version number  which identifies a `classic' or  a `new' version

    is  explicitly   written  in  the   table  header,  but   there  are

    undocumented  differences between  Microsoft's and  Apple's formats.

    It is  called a `dialect' in  the following.  There  are three cases

    which  should  be  handled:   the  new  Apple-dialect,  the  classic

    Apple-dialect,  and the classic  Microsoft-dialect.  An  analysis of

    the formats and the auto detection algorithm of gxvalid is described

    in the following.

    5-2-1. Version detection: classic and new kern

    ----------------------------------------------

      According  to Apple  TrueType  specification, there  are only  two

      differences between the classic and the new:

        - The `kern' table header starts with the version number.

          The classic version starts with 0x0000 (16bit),

          the new version starts with 0x00010000 (32bit).

        - In the  `kern' table header,  the number of  subtables follows

          the version number.

          In the classic version, it is stored as a 16bit value.

          In the new version, it is stored as a 32bit value.

      From Apple font tool's output (DumpKERN is also tested in addition

      to  the  three  Apple  font  tools in  above),  there  is  another

      undocumented difference.  In the  new version, the subtable header

      includes a 16bit variable  named `tupleIndex' which does not exist

      in the classic version.

      The new version  can store all subtable formats (0,  1, 2, and 3),

      but the Apple TrueType specification does not mention the subtable

      formats available in the classic version.

    5-2-2. Available subtable formats in classic version

    ----------------------------------------------------

      Although the  Apple TrueType  specification recommends to  use the

      classic version in  the case if the font is  designed for both the

      Apple and Microsoft platforms,  it does not document the available

      subtable formats in the classic version.

      According  to the Microsoft  TrueType specification,  the subtable

      format  assured for  Windows  and OS/2  support  is only  subtable

      format  0.  The  Microsoft TrueType  specification  also describes

      subtable format  2, but does  not mention which  platforms support

      it.  Aubtable formats 1, 3,  and higher are documented as reserved

      for future use.  Therefore, the classic version can store subtable

      formats 0 and 2, at least.  `ttfdump.exe', a font tool provided by

      Microsoft,  ignores the  subtable format  written in  the subtable

      header, and parses the table as if all subtables are in format 0.

      `kern'  subtable format  1  uses  a StateTable,  so  it cannot  be

      utilized without a GX  State Machine.  Therefore, it is reasonable

      to assume  that format 1 (and  3) were introduced  after Apple had

      introduced GX and moved to the new 32bit version.

    5-2-3. Apple and Microsoft dialects

    -----------------------------------

      The  `kern' subtable  has  a 16bit  `coverage'  field to  describe

      kerning attributes, but bit interpretations by Apple and Microsoft

      are different:  For example, Apple  uses bits 0-7 to  identify the

      subtable, while Microsoft uses bits 8-15.

      In  addition, due  to the  output of  DumpKERN  and FontValidator,

      Apple's bit interpretations of coverage in classic and new version

      are  incompatible also.   In  summary, there  are three  dialects:

      classic Apple  dialect, classic  Microsoft dialect, and  new Apple

      dialect.  The classic Microsoft  dialect and the new Apple dialect

      are documented  by each vendors' TrueType  font specification, but

      the documentation for classic Apple dialect is not available.

      For example,  in the  new Apple dialect,  bit 15 is  documented as

      `set to  1 if  the kerning  is vertical'.  On  the other  hand, in

      classic Microsoft dialect, bit 1 is documented as `set to 1 if the

      kerning  is  horizontal'.   From   the  outputs  of  DumpKERN  and

      FontValidator, classic  Apple dialect recognizes  15 as `set  to 1

      when  the kerning  is horizontal'.   From the  results  of similar

      experiments, classic Apple dialect  seems to be the Endian reverse

      of the classic Microsoft dialect.

      As a  conclusion it must be  noted that no font  tool can identify

      classic Apple dialect or classic Microsoft dialect automatically.

    5-2-4. gxvalid auto dialect detection algorithm

    -----------------------------------------------

      The first 16  bits of the `kern' table are  enough to identify the

      version:

        - if  the first  16  bits are  0x0000,  the `kern'  table is  in

          classic Apple dialect or classic Microsoft dialect

        - if the first 16 bits are  0x0001, and next 16 bits are 0x0000,

          the kern table is in new Apple dialect.

      If the `kern'  table is a classic one,  the 16bit `coverage' field

      is checked next.   Firstly, the coverage bits are  decoded for the

      classic Apple dialect using the following bit masks (this is based

      on DumpKERN output):

        0x8000: 1=horizontal, 0=vertical

        0x4000: not used

        0x2000: 1=cross-stream, 0=normal

        0x1FF0: reserved

        0x000F: subtable format

      If  any  of  reserved  bits  are  set  or  the  subtable  bits  is

      interpreted as format 1 or 3, we take it as `impossible in classic

      Apple dialect' and retry, using the classic Microsoft dialect.

        The most popular coverage in new Apple-dialect:         0x8000,

        The most popular coverage in classic Apple-dialect:     0x0000,

        The most popular coverage in classic Microsoft dialect: 0x0001.

  5-3. Tested fonts

  -----------------

    We checked  59 fonts  bundled with MacOS  and 38 fonts  bundled with

    Windows, where all font include a `kern' table.

      - fonts bundled with MacOS

        * new Apple dialect

          format 0: 18

          format 2:  1

          format 3:  1

        * classic Apple dialect

          format 0: 14

        * classic Microsoft dialect

          format 0: 15

      - fonts bundled with Windows

        * classic Microsoft dialect

          format 0: 38

    It looks strange that classic Microsoft-dialect fonts are bundled to

    MacOS: they come from MSIE for MacOS, except of MarkerFelt.dfont.

  ACKNOWLEDGEMENT

  ---------------

  Some parts of gxvalid are  derived from both the `gxlayout' module and

  the `otvalid'  module.  Development of  gxlayout was supported  by the

  Information-technology Promotion Agency(IPA), Japan.

  The detailed analysis of undefined  glyph ID utilization in `mort' and

  `morx' tables is provided by George Williams.

------------------------------------------------------------------------

Copyright 2004, 2005, 2007 by

suzuki toshiya, Masatake YAMATO, Red hat K.K.,

David Turner, Robert Wilhelm, and Werner Lemberg.

This  file is  part  of the  FreeType  project, and  may  only be  used,

modified,  and  distributed under  the  terms  of  the FreeType  project

license, LICENSE.TXT.  By continuing  to use, modify, or distribute this

file  you indicate that  you have  read the  license and  understand and

accept it fully.

--- end of README ---