The Unicode Standard�, Version 2.1

Summary

This report documents the Unicode Standard, Version 2.1.

Status of this document

This document contains informative material and normative specifications which have been considered and approved by the Unicode Technical Committee for publication as a Technical Report and as part of the Unicode Standard, Version 2.1. Any reference to version 2.1 of the Unicode Standard automatically includes this technical report. Please mail corrigenda and other comments to the author.

The content of all technical reports must be understood in the context of the appropriate version of the Unicode Standard. References in this technical report to sections of the Unicode Standard refer to the Unicode Standard, Version 2.0. See http://www.unicode.org/unicode/standard/versions for more information.

Contents

• 1 Description
  • 1.1 Conformance
• 2 Object Replacement Character
• 3 Euro Sign
• 4 Errata
  • 4.1 Math Property Characters
  • 4.2 Letter Errata
  • 4.3 Canonical Decomposition Clarification
  • 4.4 Identifier Errata
  • 4.5 Bidirectional Behavior Errata
    • 4.5.1 Basic Display Algorithm
    • 4.5.2 Bidirectional Character Types
    • 4.5.3 The Base Level
    • 4.5.4 Terminating Embeddings and Overrides
    • 4.5.5 Resolving Weak Types
    • 4.5.6 Resolving Neutral Types (1)
    • 4.5.7 Resolving Neutral Types (2)
    • 4.5.8 Resolving Implicit Levels
    • 4.5.9 Reordering Resolved Levels
    • 4.5.10 Characters with New Directional Properties
  • 4.6 Apostrophe Semantics Errata
  • 4.7 Typographic Errata
  • 4.8 Glyph Errata
  • 4.9 UTF-7 Sample Code Correction
• 5 Unicode Character Database and Properties Changes
• Revisions

1 Description

Version 2.1 of the Unicode Standard brings together two additions to the repertoire which are expected to be in wide use in a number of implementations, errata collected since the publication of Version 2.0, and a number of updates to the character properties database. The two newly added characters are the U+FFFC OBJECT REPLACEMENT CHARACTER and the U+20AC EURO SIGN. The object replacement character is already employed in multiple implementations, and the euro sign is expected to be widely used very soon as the European Monetary Union (EMU) proceeds to phase in its use as the EMU unit of currency. This modification of the Unicode Standard is made available so that implementers can proceed with their support plans knowing that their implementation of Unicode is a well-defined, conforming version. With the additions of Version 2.1, the Unicode Standard contains 38, 887 characters from the world’s scripts.

Additional characters and scripts have been accepted into the Unicode Standard since the publication of The Unicode Standard, Version 2.0. These are not included in Version 2.1 but are documented on the Unicode Web site at: http://www.unicode.org/unicode/alloc/Pipeline.html

1.1 Conformance

Overall Unicode conformance criteria as described in Chapter 3 of Version 2.0 are unchanged. Specific aspects of the bidirectional algorithm have been modified in Version 2.1, Hangul syllable decompositions have been clarified, and certain normative character property values have been changed.

2 Object Replacement Character

The U+FFFC OBJECT REPLACEMENT CHARACTER is used as an insertion point for objects located within a stream of text. All other information about the object is kept outside the character data stream. Internally it is a dummy character which acts as an anchor point for the object’s formatting information. In addition to assuring correct placement of an object in a data stream, the object replacement character also allows the use of general stream-based algorithms for any textual aspects of embedded objects

The object replacement character is classified as a Symbol, Other (So) and has a bidirectional category of Other Neutrals (ON).

Addition p 7-523. Add to the standard the following character: FFFC OBJECT REPLACEMENT CHARACTER

3 Euro Sign

The new single currency for member countries of the European Monetary Union (EMU) is the euro. The euro character is encoded in the Unicode Standard as U+20AC EURO SIGN.

To avoid confusion, the historical character U+20A0 EURO-CURRENCY SIGN has been updated with an informative note and a cross reference to U+20AC EURO SIGN.

The euro character is classified as Symbol, Currency (Sc) and has a bidirectional category of European Number Terminator (ET).

Addition p 7-161. Add to the standard the following character: 20AC EURO SIGN Add the following informative note for 20AC: "Currency sign for the European Monetary Union" Add the following cross reference for 20AC: "20A0 euro-currency sign"

4 Errata

4.1 Math Property Characters

Additional Unicode characters have been designated as having the mathematical property. Typos in the Version 2.0 list of characters with the mathematical property have also been corrected.

4.2 Letter Errata

Two characters have been removed from the alphabetics listing, U+02BC MODIFIER LETTER APOSTROPHE and U+055A ARMENIAN APOSTROPHE.

4.3 Canonical Decomposition Clarification

The status of Hangul Syllable decompositions have been clarified.

4.4 Identifier Errata

New distinctions have been made in the Unicode Character Database for use in identifiers. In addition changes have been made to the text of the standard.

4.5 Bidirectional Behavior Errata

Since the Unicode Standard Version 2, many aspects of the bidirectional behavior algorithm have been clarified or modified, including the basic display algorithm, bidirectional character types, base levels, resolving weak and neutral types, and resolving implicit levels. These changes affect pages 3-14 through 3-23 of the standard. Additionally, a few characters have been assigned new bidirectional type properties.

4.5.1 Basic Display Algorithm

The description of the scope of the algorithm within a block has been clarified, and a pointer to further information on the handling of CR and LF has been added.

4.5.2 Bidirectional Character Types

The following (together with a change to Reordering Resolved Levels) clarifies how to implement the last paragraph of page 3-16.

4.5.3 The Base Level

Several of the rules were corrected to say embedding direction rather than global direction. The first term is more explicitly defined.

4.5.4 Terminating Embeddings and Overrides

T6 incorrectly removed implicit and explicit directional formatting codes. The original purpose of T6 was to allow the use of styles or style sheets instead of embedding or override codes (see p. 3-22). T6 has been eliminated, and N4 has been changed instead (see below).

4.5.5 Resolving Weak Types

P1 has been clarified to state that it applies to single characters, and P2 more explicitly shows how to resolve a sequence of European terminators.

4.5.6 Resolving Neutral Types (1)

The wording in N2 has been modified to use the embedding direction instead of the global direction, and the confusing term "letter" has been changed to "character" which makes it clear that strong R punctuation should be included.

4.5.7 Resolving Neutral Types (2)

Since N4 describes the behavior of embedding codes, it has been moved to a more appropriate place in the algorithm. It replaces T6 and now describes the behavior of override codes as well.

4.5.8 Resolving Implicit Levels

I1 and I2 have been modified to ensure that implementers will use the embedding direction instead of the base direction. Also, although Table 3-7 refers to Sequence Type, the wording was not clear that the rules applied to sequences. This is important in the case of EN.

4.5.9 Reordering Resolved Levels

L1 incorrectly implied that there could be more than one block separator. This has been corrected and more explanation is provided.

4.5.10 Characters with New Directional Properties

Certain characters have new bidirectional property definitions. To improve the display of e-mail addresses and URLs, the directional types of U+0026 AMPERSAND and U+0040 COMMERCIAL AT have been changed from left-to-right to other neutral. The directional type of U+002E FULL STOP  has been changed from EUROPEAN NUMBER SEPARATOR   to COMMON NUMBER SEPARATOR to improve the display of decimal numbers; U+2007 FIGURE SPACE has also been changed from EUROPEAN NUMBER SEPARATOR  to COMMON NUMBER SEPARATOR for consistency.

4.6 Apostrophe Semantics Errata

The following corrigenda clarify the semantics of different apostrophes, and correct problems in the mapping tables from Windows and Macintosh code pages.

Corrigendum p 6-3. Add at the end of Loose versus Precise Semantics: "For historical reasons, U+0027 is a particularly overloaded character. In ASCII it is used to represent a punctuation mark (such as right single quotation mark, left single quotation mark, apostrophe punctuation, vertical line, or prime) or a modifier letter (such as apostrophe modifier or acute accent.) (Punctuation marks generally break words; modifier letters generally are considered part of a word.) In many systems it is always represented as a straight vertical line and can never represent a curly apostrophe or right quotation mark. In the case of an apostrophe, U+02BC MODIFIER LETTER APOSTROPHE is preferred where the character is to represent a modifier letter (for example, in transliterations to indicate a glottal stop.) In the latter case, it is also referred to as a letter apostrophe. U+2019 RIGHT SINGLE QUOTATION MARKis preferred where the character is to represent a punctuation mark, as in "We’ve been here before." In the latter case, U+2019 is also referred to as a punctuation apostrophe. In implementation, however, you cannot assume that users’ text always adheres to the distinction between these characters. The text may come from different sources, including mapping from other character sets that do not have this distinction between letter apostrophe and punctuation apostrophe/right single quotation mark. In that case, all of them will generally be represented by U+2019. Where you are parsing text where such distinctions are important, you will still need to look at the context around the characters to help disambiguate the relevant semantics."

4.7 Typographic Errata

The following are typographic errors in the text of the standard.

4.8 Glyph Errata

A number of glyphs have been corrected. The corrections are given here and can be found on the Unicode Web site at:

http://www.unicode.org/unicode/uni2errata/UnicodeTypos.html

Additional glyph corrections will be posted to this site as available.

Corrigenda
05F1		HEBREW LIGATURE YIDDISH VAV YOD
2603		SNOWMAN
3085		HIRAGANA LETTER SMALL YU
FA0E		CJK Compatibility Ideograph
FA0F		CJK Compatibility Ideograph
FA10		CJK Compatibility Ideograph
FA11		CJK Compatibility Ideograph
FA12		CJK Compatibility Ideograph
FA13		CJK Compatibility Ideograph
FA14		CJK Compatibility Ideograph
FA15		CJK Compatibility Ideograph
FA16		CJK Compatibility Ideograph
FA17		CJK Compatibility Ideograph
FA18		CJK Compatibility Ideograph
FA19		CJK Compatibility Ideograph
FA1A		CJK Compatibility Ideograph
FA1B		CJK Compatibility Ideograph
FA1C		CJK Compatibility Ideograph
FA1D		CJK Compatibility Ideograph
FA1E		CJK Compatibility Ideograph
FA1F		CJK Compatibility Ideograph
FA20		CJK Compatibility Ideograph
FA21		CJK Compatibility Ideograph
FA22		CJK Compatibility Ideograph
FA23		CJK Compatibility Ideograph
FA24		CJK Compatibility Ideograph
FA25		CJK Compatibility Ideograph
FA26		CJK Compatibility Ideograph
FA27		CJK Compatibility Ideograph
FA28		CJK Compatibility Ideograph
FA29		CJK Compatibility Ideograph
FA2A		CJK Compatibility Ideograph
FA2B		CJK Compatibility Ideograph
FA2C		CJK Compatibility Ideograph
FA2D		CJK Compatibility Ideograph

4.9 UTF-7 Sample Code Correction

The UTF-7 specification was unclear on one point, which led to an error in the sample code for converting from UCS-2 to UTF-7. The problem occurs when U+002D HYPHEN-MINUS follows a character that must be encoded. Because ASCII 0x2D is the terminating character for an encoded sequence, two 0x2D characters must be output in order to preserve the U+002D when converting back to Unicode.

RFC 2152 has been published with a revised version of the UTF-7 specifications. The file included with the CD-ROM file has been updated with this fix.

p A-5. The correction is in the code near the bottom of the page. The new text is highlighted.

if (!needshift)

{
    /* Write the explicit shift out character if
        1) The caller has requested that we always do it, or
        2) The directly encoded character is in the
        base64 set, or
        3) The directly encoded character is SHIFT_OUT.
        */

    if (verbose || ((!done) && (invbase64[r] >=0
       || r == SHIFT_OUT)))
    {
        TARGETCHECK;
        *target++ = SHIFT_OUT
    }
    shifted = 0;
}

5 Unicode Character Database and Properties Changes

In addition to including the properties for the object replacement character and the euro sign, the Unicode Technical Committee has approved changes to the Unicode Character Database to reconcile problems found in an analysis of the character categories, and to make new distinctions in the database for use in identifiers. The property changes reflect the following:

1. Encoding of U+20AC EURO SIGN and U+FFFC OBJECT REPLACEMENT CHARACTER

2. Removing space, white space and delimitation as characteristics of U+FEFF

3. Narrowing the concept of white space to avoid miscellaneous ignorable Unicode controls and the Unicode NULL.

4. Mandated changes in directional properties, expanded to compatibility forms for consistency

The details are given in the following table:

Space	Remove FEFF
White space	Remove 0000, 200C..200F,202A..202E, 206A..206F, FEFF
Punctuation	Add 00B7
Delimiter	Remove FEFF
Currency Symbol	Add 20AC
Bidi: Left-to-Right	Remove 0026, 0040, FE60, FE6B, FF06, FF20
Bidi: Eur Num Term	Add 20AC
Bidi: Eur Num Sep	Remove 002E, 2007, FE52, FF0E
Bidi: Common Sep	Add 002E, 2007, FE52, FF0E
Bidi: Other Neutrals	Add 0026, 0040, FE60, FE6B, FF06, FF20
Unassigned Code Value	Remove 20AC, FFFC

This new information is reflected in the newest version of the Unicode Character Database and the additional properties files in the Unicode 2.1 Update directory on the unicode.org ftp site:

ftp://ftp.unicode.org/Public/2.1-Update/

The 2.1 files in the update directory supersede the three 2.0 files on the CD-ROM, which is distributed with The Unicode Standard, Version 2.0, and which are also available at:

ftp://ftp.unicode.org/Public/UNIDATA

• PROPS2.TXT (superseded by PropList-2.1.1.txt)

• UNIDAT2.TXT (superseded by UnicodeData-2.1.1.txt)

• README2.TXT (superseded by ReadMe-2.1.1.txt)

A diff file cataloging the changes in the Unicode Character Database file is also available:

ftp://ftp.unicode.org/Public/2.1-Update/diff2014v211.txt

Revisions

Changes for Revision 3

Formatting corrections were made.

Changes for Revision 2

Correction of typographical and glyph errors as follows:

1. Typo in section 3.4 Identifier Errata, third line describing compatability low lines corrected to read FE33, not FF33.

2. Glyph for U+FA0E in section 3.8 corrected.

3. Under 3.4 Identifier Errata, in the small unlined table towards the bottom, under "Coverage," second entry, changed "Enclosing mark" to "Spacing combining mark."

Internal hyperlinks added at beginning of document.

Changes for Revision 1

Correction of two typographical errors as follows:

1.  In the section 3.9 "UTF-7 Sample Code Correction", in the sentence, " The problem occurs when U+200D HYPHEN-MINUS follows a character that must be encoded." "U+200D" corrected to read "U+002D".

2.  In section 3.6 the third corrigendum, "p 7-37. Change character 02BC informative notes, first bullet to:"   "first" corrected to read "third."

Copyright � 1998-1999 Unicode, Inc. All Rights Reserved. The Unicode Consortium makes no expressed or implied warranty of any kind, and assumes no liability for errors or omissions. No liability is assumed for incidental and consequential damages in connection with or arising out of the use of the information or programs contained or accompanying this technical report.

Unicode and the Unicode logo are trademarks of Unicode, Inc., and are registered in some jurisdictions.