Unicode and HTML
From Wikipedia, the free encyclopedia
| This article contains special characters. Without proper rendering support, you may see question marks, boxes, or other symbols. |
| Unicode |
|---|
| Character encodings |
| UCS |
| Mapping |
| Bi-directional text |
| BOM |
| Han unification |
| Unicode and HTML |
| Unicode and E-mail |
| Unicode typefaces |
| HTML |
|---|
 |
|
Web pages authored using hypertext markup language (HTML) may contain multilingual text represented with the Unicode universal character set.
The relationship between Unicode and HTML tends to be a difficult topic for many computer professionals, document authors, and web users alike. The accurate representation of text in web pages from different natural languages and writing systems is complicated by the details of character encoding, markup language syntax, font, and varying levels of support by web browsers.
Contents |
[edit] HTML document characters
Web pages are typically HTML or XHTML documents. Both types of documents consist, at a fundamental level, of characters, which are graphemes and grapheme-like units, independent of how they manifest in computer storage systems and networks.
An HTML document is a sequence of Unicode characters. More specifically, HTML 4.0 documents are required to consist of characters in the HTML document character set: a character repertoire wherein each character is assigned a unique, non-negative integer code point. This set is defined in the HTML 4.0 DTD, which also establishes the syntax (allowable sequences of characters) that can produce a valid HTML document. The HTML document character set for HTML 4.0 consists of most, but not all, of the characters jointly defined by Unicode and ISO/IEC 10646: the Universal Character Set (UCS).
Like HTML documents, an XHTML document is a sequence of Unicode characters. However, an XHTML document is an XML document, which, while not having an explicit "document character" layer of abstraction, nevertheless relies upon a similar definition of permissible characters that cover most, but not all, of the Unicode/UCS character definitions. The sets used by HTML and XHTML/XML are slightly different, but these differences have little effect on the average document author.
Regardless of whether the document is HTML or XHTML, when stored on a file system or transmitted over a network, the document's characters are encoded as a sequence of bit octets (bytes) according to a particular character encoding. This encoding may either be a Unicode Transformation Format, like UTF-8, that can directly encode any Unicode character, or a legacy encoding, like Windows-1252, that cannot. However, even when using encodings that do not support all Unicode characters, the encoded document may make use of numeric character references. For example ☺ (☺) is used to indicate a smiling face character in the Unicode character set.
[edit] Numeric character references
In order to work around the limitations of legacy encodings, HTML is designed such that it is possible to represent characters from the whole of Unicode inside an HTML document by using a numeric character reference: a sequence of characters that explicitly spell out the Unicode code point of the character being represented. A character reference takes the form &#N;, where N is either a decimal number for the Unicode code point, or a hexadecimal number, in which case it must be prefixed by x. The characters that compose the numeric character reference are universally representable in every encoding approved for use on the Internet.
For example, a Unicode code point like U+53F6, which corresponds to a particular Chinese character, has to be converted to a decimal number, preceded by &# and followed by ;, like this: 叶, which produces this: 叶 (if it doesn't look like a Chinese character, see the special characters note at bottom of article).
The support for hexadecimal in this context is more recent, so older browsers might have problems displaying characters referenced with hexadecimal numbers—but they will probably have a problem displaying Unicode characters above code point 255 anyway. To ensure better compatibility with older browsers, it is still a common practice to convert the hexadecimal code point into a decimal value (for example 叶 instead of 叶).
[edit] Named character entities
In HTML there is a standard set of 252 named character entities for characters — some common, some obscure — that are either not found in certain character encodings or are markup sensitive in some contexts (for example angle brackets and quotation marks). Although any Unicode character can be referenced by its numeric code point, some HTML document authors prefer to use these named entities instead, where possible, as they are less cryptic and were better supported by early browsers.
Character entities can be included in an HTML document via the use of entity references, which take the form &EntityName;, where EntityName is the name of the entity. For example, —, much like — or —, represents U+2014: the em dash character — like this — even if the character encoding used doesn't contain that character.
[edit] Character encoding determination
In order to correctly process HTML, a web browser must ascertain which Unicode characters are represented by the encoded form of an HTML document. In order to do this, the web browser must know what encoding was used. When a document is transmitted via a MIME message or a transport that uses MIME content types such as an HTTP response, the message may signal the encoding via a Content-Type header, such as Content-Type: text/html; charset=ISO-8859-1. Other external means of declaring encoding are permitted but rarely used. The encoding may also be declared within the document itself, in the form of a META element, like <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">. This requires an extension of ASCII to be used, like UTF-8. When there is no encoding declaration, the default varies depending on the localisation of the browser.
For a system set up mainly for Western European languages, it will generally be ISO-8859-1 or its close relation Windows-1252. For a browser from a location where multibyte character encodings are the norm, some form of autodetection is likely to be applied.
Because of the legacy of 8-bit text representations in programming languages and operating systems and the desire to avoid burdening users with the need to understand the nuances of encoding many text editors used by HTML authors are unable or unwilling to offer a choice of encodings when saving files to disk and often do not even allow input of characters beyond a very limited range. Consequently many HTML authors are unaware of encoding issues and may not have any idea what encoding their documents actually use. It is also a common misunderstanding that the encoding declaration effects a change in the actual encoding - whereas it is actually just a label that could be inaccurate.
Many HTML documents are served with inaccurate encoding declarations, or no declarations at all. In order to determine the encoding in such cases, many browsers allow the user to manually select one from a list. They may also employ an encoding autodetection algorithm that works in concert with the manual override. The manual override may apply to all documents, or only those for which the encoding cannot be ascertained by looking at declarations and/or byte patterns. The fact that the manual override is present and widely used hinders the adoption of accurate encoding declarations on the Web; therefore the problem is likely to persist. This has been addressed somewhat by XHTML, which, being XML, requires that encoding declarations be accurate and that no workarounds be employed when they're found to be inaccurate. Though XML does permit higher protocols to override encodings or handoff encoding information for documents without encoding declarations, HTTP only does so for resources with content-type "text/*". Therefore for XML documents and XHTML documents delivered as content-type "application/xhtml+xml" there is no danger that inaccurate HTTP header information will override a correctly declared XML (and XHTML) document.
For both serializations of HTML (content-type "text/html" and content/type "application/xhtml+xml") using UTF-16 (or UTF-32) also provides an effective way to transmit encoding information within an HTML document. Since these UTF encodings require an initial byte-order Mark character (U+FEFF), the encoding automatically declares itself to any processing application. Processing applications need only look for an initial 0x0000FEFF or 0xFEFF in the byte steam to identify the document as UTF-32 or UTF-16 encoded respectively. No additional metadata mechanisms are required for these encodings since the byte-order mark includes all of the information necessary for processing applications. In most circumstances the byte-order mark character is handled by editing applications separately from the other characters so there is little risk of an author removing or otherwise changing the byte order mark to indicate the wrong encoding (as can happen when the encoding is declared in English/Latin script). If the document lacks a byte-order mark, the fact that the first non-blank printable character in an HTML document is supposed to be < (U+003C) can be used to determine a UTF-8/UTF-16/UTF-32 encoding.
[edit] Web browser support
Many browsers are only capable of displaying a small subset of the full Unicode repertoire. Here is how your browser displays various Unicode code points:
| Character | HTML char ref | Unicode name | What your browser displays |
|---|---|---|---|
| U+0041 | A or A |
Latin capital letter A | A |
| U+00DF | ß or ß |
Latin small letter Sharp S | ß |
| U+00FE | þ or þ |
Latin small letter Thorn | þ |
| U+0394 | Δ or Δ |
Greek capital letter Delta | Δ |
| U+017D | Ž or Ž |
Latin capital letter Z with caron (used in Central Europe) | Ž |
| U+0419 | Й or Й |
Cyrillic capital letter Short I | Й |
| U+05E7 | ק or ק |
Hebrew letter Qof | ק |
| U+0645 | م or م |
Arabic letter Meem | م |
| U+0E57 | ๗ or ๗ |
Thai digit 7 | ๗ |
| U+1250 | ቐ or ቐ |
Ge'ez syllable Qha | ቐ |
| U+3042 | あ or あ |
Hiragana letter A (Japanese) | あ |
| U+53F6 | 叶 or 叶 |
CJK Unified Ideograph-53F6 (Simplified Chinese "Leaf") | 叶 |
| U+8449 | 葉 or 葉 |
CJK Unified Ideograph-8449 (Traditional Chinese "Leaf") | 葉 |
| U+B5AB | 떫 or 떫 |
Hangul syllable Tteolp (Korean "Ssangtikeut Eo Rieulbieup") | 떫 |
| U+10346 | 𐍆 or 𐍆 |
Gothic letter Faihu | 𐍆 |
| To display all of the characters above, you may need to install one or more large multilingual fonts, like Code2000 (and Code2001 for some extinct languages, for example Gothic). | |||
Some web browsers, such as Mozilla Firefox, Opera, and Safari, are able to display multilingual web pages by intelligently choosing a font to display each individual character on the page. They will correctly display any mix of Unicode blocks, as long as appropriate fonts are present in the operating system.
Internet Explorer version 6 for Windows is capable of displaying the full range of Unicode characters, but characters which are not present in the first available font specified in the web page will only display if they are present in the designated fallback font for the current international script[1] (for example, only Arial font will be considered for a block beginning with Latin text, or Arial Unicode MS if it is also installed; subsequent fonts specified in a list are ignored).[2] Otherwise, Internet Explorer will display placeholder squares. For characters not present in a web page's fonts, Web page authors must guess which other appropriate fonts might be present on users' systems, and manually specify them as the preferred choices for each block or range of text containing such characters—Microsoft recommends using CSS to specify a font for each block of text in a different language or script. The characters in the table above haven't been assigned specific fonts, yet most should render correctly if appropriate fonts have been installed.
Older browsers, such as Netscape Navigator 4.77, can only display text supported by the current font associated with the character encoding of the page, and may misinterpret numeric character references as being references to code values within the current character encoding, rather than references to Unicode code points. When you are using such a browser, it is unlikely that your computer has all of those fonts, or that the browser can use all available fonts on the same page. As a result, the browser will not display the text in the examples above correctly, though it may display a subset of them. Because they are encoded according to the standard, though, they will display correctly on any system that is compliant and does have the characters available. Further, those characters given names for use in named entity references are likely to be more commonly available than others.
For displaying characters outside the Basic Multilingual Plane, like the Gothic letter faihu in the table above, some systems (like Windows 2000) need manual adjustments of their settings.
[edit] Frequency of usage
According to internal data from Google's web index, in December 2007 the UTF-8 Unicode encoding became the most frequently used encoding on web pages, overtaking both ASCII (US) and 8859-1/1252 (Western European).[3]
[edit] References
- ^ Microsoft (2006), “Globalization Step-by-Step: Fonts” at Microsoft Global Development and Computing Portal. URL retrieved on 2006-04-26.
- ^ Girt By Net (2005), “Internet Explorer Makes Me ☹” at girtby.net. URL retrieved on 2006-04-26.
- ^ Mark Davis: Moving to Unicode 5.1 Official Google blog, 5 May 2008
[edit] See also
- Help file for using special characters on Wikipedia
- Character encodings in HTML
- Unicode character reference (wikibooks)
[edit] External links
- Unicode in XML and other Markup Languages - a W3C & Unicode Consortium joint publication that describes issues and provides guidelines relating to Unicode in markup languages
- Latin-1, "Special", and Mathematical, Greek and Symbolic named character entity definitions for HTML 4.01
- UnicodeMap.org - Browse Unicode characters, ranges, and other information
- SIL's freeware fonts, editors and documentation
- Alan Wood’s Unicode Resources - Unicode fonts and information (www.alanwood.net/unicode).
- http://www.phon.ucl.ac.uk/home/wells/ipa-unicode.htm The International Phonetic Alphabet in Unicode
- http://www.alanwood.net/unicode/cjk_compatibility_ideographs.html CJK Compatibility Ideographs
- http://www.unicode.org/charts/ Unicode character charts; hexadecimal numbers only; PDF files showing all characters independent of browser capabilities
- Table of Unicode characters from 1 to 65535 - shows how they look in one's browser
- Web tool that converts "special" characters (such as Chinese characters) to Unicode numeric character references
- Multi-lingual web pages and Unicode - how to fix display problems
