Unicode and HTML

Unicode series

Unicode

Unicode Consortium

UCS

UTF-7

UTF-8

UTF-16

UTF-32

SCSU

Punycode

Bi-directional text

BOM

Han unification

Unicode and HTML

HTML 4.0 uses ISO 10646/Unicode as its official character set. That is, an HTML document must be composed of a sequence of Unicode characters. When stored or transmitted over a network, however, these characters are typically encoded as a sequence of bits according to some character encoding.

Depending on which encoding is used, the encoded form may not be capable of representing all of the Unicode characters in the document. For example, if an 8-bit, one-byte-per-character encoding such as one of the ISO 8859 encodings is used, then at most only 256 specific characters from Unicode can be directly represented in the encoded HTML.

In order to work around this limitation, HTML is designed such that it is possible to represent characters from the whole of Unicode inside an HTML document by using a numeric character reference of the form &#N;, where N is a decimal number for the Unicode code point, or a hexadecimal number prefixed by x. The characters that comprise the numeric character reference are universally representable in every encoding approved for use on the Internet.

The support for hexadecimal in this context is more recent, so older browsers might have problems displaying those characters — but they will probably have a problem displaying Unicode characters outside the 8-bit range in the first place. It is still a common practice to convert the hexadecimal code point into a decimal value (e.g. ♠ instead of ♠).

There is also a standard set of named character entities for commonly used symbols outside of some character encodings. These entities can be included in an HTML document via the use of entity references of the form &EntityName;, where EntityName is the name of the entity. For example, —, much like — or &x2014;, represents U+2014: the em dash character—like this— even if the character encoding used doesn't contain that character. (In the Unicode standard, each code point is expressed in the notation U+hhhh, where hhhh are the hexadecimal digits).

In order to correctly process HTML, a web browser must ascertain which Unicode characters are represented by the encoded form of an HTML document. In order to do this, the web browser must know what encoding was used. When a document is transmitted via a MIME message such as an HTTP response, the message may signal the encoding via a Content-Type header, such as Content-Type: text/html;charset="iso-8859-1". Other external means of declaring encoding are permitted, but rarely used. The encoding may also be declared within the document itself, in the form of a META element, like <meta http-equiv="Content-Type" content="text/html;charset="iso-8859-1". When there is no encoding declaration, the default is either us-ascii or iso-8859-1, depending on a number of factors.

Because of the legacy of 8-bit text representations in programming languages and operating systems, and the desire to avoid burdening users with needing to understand the nuances of encoding, many text editors used by HTML authors are unable or unwilling to offer a choice of encodings when saving files to disk, and often do not even allow input of characters beyond a very limited range. Consequently, many HTML authors are unaware of encoding issues and may not have any idea what encoding their documents actually use. It is also a common misunderstanding that the encoding declaration effects a change in the actual encoding, whereas it is actually just a label that could be inaccurate.

Many HTML documents are served on the World Wide Web with inaccurate encoding declarations, or no declarations at all. In order to determine the encoding in such cases, many browsers allow the user to manually select one from a list. They may also employ an encoding autodetection algorithm that works in concert with the manual override. The manual override may apply to all documents, or only those for which the encoding cannot be ascertained by looking at declarations and/or byte patterns. The fact that the manual override is present and widely used hinders the adoption of accurate encoding declarations on the Web; therefore the problem is likely to persist. This has been addressed somewhat by XHTML, which, being XML, requires that encoding declarations be accurate and that no workarounds be employed when they're found to be inaccurate.

Many browsers are only capable of displaying a small subset of the full Unicode repertoire. Here is how your browser displays various Unicode code points:

Code	Description	What your browser displays
A	Latin capital letter "A"	A
ß	Latin small letter "Sharp S"	ß
þ	Latin small letter "Thorn"	þ
Δ	Greek capital letter "Delta"	Δ
Й	Cyrillic capital letter "Short I"	Й
ק	Hebrew letter "Qof"	ק
م	Arabic letter "Meem"	م
๗	Thai digit 7	๗
&amp#4688;	Ethiopic syllable "Qha"	ቐ
あ	Japanese Hiragana "A"	あ
叶	Simplified Chinese "Leaf"	叶
葉	Traditional Chinese "Leaf"	葉
냻	Korean Hangul syllable "Nieun Yae Rieulhieuh"	냻
𐍆	Gothic letter "faihu"	𐍆

Some multilingual web browsers that dynamically merge the required font sets on demand, e.g., Microsoft's Internet Explorer 5.0 and up on Windows, or Mozilla/Netscape 6 and up cross-platform, are capable of displaying all the Unicode characters on this page simultaneously after the appropriate "text display support packs" are downloaded. MSIE 5.5 would prompt the users if a new font were needed via its "install on demand" feature.

For displaying characters outside the Basic Multilingual Plane, like the Gothic letter faihu in the table above, some systems (like Windows 2000) need manual adjustments of their settings.

Other browsers, such as Netscape Navigator 4.77, can only display text supported by the current font associated with the character encoding of the page, and may misinterpret numeric character references as being references to code values within the current character encoding, rather than references to Unicode code points. When you are using such a browser, it is unlikely that your computer has all of those fonts, or that the browser can use all available fonts on the same page. As a result, the browser will not display the text in the examples above correctly, though it may display a subset of them. Because they are encoded according to the standard, though, they will display correctly on any system that is compliant and does have the characters available. Further, those characters given names for use in named entity references are likely to be more commonly available than others.

• Table of Unicode characters 32 to 63, 64 to 95, 96 to 127, 160 to 191.
• Table of Unicode characters, 128 to 999
• Table of Unicode characters, 1000 to 1999
• BambooWeb:Unicode and BambooWeb.