Re: Character encodings

Chris Maden (crism@oreilly.com)
Mon, 7 Dec 1998 15:56:38 -0500 (EST)

[Chris von See]
> I am relatively new to XML and am trying to develop a program that
> can generate XML in various encodings. In section 4.3.3, the XML
> spec implies that support of ISO 10646 UCS-2 encoding (i.e. Unicode)
> is valid, but in the section on autodetection of encodings (Appendix
> F) there's no mention of how to detect UCS-2 encoding. I would
> *assume* that UCS-2 would start with \x00 \x3c\ x00 \x3f ("<?") - is
> that right? If so, then is the spec wrong in not including this in
> Appendix F as valid? Is it reasonable to expect that many people
> will use UCS-2 because of its similarity to Unicode?

A UCS-2 file starts with the byte order mark (BOM), \xFEFF. The
ordering of the two bytes shows whether the document is big-endian or
little-endian.

>From 4.3.3:

Entities encoded in UTF-16 must begin with the Byte Order Mark
described by ISO/IEC 10646 Annex E and Unicode Appendix B (the ZERO
WIDTH NO-BREAK SPACE character, #xFEFF). This is an encoding
signature, not part of either the markup or the character data of the
XML document. XML processors must be able to use this character to
differentiate between UTF-8 and UTF-16 encoded documents.

What's unsaid is that UCS-2 and UTF-16 are the same.

-Chris

-- 
<!NOTATION SGML.Geek PUBLIC "-//Anonymous//NOTATION SGML Geek//EN">
<!ENTITY crism PUBLIC "-//O'Reilly//NONSGML Christopher R. Maden//EN"
"<URL>http://www.oreilly.com/people/staff/crism/ <TEL>+1.617.499.7487
<USMAIL>90 Sherman Street, Cambridge, MA 02140 USA" NDATA SGML.Geek>