js-codepage/misc/README.md.utf7
SheetJS 0121708499 version bump 0.6.0: windows support
- MakeEncoding.cs program pulls from .NET encodings (tested with Windows 7)
  (outputs stored at dotnet/out and misc/err)
- Many new codepages and improvements
- coverage + coveralls
- better environment handling (h/t @xch89820)
- cpexcel.js for the formats recognized by excel
- decode endian option removed (implicit from codepage definition)
NOTE: utf7 encode can be more space efficient
2014-04-14 13:45:36 -07:00

262 lines
14 KiB
Plaintext

+ACM Codepages for JS
+AFs-Codepages+AF0(https://en.wikipedia.org/wiki/Codepage) are character encodings. In
many contexts, single-byte character sets are used in lieu of standard multibyte
Unicode encodings. They use 256 characters with a simple mapping.
+AFs-unicode.org+AF0(http://www.unicode.org/Public/MAPPINGS/) hosts lists of mappings.
The build script automatically downloads and parses the mappings in order to
generate the full script. The +AGA-pages.csv+AGA description in +AGA-codepage.md+AGA controls
which codepages are used.
+ACMAIw Setup
In the browser:
+ADw-script src+AD0AIg-cptable.js+ACIAPgA8-/script+AD4
+ADw-script src+AD0AIg-cputils.js+ACIAPgA8-/script+AD4
The complete set of codepages is large due to some Double Byte Character Set
encodings. A much smaller file that just includes SBCS codepages is provided in
this repo (+AGA-sbcs.js+AGA), as well as a file for other projects (+AGA-cpexcel.js+AGA)
If you know which codepages you need, you can include individual scripts for
each codepage. The individual files are provided in the +AGA-bits/+AGA directory.
For example, to include only the Mac codepages:
+ADw-script src+AD0AIg-bits/10000.js+ACIAPgA8-/script+AD4
+ADw-script src+AD0AIg-bits/10006.js+ACIAPgA8-/script+AD4
+ADw-script src+AD0AIg-bits/10007.js+ACIAPgA8-/script+AD4
+ADw-script src+AD0AIg-bits/10029.js+ACIAPgA8-/script+AD4
+ADw-script src+AD0AIg-bits/10079.js+ACIAPgA8-/script+AD4
+ADw-script src+AD0AIg-bits/10081.js+ACIAPgA8-/script+AD4
All of the browser scripts define and append to the +AGA-cptable+AGA object. To rename
the object, edit the +AGA-JSVAR+AGA shell variable in +AGA-make.sh+AGA and run the script.
The utilities functions are contained in +AGA-cputils.js+AGA, which assumes that the
appropriate codepage scripts were loaded.
In node:
var cptable +AD0 require('codepage')+ADs
+ACMAIw Usage
The codepages are indexed by number. To get the unicode character for a given
codepoint, use the +AGA-dec+AGA property:
var unicode+AF8-cp10000+AF8-255 +AD0 cptable+AFs-10000+AF0.dec+AFs-255+AF0AOw // +Asc
To get the codepoint for a given character, use the +AGA-enc+AGA property:
var cp10000+AF8-711 +AD0 cptable+AFs-10000+AF0.enc+AFs-String.fromCharCode(711)+AF0AOw // 255
There are a few utilities that deal with strings and buffers:
var +bEdgOw +AD0 cptable.utils.decode(936, +AFs-0xbb,0xe3,0xd7,0xdc+AF0)+ADs
var buf +AD0 cptable.utils.encode(936, +bEdgOw)+ADs
+AGA-cptable.utils.encode(CP, data, ofmt)+AGA accepts a String or Array of characters
and returns a representation controlled by +AGA-ofmt+AGA:
- Default output is a Buffer (or Array) of bytes (integers between 0 and 255).
- If +AGA-ofmt +AD0APQ 'str'+AGA, return a String where +AGA-o.charCodeAt(i)+AGA is the ith byte
- If +AGA-ofmt +AD0APQ 'arr'+AGA, return an Array of bytes
+ACMAIw Building the script
This script uses +AFs-voc+AF0(npm.im/voc). The script to build the codepage tables and
the JS source is +AGA-codepage.md+AGA, so building is as simple as +AGA-voc codepage.md+AGA.
+ACMAIw Generated Codepages
The complete list of hardcoded codepages can be found in the file +AGA-pages.csv+AGA.
Some codepages are easier to implement algorithmically. Since these are
hardcoded in utils, there is no corresponding entry (they are +ACI-magic+ACI)
+AHw CP+ACM +AHw Information +AHw Description +AHw
+AHw --: +AHw ----------- +AHw ----------- +AHw
+AHw 37+AHw unicode.org +AHw-IBM EBCDIC US-Canada
+AHw 437+AHw unicode.org +AHw-OEM United States
+AHw 500+AHw unicode.org +AHw-IBM EBCDIC International
+AHw 708+AHw-MakeEncoding.cs+AHw-Arabic (ASMO 708)
+AHw 720+AHw-MakeEncoding.cs+AHw-Arabic (Transparent ASMO)+ADs Arabic (DOS)
+AHw 737+AHw unicode.org +AHw-OEM Greek (formerly 437G)+ADs Greek (DOS)
+AHw 775+AHw unicode.org +AHw-OEM Baltic+ADs Baltic (DOS)
+AHw 850+AHw unicode.org +AHw-OEM Multilingual Latin 1+ADs Western European (DOS)
+AHw 852+AHw unicode.org +AHw-OEM Latin 2+ADs Central European (DOS)
+AHw 855+AHw unicode.org +AHw-OEM Cyrillic (primarily Russian)
+AHw 857+AHw unicode.org +AHw-OEM Turkish+ADs Turkish (DOS)
+AHw 858+AHw-MakeEncoding.cs+AHw-OEM Multilingual Latin 1 +- Euro symbol
+AHw 860+AHw unicode.org +AHw-OEM Portuguese+ADs Portuguese (DOS)
+AHw 861+AHw unicode.org +AHw-OEM Icelandic+ADs Icelandic (DOS)
+AHw 862+AHw unicode.org +AHw-OEM Hebrew+ADs Hebrew (DOS)
+AHw 863+AHw unicode.org +AHw-OEM French Canadian+ADs French Canadian (DOS)
+AHw 864+AHw unicode.org +AHw-OEM Arabic+ADs Arabic (864)
+AHw 865+AHw unicode.org +AHw-OEM Nordic+ADs Nordic (DOS)
+AHw 866+AHw unicode.org +AHw-OEM Russian+ADs Cyrillic (DOS)
+AHw 869+AHw unicode.org +AHw-OEM Modern Greek+ADs Greek, Modern (DOS)
+AHw 870+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Multilingual/ROECE (Latin 2)
+AHw 874+AHw unicode.org +AHw-Windows Thai
+AHw 875+AHw unicode.org +AHw-IBM EBCDIC Greek Modern
+AHw 932+AHw unicode.org +AHw-Japanese Shift-JIS
+AHw 936+AHw unicode.org +AHw-Simplified Chinese GBK
+AHw 949+AHw unicode.org +AHw-Korean
+AHw 950+AHw unicode.org +AHw-Traditional Chinese Big5
+AHw 1026+AHw unicode.org +AHw-IBM EBCDIC Turkish (Latin 5)
+AHw 1047+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Latin 1/Open System
+AHw 1140+AHw-MakeEncoding.cs+AHw-IBM EBCDIC US-Canada (037 +- Euro symbol)
+AHw 1141+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Germany (20273 +- Euro symbol)
+AHw 1142+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Denmark-Norway (20277 +- Euro symbol)
+AHw 1143+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Finland-Sweden (20278 +- Euro symbol)
+AHw 1144+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Italy (20280 +- Euro symbol)
+AHw 1145+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Latin America-Spain (20284 +- Euro symbol)
+AHw 1146+AHw-MakeEncoding.cs+AHw-IBM EBCDIC United Kingdom (20285 +- Euro symbol)
+AHw 1147+AHw-MakeEncoding.cs+AHw-IBM EBCDIC France (20297 +- Euro symbol)
+AHw 1148+AHw-MakeEncoding.cs+AHw-IBM EBCDIC International (500 +- Euro symbol)
+AHw 1149+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Icelandic (20871 +- Euro symbol)
+AHw 1200+AHw magic +AHw-Unicode UTF-16, little endian (BMP of ISO 10646)
+AHw 1201+AHw magic +AHw-Unicode UTF-16, big endian
+AHw 1250+AHw unicode.org +AHw-Windows Central Europe
+AHw 1251+AHw unicode.org +AHw-Windows Cyrillic
+AHw 1252+AHw unicode.org +AHw-Windows Latin I
+AHw 1253+AHw unicode.org +AHw-Windows Green
+AHw 1254+AHw unicode.org +AHw-Windows Turkish
+AHw 1255+AHw unicode.org +AHw-Windows Hebrew
+AHw 1256+AHw unicode.org +AHw-Windows Arabic
+AHw 1257+AHw unicode.org +AHw-Windows Baltic
+AHw 1258+AHw unicode.org +AHw-Windows Vietnam
+AHw 1361+AHw-MakeEncoding.cs+AHw-Korean (Johab)
+AHw-10000+AHw unicode.org +AHw-MAC Roman
+AHw-10001+AHw-MakeEncoding.cs+AHw-Japanese (Mac)
+AHw-10002+AHw-MakeEncoding.cs+AHw-MAC Traditional Chinese (Big5)
+AHw-10003+AHw-MakeEncoding.cs+AHw-Korean (Mac)
+AHw-10004+AHw-MakeEncoding.cs+AHw-Arabic (Mac)
+AHw-10005+AHw-MakeEncoding.cs+AHw-Hebrew (Mac)
+AHw-10006+AHw unicode.org +AHw-Greek (Mac)
+AHw-10007+AHw unicode.org +AHw-Cyrillic (Mac)
+AHw-10008+AHw-MakeEncoding.cs+AHw-MAC Simplified Chinese (GB 2312)
+AHw-10010+AHw-MakeEncoding.cs+AHw-Romanian (Mac)
+AHw-10017+AHw-MakeEncoding.cs+AHw-Ukrainian (Mac)
+AHw-10021+AHw-MakeEncoding.cs+AHw-Thai (Mac)
+AHw-10029+AHw unicode.org +AHw-MAC Latin 2 (Central European)
+AHw-10079+AHw unicode.org +AHw-Icelandic (Mac)
+AHw-10081+AHw unicode.org +AHw-Turkish (Mac)
+AHw-10082+AHw-MakeEncoding.cs+AHw-Croatian (Mac)
+AHw-12000+AHw magic +AHw-Unicode UTF-32, little endian byte order
+AHw-12001+AHw magic +AHw-Unicode UTF-32, big endian byte order
+AHw-20000+AHw-MakeEncoding.cs+AHw-CNS Taiwan (Chinese Traditional)
+AHw-20001+AHw-MakeEncoding.cs+AHw-TCA Taiwan
+AHw-20002+AHw-MakeEncoding.cs+AHw-Eten Taiwan (Chinese Traditional)
+AHw-20003+AHw-MakeEncoding.cs+AHw-IBM5550 Taiwan
+AHw-20004+AHw-MakeEncoding.cs+AHw-TeleText Taiwan
+AHw-20005+AHw-MakeEncoding.cs+AHw-Wang Taiwan
+AHw-20105+AHw-MakeEncoding.cs+AHw-Western European IA5 (IRV International Alphabet 5) 7-bit
+AHw-20106+AHw-MakeEncoding.cs+AHw-IA5 German (7-bit)
+AHw-20107+AHw-MakeEncoding.cs+AHw-IA5 Swedish (7-bit)
+AHw-20108+AHw-MakeEncoding.cs+AHw-IA5 Norwegian (7-bit)
+AHw-20127+AHw magic +AHw-US-ASCII (7-bit)
+AHw-20261+AHw-MakeEncoding.cs+AHw-T.61
+AHw-20269+AHw-MakeEncoding.cs+AHw-ISO 6937 Non-Spacing Accent
+AHw-20273+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Germany
+AHw-20277+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Denmark-Norway
+AHw-20278+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Finland-Sweden
+AHw-20280+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Italy
+AHw-20284+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Latin America-Spain
+AHw-20285+AHw-MakeEncoding.cs+AHw-IBM EBCDIC United Kingdom
+AHw-20290+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Japanese Katakana Extended
+AHw-20297+AHw-MakeEncoding.cs+AHw-IBM EBCDIC France
+AHw-20420+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Arabic
+AHw-20423+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Greek
+AHw-20424+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Hebrew
+AHw-20833+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Korean Extended
+AHw-20838+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Thai
+AHw-20866+AHw-MakeEncoding.cs+AHw-Russian Cyrillic (KOI8-R)
+AHw-20871+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Icelandic
+AHw-20880+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Cyrillic Russian
+AHw-20905+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Turkish
+AHw-20924+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Latin 1/Open System (1047 +- Euro symbol)
+AHw-20932+AHw-MakeEncoding.cs+AHw-Japanese (JIS 0208-1990 and 0212-1990)
+AHw-20936+AHw-MakeEncoding.cs+AHw-Simplified Chinese (GB2312-80)
+AHw-20949+AHw-MakeEncoding.cs+AHw-Korean Wansung
+AHw-21025+AHw-MakeEncoding.cs+AHw-IBM EBCDIC Cyrillic Serbian-Bulgarian
+AHw-21866+AHw-MakeEncoding.cs+AHw-Ukrainian Cyrillic (KOI8-U)
+AHw-28591+AHw unicode.org +AHw-ISO 8859-1 Latin 1 (Western European)
+AHw-28592+AHw unicode.org +AHw-ISO 8859-2 Latin 2 (Central European)
+AHw-28593+AHw unicode.org +AHw-ISO 8859-3 Latin 3
+AHw-28594+AHw unicode.org +AHw-ISO 8859-4 Baltic
+AHw-28595+AHw unicode.org +AHw-ISO 8859-5 Cyrillic
+AHw-28596+AHw unicode.org +AHw-ISO 8859-6 Arabic
+AHw-28597+AHw unicode.org +AHw-ISO 8859-7 Greek
+AHw-28598+AHw unicode.org +AHw-ISO 8859-8 Hebrew (ISO-Visual)
+AHw-28599+AHw unicode.org +AHw-ISO 8859-9 Turkish
+AHw-28600+AHw unicode.org +AHw-ISO 8859-10 Latin 6
+AHw-28601+AHw unicode.org +AHw-ISO 8859-11 Latin (Thai)
+AHw-28603+AHw unicode.org +AHw-ISO 8859-13 Latin 7 (Estonian)
+AHw-28604+AHw unicode.org +AHw-ISO 8859-14 Latin 8 (Celtic)
+AHw-28605+AHw unicode.org +AHw-ISO 8859-15 Latin 9
+AHw-28606+AHw unicode.org +AHw-ISO 8859-15 Latin 10
+AHw-29001+AHw-MakeEncoding.cs+AHw-Europa 3
+AHw-38598+AHw-MakeEncoding.cs+AHw-ISO 8859-8 Hebrew (ISO-Logical)
+AHw-50220+AHw-MakeEncoding.cs+AHw-ISO 2022 JIS Japanese with no halfwidth Katakana
+AHw-50221+AHw-MakeEncoding.cs+AHw-ISO 2022 JIS Japanese with halfwidth Katakana
+AHw-50222+AHw-MakeEncoding.cs+AHw-ISO 2022 Japanese JIS X 0201-1989 (1 byte Kana-SO/SI)
+AHw-50225+AHw-MakeEncoding.cs+AHw-ISO 2022 Korean
+AHw-50227+AHw-MakeEncoding.cs+AHw-ISO 2022 Simplified Chinese
+AHw-51932+AHw-MakeEncoding.cs+AHw-EUC Japanese
+AHw-51936+AHw-MakeEncoding.cs+AHw-EUC Simplified Chinese
+AHw-51949+AHw-MakeEncoding.cs+AHw-EUC Korean
+AHw-52936+AHw-MakeEncoding.cs+AHw-HZ-GB2312 Simplified Chinese
+AHw-54936+AHw-MakeEncoding.cs+AHw-GB18030 Simplified Chinese (4 byte)
+AHw-57002+AHw-MakeEncoding.cs+AHw-ISCII Devanagari
+AHw-57003+AHw-MakeEncoding.cs+AHw-ISCII Bengali
+AHw-57004+AHw-MakeEncoding.cs+AHw-ISCII Tamil
+AHw-57005+AHw-MakeEncoding.cs+AHw-ISCII Telugu
+AHw-57006+AHw-MakeEncoding.cs+AHw-ISCII Assamese
+AHw-57007+AHw-MakeEncoding.cs+AHw-ISCII Oriya
+AHw-57008+AHw-MakeEncoding.cs+AHw-ISCII Kannada
+AHw-57009+AHw-MakeEncoding.cs+AHw-ISCII Malayalam
+AHw-57010+AHw-MakeEncoding.cs+AHw-ISCII Gujarati
+AHw-57011+AHw-MakeEncoding.cs+AHw-ISCII Punjabi
+AHw-65000+AHw magic +AHw-Unicode (UTF-7)
+AHw-65001+AHw magic +AHw-Unicode (UTF-8)
Note that MakeEncoding.cs deviates from unicode.org for some codepages. In the
case of direct conflicts, unicode.org takes precedence. In cases where the
unicode.org listing does not prescribe a value, MakeEncoding.cs value is used.
+ACMAIw Missing Codepages
The following codepages are not implemented. Normative references may not be
available in all cases. Furthermore, other software packages are known to hack
certain codepages (for example, Mozilla treats ASMO-708 as an alias of Arabic
ISO-8869-6 when in fact there are many differences), so all implementations
+ACo-should+ACo be cleanroom when possible.
- 709 Arabic (ASMO-449+-, BCON V4)
- 710 Arabic - Transparent Arabic
- 21027 (deprecated) +ADw--- is this necessary?
- 50229 ISO 2022 Traditional Chinese
- 50930 EBCDIC Japanese (Katakana) Extended
- 50931 EBCDIC US-Canada and Japanese
- 50933 EBCDIC Korean Extended and Korean
- 50935 EBCDIC Simplified Chinese Extended and Simplified Chinese
- 50936 EBCDIC Simplified Chinese
- 50937 EBCDIC US-Canada and Traditional Chinese
- 50939 EBCDIC Japanese (Latin) Extended and Japanese
- 51950 EUC Traditional Chinese
+ACMAIw Sources
- +AFs-Unicode Consortium Public Mappings+AF0(http://www.unicode.org/Public/MAPPINGS/)
- +AFs-Code Page Enumeration+AF0(http://msdn.microsoft.com/en-us/library/cc195051.aspx)
- +AFs-Code Page Identifiers+AF0(http://msdn.microsoft.com/en-us/library/windows/desktop/dd317756.aspx)
+ACMAIw Badges
+AFsAIQBb-githalytics.com alpha+AF0(https://cruel-carlota.pagodabox.com/afa29a5e8495a01059ee5b353f9042cb +ACI-githalytics.com+ACI)+AF0(http://githalytics.com/SheetJS/js-codepage)
+AFsAIQBb-Build Status+AF0(https://travis-ci.org/SheetJS/js-codepage.svg?branch+AD0-master)+AF0(https://travis-ci.org/SheetJS/js-codepage)
+AFsAIQBb-Coverage Status+AF0(https://coveralls.io/repos/SheetJS/js-codepage/badge.png)+AF0(https://coveralls.io/r/SheetJS/js-codepage)