💱 Codepages for JS
Go to file
2016-09-17 20:33:32 -07:00
bin update ci versions 2016-01-15 22:51:05 -05:00
bits version bump 1.4.0 2016-01-15 22:27:02 -05:00
codepages version bump 1.3.7: extra DBF encodings 2014-12-19 14:36:39 -05:00
ctest version bump 1.4.0 2016-01-15 22:27:02 -05:00
dist version bump 1.4.0 2016-01-15 22:27:02 -05:00
dotnet version bump 0.6.0: windows support 2014-04-14 13:45:36 -07:00
misc version bump 1.4.0 2016-01-15 22:27:02 -05:00
nls version bump 1.4.0 2016-01-15 22:27:02 -05:00
.jscs.json version bump 1.3.0: performance 2014-06-26 01:54:13 -04:00
.travis.yml update ci versions 2016-01-15 22:51:05 -05:00
codepage.md update ci versions 2016-01-15 22:51:05 -05:00
cpexcel.js version bump 1.4.0 2016-01-15 22:27:02 -05:00
cptable.js version bump 1.4.0 2016-01-15 22:27:02 -05:00
cputils.js version bump 1.4.0 2016-01-15 22:27:02 -05:00
dotnet.sh version bump 0.6.0: windows support 2014-04-14 13:45:36 -07:00
excel.csv version bump 1.3.4: more codepages 2014-07-13 04:30:13 -04:00
index.html version bump 1.3.6: codepage cmd fixes 2014-10-18 22:39:39 -04:00
LICENSE version bump 1.4.0 2016-01-15 22:27:02 -05:00
Makefile version bump 1.4.0 2016-01-15 22:27:02 -05:00
NOTES.md version bump 1.3.6: codepage cmd fixes 2014-10-18 22:39:39 -04:00
package.json update ci versions 2016-01-15 22:51:05 -05:00
pages.csv version bump 1.3.7: extra DBF encodings 2014-12-19 14:36:39 -05:00
README.md Added syntax highlighting to readme examples 2016-09-17 20:33:32 -07:00
sauce.html version bump 1.0.0 2014-04-22 16:02:59 -07:00
sbcs.js version bump 1.4.0 2016-01-15 22:27:02 -05:00
test.js version bump 1.3.5: codepage bin supports stdin 2014-09-27 14:05:43 -04:00

Codepages for JS

Codepages are character encodings. In many contexts, single- or double-byte character sets are used in lieu of Unicode encodings. The codepages map between characters and numbers.

unicode.org hosts lists of mappings. The build script automatically downloads and parses the mappings in order to generate the full script. The pages.csv description in codepage.md controls which codepages are used.

Setup

In node:

const cptable = require('codepage');

In the browser:

<script src="cptable.js"></script>
<script src="cputils.js"></script>

Alternatively, use the full version in the dist folder:

<script src="cptable.full.js"></script>

The complete set of codepages is large due to some Double Byte Character Set encodings. A much smaller file that just includes SBCS codepages is provided in this repo (sbcs.js), as well as a file for other projects (cpexcel.js)

If you know which codepages you need, you can include individual scripts for each codepage. The individual files are provided in the bits/ directory. For example, to include only the Mac codepages:

<script src="bits/10000.js"></script>
<script src="bits/10006.js"></script>
<script src="bits/10007.js"></script>
<script src="bits/10029.js"></script>
<script src="bits/10079.js"></script>
<script src="bits/10081.js"></script>

All of the browser scripts define and append to the cptable object. To rename the object, edit the JSVAR shell variable in make.sh and run the script.

The utilities functions are contained in cputils.js, which assumes that the appropriate codepage scripts were loaded.

Usage

The codepages are indexed by number. To get the unicode character for a given codepoint, use the dec property:

const unicode_cp10000_255 = cptable[10000].dec[255]; // ˇ

To get the codepoint for a given character, use the enc property:

const cp10000_711 = cptable[10000].enc[String.fromCharCode(711)]; // 255

There are a few utilities that deal with strings and buffers:

const 汇总 = cptable.utils.decode(936, [0xbb,0xe3,0xd7,0xdc]);
const buf =  cptable.utils.encode(936,  汇总);
const sushi= cptable.utils.decode(65001, [0xf0,0x9f,0x8d,0xa3]); // 🍣
const sbuf = cptable.utils.encode(65001, sushi);

cptable.utils.encode(CP, data, ofmt) accepts a String or Array of characters and returns a representation controlled by ofmt:

  • Default output is a Buffer (or Array) of bytes (integers between 0 and 255).
  • If ofmt == 'str', return a String where o.charCodeAt(i) is the ith byte
  • If ofmt == 'arr', return an Array of bytes

Known Excel Codepages

A much smaller script, including only the codepages known to be used in Excel, is available under the name cpexcel. It exposes the same variable cptable and is suitable as a drop-in replacement when the full codepage tables are not needed.

In node:

const cptable = require('codepage/dist/cpexcel.full');

Rolling your own script

The make.sh script in the repo can take a manifest and generate JS source.

Usage:

bash make.sh path_to_manifest output_file_name JSVAR

where

  • JSVAR is the name of the exported variable (generally cptable)
  • output_file_name is the output file (e.g. cpexcel.js, cptable.js)
  • path_to_manifest is the path to the manifest file.

The manifest file is expected to be a CSV with 3 columns:

<codepage number>,<source>,<size>

If a source is specified, it will try to download the specified file and parse. The file format is expected to follow the format from the unicode.org site. The size should be 1 for a single-byte codepage and 2 for a double-byte codepage. For mixed codepages (which use some single- and some double-byte codes), the script assumes the mapping is a prefix code and generates efficient JS code.

Generated scripts only include the mapping. cat a mapping with cputils.js to produce a complete script like cpexcel.full.js.

Building the complete script

This script uses voc. The script to build the codepage tables and the JS source is codepage.md, so building is as simple as voc codepage.md.

Generated Codepages

The complete list of hardcoded codepages can be found in the file pages.csv.

Some codepages are easier to implement algorithmically. Since these are hardcoded in utils, there is no corresponding entry (they are "magic")

CP# Information Description
37 unicode.org IBM EBCDIC US-Canada
437 unicode.org OEM United States
500 unicode.org IBM EBCDIC International
620 NLS Mazovia (Polish) MS-DOS
708 MakeEncoding.cs Arabic (ASMO 708)
720 MakeEncoding.cs Arabic (Transparent ASMO); Arabic (DOS)
737 unicode.org OEM Greek (formerly 437G); Greek (DOS)
775 unicode.org OEM Baltic; Baltic (DOS)
850 unicode.org OEM Multilingual Latin 1; Western European (DOS)
852 unicode.org OEM Latin 2; Central European (DOS)
855 unicode.org OEM Cyrillic (primarily Russian)
857 unicode.org OEM Turkish; Turkish (DOS)
858 MakeEncoding.cs OEM Multilingual Latin 1 + Euro symbol
860 unicode.org OEM Portuguese; Portuguese (DOS)
861 unicode.org OEM Icelandic; Icelandic (DOS)
862 unicode.org OEM Hebrew; Hebrew (DOS)
863 unicode.org OEM French Canadian; French Canadian (DOS)
864 unicode.org OEM Arabic; Arabic (864)
865 unicode.org OEM Nordic; Nordic (DOS)
866 unicode.org OEM Russian; Cyrillic (DOS)
869 unicode.org OEM Modern Greek; Greek, Modern (DOS)
870 MakeEncoding.cs IBM EBCDIC Multilingual/ROECE (Latin 2)
874 unicode.org Windows Thai
875 unicode.org IBM EBCDIC Greek Modern
895 NLS Kamenický (Czech) MS-DOS
932 unicode.org Japanese Shift-JIS
936 unicode.org Simplified Chinese GBK
949 unicode.org Korean
950 unicode.org Traditional Chinese Big5
1026 unicode.org IBM EBCDIC Turkish (Latin 5)
1047 MakeEncoding.cs IBM EBCDIC Latin 1/Open System
1140 MakeEncoding.cs IBM EBCDIC US-Canada (037 + Euro symbol)
1141 MakeEncoding.cs IBM EBCDIC Germany (20273 + Euro symbol)
1142 MakeEncoding.cs IBM EBCDIC Denmark-Norway (20277 + Euro symbol)
1143 MakeEncoding.cs IBM EBCDIC Finland-Sweden (20278 + Euro symbol)
1144 MakeEncoding.cs IBM EBCDIC Italy (20280 + Euro symbol)
1145 MakeEncoding.cs IBM EBCDIC Latin America-Spain (20284 + Euro symbol)
1146 MakeEncoding.cs IBM EBCDIC United Kingdom (20285 + Euro symbol)
1147 MakeEncoding.cs IBM EBCDIC France (20297 + Euro symbol)
1148 MakeEncoding.cs IBM EBCDIC International (500 + Euro symbol)
1149 MakeEncoding.cs IBM EBCDIC Icelandic (20871 + Euro symbol)
1200 magic Unicode UTF-16, little endian (BMP of ISO 10646)
1201 magic Unicode UTF-16, big endian
1250 unicode.org Windows Central Europe
1251 unicode.org Windows Cyrillic
1252 unicode.org Windows Latin I
1253 unicode.org Windows Greek
1254 unicode.org Windows Turkish
1255 unicode.org Windows Hebrew
1256 unicode.org Windows Arabic
1257 unicode.org Windows Baltic
1258 unicode.org Windows Vietnam
1361 MakeEncoding.cs Korean (Johab)
10000 unicode.org MAC Roman
10001 MakeEncoding.cs Japanese (Mac)
10002 MakeEncoding.cs MAC Traditional Chinese (Big5)
10003 MakeEncoding.cs Korean (Mac)
10004 MakeEncoding.cs Arabic (Mac)
10005 MakeEncoding.cs Hebrew (Mac)
10006 unicode.org Greek (Mac)
10007 unicode.org Cyrillic (Mac)
10008 MakeEncoding.cs MAC Simplified Chinese (GB 2312)
10010 MakeEncoding.cs Romanian (Mac)
10017 MakeEncoding.cs Ukrainian (Mac)
10021 MakeEncoding.cs Thai (Mac)
10029 unicode.org MAC Latin 2 (Central European)
10079 unicode.org Icelandic (Mac)
10081 unicode.org Turkish (Mac)
10082 MakeEncoding.cs Croatian (Mac)
12000 magic Unicode UTF-32, little endian byte order
12001 magic Unicode UTF-32, big endian byte order
20000 MakeEncoding.cs CNS Taiwan (Chinese Traditional)
20001 MakeEncoding.cs TCA Taiwan
20002 MakeEncoding.cs Eten Taiwan (Chinese Traditional)
20003 MakeEncoding.cs IBM5550 Taiwan
20004 MakeEncoding.cs TeleText Taiwan
20005 MakeEncoding.cs Wang Taiwan
20105 MakeEncoding.cs Western European IA5 (IRV International Alphabet 5) 7-bit
20106 MakeEncoding.cs IA5 German (7-bit)
20107 MakeEncoding.cs IA5 Swedish (7-bit)
20108 MakeEncoding.cs IA5 Norwegian (7-bit)
20127 magic US-ASCII (7-bit)
20261 MakeEncoding.cs T.61
20269 MakeEncoding.cs ISO 6937 Non-Spacing Accent
20273 MakeEncoding.cs IBM EBCDIC Germany
20277 MakeEncoding.cs IBM EBCDIC Denmark-Norway
20278 MakeEncoding.cs IBM EBCDIC Finland-Sweden
20280 MakeEncoding.cs IBM EBCDIC Italy
20284 MakeEncoding.cs IBM EBCDIC Latin America-Spain
20285 MakeEncoding.cs IBM EBCDIC United Kingdom
20290 MakeEncoding.cs IBM EBCDIC Japanese Katakana Extended
20297 MakeEncoding.cs IBM EBCDIC France
20420 MakeEncoding.cs IBM EBCDIC Arabic
20423 MakeEncoding.cs IBM EBCDIC Greek
20424 MakeEncoding.cs IBM EBCDIC Hebrew
20833 MakeEncoding.cs IBM EBCDIC Korean Extended
20838 MakeEncoding.cs IBM EBCDIC Thai
20866 MakeEncoding.cs Russian Cyrillic (KOI8-R)
20871 MakeEncoding.cs IBM EBCDIC Icelandic
20880 MakeEncoding.cs IBM EBCDIC Cyrillic Russian
20905 MakeEncoding.cs IBM EBCDIC Turkish
20924 MakeEncoding.cs IBM EBCDIC Latin 1/Open System (1047 + Euro symbol)
20932 MakeEncoding.cs Japanese (JIS 0208-1990 and 0212-1990)
20936 MakeEncoding.cs Simplified Chinese (GB2312-80)
20949 MakeEncoding.cs Korean Wansung
21025 MakeEncoding.cs IBM EBCDIC Cyrillic Serbian-Bulgarian
21027 NLS Extended/Ext Alpha Lowercase
21866 MakeEncoding.cs Ukrainian Cyrillic (KOI8-U)
28591 unicode.org ISO 8859-1 Latin 1 (Western European)
28592 unicode.org ISO 8859-2 Latin 2 (Central European)
28593 unicode.org ISO 8859-3 Latin 3
28594 unicode.org ISO 8859-4 Baltic
28595 unicode.org ISO 8859-5 Cyrillic
28596 unicode.org ISO 8859-6 Arabic
28597 unicode.org ISO 8859-7 Greek
28598 unicode.org ISO 8859-8 Hebrew (ISO-Visual)
28599 unicode.org ISO 8859-9 Turkish
28600 unicode.org ISO 8859-10 Latin 6
28601 unicode.org ISO 8859-11 Latin (Thai)
28603 unicode.org ISO 8859-13 Latin 7 (Estonian)
28604 unicode.org ISO 8859-14 Latin 8 (Celtic)
28605 unicode.org ISO 8859-15 Latin 9
28606 unicode.org ISO 8859-15 Latin 10
29001 MakeEncoding.cs Europa 3
38598 MakeEncoding.cs ISO 8859-8 Hebrew (ISO-Logical)
50220 MakeEncoding.cs ISO 2022 JIS Japanese with no halfwidth Katakana
50221 MakeEncoding.cs ISO 2022 JIS Japanese with halfwidth Katakana
50222 MakeEncoding.cs ISO 2022 Japanese JIS X 0201-1989 (1 byte Kana-SO/SI)
50225 MakeEncoding.cs ISO 2022 Korean
50227 MakeEncoding.cs ISO 2022 Simplified Chinese
51932 MakeEncoding.cs EUC Japanese
51936 MakeEncoding.cs EUC Simplified Chinese
51949 MakeEncoding.cs EUC Korean
52936 MakeEncoding.cs HZ-GB2312 Simplified Chinese
54936 MakeEncoding.cs GB18030 Simplified Chinese (4 byte)
57002 MakeEncoding.cs ISCII Devanagari
57003 MakeEncoding.cs ISCII Bengali
57004 MakeEncoding.cs ISCII Tamil
57005 MakeEncoding.cs ISCII Telugu
57006 MakeEncoding.cs ISCII Assamese
57007 MakeEncoding.cs ISCII Oriya
57008 MakeEncoding.cs ISCII Kannada
57009 MakeEncoding.cs ISCII Malayalam
57010 MakeEncoding.cs ISCII Gujarati
57011 MakeEncoding.cs ISCII Punjabi
65000 magic Unicode (UTF-7)
65001 magic Unicode (UTF-8)

Note that MakeEncoding.cs deviates from unicode.org for some codepages. In the case of direct conflicts, unicode.org takes precedence. In cases where the unicode.org listing does not prescribe a value, MakeEncoding.cs value is used.

NLS refers to the National Language Support files supplied in various versions of Windows. In older versions of Windows (e.g. Windows 98) these files followed the pattern CP_#.NLS, but newer versions use the pattern C_#.NLS.

Sources

Badges

Build Status Coverage Status Analytics