+ACM Getting Codepages The fields of the pages.csv manifest are +AGA-codepage,url,bytes+AGA (SBCS+AD0-1, DBCS+AD0-2) +AGAAYABgAD4-pages.csv 37,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/EBCDIC/CP037.TXT,1 437,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP437.TXT,1 500,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/EBCDIC/CP500.TXT,1 737,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP737.TXT,1 775,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP775.TXT,1 850,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP850.TXT,1 852,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP852.TXT,1 855,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP855.TXT,1 857,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP857.TXT,1 860,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP860.TXT,1 861,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP861.TXT,1 862,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP862.TXT,1 863,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP863.TXT,1 864,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP864.TXT,1 865,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP865.TXT,1 866,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP866.TXT,1 869,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP869.TXT,1 874,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP874.TXT,1 875,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/EBCDIC/CP875.TXT,1 932,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP932.TXT,2 936,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP936.TXT,2 949,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP949.TXT,2 950,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP950.TXT,2 1026,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/EBCDIC/CP1026.TXT,1 1250,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1250.TXT,1 1251,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1251.TXT,1 1252,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1252.TXT,1 1253,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1253.TXT,1 1254,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1254.TXT,1 1255,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1255.TXT,1 1256,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1256.TXT,1 1257,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1257.TXT,1 1258,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP1258.TXT,1 +AGAAYABg Note that the Windows rendering is used for the Mac code pages. The primary difference is the use of the private +AGA-0xF8FF+AGA code (which renders as an Apple logo on macs but as garbage on other operating systems). It may be desirable to fall back to the behavior, in which case the files are under APPLE and not MICSFT. Codepages are an absolute pain :/ +AGAAYABgAD4-pages.csv 10000,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/MAC/ROMAN.TXT,1 10006,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/MAC/GREEK.TXT,1 10007,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/MAC/CYRILLIC.TXT,1 10029,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/MAC/LATIN2.TXT,1 10079,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/MAC/ICELAND.TXT,1 10081,http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/MAC/TURKISH.TXT,1 +AGAAYABg The numbering scheme for the +AGA-ISO-8859-X+AGA series is +AGA-28590 +- X+AGA: +AGAAYABgAD4-pages.csv 28591,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-1.TXT,1 28592,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-2.TXT,1 28593,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-3.TXT,1 28594,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-4.TXT,1 28595,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-5.TXT,1 28596,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-6.TXT,1 28597,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-7.TXT,1 28598,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-8.TXT,1 28599,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-9.TXT,1 28600,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-10.TXT,1 28601,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-11.TXT,1 28603,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-13.TXT,1 28604,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-14.TXT,1 28605,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-15.TXT,1 28606,http://www.unicode.org/Public/MAPPINGS/ISO8859/8859-16.TXT,1 +AGAAYABg +ACMAIw Generated Codepages The following codepages are available in .NET on Windows: - 708 Arabic (ASMO 708) - 720 Arabic (Transparent ASMO)+ADs Arabic (DOS) - 858 OEM Multilingual Latin 1 +- Euro symbol - 870 IBM EBCDIC Multilingual/ROECE (Latin 2)+ADs IBM EBCDIC Multilingual Latin 2 - 1047 IBM EBCDIC Latin 1/Open System - 1140 IBM EBCDIC US-Canada (037 +- Euro symbol)+ADs IBM EBCDIC (US-Canada-Euro) - 1141 IBM EBCDIC Germany (20273 +- Euro symbol)+ADs IBM EBCDIC (Germany-Euro) - 1142 IBM EBCDIC Denmark-Norway (20277 +- Euro symbol)+ADs IBM EBCDIC (Denmark-Norway-Euro) - 1143 IBM EBCDIC Finland-Sweden (20278 +- Euro symbol)+ADs IBM EBCDIC (Finland-Sweden-Euro) - 1144 IBM EBCDIC Italy (20280 +- Euro symbol)+ADs IBM EBCDIC (Italy-Euro) - 1145 IBM EBCDIC Latin America-Spain (20284 +- Euro symbol)+ADs IBM EBCDIC (Spain-Euro) - 1146 IBM EBCDIC United Kingdom (20285 +- Euro symbol)+ADs IBM EBCDIC (UK-Euro) - 1147 IBM EBCDIC France (20297 +- Euro symbol)+ADs IBM EBCDIC (France-Euro) - 1148 IBM EBCDIC International (500 +- Euro symbol)+ADs IBM EBCDIC (International-Euro) - 1149 IBM EBCDIC Icelandic (20871 +- Euro symbol)+ADs IBM EBCDIC (Icelandic-Euro) - 1361 Korean (Johab) - 10001 Japanese (Mac) - 10002 MAC Traditional Chinese (Big5)+ADs Chinese Traditional (Mac) - 10003 Korean (Mac) - 10004 Arabic (Mac) - 10005 Hebrew (Mac) - 10008 MAC Simplified Chinese (GB 2312)+ADs Chinese Simplified (Mac) - 10010 Romanian (Mac) - 10017 Ukrainian (Mac) - 10021 Thai (Mac) - 10082 Croatian (Mac) - 20000 CNS Taiwan+ADs Chinese Traditional (CNS) - 20001 TCA Taiwan - 20002 Eten Taiwan+ADs Chinese Traditional (Eten) - 20003 IBM5550 Taiwan - 20004 TeleText Taiwan - 20005 Wang Taiwan - 20105 IA5 (IRV International Alphabet No. 5, 7-bit)+ADs Western European (IA5) - 20106 IA5 German (7-bit) - 20107 IA5 Swedish (7-bit) - 20108 IA5 Norwegian (7-bit) - 20261 T.61 - 20269 ISO 6937 Non-Spacing Accent - 20273 IBM EBCDIC Germany - 20277 IBM EBCDIC Denmark-Norway - 20278 IBM EBCDIC Finland-Sweden - 20280 IBM EBCDIC Italy - 20284 IBM EBCDIC Latin America-Spain - 20285 IBM EBCDIC United Kingdom - 20290 IBM EBCDIC Japanese Katakana Extended - 20297 IBM EBCDIC France - 20420 IBM EBCDIC Arabic - 20423 IBM EBCDIC Greek - 20424 IBM EBCDIC Hebrew - 20833 IBM EBCDIC Korean Extended - 20838 IBM EBCDIC Thai - 20866 Russian (KOI8-R)+ADs Cyrillic (KOI8-R) - 20871 IBM EBCDIC Icelandic - 20880 IBM EBCDIC Cyrillic Russian - 20905 IBM EBCDIC Turkish - 20924 IBM EBCDIC Latin 1/Open System (1047 +- Euro symbol) - 20932 Japanese (JIS 0208-1990 and 0212-1990) - 20936 Simplified Chinese (GB2312)+ADs Chinese Simplified (GB2312-80) - 20949 Korean Wansung - 21025 IBM EBCDIC Cyrillic Serbian-Bulgarian - 21866 Ukrainian (KOI8-U)+ADs Cyrillic (KOI8-U) - 29001 Europa 3 - 38598 ISO 8859-8 Hebrew+ADs Hebrew (ISO-Logical) - 50220 ISO 2022 Japanese with no halfwidth Katakana+ADs Japanese (JIS) - 50221 ISO 2022 Japanese with halfwidth Katakana+ADs Japanese (JIS-Allow 1 byte Kana) - 50222 ISO 2022 Japanese JIS X 0201-1989+ADs Japanese (JIS-Allow 1 byte Kana - SO/SI) - 50225 ISO 2022 Korean - 50227 ISO 2022 Simplified Chinese+ADs Chinese Simplified (ISO 2022) - 51932 EUC Japanese - 51936 EUC Simplified Chinese+ADs Chinese Simplified (EUC) - 51949 EUC Korean - 52936 HZ-GB2312 Simplified Chinese+ADs Chinese Simplified (HZ) - 54936 Windows XP and later: GB18030 Simplified Chinese (4 byte)+ADs Chinese Simplified (GB18030) - 57002 ISCII Devanagari - 57003 ISCII Bengali - 57004 ISCII Tamil - 57005 ISCII Telugu - 57006 ISCII Assamese - 57007 ISCII Oriya - 57008 ISCII Kannada - 57009 ISCII Malayalam - 57010 ISCII Gujarati - 57011 ISCII Punjabi +AGAAYABgAD4-pages.csv 708,,1 720,,1 858,,1 870,,1 1047,,1 1140,,1 1141,,1 1142,,1 1143,,1 1144,,1 1145,,1 1146,,1 1147,,1 1148,,1 1149,,1 1361,,2 10001,,2 10002,,2 10003,,2 10004,,1 10005,,1 10008,,2 10010,,1 10017,,1 10021,,1 10082,,1 20000,,2 20001,,2 20002,,2 20003,,2 20004,,2 20005,,2 20105,,1 20106,,1 20107,,1 20108,,1 20261,,2 20269,,1 20273,,1 20277,,1 20278,,1 20280,,1 20284,,1 20285,,1 20290,,1 20297,,1 20420,,1 20423,,1 20424,,1 20833,,1 20838,,1 20866,,1 20871,,1 20880,,1 20905,,1 20924,,1 20932,,2 20936,,2 20949,,2 21025,,1 21866,,1 29001,,1 38598,,1 50220,,2 50221,,2 50222,,2 50225,,2 50227,,2 51932,,2 51936,,2 51949,,2 52936,,2 54936,,2 57002,,2 57003,,2 57004,,2 57005,,2 57006,,2 57007,,2 57008,,2 57009,,2 57010,,2 57011,,2 +AGAAYABg The known missing codepages are enumerated in the README. +ACMAIw Building Notes The script +AGA-make.sh+AGA (described later) will get these files and massage the data (printing code-unicode pairs). The eventual tables are dropped in the paths +AGA./codepages/+ADw-CODEPAGE+AD4.TBL+AGA. For example, the last 10 lines of +AGA-10000.TBL+AGA are +AGAAYABgAD4 0xF6 0x02C6 0xF7 0x02DC 0xF8 0x00AF 0xF9 0x02D8 0xFA 0x02D9 0xFB 0x02DA 0xFC 0x00B8 0xFD 0x02DD 0xFE 0x02DB 0xFF 0x02C7 +AGAAYABg which implies that code 0xF6 is +AGA-String.fromCharCode(0x02C6)+AGA and vice versa. +ACMAIw Windows-dependent build step To build the sources on windows, consult +AGA-dotnet/MakeEncoding.cs+AGA. After saving the standard output to +AGA-out+AGA, a simple awk script (+AGA-dotnet.sh+AGA) takes care of the rest: +AGAAYABgAD4-dotnet.sh +ACMAIQ-/bin/bash if +AFs +ACE -e dotnet/out +AF0AOw then exit+ADs fi +ADw-dotnet/out tr -s ' ' '+AFw-t' +AHw awk 'NF+AD4-2 +AHs-if(outfile) close(outfile)+ADs outfile+AD0AIg-codepages/+ACI +ACQ-1 +ACI.TBL+ACIAfQ NF+AD0APQ-2 +AHs-print +AD4 outfile+AH0' +AGAAYABg +ACM Building the script +AGA-make.njs+AGA takes a codepage argument, reads the corresponding table file and generates JS code for encoding and decoding: +ACMAIw Raw Codepages +AGAAYABgAD4-make.njs +ACMAIQ-/usr/bin/env node var argv +AD0 process.argv.slice(1), fs +AD0 require('fs')+ADs if(argv.length +ADw 2) +AHs console.error(+ACI-usage: make.njs +ADw-codepage+AF8-index+AD4 +AFs-variable+AF0AIg)+ADs process.exit(22)+ADs /+ACo EINVAL +ACo-/ +AH0 var cp +AD0 argv+AFs-1+AF0AOw var jsvar +AD0 argv+AFs-2+AF0 +AHwAfA +ACI-cptable+ACIAOw var x +AD0 fs.readFileSync(+ACI-codepages/+ACI +- cp +- +ACI.TBL+ACI,+ACI-utf8+ACI)+ADs var maxcp +AD0 0+ADs var y +AD0 x.split(+ACIAXA-n+ACI).map(function(z) +AHs var w +AD0 z.split(+ACIAXA-t+ACI)+ADs if(w.length +ADw 2) return w+ADs return +AFs-Number(w+AFs-0+AF0), Number(w+AFs-1+AF0)+AF0AOw +AH0).filter(function(z) +AHs return z.length +AD4 1+ADs +AH0)+ADs +AGAAYABg The DBCS and SBCS code generation strategies are different. The maximum code is used to distinguish (max 0xFF for SBCS). +AGAAYABg for(var i +AD0 0+ADs i +ACEAPQ y.length+ADs +-+-i) if(y+AFs-i+AF0AWw-0+AF0 +AD4 maxcp) maxcp +AD0 y+AFs-i+AF0AWw-0+AF0AOw var enc +AD0 +AHsAfQ, dec +AD0 (maxcp +ADw 256 ? +AFsAXQ : +AHsAfQ)+ADs for(var i +AD0 0+ADs i +ACEAPQ y.length+ADs +-+-i) +AHs dec+AFs-y+AFs-i+AF0AWw-0+AF0AXQ +AD0 String.fromCharCode(y+AFs-i+AF0AWw-1+AF0)+ADs enc+AFs-String.fromCharCode(y+AFs-i+AF0AWw-1+AF0)+AF0 +AD0 y+AFs-i+AF0AWw-0+AF0AOw +AH0 var odec, oenc, outstr+ADs if(maxcp +ADw 256) +AHs +AGAAYABg The unicode character +AGA-0xFFFD+AGA (REPLACEMENT CHARACTER) is used as a placeholder for characters that are not specified in the map (for example, +AGA-0xF0+AGA is not in code page 10000). For SBCS, the idea is to embed a raw string with the contents of the 256 codes. The +AGA-dec+AGA field is merely a split of the string, and +AGA-enc+AGA is an eversion: +AGAAYABg for(var i +AD0 0+ADs i +ACEAPQ 256+ADs +-+-i) if(typeof dec+AFs-i+AF0 +AD0APQA9 +ACI-undefined+ACI) dec+AFs-i+AF0 +AD0 String.fromCharCode(0xFFFD)+ADs odec +AD0 JSON.stringify(dec.join(+ACIAIg))+ADs outstr +AD0 '(function()+AHs var d +AD0 ' +- odec +- ', e +AD0 +AHsAfQA7 for(var i+AD0-0+ADs-i+ACEAPQ-d.length+ADsAKwAr-i) if(d.charCodeAt(i) +ACEAPQA9 0xFFFD)e+AFs-d+AFs-i+AF0AXQ +AD0 i+ADs return +AHsAIg-enc+ACI: e, +ACI-dec+ACI: d.split(+ACIAIg) +AH0AOw +AH0)()+ADs'+ADs +AH0 else +AHs +AGAAYABg DBCS is similar, except that the space is sliced into 256-byte chunks (strings are only generated for those high-bytes represented in the codepage). The strategy is to construct an array-of-arrays so that +AGA-dd+AFs-high+AF0AWw-low+AF0AYA is the character associated with the code. This array is combined at runtime to yield the complete decoding object (and the encoding object is an eversion): +AGAAYABg var dd +AD0 +AFsAXQA7 for(var i in dec) if(dec.hasOwnProperty(i)) +AHs if(typeof dd+AFs-i +AD4APg 8+AF0 +AD0APQA9 +ACI-undefined+ACI) dd+AFs-i +AD4APg 8+AF0 +AD0 +AFsAXQA7 dd+AFs-i +AD4APg 8+AF0AWw-i +ACU 256+AF0 +AD0 dec+AFs-i+AF0AOw +AH0 outstr +AD0 '(function()+AHs var d +AD0 +AFsAXQ, e +AD0 +AHsAfQ, D +AD0 +AFsAXQ, j+ADsAXA-n'+ADs for(var i +AD0 0+ADs i +ACEAPQ 256+ADs +-+-i) if(dd+AFs-i+AF0) +AHs for(var j +AD0 0+ADs j +ACEAPQ 256+ADs +-+-j) if(typeof dd+AFs-i+AF0AWw-j+AF0 +AD0APQA9 +ACI-undefined+ACI) dd+AFs-i+AF0AWw-j+AF0 +AD0 String.fromCharCode(0xFFFD)+ADs outstr +-+AD0 'D+AFs' +- i +- '+AF0 +AD0 ' +- JSON.stringify(dd+AFs-i+AF0.join(+ACIAIg)) +- '.split(+ACIAIg)+ADsAXA-n'+ADs outstr +-+AD0 'for(j +AD0 0+ADs j +ACEAPQ D+AFs' +- i +- '+AF0.length+ADs +-+-j) if(D+AFs' +- i +- '+AF0AWw-j+AF0.charCodeAt(0) +ACEAPQA9 0xFFFD) +AHs e+AFs-D+AFs' +- i +- '+AF0AWw-j+AF0AXQ +AD0 ' +- (i+ACo-256) +- ' +- j+ADs d+AFs' +- (i+ACo-256) +- ' +- j+AF0 +AD0 D+AFs' +- i +- '+AF0AWw-j+AF0AOwB9AFw-n' +AH0 outstr +-+AD0 'return +AHsAIg-enc+ACI: e, +ACI-dec+ACI: d +AH0AOw +AH0)()+ADs'+ADs +AH0 console.log(jsvar +- +ACIAWwAi +- cp +- +ACIAXQ +AD0 +ACI +- outstr)+ADs +AGAAYABg +AGA-make.sh+AGA generates the tables used by +AGA-make.njs+AGA. The raw unicode TXT files are columnar: +AGA-code unicode +ACM-comments+AGA. For example, the last 10 lines of the text file ROMAN.TXT (for CP 10000) are: +AGAAYABgAD4 0xF6 0x02C6 +ACM-MODIFIER LETTER CIRCUMFLEX ACCENT 0xF7 0x02DC +ACM-SMALL TILDE 0xF8 0x00AF +ACM-MACRON 0xF9 0x02D8 +ACM-BREVE 0xFA 0x02D9 +ACM-DOT ABOVE 0xFB 0x02DA +ACM-RING ABOVE 0xFC 0x00B8 +ACM-CEDILLA 0xFD 0x02DD +ACM-DOUBLE ACUTE ACCENT 0xFE 0x02DB +ACM-OGONEK 0xFF 0x02C7 +ACM-CARON +AGAAYABg In processing the data, the comments (after the +AGAAIwBg) are stripped and undefined elements (like +AGA-0x7F+AGA for CP 10000) are removed. +AGAAYABgAD4-make.sh +ACMAIQ-/bin/bash INFILE+AD0AJAB7-1:-pages.csv+AH0 OUTFILE+AD0AJAB7-2:-cptable.js+AH0 JSVAR+AD0AJAB7-3:-cptable+AH0 mkdir -p codepages bits rm -f +ACQ-OUTFILE +ACQ-OUTFILE.tmp echo +ACI-/+ACo +ACQ-OUTFILE (C) 2013-2014 SheetJS -- http://sheetjs.com +ACo-/+ACI +AD4 +ACQ-OUTFILE.tmp echo +ACI-/+ACo-jshint -W100 +ACo-/+ACI +AD4APg +ACQ-OUTFILE.tmp echo +ACI-var +ACQ-JSVAR +AD0 +AHsAfQA7ACI +AD4APg +ACQ-OUTFILE.tmp if +AFs -e dotnet.sh +AF0AOw then bash dotnet.sh+ADs fi awk -F, '+AHs-print +ACQ-1, +ACQ-2, +ACQ-3+AH0' +ACQ-INFILE +AHw while read cp url cptype+ADs do echo +ACQ-cp +ACQ-url if +AFs +ACE -e codepages/+ACQ-cp.TBL +AF0AOw then curl +ACQ-url +AHw sed 's/+ACM.+ACo-//g' +AHw awk 'NF+AD0APQ-2' +AD4 codepages/+ACQ-cp.TBL fi echo +ACI-if(typeof +ACQ-JSVAR +AD0APQA9 'undefined') +ACQ-JSVAR +AD0 +AHsAfQA7ACI +AD4 bits/+ACQ-cp.js.tmp node make.njs +ACQ-cp +ACQ-JSVAR +AHw tee -a bits/+ACQ-cp.js.tmp +AD4APg +ACQ-OUTFILE.tmp sed 's/+ACIAXA(+AFs-0-9+AF0AKwBc)+ACI:/+AFw-1:/g' +ADw-bits/+ACQ-cp.js.tmp +AD4-bits/+ACQ-cp.js rm -f bits/+ACQ-cp.js.tmp done echo +ACI-if (typeof module +ACEAPQA9 'undefined' +ACYAJg module.exports) module.exports +AD0 +ACQ-JSVAR+ADsAIg +AD4APg +ACQ-OUTFILE.tmp sed 's/+ACIAXA(+AFs-0-9+AF0AKwBc)+ACI:/+AFw-1:/g' +ADwAJA-OUTFILE.tmp +AD4AJA-OUTFILE rm -f +ACQ-OUTFILE.tmp +AGAAYABg +ACMAIw Utilities The encode and decode functions are kept in a separate script (cputils.js). Both encode and decode deal with data represented as: - String (encode expects JS string, decode interprets UCS2 chars as codes) - Array (encode expects array of JS String characters, decode expects numbers) - Buffer (encode expects UTF-8 string, decode expects codepoints/bytes). The +AGA-ofmt+AGA variable controls +AGA-encode+AGA output (+AGA-str+AGA, +AGA-arr+AGA respectively) while the input format is automatically determined. +ACM Tests The tests include JS validity tests (requiring or eval'ing code): +AGAAYABgAD4-test.js var fs +AD0 require('fs'), assert +AD0 require('assert'), vm +AD0 require('vm')+ADs var cptable, sbcs+ADs describe('source', function() +AHs it('should load node', function() +AHs cptable +AD0 require('./')+ADs +AH0)+ADs it('should load sbcs', function() +AHs sbcs +AD0 require('./sbcs')+ADs +AH0)+ADs it('should load excel', function() +AHs excel +AD0 require('./cpexcel')+ADs +AH0)+ADs it('should process bits', function() +AHs var files +AD0 fs.readdirSync('bits').filter(function(x)+AHs-return x.substr(-3)+AD0APQAi.js+ACIAOwB9)+ADs files.forEach(function(x) +AHs vm.runInThisContext(fs.readFileSync('./bits/' +- x))+ADs +AH0)+ADs +AH0)+ADs +AH0)+ADs +AGAAYABg The README tests verify the snippets in the README: +AGAAYABgAD4-test.js describe('README', function() +AHs var readme +AD0 function() +AHs var unicode+AF8-cp10000+AF8-255 +AD0 cptable+AFs-10000+AF0.dec+AFs-255+AF0AOw // +Asc assert.equal(unicode+AF8-cp10000+AF8-255, +ACICxwAi)+ADs var cp10000+AF8-711 +AD0 cptable+AFs-10000+AF0.enc+AFs-String.fromCharCode(711)+AF0AOw // 255 assert.equal(cp10000+AF8-711, 255)+ADs var b1 +AD0 +AFs-0xbb,0xe3,0xd7,0xdc+AF0AOw var +bEdgOw +AD0 cptable.utils.decode(936, b1)+ADs var buf +AD0 cptable.utils.encode(936, +bEdgOw)+ADs assert.equal(+bEdgOw,+ACJsR2A7ACI)+ADs assert.equal(buf.length, 4)+ADs for(var i +AD0 0+ADs i +ACEAPQ 4+ADs +-+-i) assert.equal(b1+AFs-i+AF0, buf+AFs-i+AF0)+ADs var b2 +AD0 +AFs-0xf0,0x9f,0x8d,0xa3+AF0AOw var sushi+AD0 cptable.utils.decode(65001, b2)+ADs var sbuf +AD0 cptable.utils.encode(65001, sushi)+ADs assert.equal(sushi,+ACLYPN9jACI)+ADs assert.equal(sbuf.length, 4)+ADs for(var i +AD0 0+ADs i +ACEAPQ 4+ADs +-+-i) assert.equal(b2+AFs-i+AF0, sbuf+AFs-i+AF0)+ADs +AH0AOw it('should be correct', function() +AHs cptable.utils.cache.encache()+ADs readme()+ADs cptable.utils.cache.decache()+ADs readme()+ADs +AH0)+ADs +AH0)+ADs +AGAAYABg The consistency tests make sure that encoding and decoding are pseudo inverses: +AGAAYABgAD4-test.js describe('consistency', function() +AHs cptable +AD0 require('./')+ADs U +AD0 cptable.utils+ADs var chk +AD0 function(cptable, cacheit) +AHs return function(x) +AHs it('should consistently process CP ' +- x +- ' ' +- cacheit, function() +AHs var cp +AD0 cptable+AFs-x+AF0, D +AD0 cp.dec, E +AD0 cp.enc+ADs if(cacheit) cptable.utils.cache.encache()+ADs else cptable.utils.cache.decache()+ADs Object.keys(D).forEach(function(d) +AHs if(E+AFs-D+AFs-d+AF0AXQ +ACEAPQ d) +AHs if(typeof E+AFs-D+AFs-d+AF0AXQ +ACEAPQA9 +ACI-undefined+ACI) return+ADs if(D+AFs-d+AF0.charCodeAt(0) +AD0APQ 0xFFFD) return+ADs if(D+AFs-E+AFs-D+AFs-d+AF0AXQBd +AD0APQA9 D+AFs-d+AF0) return+ADs throw new Error(x +- +ACI e.d+AFsAIg +- d +- +ACIAXQ +AD0 +ACI +- E+AFs-D+AFs-d+AF0AXQ +- +ACIAOw d+AFsAIg +- d +- +ACIAXQA9ACI +- D+AFs-d+AF0 +- +ACIAOw d.e.d+AFsAIg +- d +- +ACIAXQ +AD0 +ACI +- D+AFs-E+AFs-D+AFs-d+AF0AXQBd)+ADs +AH0 +AH0)+ADs Object.keys(E).forEach(function(e) +AHs if(D+AFs-E+AFs-e+AF0AXQ +ACEAPQ e) +AHs throw new Error(x +- +ACI d.e+AFsAIg +- e +- +ACIAXQ +AD0 +ACI +- D+AFs-E+AFs-e+AF0AXQ +- +ACIAOw e+AFsAIg +- e +- +ACIAXQA9ACI +- E+AFs-e+AF0 +- +ACIAOw e.d.e+AFsAIg +- e +- +ACIAXQ +AD0 +ACI +- E+AFs-D+AFs-E+AFs-e+AF0AXQBd)+ADs +AH0 +AH0)+ADs var corpus +AD0 +AFsAIg-foobar+ACIAXQA7 corpus.forEach(function(w)+AHs assert.equal(U.decode(x,U.encode(x,w)),w)+ADs +AH0)+ADs cptable.utils.cache.encache()+ADs +AH0)+ADs +AH0AOw +AH0AOw Object.keys(cptable).filter(function(w) +AHs return w +ACEAPQ +ACI-utils+ACIAOw +AH0).forEach(chk(cptable, true))+ADs Object.keys(cptable).filter(function(w) +AHs return w +ACEAPQ +ACI-utils+ACIAOw +AH0).forEach(chk(cptable, false))+ADs +AH0)+ADs +AGAAYABg The next tests look at possible entry conditions: +AGAAYABg describe('entry conditions', function() +AHs it('should fail to load utils if cptable unavailable', function() +AHs var sandbox +AD0 +AHsAfQA7 var ctx +AD0 vm.createContext(sandbox)+ADs assert.throws(function() +AHs vm.runInContext(fs.readFileSync('cputils.js','utf8'),ctx)+ADs +AH0)+ADs +AH0)+ADs it('should load utils if cptable is available', function() +AHs var sandbox +AD0 +AHsAfQA7 var ctx +AD0 vm.createContext(sandbox)+ADs vm.runInContext(fs.readFileSync('cpexcel.js','utf8'),ctx)+ADs vm.runInContext(fs.readFileSync('cputils.js','utf8'),ctx)+ADs +AH0)+ADs var chken +AD0 function(cp, i) +AHs var c +AD0 function(cp, i, e) +AHs var str +AD0 cptable.utils.encode(cp,i,e)+ADs var arr +AD0 cptable.utils.encode(cp,i.split(+ACIAIg),e)+ADs assert.deepEqual(str,arr)+ADs if(typeof Buffer +AD0APQA9 'undefined') return+ADs var buf +AD0 cptable.utils.encode(cp,new Buffer(i),e)+ADs assert.deepEqual(str,buf)+ADs +AH0AOw cptable.utils.cache.encache()+ADs c(cp,i)+ADs c(cp,i,'buf')+ADs c(cp,i,'arr')+ADs c(cp,i,'str')+ADs cptable.utils.cache.decache()+ADs c(cp,i)+ADs c(cp,i,'buf')+ADs c(cp,i,'arr')+ADs c(cp,i,'str')+ADs +AH0AOw describe('encode', function() +AHs it('CP 1252 : sbcs', function() +AHs chken(1252,+ACI-foobar+ACI)+ADs +AH0)+ADs it('CP 708 : sbcs', function() +AHs chken(708,+ACIGKg and +Bis smiley faces+ACI)+ADsAfQ)+ADs it('CP 936 : dbcs', function() +AHs chken(936, +ACKP2WYvTi1lh1tXeyZtS4vVACI)+ADsAfQ)+ADs +AH0)+ADs var chkde +AD0 function(cp, i) +AHs var c +AD0 function(cp, i) +AHs var s+ADs if(typeof Buffer +ACEAPQA9 'undefined' +ACYAJg i instanceof Buffer) s +AD0 +AFsAXQ.map.call(i, function(s)+AHs-return String.fromCharCode(s)+ADs +AH0)+ADs else s+AD0(i.map) ? i.map(function(s)+AHs-return String.fromCharCode(s)+ADs +AH0) : i+ADs var str +AD0 cptable.utils.decode(cp,i)+ADs var arr +AD0 cptable.utils.decode(cp,s.join?s.join(+ACIAIg):s)+ADs assert.deepEqual(str,arr)+ADs if(typeof Buffer +AD0APQA9 'undefined') return+ADs var buf +AD0 cptable.utils.decode(cp,new Buffer(i))+ADs assert.deepEqual(str,buf)+ADs +AH0AOw cptable.utils.cache.encache()+ADs c(cp,i)+ADs cptable.utils.cache.decache()+ADs c(cp,i)+ADs +AH0AOw describe('decode', function() +AHs it('CP 1252 : sbcs', function() +AHs chkde(1252,+AFs-0x66, 0x6f, 0x6f, 0x62, 0x61, 0x72+AF0)+ADs +AH0)+ADs /+ACo +ACI-foobar+ACI +ACo-/ if(typeof Buffer +ACEAPQA9 'undefined') it('CP 708 : sbcs', function() +AHs chkde(708, new Buffer(+AFs-0xca, 0x20, 0x61, 0x6e, 0x64, 0x20, 0xcb, 0x20, 0x73, 0x6d, 0x69, 0x6c, 0x65, 0x79, 0x20, 0x66, 0x61, 0x63, 0x65, 0x73+AF0))+ADs +AH0)+ADs /+ACo (+ACIGKg and +Bis smiley faces+ACI) +ACo-/ it('CP 936 : dbcs', function() +AHs chkde(936, +AFs-0xd5, 0xe2, 0xca, 0xc7, 0xd6, 0xd0, 0xce, 0xc4, 0xd7, 0xd6, 0xb7, 0xfb, 0xb2, 0xe2, 0xca, 0xd4+AF0)+ADsAfQ)+ADs /+ACo +ACKP2WYvTi1lh1tXeyZtS4vVACI +ACo-/ +AH0)+ADs +AH0)+ADs +AGAAYABg The +AGA-testfile+AGA helper function reads a file and compares to node's read facilities: +AGAAYABgAD4-test.js function testfile(f,cp,type,skip) +AHs var d +AD0 fs.readFileSync(f)+ADs var x +AD0 fs.readFileSync(f, type)+ADs var a +AD0 x.split(+ACIAIg)+ADs var chk +AD0 function(cp) +AHs var y +AD0 cptable.utils.decode(cp, d)+ADs assert.equal(x,y)+ADs var z +AD0 cptable.utils.encode(cp, x)+ADs if(z.length +ACEAPQ d.length) throw new Error(f +- +ACI +ACI +- JSON.stringify(z) +- +ACI +ACEAPQ +ACI +- JSON.stringify(d) +- +ACI : +ACI +- z.length +- +ACI +ACI +- d.length)+ADs for(var i +AD0 0+ADs i +ACEAPQ d.length+ADs +-+-i) if(d+AFs-i+AF0 +ACEAPQA9 z+AFs-i+AF0) throw new Error(+ACIAIg +- i +- +ACI +ACI +- d+AFs-i+AF0 +- +ACIAIQA9ACI +- z+AFs-i+AF0)+ADs if(skip) return+ADs z +AD0 cptable.utils.encode(cp, a)+ADs if(z.length +ACEAPQ d.length) throw new Error(f +- +ACI +ACI +- JSON.stringify(z) +- +ACI +ACEAPQ +ACI +- JSON.stringify(d) +- +ACI : +ACI +- z.length +- +ACI +ACI +- d.length)+ADs for(var i +AD0 0+ADs i +ACEAPQ d.length+ADs +-+-i) if(d+AFs-i+AF0 +ACEAPQA9 z+AFs-i+AF0) throw new Error(+ACIAIg +- i +- +ACI +ACI +- d+AFs-i+AF0 +- +ACIAIQA9ACI +- z+AFs-i+AF0)+ADs +AH0 cptable.utils.cache.encache()+ADs chk(cp)+ADs if(skip) return+ADs cptable.utils.cache.decache()+ADs chk(cp)+ADs cptable.utils.cache.encache()+ADs +AH0 +AGAAYABg The +AGA-utf8+AGA tests verify utf8 encoding of the actual JS sources: +AGAAYABgAD4-test.js describe('node natives', function() +AHs var node +AD0 +AFsAWw-65001, 'utf8',1+AF0, +AFs-1200, 'utf16le',1+AF0, +AFs-20127, 'ascii',0+AF0AXQA7 var unicodefiles +AD0 +AFs'codepage.md','README.md','cptable.js'+AF0AOw var asciifiles +AD0 +AFs'cputils.js'+AF0AOw node.forEach(function(w) +AHs describe(w+AFs-1+AF0, function() +AHs cptable +AD0 require('./')+ADs asciifiles.forEach(function(f) +AHs it('should process ' +- f, function() +AHs testfile('./misc/'+-f+-'.'+-w+AFs-1+AF0,w+AFs-0+AF0,w+AFs-1+AF0)+ADs +AH0)+ADs +AH0)+ADs if(+ACE-w+AFs-2+AF0) return+ADs unicodefiles.forEach(function(f) +AHs it('should process ' +- f, function() +AHs testfile('./misc/'+-f+-'.'+-w+AFs-1+AF0,w+AFs-0+AF0,w+AFs-1+AF0)+ADs +AH0)+ADs +AH0)+ADs if(w+AFs-1+AF0 +AD0APQA9 'utf8') it('should process bits', function() +AHs var files +AD0 fs.readdirSync('bits').filter(function(x)+AHs-return x.substr(-3)+AD0APQAi.js+ACIAOwB9)+ADs files.forEach(function(f) +AHs testfile('./bits/' +- f,w+AFs-0+AF0,w+AFs-1+AF0,true)+ADs +AH0)+ADs +AH0)+ADs +AH0)+ADs +AH0)+ADs +AH0)+ADs +AGAAYABg The utf+ACo and ascii tests attempt to test other magic formats: +AGAAYABgAD4-test.js var m +AD0 cptable.utils.magic+ADs function cmp(x,z) +AHs assert.equal(x.length, z.length)+ADs for(var i +AD0 0+ADs i +ACEAPQ z.length+ADs +-+-i) assert.equal(i+-+ACI-/+ACIAKw-x.length+-+ACIAIgAr-x+AFs-i+AF0, i+-+ACI-/+ACIAKw-z.length+-+ACIAIgAr-z+AFs-i+AF0)+ADs +AH0 Object.keys(m).forEach(function(t)+AHs-if(t +ACEAPQ 16969) describe(m+AFs-t+AF0, function() +AHs it(+ACI-should process codepage.md.+ACI +- m+AFs-t+AF0, fs.existsSync('./misc/codepage.md.' +- m+AFs-t+AF0) ? function() +AHs var b +AD0 fs.readFileSync('./misc/codepage.md.utf8', +ACI-utf8+ACI)+ADs if(m+AFs-t+AF0 +AD0APQA9 +ACI-ascii+ACI) b +AD0 b.replace(/+AFsAXA-u0080-+AFw-uffff+AF0AKg-/g,+ACIAIg)+ADs var x +AD0 fs.readFileSync('./misc/codepage.md.' +- m+AFs-t+AF0)+ADs var y, z+ADs cptable.utils.cache.encache()+ADs y +AD0 cptable.utils.decode(t, x)+ADs assert.equal(y,b)+ADs z +AD0 cptable.utils.encode(t, y)+ADs if(t +ACEAPQ 65000) cmp(x,z)+ADs else +AHs assert.equal(y, cptable.utils.decode(t, z))+ADs +AH0 cptable.utils.cache.decache()+ADs y +AD0 cptable.utils.decode(t, x)+ADs assert.equal(y,b)+ADs z +AD0 cptable.utils.encode(t, y)+ADs if(t +ACEAPQ 65000) cmp(x,z)+ADs else +AHs assert.equal(y, cptable.utils.decode(t, z))+ADs +AH0 cptable.utils.cache.encache()+ADs +AH0 : null)+ADs it(+ACI-should process README.md.+ACI +- m+AFs-t+AF0, fs.existsSync('./misc/README.md.' +- m+AFs-t+AF0) ? function() +AHs var b +AD0 fs.readFileSync('./misc/README.md.utf8', +ACI-utf8+ACI)+ADs if(m+AFs-t+AF0 +AD0APQA9 +ACI-ascii+ACI) b +AD0 b.replace(/+AFsAXA-u0080-+AFw-uffff+AF0AKg-/g,+ACIAIg)+ADs var x +AD0 fs.readFileSync('./misc/README.md.' +- m+AFs-t+AF0)+ADs x +AD0 +AFsAXQ.slice.call(x)+ADs cptable.utils.cache.encache()+ADs var y +AD0 cptable.utils.decode(t, x)+ADs assert.equal(y,b)+ADs cptable.utils.cache.decache()+ADs var y +AD0 cptable.utils.decode(t, x)+ADs assert.equal(y,b)+ADs cptable.utils.cache.encache()+ADs +AH0 : null)+ADs +AH0)+ADsAfQ)+ADs +AGAAYABg The codepage +AGA-6969+AGA is not defined, so operations should fail: +AGAAYABgAD4-test.js describe('failures', function() +AHs it('should fail to find CP 6969', function() +AHs assert.throws(function()+AHs-cptable+AFs-6969+AF0.dec+AH0)+ADs assert.throws(function()+AHs-cptable+AFs-6969+AF0.enc+AH0)+ADs +AH0)+ADs it('should fail using utils', function() +AHs assert(+ACE-cptable.utils.hascp(6969))+ADs assert.throws(function()+AHs-return cptable.utils.encode(6969, +ACI-foobar+ACI)+ADs +AH0)+ADs assert.throws(function()+AHs-return cptable.utils.decode(6969, +AFs-0x20+AF0)+ADs +AH0)+ADs +AH0)+ADs it('should fail with black magic', function() +AHs assert(cptable.utils.hascp(16969))+ADs assert.throws(function()+AHs-return cptable.utils.encode(16969, +ACI-foobar+ACI)+ADs +AH0)+ADs assert.throws(function()+AHs-return cptable.utils.decode(16969, +AFs-0x20+AF0)+ADs +AH0)+ADs +AH0)+ADs it('should fail when presented with invalid char codes', function() +AHs assert.throws(function()+AHs-cptable.utils.cache.decache()+ADs return cptable.utils.encode(20127, +AFs-String.fromCharCode(0xAA)+AF0)+ADsAfQ)+ADs +AH0)+ADs +AH0)+ADs +AGAAYABg +ACM Nitty Gritty +AGAAYABg-json+AD4-package.json +AHs +ACI-name+ACI: +ACI-codepage+ACI, +ACI-version+ACI: +ACI-1.3.0+ACI, +ACI-author+ACI: +ACI-SheetJS+ACI, +ACI-description+ACI: +ACI-pure-JS library to handle codepages+ACI, +ACI-keywords+ACI: +AFs +ACI-codepage+ACI, +ACI-iconv+ACI, +ACI-convert+ACI, +ACI-strings+ACI +AF0, +ACI-bin+ACI: +AHs +ACI-codepage+ACI: +ACI./bin/codepage.njs+ACI +AH0, +ACI-main+ACI: +ACI-cputils.js+ACI, +ACI-dependencies+ACI: +AHs +ACI-voc+ACI:+ACIAIg, +ACI-commander+ACI:+ACIAIg +AH0, +ACI-devDependencies+ACI: +AHs +ACI-mocha+ACI:+ACIAIg +AH0, +ACI-repository+ACI: +AHs +ACI-type+ACI:+ACI-git+ACI, +ACI-url+ACI:+ACI-git://github.com/SheetJS/js-codepage.git+ACIAfQ, +ACI-scripts+ACI: +AHs +ACI-pretest+ACI: +ACI-git submodule init +ACYAJg git submodule update+ACI, +ACI-test+ACI: +ACI-make test+ACI, +ACI-build+ACI: +ACI-make js+ACI +AH0, +ACI-config+ACI: +AHs +ACI-blanket+ACI: +AHs +ACI-pattern+ACI: +ACIAWw-cptable.js,cputils.js,cpexcel.js+AF0AIg +AH0 +AH0, +ACI-bugs+ACI: +AHs +ACI-url+ACI: +ACI-https://github.com/SheetJS/js-codepage/issues+ACI +AH0, +ACI-license+ACI: +ACI-Apache-2.0+ACI, +ACI-engines+ACI: +AHs +ACI-node+ACI: +ACIAPgA9-0.8+ACI +AH0 +AH0 +AGAAYABg +AGAAYABgAD4.vocrc +AHs +ACI-post+ACI: +ACI-make js+ACI +AH0 +AGAAYABg +AGAAYABgAD4.gitignore .gitignore codepages/ .vocrc node+AF8-modules/ make.sh make.njs misc/coverage.html +AGAAYABg