Writing large excel files causes out of memory in IE11 and web page crashes in chrome #540

New Issue

2017-01-25T10:59:50Z

rahulstomar commented

2017-01-25 10:59:50 +00:00

(Migrated from github.com)

Hi,

I am trying to write an excel file from a 2D array. My array contains approximately 176000 records (total cells will be 176000 * 17, where 17 is column count). While writing IE11 gives Out of Memory in console and Chrome gives "Aw Snap" page saying "Chrome ran out of memory". One time it ran successfully to write the same data file and it's size came to be around 127MB.

Is there a different way of writing these kinds of large excel files? like as an stream.

I am using the same code provided in https://github.com/SheetJS/js-xlsx/blob/master/tests/write.js, apart from this I am using jszip.js and FileSaver.js for saving the file.

Hi, I am trying to write an excel file from a 2D array. My array contains approximately 176000 records (total cells will be 176000 * 17, where 17 is column count). While writing IE11 gives Out of Memory in console and Chrome gives "Aw Snap" page saying "Chrome ran out of memory". One time it ran successfully to write the same data file and it's size came to be around 127MB. Is there a different way of writing these kinds of large excel files? like as an stream. I am using the same code provided in https://github.com/SheetJS/js-xlsx/blob/master/tests/write.js, apart from this I am using jszip.js and FileSaver.js for saving the file.

alejandrolsca commented

2017-02-01 22:01:37 +00:00

(Migrated from github.com)

Same problem with 65000 records, any thoughts/ideas?

rahulstomar commented

2017-02-03 06:20:18 +00:00

(Migrated from github.com)

Not much, the code breaks in xlsx.js when jszip.js' generate function is called. I tried updating the code to use jszip's 3.x version, which has generateAsync function. Since I don't know much about web worker or StreamHelper I couldn't do anything with it.

Finally I dropped out of generating excel file, now I am just creating CSVs.

Not much, the code breaks in **xlsx.js** when j**szip.js**' **generate** function is called. I tried updating the code to use jszip's 3.x version, which has **generateAsync** function. Since I don't know much about web worker or **StreamHelper** I couldn't do anything with it. Finally I dropped out of generating excel file, now I am just creating CSVs.

reviewher commented

2017-02-03 07:45:20 +00:00

(Migrated from github.com)

@rahulstomar consider a simple file of the form

|     1 |     2 |     3 | ... | ncol |
| =A1+1 | =B1+1 | =C1+1 | ... |  ... |
| =A2+1 | =B2+1 | =C2+1 | ... |  ... |
| ..... | ..... | ..... | ... |  ... |

For 176k rows and 17 columns, Excel 2016 for Windows generates files of the following sizes:

format	size	ws size
xlsx	27 MB	180 MB
xlsb	16 MB	119 MB
xls	35 MB	34 MB
csv	18 MB	18 MB
xml	224 MB	224 MB

IMHO at this scale if you are just passing data you can't really beat the CSV.

Exposing the jszip compression option would reduce the file size but extend the processing time.

@alejandrolsca if generating large files is of particular interest, we would need a different approach. The Excel 2003 XML format is incredibly easy to stream as it does not involve any sort of post-processing, but has the downside of bloat. Generating an old form of XLS (e.g. BIFF2) would also have a small file size but you would not be able to open the file with all third-party programs.

@rahulstomar consider a simple file of the form ``` | 1 | 2 | 3 | ... | ncol | | =A1+1 | =B1+1 | =C1+1 | ... | ... | | =A2+1 | =B2+1 | =C2+1 | ... | ... | | ..... | ..... | ..... | ... | ... | ``` For 176k rows and 17 columns, Excel 2016 for Windows generates files of the following sizes: | format | size | ws size | |--------|-------:|--------:| | xlsx | 27 MB | 180 MB | | xlsb | 16 MB | 119 MB | | xls | 35 MB | 34 MB | | csv | 18 MB | 18 MB | | xml | 224 MB | 224 MB | IMHO at this scale if you are just passing data you can't really beat the CSV. Exposing the jszip compression option would reduce the file size but extend the processing time. @alejandrolsca if generating large files is of particular interest, we would need a different approach. The Excel 2003 XML format is incredibly easy to stream as it does not involve any sort of post-processing, but has the downside of bloat. Generating an old form of XLS (e.g. BIFF2) would also have a small file size but you would not be able to open the file with all third-party programs.

the-spyke commented

2017-07-04 11:50:38 +00:00

(Migrated from github.com)

I'm having the same issue. Quick googling gave me this Stuk/jszip#135.

😆 1

reviewher commented

2021-09-19 00:48:28 +00:00

(Migrated from github.com)

Grouping large file write issues into #77

Sign in to join this conversation.