XLSX.write generates a memory error while dealing with large json data leaving JavaScript heap out of memory error #798
Labels
No Label
DBF
Dates
Defined Names
Features
Formula
HTML
Images
Infrastructure
Integration
International
ODS
Operations
Performance
PivotTables
Pro
Protection
Read Bug
SSF
SYLK
Style
Write Bug
good first issue
No Milestone
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: sheetjs/sheetjs#798
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
So i have a large json of array object and length of this json array is around 100000 , which i first convert it into a sheet using
json_to_sheet
, push it into a workbook and then write the given workbook into xlsx format usingXLSX.write()
, which takes a lot memory and eventually nodejs terminates in between process displayingFATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
error ..i have tried to increase heap memory along the node process with
--max-old-space-size=8100
which results the same giving the same errortried it with
code i m using :
and the error generated
I am getting the same issue but for method writeFile.
My code is:
let workbook = { SheetNames: [], Sheets: {} };
let result = await makeGetRequest(url, headers, null, cookies);
let data = result.body;
let wb = XLSX.read(data, {raw: true});
let ws = wb.Sheets[wb.SheetNames[0]];
/* Add the sheet name to the list */
let sheetName = key.replace(/\/*\\*\[*\]*\:*\**\?*/g, '').substr(0,30);
workbook.SheetNames.push(sheetName);
/* Load the worksheet object */
workbook.Sheets[key] = ws;
//Synchronous version of temp file creation
const tmpobj = tmp.fileSync({mode: '0644', postfix: '_tab.xlsx'});
XLSX.writeFile(workbook, tmpobj.name);
try using
buffer
instead ofbase64
var wb = XLSX.read(bstr, { type: "buffer" });
or
var wbbuff = XLSX.write(wb, { type: 'buffer' });
@wlodi83 XLSX eventually has to go through XML text when reading and writing, and V8 has some limitations that you may be encountering. In addition to @shubham-kumar 's suggestion to use the
buffer
type in nodejs, can you also try writing toxlsb
orxls
to see if the issue occurs? to control the output format, just change the filename:Hi @reviewher, @shubham-kumar,
Neither buffer nor xls/xlsb work. I am getting following error:
<--- Last few GCs --->
364747 ms: Mark-sweep 1376.6 (1435.2) -> 1376.6 (1435.2) MB, 2993.1 / 0.0 ms [allocation failure] [GC in old space requested].
367677 ms: Mark-sweep 1376.6 (1435.2) -> 1376.6 (1435.2) MB, 2929.6 / 0.0 ms [allocation failure] [GC in old space requested].
371880 ms: Mark-sweep 1376.6 (1435.2) -> 1380.2 (1419.2) MB, 4203.1 / 0.0 ms [last resort gc].
374800 ms: Mark-sweep 1380.2 (1419.2) -> 1384.0 (1419.2) MB, 2919.7 / 0.0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x27414f53fa99
1: encode_row [/Users/lukasz/Documents/Projects/project/node_modules/xlsx/xlsx.js:~2579] [pc=0x2b890a89414e] (this=0xd21b423e2d1 ,row=75644)
2: write_ws_biff8 [/Users/lukasz/Documents/Projects/-reports/node_modules/xlsx/xlsx.js:~17106] [pc=0x2b890a44369e] (this=0xd21b423e2d1 ,idx=0,opts=0x647d1be2869 <an Object with map 0x52c36a...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/usr/local/bin/node]
2: node::FatalException(v8::Isolate*, v8::Localv8::Value, v8::Localv8::Message) [/usr/local/bin/node]
3: v8::Utils::ReportApiFailure(char const*, char const*) [/usr/local/bin/node]
4: v8::Utils::ApiCheck(bool, char const*, char const*) [/usr/local/bin/node]
5: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
6: v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
7: v8::internal::Factory::NewStringFromOneByte(v8::internal::Vector, v8::internal::PretenureFlag) [/usr/local/bin/node]
8: v8::internal::Factory::NumberToString(v8::internal::Handlev8::internal::Object, bool) [/usr/local/bin/node]
9: v8::internal::Runtime_NumberToStringSkipCache(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
10: 0x2b8909e060c7
11: 0x2b890a89414e
sh: line 1: 7952 Abort trap: 6 node dist/main $PWD/settings.json
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! Exit status 134
npm ERR!
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/wlodi83/.npm/_logs/2018-01-12T13_06_28_815Z-debug.log
File is quite big. It has around 300k rows. I wonder if I can handle it easily.
get this issue too. allocation failed.
I too get the error. Specifically in Internet Explorer. Chrome seems to handle it just fine. Watching the memory as the function runs just eats and eats memory. IE can't handle and just bombs out with an out of memory error.
"buffer" is used only for NodeJS so "base64" should be used
Actually I have issues with reading .xlsx files more than 7Mb using either "base64" or "binary"
Did anyone find resolution to this issue?
Same problem
Having the same problem too. Using SheetJS in a browser
having same issue, anyone found a solution?
Coming back to this my issue turned out I had memory leak bad function loop. Chrome could handle the leak while IE could not. Fixing the leak solved the error for me. Make sure you are handling the data you are processing clean.
Is it possible to append to a sheet in a file without opening the whole file into memory?
I just use
xls
file and it works fine with me on node, here's my functionGrouping large file write issues into #77