sheet_to_json use lots of memory #764
Labels
No Label
DBF
Dates
Defined Names
Features
Formula
HTML
Images
Infrastructure
Integration
International
ODS
Operations
Performance
PivotTables
Pro
Protection
Read Bug
SSF
SYLK
Style
Write Bug
good first issue
No Milestone
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: sheetjs/sheetjs#764
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
i upload a 11 kb xlsx file and want to convert it to json in my server. my server only has 512mb memory, and i got node process killed when execute
sheet_to_json
method. the scripts works well in my pc which has 8gb memory.is
sheet_to_json
function use a lot of memory which is much bigger than file itself ? is there any tips to solve memory problems ? thanks@smallst can you share a sample file?
Inspire-single.xlsx
thanks for replying. this is a sample file.
The root cause of the issue is a bad reported sheet size: If you unzip the file (XLSX is really a zip format) and look at the xml in
xl/worksheets/sheet1.xml
you'll see:The first line says that the file has over 1M records and the second line is completely unnecessary (guessing it was exported from a third party tool?)
Can you try overwriting the range using the
update_sheet_range
function from the wiki:Call that function on the worksheet just before passing it to
sheet_to_json
. It scans the cells to recalculate the range.how foolish i am! i edit the xlsx with libreoffice in linux, i nerver consider about file format problems.
problem is solved by call the
update_sheet_range
function. great job!@smallst some third party tools try to cut corners :( But this is really not your fault -- we came across a V8 / nodejs bug! https://github.com/nodejs/node/issues/14652
I understand that the bug is from nodejs/v8 but is there some workaround ? I tried using the update_sheet_range function before sheet_to_csv but without success.
I get the high memory bug in all recent nodejs versions (latest 8.x 10.x and 11.x). A 40mb xls files goes over the heap size limit with max-old-space-size=1024.
It seems to be beter by using type Map; (node verison v14.9.0)
this may be helpful:
reference link:
node issue
v8 issue