Does sheetjs support streaming write large xlsx file in browser #3120

New Issue

AlwaysNoobCoder · 2024-04-25T06:51:23Z

AlwaysNoobCoder commented

2024-04-25 06:51:23 +00:00

I have tested with csv:

const exportCsv = async () => {
  if (!window.showSaveFilePicker) {
    console.log("saving with streamsaver")
    const te = new TextEncoder()
    const fileStream = createWriteStream("users.csv", {
      size: 164224, // (optional filesize) Will show progress
      writableStrategy: undefined, // (optional)
      readableStrategy: undefined, // (optional)
    })
    const writer = fileStream.getWriter()

    const ws = new WebSocket("wss://portal-local.itps.xxx.com:3000/ws/export/sea/users")

    // ws.binaryType = "arraybuffer"

    ws.onopen = () => {
      console.log("websocket connected")
      ws.send("{}")
    }
    ws.onmessage = (event) => {
      console.log("websocket message")
      writer.write(te.encode(event.data))
    }
    ws.onclose = (event) => {
      console.log("websocket closed", event)
      writer.close()
    }

    ws.onerror = (event) => {
      console.log("websocket error", event)
    }
  } else {
    console.log("saving with file system api")

    const handle = await window.showSaveFilePicker({
      types: [
        {
          accept: {
            // Omitted
          },
        },
      ],
    })
    const writable = await handle.createWritable()

    const ws = new WebSocket("wss://portal-local.itps.xxxx.com:3000/ws/export/sea/users")
    ws.onopen = () => {
      console.log("websocket connected")
      ws.send("{}")
    }
    ws.onmessage = async (event) => {
      console.log("websocket message", event)
      await writable.write(event.data)
    }
    ws.onclose = async (event) => {
      console.log("websocket closed", event)
      await writable.close()
    }

    ws.onerror = (event) => {
      console.log("websocket error", event)
    }
  }
}

for xlsx, is it possible ? I tried exceljs, seems it's streaming support is only for nodejs environment, while that's confusing me, because web streams is already supported by major browsers, I asked help at exceljs's repo but no response.

what about sheetjs ? do you suport web streams ?

I have tested with csv: ``` const exportCsv = async () => { if (!window.showSaveFilePicker) { console.log("saving with streamsaver") const te = new TextEncoder() const fileStream = createWriteStream("users.csv", { size: 164224, // (optional filesize) Will show progress writableStrategy: undefined, // (optional) readableStrategy: undefined, // (optional) }) const writer = fileStream.getWriter() const ws = new WebSocket("wss://portal-local.itps.xxx.com:3000/ws/export/sea/users") // ws.binaryType = "arraybuffer" ws.onopen = () => { console.log("websocket connected") ws.send("{}") } ws.onmessage = (event) => { console.log("websocket message") writer.write(te.encode(event.data)) } ws.onclose = (event) => { console.log("websocket closed", event) writer.close() } ws.onerror = (event) => { console.log("websocket error", event) } } else { console.log("saving with file system api") const handle = await window.showSaveFilePicker({ types: [ { accept: { // Omitted }, }, ], }) const writable = await handle.createWritable() const ws = new WebSocket("wss://portal-local.itps.xxxx.com:3000/ws/export/sea/users") ws.onopen = () => { console.log("websocket connected") ws.send("{}") } ws.onmessage = async (event) => { console.log("websocket message", event) await writable.write(event.data) } ws.onclose = async (event) => { console.log("websocket closed", event) await writable.close() } ws.onerror = (event) => { console.log("websocket error", event) } } } ``` for xlsx, is it possible ? I tried exceljs, seems it's streaming support is only for nodejs environment, while that's confusing me, because web streams is already supported by major browsers, I asked help at exceljs's repo but no response. what about sheetjs ? do you suport web streams ?

sheetjs commented

2024-04-25 09:22:24 +00:00

We looked into CSV streaming in the browser and found that it was much slower than writing at once.

https://docs.sheetjs.com/docs/demos/bigdata/worker#streaming-write expand the "Live Demo" section at the end.

The live demo will process data in a web worker and use the browser streaming APIs to write to file.

If you do not check "Commit each row", the data is collected and saved to file at the end.

If you check "Commit each row", each row is written in order.

The write is significantly slower even at 10K rows. This is due to the forced await. The current implementation of Web Streams API in Chomium does not guarantee write order, so "just don't await" is not viable.

If there is some serious improvement in the web APIs, we will revisit the issue.

We looked into CSV streaming in the browser and found that it was much slower than writing at once. https://docs.sheetjs.com/docs/demos/bigdata/worker#streaming-write expand the "Live Demo" section at the end. The live demo will process data in a web worker and use the browser streaming APIs to write to file. If you do not check "Commit each row", the data is collected and saved to file at the end. ![i3120-end.png](/attachments/07f46fde-f81a-4fdd-8ca7-c2122e2a16e3) If you check "Commit each row", each row is written in order. ![i3120-each.png](/attachments/6862ce6e-a027-43f6-8360-5d6edcbda0fe) The write is significantly slower even at 10K rows. This is due to the forced `await`. The current implementation of Web Streams API in Chomium does not guarantee write order, so "just don't await" is not viable. If there is some serious improvement in the web APIs, we will revisit the issue.

i3120-end.png

54 KiB

i3120-each.png

53 KiB

Sign in to join this conversation.