13 KiB
title | pagination_prev |
---|---|
Browser Automation | demos/net/email/index |
import current from '/version.js'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock';
Headless automation involves controlling "headless browsers" to access websites and submit or download data. It is also possible to automate browsers using custom browser extensions.
The SheetJS standalone scripts
can be added to any website by inserting a SCRIPT
tag. Headless browsers
usually provide utility functions for running custom snippets in the browser and
passing data back to the automation script.
Use Case
This demo focuses on exporting table data to a workbook. Headless browsers do not generally support passing objects between the browser context and the automation script, so the file data must be generated in the browser context and sent back to the automation script for saving in the file system.
Key Steps (click to hide)
-
Launch the headless browser and load the target site.
-
Add the standalone SheetJS build to the page in a
SCRIPT
tag. -
Add a script to the page (in the browser context) that will:
- Make a workbook object from the first table using
XLSX.utils.table_to_book
- Generate the bytes for an XLSB file using
XLSX.write
- Send the bytes back to the automation script
- When the automation context receives data, save to a file
This demo exports data from https://sheetjs.com/demos/table.
:::note pass
It is also possible to parse files from the browser context, but parsing from the automation context is more efficient and strongly recommended.
:::
Puppeteer
Puppeteer enables headless Chromium automation for NodeJS. Releases ship with an installer script that prepares a compatible browser version.
Binary strings are the favored data type. They can be safely passed from the
browser context to the automation script. NodeJS provides an API to write
binary strings to file (fs.writeFileSync
using encoding binary
).
The key steps are commented below:
{\ const fs = require("fs"); const puppeteer = require('puppeteer'); (async () => { /* (1) Load the target page */ const browser = await puppeteer.launch(); const page = await browser.newPage(); page.on("console", msg => console.log("PAGE LOG:", msg.text())); await page.setViewport({width: 1920, height: 1080}); await page.goto('https://sheetjs.com/demos/table'); \n\ /* (2) Load the standalone SheetJS build from the CDN */ await page.addScriptTag({ url: 'https://cdn.sheetjs.com/xlsx-${current}/package/dist/xlsx.full.min.js' }); \n\ /* (3) Run the snippet in browser and return data */ const bin = await page.evaluate(() => { /* NOTE: this function will be evaluated in the browser context. \
page`, `fs` and `puppeteer` are not available.
`XLSX` will be available thanks to step 2 /
\n
/ find first table /
var table = document.body.getElementsByTagName('table')[0];
\n
/ call table_to_book on first table /
var wb = XLSX.utils.table_to_book(table);
\n
/ generate XLSB and return binary string /
return XLSX.write(wb, {type: "binary", bookType: "xlsb"});
});
\n
/ (4) write data to file */
fs.writeFileSync("SheetJSPuppeteer.xlsb", bin, { encoding: "binary" });
\n
await browser.close();
})();`}
Demo
:::note
This demo was last tested on 2023 September 14 against Puppeteer 21.2.1.
:::
- Install SheetJS and Puppeteer:
{\ npm i --save https://cdn.sheetjs.com/xlsx-${current}/xlsx-${current}.tgz puppeteer@21.2.1
}
-
Save the
SheetJSPuppeteer.js
code snippet toSheetJSPuppeteer.js
. -
Run the script:
node SheetJSPuppeteer.js
When the script finishes, the file SheetJSPuppeteer.xlsb
will be created.
This file can be opened with Excel.
:::caution pass
Deno Puppeteer is a fork. It is not officially supported by the Puppeteer team.
:::
Base64 strings are the favored data type. They can be safely passed from the
browser context to the automation script. Deno can decode the Base64 strings
and write the decoded Uint8Array
data to file with Deno.writeFileSync
The key steps are commented below:
{\ import puppeteer from "https://deno.land/x/puppeteer@16.2.0/mod.ts"; import { decode } from "https://deno.land/std/encoding/base64.ts" \n\ /* (1) Load the target page */ const browser = await puppeteer.launch(); const page = await browser.newPage(); page.on("console", msg => console.log("PAGE LOG:", msg.text())); await page.setViewport({width: 1920, height: 1080}); await page.goto('https://sheetjs.com/demos/table'); \n\ /* (2) Load the standalone SheetJS build from the CDN */ await page.addScriptTag({ url: 'https://cdn.sheetjs.com/xlsx-${current}/package/dist/xlsx.full.min.js' }); \n\ /* (3) Run the snippet in browser and return data */ const b64 = await page.evaluate(() => { /* NOTE: this function will be evaluated in the browser context. \
page`, `fs` and `puppeteer` are not available.
`XLSX` will be available thanks to step 2 /
\n
/ find first table /
var table = document.body.getElementsByTagName('table')[0];
\n
/ call table_to_book on first table /
var wb = XLSX.utils.table_to_book(table);
\n
/ generate XLSB and return binary string /
return XLSX.write(wb, {type: "base64", bookType: "xlsb"});
});
/ (4) write data to file */
Deno.writeFileSync("SheetJSPuppeteer.xlsb", decode(b64));
\n
await browser.close();`}
Demo
:::note
This demo was last tested on 2023 September 14 against deno-puppeteer 16.2.0.
:::
- Install deno-puppeteer:
env PUPPETEER_PRODUCT=chrome deno run -A --unstable https://deno.land/x/puppeteer@16.2.0/install.ts
:::note pass
In PowerShell, the environment variable should be set separately:
[Environment]::SetEnvironmentVariable('PUPPETEER_PRODUCT', 'chrome')
deno run -A --unstable https://deno.land/x/puppeteer@16.2.0/install.ts
:::
-
Save the
SheetJSPuppeteer.ts
code snippet toSheetJSPuppeteer.ts
. -
Run the script:
deno run -A --unstable SheetJSPuppeteer.ts
When the script finishes, the file SheetJSPuppeteer.xlsb
will be created.
This file can be opened with Excel.
Playwright
Playwright presents a unified scripting framework for Chromium, WebKit, and other browsers. It draws inspiration from Puppeteer. In fact, the example code is almost identical!
Differences from the Puppeteer example are highlighted below:
{\ const fs = require("fs"); // highlight-next-line const { webkit } = require('playwright'); // import desired browser (async () => { /* (1) Load the target page */ // highlight-next-line const browser = await webkit.launch(); // launch desired browser const page = await browser.newPage(); page.on("console", msg => console.log("PAGE LOG:", msg.text())); // highlight-next-line await page.setViewportSize({width: 1920, height: 1080}); // different name :( await page.goto('https://sheetjs.com/demos/table'); \n\ /* (2) Load the standalone SheetJS build from the CDN */ await page.addScriptTag({ url: 'https://cdn.sheetjs.com/xlsx-${current}/package/dist/xlsx.full.min.js' }); \n\ /* (3) Run the snippet in browser and return data */ const bin = await page.evaluate(() => { /* NOTE: this function will be evaluated in the browser context. \
page`, `fs` and the browser engine are not available.
`XLSX` will be available thanks to step 2 /
\n
/ find first table /
var table = document.body.getElementsByTagName('table')[0];
\n
/ call table_to_book on first table /
var wb = XLSX.utils.table_to_book(table);
\n
/ generate XLSB and return binary string /
return XLSX.write(wb, {type: "binary", bookType: "xlsb"});
});
\n
/ (4) write data to file */
fs.writeFileSync("SheetJSPlaywright.xlsb", bin, { encoding: "binary" });
\n
await browser.close();
})();`}
Demo
:::note
This demo was last tested on 2023 September 14 against Playwright 1.38.0.
:::
- Install SheetJS and Playwright:
{\ npm i --save https://cdn.sheetjs.com/xlsx-${current}/xlsx-${current}.tgz playwright@1.33.0
}
-
Save the
SheetJSPlaywright.js
code snippet toSheetJSPlaywright.js
. -
Run the script
node SheetJSPlaywright.js
When the script finishes, the file SheetJSPlaywright.xlsb
will be created.
This file can be opened with Excel.
:::caution pass
In the latest Windows 10 test, the commmand failed with a clear error message:
╔═════════════════════════════════════════════════════════════════════════╗
║ Looks like Playwright Test or Playwright was just installed or updated. ║
║ Please run the following command to download new browsers: ║
║ ║
║ npx playwright install ║
║ ║
║ <3 Playwright Team ║
╚═════════════════════════════════════════════════════════════════════════╝
As recommended, the command
npx playwright install
will download and install the browsers.
:::
PhantomJS
PhantomJS is a headless web browser powered by WebKit.
:::warning pass
This information is provided for legacy deployments. PhantomJS development has been suspended and there are known vulnerabilities, so new projects should use alternatives. For WebKit automation, new projects should use Playwright.
:::
Binary strings are the favored data type. They can be safely passed from the
browser context to the automation script. PhantomJS provides an API to write
binary strings to file (fs.write
using mode wb
).
Integration Details and Demo (click to show)
The steps are marked in the comments:
{\ var page = require('webpage').create(); page.onConsoleMessage = function(msg) { console.log(msg); }; \n\ /* (1) Load the target page */ page.open('https://sheetjs.com/demos/table', function() { \n\ /* (2) Load the standalone SheetJS build from the CDN */ page.includeJs("https://cdn.sheetjs.com/xlsx-${current}/package/dist/xlsx.full.min.js", function() { \n\ /* (3) Run the snippet in browser and return data */ var bin = page.evaluateJavaScript([ "function(){", \n\ /* find first table */ "var table = document.body.getElementsByTagName('table')[0];", \n\ /* call table_to_book on first table */ "var wb = XLSX.utils.table_to_book(table);", \n\ /* generate XLSB file and return binary string */ "return XLSX.write(wb, {type: 'binary', bookType: 'xlsb'});", "}" ].join("")); \n\ /* (4) write data to file */ require("fs").write("SheetJSPhantomJS.xlsb", bin, "wb"); \n\ phantom.exit(); }); });
}
:::caution pass
PhantomJS is very finicky and will hang if there are script errors. It is strongly recommended to add verbose logging and to lint scripts before use.
:::
Demo
:::note
This demo was last tested on 2023 September 14 against PhantomJS 2.1.1
:::
-
Download and unzip the PhantomJS release from the official website1.
-
Save the
SheetJSPhantom.js
code snippet toSheetJSPhantom.js
. -
Run the
phantomjs
program and pass the script as the first argument.
For example, if the macOS Archive Utility unzipped the 2.1.1
release, binaries
will be placed in phantomjs-2.1.1-macosx/bin/
and the command will be:
./phantomjs-2.1.1-macosx/bin/phantomjs SheetJSPhantom.js
When the script finishes, the file SheetJSPhantomJS.xlsb
will be created.
This file can be opened with Excel.
-
Downloads available at https://phantomjs.org/download.html ↩︎