diff --git a/.spelling b/.spelling index 0513a27..f23f296 100644 --- a/.spelling +++ b/.spelling @@ -160,6 +160,7 @@ GBK GatsbyJS GitLab Goja +GraphiQL HTML HTML5 HTTP diff --git a/docz/docs/03-demos/06-content.md b/docz/docs/03-demos/06-content.md index 8030167..1b3fcca 100644 --- a/docz/docs/03-demos/06-content.md +++ b/docz/docs/03-demos/06-content.md @@ -2,6 +2,8 @@ title: Content and Site Generation --- +import current from '/version.js'; + With the advent of server-side frameworks and content management systems, it is possible to build sites whose source of truth is a spreadsheet! This demo explores a number of approaches. @@ -100,6 +102,268 @@ in the parsing logic, issues should then be raised with the SheetJS project. ::: +
GraphQL details (click to show) + +`gatsby-transformer-excel` generates nodes for each data row of each worksheet. +Under the hood, it uses [`sheet_to_json`](/docs/api/utilities#array-output) +to generate row objects using the headers in the first row as keys. + +![pres.xlsx](pathname:///pres.png) + +Assuming the file name is `pres.xlsx` and the data is stored in "Sheet1", the +following nodes will be created: + +```js +[ + { Name: "Bill Clinton", Index: 42, type: "PresXlsxSheet1" }, + { Name: "GeorgeW Bush", Index: 43, type: "PresXlsxSheet1" }, + { Name: "Barack Obama", Index: 44, type: "PresXlsxSheet1" }, + { Name: "Donald Trump", Index: 45, type: "PresXlsxSheet1" }, + { Name: "Joseph Biden", Index: 46, type: "PresXlsxSheet1" }, +] +``` + +The type is a proper casing of the file name concatenated with the sheet name. + +The following query pulls the `Name` and `Index` fields from each row: + +```graphql +{ + allPresXlsxSheet1 { # "all" followed by type + edges { + node { # each line in this block should be a field in the data + Name + Index + } + } + } +} +``` + +
+ +:::caution + +`gatsby-transformer-excel` uses an older version of the library. It can be +overridden through a `package.json` override in the latest versions of NodeJS: + +
{`\
+{
+  "overrides": {
+    "xlsx": "https://cdn.sheetjs.com/xlsx-${current}/xlsx-${current}.tgz"
+  }
+}`}
+
+ +::: + +
Complete Example (click to show) + +:::note + +This demo was tested on 2022 November 11 against `create-gatsby@3.0.0`. The +generated project used `gatsby@5.0.0` and `react@18.2.0`. + +::: + +1) Run `npm init gatsby -- -y sheetjs-gatsby` to create the template site. + +2) Follow the on-screen instructions for starting the local development server: + +```bash +cd sheetjs-gatsby +npm run develop +``` + +Open a web browser to the displayed URL (typically `http://localhost:8000/`) + +3) Edit `package.json` and add the highlighted lines in the JSON object: + +
{`\
+{
+  // highlight-start
+  "overrides": {
+    "xlsx": "https://cdn.sheetjs.com/xlsx-${current}/xlsx-${current}.tgz"
+  },
+  // highlight-end
+  "name": "sheetjs-gatsby",
+  "version": "1.0.0",
+`}
+
+ +4) Install the library and plugins: + +
{`\
+npm i --save https://cdn.sheetjs.com/xlsx-${current}/xlsx-${current}.tgz
+npm i --save gatsby-transformer-excel gatsby-source-filesystem
+`}
+
+ +5) Edit `gatsby-config.js` and add the following lines to the `plugins` array: + +```js + plugins: [ + { + resolve: `gatsby-source-filesystem`, + options: { + name: `data`, + path: `${__dirname}/src/data/`, + }, + }, + `gatsby-transformer-excel`, + ], +``` + +Stop and restart the development server process (`npm run develop`). + +6) Make a `src/data` directory, download , and +move the downloaded file into the new folder: + +```bash +mkdir -p src/data +curl -LO https://sheetjs.com/pres.xlsx +mv pres.xlsx src/data +``` + +7) To verify, open the GraphiQL editor at `http://localhost:8000/___graphql`. + +There is an editor in the left pane. Paste the following query into the editor: + +```graphql +{ + allPresXlsxSheet1 { + edges { + node { + Name + Index + } + } + } +} +``` + +Press the Execute Query button and data should show up in the right pane: + +![GraphiQL Screenshot](pathname:///gatsby/graphiql.png) + +8) Create a new file `src/pages/pres.js` that uses the query and displays the result: + +```jsx title="src/pages/pres.js" +import { graphql } from "gatsby" +import * as React from "react" + +export const query = graphql`query { + allPresXlsxSheet1 { + edges { + node { + Name + Index + } + } + } +}`; + +const PageComponent = ({data}) => { + return (
{JSON.stringify(data, 2, 2)}
); +}; +export default PageComponent; +``` + +After saving the file, access `http://localhost:8000/pres`. The displayed JSON +is the data that the component receives: + +```js +{ + "allPresXlsxSheet1": { + "edges": [ + { + "node": { + "Name": "Bill Clinton", + "Index": 42 + } + }, + // .... +``` + +9) Change `PageComponent` to display a table based on the data: + +```jsx title="src/pages/pres.js" +import { graphql } from "gatsby" +import * as React from "react" + +export const query = graphql`query { + allPresXlsxSheet1 { + edges { + node { + Name + Index + } + } + } +}`; + +// highlight-start +const PageComponent = ({data}) => { + const rows = data.allPresXlsxSheet1.edges.map(r => r.node); + return ( + + {rows.map(row => ( + + + ))} +
NameIndex
{row.Name}{row.Index}
); +}; +// highlight-end + +export default PageComponent; +``` + +Going back to the browser, `http://localhost:8000/pres` will show a table: + +![Data in Table](pathname:///gatsby/table1.png) + +10) Open the file `src/data/pres.xlsx` in Excel or LibreOffice or Numbers. +Add a new row at the end of the file: + +![New Row in File](pathname:///gatsby/pres2.png) + +Save the file and notice that the table has refreshed with the new data: + +![Updated Table](pathname:///gatsby/table2.png) + +11) Stop the development server and run `npm run build`. Once the build is +finished, the display will confirm that the `/pres` route is static: + +``` +Pages + +┌ src/pages/404.js +│ ├ /404/ +│ └ /404.html +├ src/pages/index.js +│ └ / +└ src/pages/pres.js + └ /pres/ + + ╭────────────────────────────────────────────────────────────────╮ + │ │ + │ (SSG) Generated at build time │ + │ D (DSG) Deferred static generation - page generated at runtime │ + │ ∞ (SSR) Server-side renders at runtime (uses getServerData) │ + │ λ (Function) Gatsby function │ + │ │ + ╰────────────────────────────────────────────────────────────────╯ +``` + +The built page will be placed in `public/pres/index.html`. Open the page with a +text editor and search for "SheetJS" to verify raw HTML was generated: + +```html +SheetJS Dev47 +``` + +
+ ## ViteJS :::note diff --git a/docz/docs/03-demos/13-vue.md b/docz/docs/03-demos/13-vue.md index c13b92a..649deb9 100644 --- a/docz/docs/03-demos/13-vue.md +++ b/docz/docs/03-demos/13-vue.md @@ -115,27 +115,17 @@ function exportFile() { :::note -This demo was last run on 2022 November 23 using `vue@3.2.41`. When running +This demo was last run on 2022 November 11 using `vue@3.2.41`. When running `npm init`, the package `create-vue@3.4.0` was installed. ::: -1) Run `npm init vue@latest`. When prompted: - -- Project name: `SheetJSVue` -- Package name: `sheetjsvue` -- Add TypeScript? `No` (default) -- Add JSX Support? `No` (default) -- Add Vue Router for Single Page Application development? `No` (default) -- Add Pinia for state management? `No` (default) -- Add Vitest for Unit Testing? `No` (default) -- Add an End-to-End Testing Solution? `No` (default) -- Add ESLint for code quality? `No` (default) +1) Run `npm init vue@latest -- sheetjs-vue --default`. 2) Install the SheetJS dependency and start the dev server: ```bash -cd SheetJSVue +cd sheetjs-vue npm install npm i --save https://cdn.sheetjs.com/xlsx-latest/xlsx-latest.tgz npm run dev @@ -145,6 +135,13 @@ npm run dev 4) Replace `src/App.vue` with the `src/SheetJSVueAoO.vue` example. +The page will refresh and show a table with an Export button. Click the button +and the page will attempt to download `SheetJSVueAoA.xlsx`. There may be a delay +since Vite will try to optimize the SheetJS library on the fly. + +5) Build the site with `npm run build`, then test with `npx http-server dist`. +Access `http://localhost:8080` with a web browser to test the bundled site. + ### HTML @@ -194,27 +191,17 @@ function exportFile() { :::note -This demo was last run on 2022 November 23 using `vue@3.2.41`. When running +This demo was last run on 2022 November 11 using `vue@3.2.41`. When running `npm init`, the package `create-vue@3.4.0` was installed. ::: -1) Run `npm init vue@latest`. When prompted: - -- Project name: `SheetJSVue` -- Package name: `sheetjsvue` -- Add TypeScript? `No` (default) -- Add JSX Support? `No` (default) -- Add Vue Router for Single Page Application development? `No` (default) -- Add Pinia for state management? `No` (default) -- Add Vitest for Unit Testing? `No` (default) -- Add an End-to-End Testing Solution? `No` (default) -- Add ESLint for code quality? `No` (default) +1) Run `npm init vue@latest -- sheetjs-vue --default`. 2) Install the SheetJS dependency and start the dev server: ```bash -cd SheetJSVue +cd sheetjs-vue npm install npm i --save https://cdn.sheetjs.com/xlsx-latest/xlsx-latest.tgz npm run dev @@ -224,6 +211,13 @@ npm run dev 4) Replace `src/App.vue` with the `src/SheetJSVueHTML.vue` example. +The page will refresh and show a table with an Export button. Click the button +and the page will attempt to download `SheetJSVueHTML.xlsx`. There may be a delay +since Vite will try to optimize the SheetJS library on the fly. + +5) Build the site with `npm run build`, then test with `npx http-server dist`. +Access `http://localhost:8080` with a web browser to test the bundled site. + ### Rows and Columns diff --git a/docz/docs/03-demos/34-network.mdx b/docz/docs/03-demos/34-network.mdx index 87070ff..2f594f5 100644 --- a/docz/docs/03-demos/34-network.mdx +++ b/docz/docs/03-demos/34-network.mdx @@ -482,10 +482,48 @@ function SheetJSSuperAgentUL() { ## NodeJS Demos -This demo focuses on a number of strategies. Some of these demos were written -before NodeJS added `fetch`. +These examples show how to download data in NodeJS. -### request +### HTTPS GET + +The `https` module provides a low-level `get` method for HTTPS GET requests: + +```js +var https = require("https"), XLSX = require("xlsx"); + +https.get('https://sheetjs.com/pres.numbers', function(res) { + var bufs = []; + res.on('data', function(chunk) { bufs.push(chunk); }); + res.on('end', function() { + var buf = Buffer.concat(bufs); + var wb = XLSX.read(buf); + /* print the first worksheet to console */ + var ws = wb.Sheets[wb.SheetNames[0]]; + console.log(XLSX.utils.sheet_to_csv(ws)); + }); +}); +``` + +### fetch + +The `fetch` implementation has the same return types as the browser version: + +```js +async function parse_from_url(url) { + const res = await fetch(url); + if(!res.ok) throw new Error("fetch failed"); + const ab = await res.arrayBuffer(); + const workbook = XLSX.read(ab); + return workbook; +} +``` + +### Wrapper Libraries + +The latest releases of NodeJS support `fetch` natively. Before `fetch` support +was added to the platform, third party modules wrapped the native APIs. + +#### request :::warning @@ -510,6 +548,22 @@ request(url, {encoding: null}, function(err, res, data) { /* print the first worksheet to console */ var ws = wb.Sheets[wb.SheetNames[0]]; - console.log(XLSX.utils.sheet_to_csv(ws, {blankrows:false})); + console.log(XLSX.utils.sheet_to_csv(ws)); }); -``` \ No newline at end of file +``` + +#### axios + +When the `responseType` is `"arraybuffer"`, `axios` actually captures the data +in a NodeJS Buffer. `XLSX.read` will transparently handle Buffers: + +```js +const XLSX = require("xlsx"), axios = require("axios"); + +async function workbook_dl_axios(url) { + const res = await axios(url, {responseType:'arraybuffer'}); + /* at this point, res.data is a Buffer */ + const workbook = XLSX.read(res.data); + return workbook; +} +``` diff --git a/docz/docs/03-demos/43-ml.mdx b/docz/docs/03-demos/43-ml.mdx index b0a2119..fcef16f 100644 --- a/docz/docs/03-demos/43-ml.mdx +++ b/docz/docs/03-demos/43-ml.mdx @@ -11,14 +11,15 @@ not JS Arrays! With some data wrangling, translating between SheetJS worksheets and typed arrays is straightforward. This demo covers conversions between worksheets and Typed Arrays for use with -[TensorFlow.js](https://js.tensorflow.org/js/) and other ML libraries. +TensorFlow.js and other ML libraries. :::note -The live code blocks in this demo load the standalone TensorFlow.js build from -version `3.18.0`. It is shipped in the NodeJS package at `/dist/tf.min.js`. +Live code blocks in this page load the standalone build from version `3.18.0`. -[Official installation notes](https://www.tensorflow.org/js/tutorials/setup) +For use in web frameworks, the `@tensorflow/tfjs` module should be used. + +For use in NodeJS, the native bindings module is `@tensorflow/tfjs-node`. ::: diff --git a/docz/static/gatsby/graphiql.png b/docz/static/gatsby/graphiql.png new file mode 100644 index 0000000..5ab73fa Binary files /dev/null and b/docz/static/gatsby/graphiql.png differ diff --git a/docz/static/gatsby/pres2.png b/docz/static/gatsby/pres2.png new file mode 100644 index 0000000..8dbf8a7 Binary files /dev/null and b/docz/static/gatsby/pres2.png differ diff --git a/docz/static/gatsby/table1.png b/docz/static/gatsby/table1.png new file mode 100644 index 0000000..1cc497d Binary files /dev/null and b/docz/static/gatsby/table1.png differ diff --git a/docz/static/gatsby/table2.png b/docz/static/gatsby/table2.png new file mode 100644 index 0000000..5615ea9 Binary files /dev/null and b/docz/static/gatsby/table2.png differ