Loader tutorial

docz/docs/02-getting-started/02-examples/02-export.mdx

@@ -222,11 +222,13 @@ const prez = raw_data.filter(row => row.terms.some(term => term.type == "prez"))
 ### Sorting by First Term
 
 The dataset is sorted in chronological order by the first presidential or vice
-presidential term.  The Vice President and President in a given term are sorted
-alphabetically. Joe Biden and Barack Obama were Vice President and President
-respectively in 2009. Since "Biden" is alphabetically before "Obama", Biden's
-data point appears first. The goal is to sort the presidents in order of their
-presidential term.
+presidential term. The Vice President and President in a given term are sorted
+alphabetically.
+
+Barack Obama became President and Joseph Biden became Vice President in 2009.
+Since "Biden" is alphabetically before "Obama", Biden's data appears first.
+
+The goal is to sort the presidents in order of their initial presidential term.
 
 The first step is adding the first presidential term start date to the dataset.
 The following code looks at each president and creates a `start` property that

docz/docs/02-getting-started/02-examples/06-loader.md

@@ -0,0 +1,707 @@
+---
+title: Loader Tutorial
+pagination_prev: getting-started/installation/index
+pagination_next: getting-started/roadmap
+sidebar_position: 6
+---
+
+import current from '/version.js';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+Many existing systems and platforms include support for loading data from CSV
+files. Many users prefer to work in spreadsheet software and multi-sheet file
+formats including XLSX. SheetJS libraries help bridge the gap by translating
+complex workbooks to simple CSV data.
+
+The goal of this example is to load spreadsheet data into a vector store and use
+a large language model to generate queries based on English language input. The
+existing tooling supports CSV but does not support real spreadsheets.
+
+In ["SheetJS Conversion"](#sheetjs-conversion), we will use SheetJS libraries to
+generate CSV files for the LangChain CSV loader. These conversions can be run in
+a preprocessing step without disrupting existing CSV workflows.
+
+In ["SheetJS Loader"](#sheetjs-loader), we will use SheetJS libraries in a custom
+loader to directly generate documents and metadata.
+
+:::note Tested Deployments
+
+This demo was tested in the following configurations:
+
+| Date       | Platform                                                      |
+|:-----------|:--------------------------------------------------------------|
+| 2024-06-19 | Apple M2 Max 12-Core CPU + 30-Core GPU (32 GB unified memory) |
+| 2024-06-19 | NVIDIA RTX 4080 SUPER (16 GB VRAM) + i9-10910 (128 GB RAM)    |
+
+This explanation was verified against LangChain 0.2.
+
+:::
+
+## CSV Loader
+
+Document loaders generate data objects ("documents") and associated metadata
+from data sources.
+
+LangChain offers a `CSVLoader`[^1] component for loading CSV data from a file:
+
+```js title="Generating Documents from a CSV file"
+import { CSVLoader } from "@langchain/community/document_loaders/fs/csv";
+
+const loader = new CSVLoader("pres.csv");
+const docs = await loader.load();
+
+console.log(docs);
+```
+
+The CSV loader uses the first row to determine column headers and generates one
+document per data row. For example, the following CSV holds Presidential data:
+
+```csv
+Name,Index
+Bill Clinton,42
+GeorgeW Bush,43
+Barack Obama,44
+Donald Trump,45
+Joseph Biden,46
+```
+
+Each data row is translated to a document whose content is a list of attributes
+and values. For example, the third data row is shown below:
+
+<table>
+  <thead><tr><th>CSV Row</th><th>Document Content</th></tr></thead>
+  <tbody><tr><td>
+
+```
+Name,Index
+Barack Obama,44
+```
+
+  </td><td>
+
+```
+Name: Barack Obama
+Index: 44
+```
+
+  </td></tr></tbody>
+</table>
+
+The LangChain CSV loader will include source metadata in the document:
+
+```js title="Document generated by the CSV loader"
+Document {
+  pageContent: 'Name: Barack Obama\nIndex: 44',
+  metadata: { source: 'pres.csv', line: 3 }
+}
+```
+
+## SheetJS Conversion
+
+The [SheetJS NodeJS module](/docs/getting-started/installation/nodejs) can be
+imported in NodeJS scripts that use LangChain and other JavaScript libraries.
+
+A simple pre-processing step can convert workbooks to spreadsheets
+
+```mermaid
+flowchart LR
+  file[(Workbook\nXLSX/XLS)]
+  subgraph SheetJS Structures
+    wb(((SheetJS\nWorkbook)))
+    ws((SheetJS\nWorksheet))
+  end
+  csv(CSV\nstring)
+  docs[[Documents\nArray]]
+  file --> |readFile\n\n| wb
+  wb --> |wb.Sheets\nselect sheet| ws
+  ws --> |sheet_to_csv\n\n| csv
+  csv --> |CSVLoader\n\n| docs
+  linkStyle 0,1,2 color:blue,stroke:blue;
+```
+
+The SheetJS `readFile` method[^2] can read general workbooks. The method returns
+a workbook object that conforms to the SheetJS data model[^3].
+
+Workbook objects represent multi-sheet workbook files. They store individual
+worksheet objects and other metadata.
+
+Each worksheet in the workbook can be written to CSV text using the SheetJS
+`sheet_to_csv`[^4] method.
+
+For example, the following NodeJS script reads `pres.xlsx` and displays CSV rows
+from the first worksheet:
+
+```js title="Print CSV data from the first worksheet"
+/* Load SheetJS Libraries */
+import { readFile, set_fs, utils } from 'xlsx';
+
+/* Load 'fs' for readFile support */
+import * as fs from 'fs';
+set_fs(fs);
+
+/* Parse `pres.xlsx` */
+const wb = readFile("pres.xlsx");
+
+/* Print CSV rows from first worksheet */
+const first_ws = wb.Sheets[wb.SheetNames[0]];
+const csv = utils.sheet_to_csv(first_ws);
+console.log(csv);
+```
+
+### Single Worksheet
+
+For a single worksheet, a SheetJS pre-processing step can write the CSV rows to
+file and the `CSVLoader` can load the newly written file.
+
+<details open>
+  <summary><b>Code example</b> (click to hide)</summary>
+
+```js title="Pulling data from the first worksheet of a workbook"
+import { CSVLoader } from "@langchain/community/document_loaders/fs/csv";
+import { readFile, set_fs, utils } from 'xlsx';
+
+/* Load 'fs' for readFile support */
+import * as fs from 'fs';
+set_fs(fs);
+
+/* Parse `pres.xlsx`` */
+const wb = readFile("pres.xlsx");
+
+/* Generate CSV and write to `pres.xlsx.csv` */
+const first_ws = wb.Sheets[wb.SheetNames[0]];
+const csv = utils.sheet_to_csv(first_ws);
+fs.writeFileSync("pres.xlsx.csv", csv);
+
+/* Create documents with CSVLoader */
+const loader = new CSVLoader("pres.xlsx.csv");
+const docs = await loader.load();
+
+console.log(docs);
+// ...
+```
+
+</details>
+
+### Workbook
+
+A workbook is a collection of worksheets. Each worksheet can be exported to a
+separate CSV. If the CSVs are written to a subfolder, a `DirectoryLoader`[^5]
+can process the files in one step.
+
+<details open>
+  <summary><b>Code example</b> (click to hide)</summary>
+
+In this example, the script creates a subfolder named `csv`. Each worksheet in
+the workbook will be processed and the generated CSV will be stored to numbered
+files. The first worksheet will be stored to `csv/0.csv`.
+
+```js title="Pulling data from the each worksheet of a workbook"
+import { CSVLoader } from "@langchain/community/document_loaders/fs/csv";
+import { DirectoryLoader } from "langchain/document_loaders/fs/directory";
+import { readFile, set_fs, utils } from 'xlsx';
+
+/* Load 'fs' for readFile support */
+import * as fs from 'fs';
+set_fs(fs);
+
+/* Parse `pres.xlsx`` */
+const wb = readFile("pres.xlsx");
+
+/* Create a folder `csv` */
+try { fs.mkdirSync("csv"); } catch(e) {}
+
+/* Generate CSV data for each worksheet */
+wb.SheetNames.forEach((name, idx) => {
+  const ws = wb.Sheets[name];
+  const csv = utils.sheet_to_csv(ws);
+  fs.writeFileSync(`csv/${idx}.csv`, csv);
+});
+
+/* Create documents with DirectoryLoader */
+const loader = new DirectoryLoader("csv", {
+  ".csv": (path) => new CSVLoader(path)
+});
+const docs = await loader.load();
+
+console.log(docs);
+// ...
+```
+
+</details>
+
+## SheetJS Loader
+
+The `CSVLoader` that ships with LangChain does not add any Document metadata and
+does not generate any attributes. A custom loader can work around limitations in
+the CSV tooling and potentially include metadata that has no CSV equivalent.
+
+```mermaid
+flowchart LR
+  file[(Workbook\nXLSX/XLS)]
+  subgraph SheetJS Structures
+    wb(((SheetJS\nWorkbook)))
+    ws((SheetJS\nWorksheet))
+  end
+  aoo[(Array of\nObjects)]
+  docs[[Documents\nArray]]
+  file --> |readFile\n\n| wb
+  wb --> |wb.Sheets\nEach worksheet| ws
+  ws --> |sheet_to_json\n\n| aoo
+  aoo --> |new Document\nEach Row| docs
+  linkStyle 0,1,2 color:blue,stroke:blue;
+```
+
+The demo [`LoadOfSheet` loader](pathname:///loadofsheet/loadofsheet.mjs) will
+generate one Document per data row across all worksheets. It will also attempt
+to build metadata and attributes for use in self-querying retrievers.
+
+<details>
+  <summary><b>Sample SheetJS Loader</b> (click to show)</summary>
+
+This example loader pulls data from each worksheet. It assumes each worksheet
+includes one header row and a number of data rows.
+
+```js title="loadofsheet.mjs"
+import { Document } from "@langchain/core/documents";
+import { BufferLoader } from "langchain/document_loaders/fs/buffer";
+import { read, utils } from "xlsx";
+
+/**
+ * Document loader that uses SheetJS to load documents.
+ *
+ * Each worksheet is parsed into an array of row objects using the SheetJS
+ * `sheet_to_json` method and projected to a `Document`. Metadata includes
+ * original sheet name, row data, and row index
+ */
+export default class LoadOfSheet extends BufferLoader {
+  /** @type {import("langchain/chains/query_constructor").AttributeInfo[]}  */
+  attributes = [];
+
+  /**
+   * Document loader that uses SheetJS to load documents.
+   *
+   * @param {string|Blob} filePathOrBlob Source Data
+   */
+  constructor(filePathOrBlob) {
+    super(filePathOrBlob);
+    this.attributes = [];
+  }
+
+  /**
+   * Parse document
+   *
+   * NOTE: column labels in multiple sheets are not disambiguated!
+   *
+   * @param {Buffer} raw Raw data Buffer
+   * @param {Document["metadata"]} metadata Document metadata
+   * @returns {Promise<Document[]>} Array of Documents
+   */
+  async parse(raw, metadata) {
+    /** @type {Document[]} */
+    const result = [];
+
+    this.attributes = [
+      { name: "worksheet", description: "Sheet or Worksheet Name", type: "string" },
+      { name: "rowNum", description: "Row index", type: "number" }
+    ];
+
+    const wb = read(raw, {type: "buffer", WTF:1});
+    for(let name of wb.SheetNames) {
+      const fields = {};
+      const ws = wb.Sheets[name];
+      if(!ws) return;
+
+      const aoo = utils.sheet_to_json(ws);
+      aoo.forEach((row, idx) => {
+        result.push({
+          pageContent: "Row " + (idx + 1) + " has the following content: \n" + Object.entries(row).map(kv => `- ${kv[0]}: ${kv[1]}`).join("\n") + "\n",
+          metadata: {
+            worksheet: name,
+            rowNum: row["__rowNum__"],
+            ...metadata,
+            ...row
+          }
+        });
+        Object.entries(row).forEach(([k,v]) => { if(v != null) (fields[k] || (fields[k] = {}))[v instanceof Date ? "date" : typeof v] = true } );
+      });
+      Object.entries(fields).forEach(([k,v]) => this.attributes.push({
+        name: k, description: k, type: Object.keys(v).join(" or ")
+      }));
+    }
+
+    return result;
+  }
+};
+```
+
+</details>
+
+### From Text to Binary
+
+Many libraries and platforms offer generic "text" loaders that process files
+assuming the UTF8 encoding. This corrupts many spreadsheet formats including
+XLSX, XLSB, XLSM and XLS.
+
+:::note pass
+
+This issue affects many JavaScript tools. Various demos cover workarounds:
+
+- [ViteJS plugins](/docs/demos/static/vitejs#plugins) receive the relative path
+to the workbook file and can read the file directly.
+
+- [Webpack Plugins](/docs/demos/static/webpack#sheetjs-loader) have a special
+option to instruct the library to pass raw binary data rather than text.
+
+:::
+
+The `CSVLoader` extends a special `TextLoader` that forces UTF8 text parsing.
+
+There is a separate `BufferLoader` class, used by the PDF loader, that passes
+the raw data using NodeJS `Buffer` objects.
+
+<table>
+  <thead><tr><th>Binary</th><th>Text</th></tr></thead>
+  <tbody><tr><td>
+
+```ts title="pdf.ts (structure)"
+export class PDFLoader extends BufferLoader {
+  // ...
+  public async parse(
+    raw: Buffer,
+    metadata: Document["metadata"]
+  ): Promise<Document[]> {
+    // ...
+  }
+  // ...
+}
+```
+
+  </td><td>
+
+```ts title="csv.ts (structure)"
+export class CSVLoader extends TextLoader {
+  // ...
+  protected async parse(
+    raw: string
+
+  ): Promise<string[]> {
+    // ...
+  }
+  // ...
+}
+```
+
+  </td></tr></tbody>
+</table>
+
+### NodeJS Buffers
+
+The SheetJS `read` method supports NodeJS Buffer objects directly[^6]:
+
+```js title="Parsing a workbook in a BufferLoader"
+import { BufferLoader } from "langchain/document_loaders/fs/buffer";
+import { read, utils } from "xlsx";
+
+export default class LoadOfSheet extends BufferLoader {
+  // ...
+  async parse(raw, metadata) {
+    // highlight-next-line
+    const wb = read(raw, {type: "buffer"});
+    // At this point, `wb` is a SheetJS workbook object
+    // ...
+  }
+}
+```
+
+The `read` method returns a SheetJS workbook object[^7].
+
+### Generating Content
+
+The SheetJS `sheet_to_json` method[^8] returns an array of data objects whose
+keys are drawn from the first row of the worksheet.
+
+<table>
+  <thead><tr><th>Spreadsheet</th><th>Array of Objects</th></tr></thead>
+  <tbody><tr><td>
+
+![`pres.xlsx` data](pathname:///pres.png)
+
+</td><td>
+
+```js
+[
+  { Name: "Bill Clinton", Index: 42 },
+  { Name: "GeorgeW Bush", Index: 43 },
+  { Name: "Barack Obama", Index: 44 },
+  { Name: "Donald Trump", Index: 45 },
+  { Name: "Joseph Biden", Index: 46 }
+]
+```
+
+</td></tr></tbody></table>
+
+The original `CSVLoader` wrote one row for each key-value pair. This text can be
+generated by looping over the keys and values of the data row object. The
+`Object.entries` helper function simplifies the conversion:
+
+```js
+function make_csvloader_doc_from_row_object(row) {
+  return Object.entries(row).map(([k,v]) => `${k}: ${v}`).join("\n");
+}
+```
+
+### Generating Documents
+
+The loader must generate row objects for each worksheet in the workbook.
+
+In the SheetJS data model, the workbook object has two relevant fields:
+
+- `SheetNames` is an array of sheet names
+- `Sheets` is an object whose keys are sheet names and values are sheet objects.
+
+A `for..of` loop can iterate across the worksheets:
+
+```js title="Looping over a workbook (skeleton)"
+    const wb = read(raw, {type: "buffer", WTF:1});
+    for(let name of wb.SheetNames) {
+      const ws = wb.Sheets[name];
+      const aoa = utils.sheet_to_json(ws);
+      // at this point, `aoa` is an array of objects
+    }
+```
+
+This simplified `parse` function uses the snippet from the previous section:
+
+```js title="BufferLoader parse function (skeleton)"
+  async parse(raw, metadata) {
+    /* array to hold generated documents */
+    const result = [];
+
+    /* read workbook */
+    const wb = read(raw, {type: "buffer", WTF:1});
+
+    /* loop over worksheets */
+    for(let name of wb.SheetNames) {
+      const ws = wb.Sheets[name];
+      const aoa = utils.sheet_to_json(ws);
+
+      /* loop over data rows */
+      aoa.forEach((row, idx) => {
+        /* generate a new document and add to the result array */
+        result.push({
+          pageContent: Object.entries(row).map(([k,v]) => `${k}: ${v}`).join("\n")
+        });
+      });
+    }
+
+    return result;
+  }
+```
+
+### Metadata and Attributes
+
+It is strongly recommended to generate additional metadata and attributes for
+self-query retrieval applications.
+
+<details>
+  <summary><b>Implementation Details</b> (click to show)</summary>
+
+**Metadata**
+
+Metadata is attached to each document object. The following example appends the
+raw row data to the document metadata:
+
+```js title="Document with metadata (snippet)"
+        /* generate a new document and add to the result array */
+        result.push({
+          pageContent: Object.entries(row).map(([k,v]) => `${k}: ${v}`).join("\n"),
+          metadata: {
+            worksheet: name, // name of the worksheet
+            rowNum: idx, // data row index
+            ...row // raw row data
+          }
+        });
+```
+
+**Attributes**
+
+Each attribute object specifies three properties:
+
+- `name` corresponds to the field in the document metadata
+- `description` is a description of the field
+- `type` is a description of the data type.
+
+While looping through data rows, a simple type check can keep track of the data
+type for each column:
+
+```js title="Tracking column types (sketch)"
+    for(let name of wb.SheetNames) {
+      /* track column types */
+      const fields = {};
+      // ...
+
+      aoo.forEach((row, idx) => {
+        result.push({/* ... */});
+        /* Check each property */
+        Object.entries(row).forEach(([k,v]) => {
+          /* Update fields entry to reflect the new data point */
+          if(v != null) (fields[k] || (fields[k] = {}))[v instanceof Date ? "date" : typeof v] = true
+        });
+      });
+      // ...
+    }
+```
+
+Attributes can be generated after writing the worksheet data. Storing attributes
+in a loader property will make it accessible to scripts that use the loader.
+
+```js title="Adding Attributes to a Loader (sketch)"
+export default class LoadOfSheet extends BufferLoader {
+  // highlight-next-line
+  attributes = [];
+  // ...
+
+  async parse(raw, metadata) {
+    // Add the worksheet name and row index attributes
+    // highlight-start
+    this.attributes = [
+      { name: "worksheet", description: "Sheet or Worksheet Name", type: "string" },
+      { name: "rowNum", description: "Row index", type: "number" }
+    ];
+    // highlight-end
+    const wb = read(raw, {type: "buffer", WTF:1});
+    for(let name of wb.SheetNames) {
+      // highlight-next-line
+      const fields = {};
+      // ...
+      const aoo = utils.sheet_to_json(ws);
+      aoo.forEach((row, idx) => {
+        result.push({/* ... */});
+        /* Check each property */
+        Object.entries(row).forEach(([k,v]) => {
+          /* Update fields entry to reflect the new data point */
+          if(v != null) (fields[k] || (fields[k] = {}))[v instanceof Date ? "date" : typeof v] = true
+        });
+      });
+      /* Add one attribute per metadata field */
+      // highlight-start
+      Object.entries(fields).forEach(([k,v]) => this.attributes.push({
+        name: k, description: k,
+        /* { number: true, string: true } -> "number or string" */
+        type: Object.keys(v).join(" or ")
+      }));
+      // highlight-end
+    }
+    // ...
+  }
+```
+
+</details>
+
+## SheetJS Loader Demo
+
+The demo performs the query "Which rows have over 40 miles per gallon?" against
+a [sample cars dataset](pathname:///cd.xls) and displays the results.
+
+:::caution pass
+
+This demo was tested using the ChatQA-1.5 model[^9] in Ollama[^10].
+
+The tested model requires 9.2GB VRAM. It is strongly recommended to run the demo
+on a newer Apple Silicon Mac or a PC with an Nvidia GPU with at least 12GB VRAM.
+
+:::
+
+0) Create a new project:
+
+```bash
+mkdir sheetjs-loader
+cd sheetjs-loader
+npm init -y
+```
+
+1) Download the demo scripts:
+
+- [`loadofsheet.mjs`](pathname:///loadofsheet/loadofsheet.mjs)
+- [`query.mjs`](pathname:///loadofsheet/query.mjs)
+
+```bash
+curl -LO https://docs.sheetjs.com/loadofsheet/query.mjs
+curl -LO https://docs.sheetjs.com/loadofsheet/loadofsheet.mjs
+```
+
+2) Install the SheetJS NodeJS module:
+
+<CodeBlock language="bash">{`\
+npm i --save https://cdn.sheetjs.com/xlsx-${current}/xlsx-${current}.tgz`}
+</CodeBlock>
+
+3) Install LangChain and HNSWLib dependencies:
+
+```bash
+npm i --save @langchain/community@0.2.0 @langchain/core@0.2.6 langchain@0.2.5 hnswlib-node@3.0.0 peggy@3.0.2
+```
+
+4) Download the [cars dataset](pathname:///cd.xls):
+
+```bash
+curl -LO https://docs.sheetjs.com/cd.xls
+```
+
+5) Install the `llama3-chatqa:8b-v1.5-q8_0` model using Ollama:
+
+```bash
+ollama pull llama3-chatqa:8b-v1.5-q8_0
+```
+
+:::note pass
+
+If the command cannot be found, install Ollama[^10] and run the command in a new
+terminal window.
+
+:::
+
+6) Run the demo script
+
+```bash
+node query.mjs
+```
+
+The demo performs the query "Which rows have over 40 miles per gallon?". It will
+print the following nine results:
+
+```js title="Expected output"
+{ Name: 'volkswagen rabbit custom diesel', MPG: 43.1 }
+{ Name: 'vw rabbit c (diesel)', MPG: 44.3 }
+{ Name: 'renault lecar deluxe', MPG: 40.9 }
+{ Name: 'honda civic 1500 gl', MPG: 44.6 }
+{ Name: 'datsun 210', MPG: 40.8 }
+{ Name: 'vw pickup', MPG: 44 }
+{ Name: 'mazda glc', MPG: 46.6 }
+{ Name: 'vw dasher (diesel)', MPG: 43.4 }
+{ Name: 'vw rabbit', MPG: 41.5 }
+```
+
+To find the expected results:
+
+- Open the `cd.xls` spreadsheet in Excel
+- Select Home > Sort &amp; Filter > Filter in the Ribbon
+- Select the filter option for column B (`Miles_per_Gallon`)
+- In the popup, select "Greater Than" in the Filter dropdown and type 40
+
+The filtered results should match the following screenshot:
+
+![Expected Results](pathname:///loadofsheet/expected.png)
+
+[^1]: See ["How to load CSV data"](https://js.langchain.com/v0.2/docs/how_to/document_loader_csv) in the LangChain documentation
+[^2]: See [`readFile` in "Reading Files"](/docs/api/parse-options)
+[^3]: See ["SheetJS Data Model"](/docs/csf/)
+[^4]: See [`sheet_to_csv` in "CSV and Text"](/docs/api/utilities/csv#delimiter-separated-output)
+[^5]: See ["Folders with multiple files"](https://js.langchain.com/v0.2/docs/integrations/document_loaders/file_loaders/directory/) in the LangChain documentation
+[^6]: See ["Supported Output Formats" type in "Writing Files"](/docs/api/write-options#supported-output-formats)
+[^7]: See ["Workbook Object"](/docs/csf/book)
+[^8]: See [`sheet_to_json` in "Utilities"](/docs/api/utilities/array#array-output)
+[^9]: See [the official ChatQA website](https://chatqa-project.github.io/) for the ChatQA paper and other model details.
+[^10]: See [the official Ollama website](https://ollama.com/download) for installation instructions.

docz/docs/02-getting-started/02-examples/index.md

@@ -23,3 +23,8 @@ The ["Import Tutorial"](/docs/getting-started/examples/import) examines the data
 import process. A legacy file is downloaded and parsed. The underlying data is
 ultimately displayed to the user in a HTML table.
 
+## Loading Sheets
+
+["Loading Sheets"](/docs/getting-started/examples/loader) explores deep SheetJS
+integrations. Based on the existing CSV and binary loaders, a spreadsheet loader
+for LangChain is developed and tested.

docz/docs/03-demos/03-net/04-email/11-pst.md

@@ -258,23 +258,23 @@ saving file 0 |RedRockA.xls| to file0.xls
 ```
 
 Lines starting with `####` show the attachment file name and the worksheet name.
-The following line explains that there is a worksheet named `"Oct 26, 2001"` in
+The following line explains that there is a worksheet named `"Oneok at 2500"` in
 the file `RedRockA.xls`:
 
 ```
-#### RedRockA.xls ! Oct 26, 2001
+#### RedRockA.xls ! Oneok at 2500
 ```
 
 Every other line is a CSV row from the named worksheet. For example, the first
-four lines of worksheet `"Oct 26, 2001"` in `RedRockA.xls` are shown below:
+four lines of worksheet `"Oneok at 2500"` in `RedRockA.xls` are shown below:
 
 ```text
-#### RedRockA.xls ! Oct 26, 2001
+#### RedRockA.xls ! Oneok at 2500
 // highlight-start
-RED ROCK EXPANSION PROJECT,,,,,,,,,,,,,,,,,,
-,,,,,,,,,,,,,,,,,,
-,,,, , , ,,,,,,,,,,,,
-SHIPPER,CONTRACT #,Term,MMBtu/d,RECEIPT POINT,DELIVERY POINT,MMBtu/d,,,,,,,,,,,,
+RED ROCK EXPANSION PROJECT,,,,,,,,
+,,,,,,,,
+,,REQUESTED,REQUESTED,,,,,
+,,RECEIPT,DELIVERY,,,Allocation,,
 // highlight-end
 ```
 

docz/docs/03-demos/17-mobile/03-quasar.md

@@ -491,6 +491,29 @@ quasar dev -m android
 
 If prompted to select an external IP, press <kbd>Enter</kbd>.
 
+:::caution pass
+
+If the app is blank or not refreshing, delete the app and close the simulator,
+then restart the development process.
+
+:::
+
+:::warning pass
+
+On Windows, the command failed with a Gradle error
+
+```
+Could not find an installed version of Gradle either in Android Studio,
+or on your system to install the gradle wrapper. Please include gradle
+in your path, or install Android Studio
+```
+
+[Gradle](https://gradle.org/) (the complete version) must be extracted and the
+`bin` folder must be added to the user PATH variable. After adding to PATH,
+launch a new PowerShell or CMD window and run the command.
+
+:::
+
 To test that reading works:
 
 - Click and drag `pres.numbers` from a Finder window into the simulator.

docz/docs/03-demos/19-desktop/04-tauri.md

@@ -514,14 +514,13 @@ export default defineConfig(async () => ({
 // highlight-end
 ```
 
-- Add the highlighted line to `tsconfig.json`:
+- Edit `tsconfig.json`. In `compilerOptions` add the option `"jsx": "preserve"`:
 
 ```js title="tsconfig.json (add highlighted line)"
 {
   "compilerOptions": {
 // highlight-next-line
     "jsx": "preserve",
-    "target": "ES2020",
 ```
 
 - Replace `index.html` with the following codeblock:

docz/docs/03-demos/37-bigdata/01-stream.md

@@ -61,7 +61,7 @@ NodeJS push streams were introduced in 2012. The text streaming methods `to_csv`
 and `to_html` are supported in NodeJS v0.10 and later while the object streaming
 method `to_json` is supported in NodeJS v0.12 and later.
 
-The first streaming write function, `to_csv`, was introduced in early 2017. It
+The first SheetJS streaming write function, `to_csv`, was introduced in 2017. It
 used and still uses the same NodeJS streaming API.
 
 Years later, browser vendors are settling on a different stream API.

docz/docs/07-csf/07-features/01-dates.md

@@ -585,8 +585,8 @@ string rather than a proper Date object.
 ```mermaid
 flowchart LR
   ws(("new Date#40;#41;"\nObject))
-  jstr[["'#quot;2001-01-01T00:00:00.000Z#quot;'"\nEncoded String]]
-  js[["#quot;2001-01-01T00:00:00.000Z#quot;"\nJavaScript String]]
+  jstr[["'#quot;2048-10-06T00:00:00.000Z#quot;'"\nEncoded String]]
+  js[["#quot;2048-10-06T00:00:00.000Z#quot;"\nJavaScript String]]
   ws --> |JSON\nstringify| jstr
   jstr --> |JSON.parse\n\n| js
   js --> |\nJSON.stringify| jstr
@@ -601,8 +601,8 @@ while the "Date Obj" data will be converted to a spreadsheet date.
 ```jsx live
 function SheetJSONDates() { return ( <button onClick={() => {
   const aoa = [
-    ["ISO Text", "2001-01-01T00:00:00.000Z"],          // B1 will be text
-    ["Date Obj", new Date("2001-01-01T00:00:00.000Z")] // B2 will be a date
+    ["ISO Text", "2048-10-06T00:00:00.000Z"],          // B1 will be text
+    ["Date Obj", new Date("2048-10-06T00:00:00.000Z")] // B2 will be a date
   ];
   const ws = XLSX.utils.aoa_to_sheet(aoa);
   const wb = XLSX.utils.book_new(ws, "Data");

docz/static/cd.csv

@@ -0,0 +1,407 @@
+Name,Miles_per_Gallon,Cylinders,Displacement,Horsepower,Weight_in_lbs,Acceleration,Year,Origin
+chevrolet chevelle malibu,18,8,307,130,3504,12,1970-01-01,USA
+buick skylark 320,15,8,350,165,3693,11.5,1970-01-01,USA
+plymouth satellite,18,8,318,150,3436,11,1970-01-01,USA
+amc rebel sst,16,8,304,150,3433,12,1970-01-01,USA
+ford torino,17,8,302,140,3449,10.5,1970-01-01,USA
+ford galaxie 500,15,8,429,198,4341,10,1970-01-01,USA
+chevrolet impala,14,8,454,220,4354,9,1970-01-01,USA
+plymouth fury iii,14,8,440,215,4312,8.5,1970-01-01,USA
+pontiac catalina,14,8,455,225,4425,10,1970-01-01,USA
+amc ambassador dpl,15,8,390,190,3850,8.5,1970-01-01,USA
+citroen ds-21 pallas,,4,133,115,3090,17.5,1970-01-01,Europe
+chevrolet chevelle concours (sw),,8,350,165,4142,11.5,1970-01-01,USA
+ford torino (sw),,8,351,153,4034,11,1970-01-01,USA
+plymouth satellite (sw),,8,383,175,4166,10.5,1970-01-01,USA
+amc rebel sst (sw),,8,360,175,3850,11,1970-01-01,USA
+dodge challenger se,15,8,383,170,3563,10,1970-01-01,USA
+plymouth 'cuda 340,14,8,340,160,3609,8,1970-01-01,USA
+ford mustang boss 302,,8,302,140,3353,8,1970-01-01,USA
+chevrolet monte carlo,15,8,400,150,3761,9.5,1970-01-01,USA
+buick estate wagon (sw),14,8,455,225,3086,10,1970-01-01,USA
+toyota corona mark ii,24,4,113,95,2372,15,1970-01-01,Japan
+plymouth duster,22,6,198,95,2833,15.5,1970-01-01,USA
+amc hornet,18,6,199,97,2774,15.5,1970-01-01,USA
+ford maverick,21,6,200,85,2587,16,1970-01-01,USA
+datsun pl510,27,4,97,88,2130,14.5,1970-01-01,Japan
+volkswagen 1131 deluxe sedan,26,4,97,46,1835,20.5,1970-01-01,Europe
+peugeot 504,25,4,110,87,2672,17.5,1970-01-01,Europe
+audi 100 ls,24,4,107,90,2430,14.5,1970-01-01,Europe
+saab 99e,25,4,104,95,2375,17.5,1970-01-01,Europe
+bmw 2002,26,4,121,113,2234,12.5,1970-01-01,Europe
+amc gremlin,21,6,199,90,2648,15,1970-01-01,USA
+ford f250,10,8,360,215,4615,14,1970-01-01,USA
+chevy c20,10,8,307,200,4376,15,1970-01-01,USA
+dodge d200,11,8,318,210,4382,13.5,1970-01-01,USA
+hi 1200d,9,8,304,193,4732,18.5,1970-01-01,USA
+datsun pl510,27,4,97,88,2130,14.5,1971-01-01,Japan
+chevrolet vega 2300,28,4,140,90,2264,15.5,1971-01-01,USA
+toyota corona,25,4,113,95,2228,14,1971-01-01,Japan
+ford pinto,25,4,98,,2046,19,1971-01-01,USA
+volkswagen super beetle 117,,4,97,48,1978,20,1971-01-01,Europe
+amc gremlin,19,6,232,100,2634,13,1971-01-01,USA
+plymouth satellite custom,16,6,225,105,3439,15.5,1971-01-01,USA
+chevrolet chevelle malibu,17,6,250,100,3329,15.5,1971-01-01,USA
+ford torino 500,19,6,250,88,3302,15.5,1971-01-01,USA
+amc matador,18,6,232,100,3288,15.5,1971-01-01,USA
+chevrolet impala,14,8,350,165,4209,12,1971-01-01,USA
+pontiac catalina brougham,14,8,400,175,4464,11.5,1971-01-01,USA
+ford galaxie 500,14,8,351,153,4154,13.5,1971-01-01,USA
+plymouth fury iii,14,8,318,150,4096,13,1971-01-01,USA
+dodge monaco (sw),12,8,383,180,4955,11.5,1971-01-01,USA
+ford country squire (sw),13,8,400,170,4746,12,1971-01-01,USA
+pontiac safari (sw),13,8,400,175,5140,12,1971-01-01,USA
+amc hornet sportabout (sw),18,6,258,110,2962,13.5,1971-01-01,USA
+chevrolet vega (sw),22,4,140,72,2408,19,1971-01-01,USA
+pontiac firebird,19,6,250,100,3282,15,1971-01-01,USA
+ford mustang,18,6,250,88,3139,14.5,1971-01-01,USA
+mercury capri 2000,23,4,122,86,2220,14,1971-01-01,USA
+opel 1900,28,4,116,90,2123,14,1971-01-01,Europe
+peugeot 304,30,4,79,70,2074,19.5,1971-01-01,Europe
+fiat 124b,30,4,88,76,2065,14.5,1971-01-01,Europe
+toyota corolla 1200,31,4,71,65,1773,19,1971-01-01,Japan
+datsun 1200,35,4,72,69,1613,18,1971-01-01,Japan
+volkswagen model 111,27,4,97,60,1834,19,1971-01-01,Europe
+plymouth cricket,26,4,91,70,1955,20.5,1971-01-01,USA
+toyota corona hardtop,24,4,113,95,2278,15.5,1972-01-01,Japan
+dodge colt hardtop,25,4,97.5,80,2126,17,1972-01-01,USA
+volkswagen type 3,23,4,97,54,2254,23.5,1972-01-01,Europe
+chevrolet vega,20,4,140,90,2408,19.5,1972-01-01,USA
+ford pinto runabout,21,4,122,86,2226,16.5,1972-01-01,USA
+chevrolet impala,13,8,350,165,4274,12,1972-01-01,USA
+pontiac catalina,14,8,400,175,4385,12,1972-01-01,USA
+plymouth fury iii,15,8,318,150,4135,13.5,1972-01-01,USA
+ford galaxie 500,14,8,351,153,4129,13,1972-01-01,USA
+amc ambassador sst,17,8,304,150,3672,11.5,1972-01-01,USA
+mercury marquis,11,8,429,208,4633,11,1972-01-01,USA
+buick lesabre custom,13,8,350,155,4502,13.5,1972-01-01,USA
+oldsmobile delta 88 royale,12,8,350,160,4456,13.5,1972-01-01,USA
+chrysler newport royal,13,8,400,190,4422,12.5,1972-01-01,USA
+mazda rx2 coupe,19,3,70,97,2330,13.5,1972-01-01,Japan
+amc matador (sw),15,8,304,150,3892,12.5,1972-01-01,USA
+chevrolet chevelle concours (sw),13,8,307,130,4098,14,1972-01-01,USA
+ford gran torino (sw),13,8,302,140,4294,16,1972-01-01,USA
+plymouth satellite custom (sw),14,8,318,150,4077,14,1972-01-01,USA
+volvo 145e (sw),18,4,121,112,2933,14.5,1972-01-01,Europe
+volkswagen 411 (sw),22,4,121,76,2511,18,1972-01-01,Europe
+peugeot 504 (sw),21,4,120,87,2979,19.5,1972-01-01,Europe
+renault 12 (sw),26,4,96,69,2189,18,1972-01-01,Europe
+ford pinto (sw),22,4,122,86,2395,16,1972-01-01,USA
+datsun 510 (sw),28,4,97,92,2288,17,1972-01-01,Japan
+toyouta corona mark ii (sw),23,4,120,97,2506,14.5,1972-01-01,Japan
+dodge colt (sw),28,4,98,80,2164,15,1972-01-01,USA
+toyota corolla 1600 (sw),27,4,97,88,2100,16.5,1972-01-01,Japan
+buick century 350,13,8,350,175,4100,13,1973-01-01,USA
+amc matador,14,8,304,150,3672,11.5,1973-01-01,USA
+chevrolet malibu,13,8,350,145,3988,13,1973-01-01,USA
+ford gran torino,14,8,302,137,4042,14.5,1973-01-01,USA
+dodge coronet custom,15,8,318,150,3777,12.5,1973-01-01,USA
+mercury marquis brougham,12,8,429,198,4952,11.5,1973-01-01,USA
+chevrolet caprice classic,13,8,400,150,4464,12,1973-01-01,USA
+ford ltd,13,8,351,158,4363,13,1973-01-01,USA
+plymouth fury gran sedan,14,8,318,150,4237,14.5,1973-01-01,USA
+chrysler new yorker brougham,13,8,440,215,4735,11,1973-01-01,USA
+buick electra 225 custom,12,8,455,225,4951,11,1973-01-01,USA
+amc ambassador brougham,13,8,360,175,3821,11,1973-01-01,USA
+plymouth valiant,18,6,225,105,3121,16.5,1973-01-01,USA
+chevrolet nova custom,16,6,250,100,3278,18,1973-01-01,USA
+amc hornet,18,6,232,100,2945,16,1973-01-01,USA
+ford maverick,18,6,250,88,3021,16.5,1973-01-01,USA
+plymouth duster,23,6,198,95,2904,16,1973-01-01,USA
+volkswagen super beetle,26,4,97,46,1950,21,1973-01-01,Europe
+chevrolet impala,11,8,400,150,4997,14,1973-01-01,USA
+ford country,12,8,400,167,4906,12.5,1973-01-01,USA
+plymouth custom suburb,13,8,360,170,4654,13,1973-01-01,USA
+oldsmobile vista cruiser,12,8,350,180,4499,12.5,1973-01-01,USA
+amc gremlin,18,6,232,100,2789,15,1973-01-01,USA
+toyota carina,20,4,97,88,2279,19,1973-01-01,Japan
+chevrolet vega,21,4,140,72,2401,19.5,1973-01-01,USA
+datsun 610,22,4,108,94,2379,16.5,1973-01-01,Japan
+maxda rx3,18,3,70,90,2124,13.5,1973-01-01,Japan
+ford pinto,19,4,122,85,2310,18.5,1973-01-01,USA
+mercury capri v6,21,6,155,107,2472,14,1973-01-01,USA
+fiat 124 sport coupe,26,4,98,90,2265,15.5,1973-01-01,Europe
+chevrolet monte carlo s,15,8,350,145,4082,13,1973-01-01,USA
+pontiac grand prix,16,8,400,230,4278,9.5,1973-01-01,USA
+fiat 128,29,4,68,49,1867,19.5,1973-01-01,Europe
+opel manta,24,4,116,75,2158,15.5,1973-01-01,Europe
+audi 100ls,20,4,114,91,2582,14,1973-01-01,Europe
+volvo 144ea,19,4,121,112,2868,15.5,1973-01-01,Europe
+dodge dart custom,15,8,318,150,3399,11,1973-01-01,USA
+saab 99le,24,4,121,110,2660,14,1973-01-01,Europe
+toyota mark ii,20,6,156,122,2807,13.5,1973-01-01,Japan
+oldsmobile omega,11,8,350,180,3664,11,1973-01-01,USA
+plymouth duster,20,6,198,95,3102,16.5,1974-01-01,USA
+ford maverick,21,6,200,,2875,17,1974-01-01,USA
+amc hornet,19,6,232,100,2901,16,1974-01-01,USA
+chevrolet nova,15,6,250,100,3336,17,1974-01-01,USA
+datsun b210,31,4,79,67,1950,19,1974-01-01,Japan
+ford pinto,26,4,122,80,2451,16.5,1974-01-01,USA
+toyota corolla 1200,32,4,71,65,1836,21,1974-01-01,Japan
+chevrolet vega,25,4,140,75,2542,17,1974-01-01,USA
+chevrolet chevelle malibu classic,16,6,250,100,3781,17,1974-01-01,USA
+amc matador,16,6,258,110,3632,18,1974-01-01,USA
+plymouth satellite sebring,18,6,225,105,3613,16.5,1974-01-01,USA
+ford gran torino,16,8,302,140,4141,14,1974-01-01,USA
+buick century luxus (sw),13,8,350,150,4699,14.5,1974-01-01,USA
+dodge coronet custom (sw),14,8,318,150,4457,13.5,1974-01-01,USA
+ford gran torino (sw),14,8,302,140,4638,16,1974-01-01,USA
+amc matador (sw),14,8,304,150,4257,15.5,1974-01-01,USA
+audi fox,29,4,98,83,2219,16.5,1974-01-01,Europe
+volkswagen dasher,26,4,79,67,1963,15.5,1974-01-01,Europe
+opel manta,26,4,97,78,2300,14.5,1974-01-01,Europe
+toyota corona,31,4,76,52,1649,16.5,1974-01-01,Japan
+datsun 710,32,4,83,61,2003,19,1974-01-01,Japan
+dodge colt,28,4,90,75,2125,14.5,1974-01-01,USA
+fiat 128,24,4,90,75,2108,15.5,1974-01-01,Europe
+fiat 124 tc,26,4,116,75,2246,14,1974-01-01,Europe
+honda civic,24,4,120,97,2489,15,1974-01-01,Japan
+subaru,26,4,108,93,2391,15.5,1974-01-01,Japan
+fiat x1.9,31,4,79,67,2000,16,1974-01-01,Europe
+plymouth valiant custom,19,6,225,95,3264,16,1975-01-01,USA
+chevrolet nova,18,6,250,105,3459,16,1975-01-01,USA
+mercury monarch,15,6,250,72,3432,21,1975-01-01,USA
+ford maverick,15,6,250,72,3158,19.5,1975-01-01,USA
+pontiac catalina,16,8,400,170,4668,11.5,1975-01-01,USA
+chevrolet bel air,15,8,350,145,4440,14,1975-01-01,USA
+plymouth grand fury,16,8,318,150,4498,14.5,1975-01-01,USA
+ford ltd,14,8,351,148,4657,13.5,1975-01-01,USA
+buick century,17,6,231,110,3907,21,1975-01-01,USA
+chevroelt chevelle malibu,16,6,250,105,3897,18.5,1975-01-01,USA
+amc matador,15,6,258,110,3730,19,1975-01-01,USA
+plymouth fury,18,6,225,95,3785,19,1975-01-01,USA
+buick skyhawk,21,6,231,110,3039,15,1975-01-01,USA
+chevrolet monza 2+2,20,8,262,110,3221,13.5,1975-01-01,USA
+ford mustang ii,13,8,302,129,3169,12,1975-01-01,USA
+toyota corolla,29,4,97,75,2171,16,1975-01-01,Japan
+ford pinto,23,4,140,83,2639,17,1975-01-01,USA
+amc gremlin,20,6,232,100,2914,16,1975-01-01,USA
+pontiac astro,23,4,140,78,2592,18.5,1975-01-01,USA
+toyota corona,24,4,134,96,2702,13.5,1975-01-01,Japan
+volkswagen dasher,25,4,90,71,2223,16.5,1975-01-01,Europe
+datsun 710,24,4,119,97,2545,17,1975-01-01,Japan
+ford pinto,18,6,171,97,2984,14.5,1975-01-01,USA
+volkswagen rabbit,29,4,90,70,1937,14,1975-01-01,Europe
+amc pacer,19,6,232,90,3211,17,1975-01-01,USA
+audi 100ls,23,4,115,95,2694,15,1975-01-01,Europe
+peugeot 504,23,4,120,88,2957,17,1975-01-01,Europe
+volvo 244dl,22,4,121,98,2945,14.5,1975-01-01,Europe
+saab 99le,25,4,121,115,2671,13.5,1975-01-01,Europe
+honda civic cvcc,33,4,91,53,1795,17.5,1975-01-01,Japan
+fiat 131,28,4,107,86,2464,15.5,1976-01-01,Europe
+opel 1900,25,4,116,81,2220,16.9,1976-01-01,Europe
+capri ii,25,4,140,92,2572,14.9,1976-01-01,USA
+dodge colt,26,4,98,79,2255,17.7,1976-01-01,USA
+renault 12tl,27,4,101,83,2202,15.3,1976-01-01,Europe
+chevrolet chevelle malibu classic,17.5,8,305,140,4215,13,1976-01-01,USA
+dodge coronet brougham,16,8,318,150,4190,13,1976-01-01,USA
+amc matador,15.5,8,304,120,3962,13.9,1976-01-01,USA
+ford gran torino,14.5,8,351,152,4215,12.8,1976-01-01,USA
+plymouth valiant,22,6,225,100,3233,15.4,1976-01-01,USA
+chevrolet nova,22,6,250,105,3353,14.5,1976-01-01,USA
+ford maverick,24,6,200,81,3012,17.6,1976-01-01,USA
+amc hornet,22.5,6,232,90,3085,17.6,1976-01-01,USA
+chevrolet chevette,29,4,85,52,2035,22.2,1976-01-01,USA
+chevrolet woody,24.5,4,98,60,2164,22.1,1976-01-01,USA
+vw rabbit,29,4,90,70,1937,14.2,1976-01-01,Europe
+honda civic,33,4,91,53,1795,17.4,1976-01-01,Japan
+dodge aspen se,20,6,225,100,3651,17.7,1976-01-01,USA
+ford granada ghia,18,6,250,78,3574,21,1976-01-01,USA
+pontiac ventura sj,18.5,6,250,110,3645,16.2,1976-01-01,USA
+amc pacer d/l,17.5,6,258,95,3193,17.8,1976-01-01,USA
+volkswagen rabbit,29.5,4,97,71,1825,12.2,1976-01-01,Europe
+datsun b-210,32,4,85,70,1990,17,1976-01-01,Japan
+toyota corolla,28,4,97,75,2155,16.4,1976-01-01,Japan
+ford pinto,26.5,4,140,72,2565,13.6,1976-01-01,USA
+volvo 245,20,4,130,102,3150,15.7,1976-01-01,Europe
+plymouth volare premier v8,13,8,318,150,3940,13.2,1976-01-01,USA
+peugeot 504,19,4,120,88,3270,21.9,1976-01-01,Europe
+toyota mark ii,19,6,156,108,2930,15.5,1976-01-01,Japan
+mercedes-benz 280s,16.5,6,168,120,3820,16.7,1976-01-01,Europe
+cadillac seville,16.5,8,350,180,4380,12.1,1976-01-01,USA
+chevy c10,13,8,350,145,4055,12,1976-01-01,USA
+ford f108,13,8,302,130,3870,15,1976-01-01,USA
+dodge d100,13,8,318,150,3755,14,1976-01-01,USA
+honda Accelerationord cvcc,31.5,4,98,68,2045,18.5,1977-01-01,Japan
+buick opel isuzu deluxe,30,4,111,80,2155,14.8,1977-01-01,USA
+renault 5 gtl,36,4,79,58,1825,18.6,1977-01-01,Europe
+plymouth arrow gs,25.5,4,122,96,2300,15.5,1977-01-01,USA
+datsun f-10 hatchback,33.5,4,85,70,1945,16.8,1977-01-01,Japan
+chevrolet caprice classic,17.5,8,305,145,3880,12.5,1977-01-01,USA
+oldsmobile cutlass supreme,17,8,260,110,4060,19,1977-01-01,USA
+dodge monaco brougham,15.5,8,318,145,4140,13.7,1977-01-01,USA
+mercury cougar brougham,15,8,302,130,4295,14.9,1977-01-01,USA
+chevrolet concours,17.5,6,250,110,3520,16.4,1977-01-01,USA
+buick skylark,20.5,6,231,105,3425,16.9,1977-01-01,USA
+plymouth volare custom,19,6,225,100,3630,17.7,1977-01-01,USA
+ford granada,18.5,6,250,98,3525,19,1977-01-01,USA
+pontiac grand prix lj,16,8,400,180,4220,11.1,1977-01-01,USA
+chevrolet monte carlo landau,15.5,8,350,170,4165,11.4,1977-01-01,USA
+chrysler cordoba,15.5,8,400,190,4325,12.2,1977-01-01,USA
+ford thunderbird,16,8,351,149,4335,14.5,1977-01-01,USA
+volkswagen rabbit custom,29,4,97,78,1940,14.5,1977-01-01,Europe
+pontiac sunbird coupe,24.5,4,151,88,2740,16,1977-01-01,USA
+toyota corolla liftback,26,4,97,75,2265,18.2,1977-01-01,Japan
+ford mustang ii 2+2,25.5,4,140,89,2755,15.8,1977-01-01,USA
+chevrolet chevette,30.5,4,98,63,2051,17,1977-01-01,USA
+dodge colt m/m,33.5,4,98,83,2075,15.9,1977-01-01,USA
+subaru dl,30,4,97,67,1985,16.4,1977-01-01,Japan
+volkswagen dasher,30.5,4,97,78,2190,14.1,1977-01-01,Europe
+datsun 810,22,6,146,97,2815,14.5,1977-01-01,Japan
+bmw 320i,21.5,4,121,110,2600,12.8,1977-01-01,Europe
+mazda rx-4,21.5,3,80,110,2720,13.5,1977-01-01,Japan
+volkswagen rabbit custom diesel,43.1,4,90,48,1985,21.5,1978-01-01,Europe
+ford fiesta,36.1,4,98,66,1800,14.4,1978-01-01,USA
+mazda glc deluxe,32.8,4,78,52,1985,19.4,1978-01-01,Japan
+datsun b210 gx,39.4,4,85,70,2070,18.6,1978-01-01,Japan
+honda civic cvcc,36.1,4,91,60,1800,16.4,1978-01-01,Japan
+oldsmobile cutlass salon brougham,19.9,8,260,110,3365,15.5,1978-01-01,USA
+dodge diplomat,19.4,8,318,140,3735,13.2,1978-01-01,USA
+mercury monarch ghia,20.2,8,302,139,3570,12.8,1978-01-01,USA
+pontiac phoenix lj,19.2,6,231,105,3535,19.2,1978-01-01,USA
+chevrolet malibu,20.5,6,200,95,3155,18.2,1978-01-01,USA
+ford fairmont (auto),20.2,6,200,85,2965,15.8,1978-01-01,USA
+ford fairmont (man),25.1,4,140,88,2720,15.4,1978-01-01,USA
+plymouth volare,20.5,6,225,100,3430,17.2,1978-01-01,USA
+amc concord,19.4,6,232,90,3210,17.2,1978-01-01,USA
+buick century special,20.6,6,231,105,3380,15.8,1978-01-01,USA
+mercury zephyr,20.8,6,200,85,3070,16.7,1978-01-01,USA
+dodge aspen,18.6,6,225,110,3620,18.7,1978-01-01,USA
+amc concord d/l,18.1,6,258,120,3410,15.1,1978-01-01,USA
+chevrolet monte carlo landau,19.2,8,305,145,3425,13.2,1978-01-01,USA
+buick regal sport coupe (turbo),17.7,6,231,165,3445,13.4,1978-01-01,USA
+ford futura,18.1,8,302,139,3205,11.2,1978-01-01,USA
+dodge magnum xe,17.5,8,318,140,4080,13.7,1978-01-01,USA
+chevrolet chevette,30,4,98,68,2155,16.5,1978-01-01,USA
+toyota corona,27.5,4,134,95,2560,14.2,1978-01-01,Japan
+datsun 510,27.2,4,119,97,2300,14.7,1978-01-01,Japan
+dodge omni,30.9,4,105,75,2230,14.5,1978-01-01,USA
+toyota celica gt liftback,21.1,4,134,95,2515,14.8,1978-01-01,Japan
+plymouth sapporo,23.2,4,156,105,2745,16.7,1978-01-01,USA
+oldsmobile starfire sx,23.8,4,151,85,2855,17.6,1978-01-01,USA
+datsun 200-sx,23.9,4,119,97,2405,14.9,1978-01-01,Japan
+audi 5000,20.3,5,131,103,2830,15.9,1978-01-01,Europe
+volvo 264gl,17,6,163,125,3140,13.6,1978-01-01,Europe
+saab 99gle,21.6,4,121,115,2795,15.7,1978-01-01,Europe
+peugeot 604sl,16.2,6,163,133,3410,15.8,1978-01-01,Europe
+volkswagen scirocco,31.5,4,89,71,1990,14.9,1978-01-01,Europe
+honda Accelerationord lx,29.5,4,98,68,2135,16.6,1978-01-01,Japan
+pontiac lemans v6,21.5,6,231,115,3245,15.4,1979-01-01,USA
+mercury zephyr 6,19.8,6,200,85,2990,18.2,1979-01-01,USA
+ford fairmont 4,22.3,4,140,88,2890,17.3,1979-01-01,USA
+amc concord dl 6,20.2,6,232,90,3265,18.2,1979-01-01,USA
+dodge aspen 6,20.6,6,225,110,3360,16.6,1979-01-01,USA
+chevrolet caprice classic,17,8,305,130,3840,15.4,1979-01-01,USA
+ford ltd landau,17.6,8,302,129,3725,13.4,1979-01-01,USA
+mercury grand marquis,16.5,8,351,138,3955,13.2,1979-01-01,USA
+dodge st. regis,18.2,8,318,135,3830,15.2,1979-01-01,USA
+buick estate wagon (sw),16.9,8,350,155,4360,14.9,1979-01-01,USA
+ford country squire (sw),15.5,8,351,142,4054,14.3,1979-01-01,USA
+chevrolet malibu classic (sw),19.2,8,267,125,3605,15,1979-01-01,USA
+chrysler lebaron town @ country (sw),18.5,8,360,150,3940,13,1979-01-01,USA
+vw rabbit custom,31.9,4,89,71,1925,14,1979-01-01,Europe
+maxda glc deluxe,34.1,4,86,65,1975,15.2,1979-01-01,Japan
+dodge colt hatchback custom,35.7,4,98,80,1915,14.4,1979-01-01,USA
+amc spirit dl,27.4,4,121,80,2670,15,1979-01-01,USA
+mercedes benz 300d,25.4,5,183,77,3530,20.1,1979-01-01,Europe
+cadillac eldorado,23,8,350,125,3900,17.4,1979-01-01,USA
+peugeot 504,27.2,4,141,71,3190,24.8,1979-01-01,Europe
+oldsmobile cutlass salon brougham,23.9,8,260,90,3420,22.2,1979-01-01,USA
+plymouth horizon,34.2,4,105,70,2200,13.2,1979-01-01,USA
+plymouth horizon tc3,34.5,4,105,70,2150,14.9,1979-01-01,USA
+datsun 210,31.8,4,85,65,2020,19.2,1979-01-01,Japan
+fiat strada custom,37.3,4,91,69,2130,14.7,1979-01-01,Europe
+buick skylark limited,28.4,4,151,90,2670,16,1979-01-01,USA
+chevrolet citation,28.8,6,173,115,2595,11.3,1979-01-01,USA
+oldsmobile omega brougham,26.8,6,173,115,2700,12.9,1979-01-01,USA
+pontiac phoenix,33.5,4,151,90,2556,13.2,1979-01-01,USA
+vw rabbit,41.5,4,98,76,2144,14.7,1980-01-01,Europe
+toyota corolla tercel,38.1,4,89,60,1968,18.8,1980-01-01,Japan
+chevrolet chevette,32.1,4,98,70,2120,15.5,1980-01-01,USA
+datsun 310,37.2,4,86,65,2019,16.4,1980-01-01,Japan
+chevrolet citation,28,4,151,90,2678,16.5,1980-01-01,USA
+ford fairmont,26.4,4,140,88,2870,18.1,1980-01-01,USA
+amc concord,24.3,4,151,90,3003,20.1,1980-01-01,USA
+dodge aspen,19.1,6,225,90,3381,18.7,1980-01-01,USA
+audi 4000,34.3,4,97,78,2188,15.8,1980-01-01,Europe
+toyota corona liftback,29.8,4,134,90,2711,15.5,1980-01-01,Japan
+mazda 626,31.3,4,120,75,2542,17.5,1980-01-01,Japan
+datsun 510 hatchback,37,4,119,92,2434,15,1980-01-01,Japan
+toyota corolla,32.2,4,108,75,2265,15.2,1980-01-01,Japan
+mazda glc,46.6,4,86,65,2110,17.9,1980-01-01,Japan
+dodge colt,27.9,4,156,105,2800,14.4,1980-01-01,USA
+datsun 210,40.8,4,85,65,2110,19.2,1980-01-01,Japan
+vw rabbit c (diesel),44.3,4,90,48,2085,21.7,1980-01-01,Europe
+vw dasher (diesel),43.4,4,90,48,2335,23.7,1980-01-01,Europe
+audi 5000s (diesel),36.4,5,121,67,2950,19.9,1980-01-01,Europe
+mercedes-benz 240d,30,4,146,67,3250,21.8,1980-01-01,Europe
+honda civic 1500 gl,44.6,4,91,67,1850,13.8,1980-01-01,Japan
+renault lecar deluxe,40.9,4,85,,1835,17.3,1980-01-01,Europe
+subaru dl,33.8,4,97,67,2145,18,1980-01-01,Japan
+vokswagen rabbit,29.8,4,89,62,1845,15.3,1980-01-01,Europe
+datsun 280-zx,32.7,6,168,132,2910,11.4,1980-01-01,Japan
+mazda rx-7 gs,23.7,3,70,100,2420,12.5,1980-01-01,Japan
+triumph tr7 coupe,35,4,122,88,2500,15.1,1980-01-01,Europe
+ford mustang cobra,23.6,4,140,,2905,14.3,1980-01-01,USA
+honda Accelerationord,32.4,4,107,72,2290,17,1980-01-01,Japan
+plymouth reliant,27.2,4,135,84,2490,15.7,1982-01-01,USA
+buick skylark,26.6,4,151,84,2635,16.4,1982-01-01,USA
+dodge aries wagon (sw),25.8,4,156,92,2620,14.4,1982-01-01,USA
+chevrolet citation,23.5,6,173,110,2725,12.6,1982-01-01,USA
+plymouth reliant,30,4,135,84,2385,12.9,1982-01-01,USA
+toyota starlet,39.1,4,79,58,1755,16.9,1982-01-01,Japan
+plymouth champ,39,4,86,64,1875,16.4,1982-01-01,USA
+honda civic 1300,35.1,4,81,60,1760,16.1,1982-01-01,Japan
+subaru,32.3,4,97,67,2065,17.8,1982-01-01,Japan
+datsun 210,37,4,85,65,1975,19.4,1982-01-01,Japan
+toyota tercel,37.7,4,89,62,2050,17.3,1982-01-01,Japan
+mazda glc 4,34.1,4,91,68,1985,16,1982-01-01,Japan
+plymouth horizon 4,34.7,4,105,63,2215,14.9,1982-01-01,USA
+ford escort 4w,34.4,4,98,65,2045,16.2,1982-01-01,USA
+ford escort 2h,29.9,4,98,65,2380,20.7,1982-01-01,USA
+volkswagen jetta,33,4,105,74,2190,14.2,1982-01-01,Europe
+renault 18i,34.5,4,100,,2320,15.8,1982-01-01,Europe
+honda prelude,33.7,4,107,75,2210,14.4,1982-01-01,Japan
+toyota corolla,32.4,4,108,75,2350,16.8,1982-01-01,Japan
+datsun 200sx,32.9,4,119,100,2615,14.8,1982-01-01,Japan
+mazda 626,31.6,4,120,74,2635,18.3,1982-01-01,Japan
+peugeot 505s turbo diesel,28.1,4,141,80,3230,20.4,1982-01-01,Europe
+saab 900s,,4,121,110,2800,15.4,1982-01-01,Europe
+volvo diesel,30.7,6,145,76,3160,19.6,1982-01-01,Europe
+toyota cressida,25.4,6,168,116,2900,12.6,1982-01-01,Japan
+datsun 810 maxima,24.2,6,146,120,2930,13.8,1982-01-01,Japan
+buick century,22.4,6,231,110,3415,15.8,1982-01-01,USA
+oldsmobile cutlass ls,26.6,8,350,105,3725,19,1982-01-01,USA
+ford granada gl,20.2,6,200,88,3060,17.1,1982-01-01,USA
+chrysler lebaron salon,17.6,6,225,85,3465,16.6,1982-01-01,USA
+chevrolet cavalier,28,4,112,88,2605,19.6,1982-01-01,USA
+chevrolet cavalier wagon,27,4,112,88,2640,18.6,1982-01-01,USA
+chevrolet cavalier 2-door,34,4,112,88,2395,18,1982-01-01,USA
+pontiac j2000 se hatchback,31,4,112,85,2575,16.2,1982-01-01,USA
+dodge aries se,29,4,135,84,2525,16,1982-01-01,USA
+pontiac phoenix,27,4,151,90,2735,18,1982-01-01,USA
+ford fairmont futura,24,4,140,92,2865,16.4,1982-01-01,USA
+amc concord dl,23,4,151,,3035,20.5,1982-01-01,USA
+volkswagen rabbit l,36,4,105,74,1980,15.3,1982-01-01,Europe
+mazda glc custom l,37,4,91,68,2025,18.2,1982-01-01,Japan
+mazda glc custom,31,4,91,68,1970,17.6,1982-01-01,Japan
+plymouth horizon miser,38,4,105,63,2125,14.7,1982-01-01,USA
+mercury lynx l,36,4,98,70,2125,17.3,1982-01-01,USA
+nissan stanza xe,36,4,120,88,2160,14.5,1982-01-01,Japan
+honda Accelerationord,36,4,107,75,2205,14.5,1982-01-01,Japan
+toyota corolla,34,4,108,70,2245,16.9,1982-01-01,Japan
+honda civic,38,4,91,67,1965,15,1982-01-01,Japan
+honda civic (auto),32,4,91,67,1965,15.7,1982-01-01,Japan
+datsun 310 gx,38,4,91,67,1995,16.2,1982-01-01,Japan
+buick century limited,25,6,181,110,2945,16.4,1982-01-01,USA
+oldsmobile cutlass ciera (diesel),38,6,262,85,3015,17,1982-01-01,USA
+chrysler lebaron medallion,26,4,156,92,2585,14.5,1982-01-01,USA
+ford granada l,22,6,232,112,2835,14.7,1982-01-01,USA
+toyota celica gt,32,4,144,96,2665,13.9,1982-01-01,Japan
+dodge charger 2.2,36,4,135,84,2370,13,1982-01-01,USA
+chevrolet camaro,27,4,151,90,2950,17.3,1982-01-01,USA
+ford mustang gl,27,4,140,86,2790,15.6,1982-01-01,USA
+vw pickup,44,4,97,52,2130,24.6,1982-01-01,Europe
+dodge rampage,32,4,135,84,2295,11.6,1982-01-01,USA
+ford ranger,28,4,120,79,2625,18.6,1982-01-01,USA
+chevy s-10,31,4,119,82,2720,19.4,1982-01-01,USA

docz/static/loadofsheet/loadofsheet.mjs

@@ -0,0 +1,70 @@
+import { Document } from "@langchain/core/documents";
+import { BufferLoader } from "langchain/document_loaders/fs/buffer";
+import { read, utils } from "xlsx";
+
+/**
+ * Document loader that uses SheetJS to load documents.
+ *
+ * Each worksheet is parsed into an array of row objects using the SheetJS
+ * `sheet_to_json` method and projected to a `Document`. Metadata includes
+ * original sheet name, row data, and row index
+ */
+export default class LoadOfSheet extends BufferLoader {
+  /** @type {import("langchain/chains/query_constructor").AttributeInfo[]}  */
+  attributes = [];
+
+  /**
+   * Document loader that uses SheetJS to load documents.
+   *
+   * @param {string|Blob} filePathOrBlob Source Data
+   */
+  constructor(filePathOrBlob) {
+    super(filePathOrBlob);
+    this.attributes = [];
+  }
+
+  /**
+   * Parse document
+   *
+   * NOTE: column labels in multiple sheets are not disambiguated!
+   *
+   * @param {Buffer} raw Raw data Buffer
+   * @param {Document["metadata"]} metadata Document metadata
+   * @returns {Promise<Document[]>} Array of Documents
+   */
+  async parse(raw, metadata) {
+    /** @type {Document[]} */
+    const result = [];
+
+    this.attributes = [
+      { name: "worksheet", description: "Sheet or Worksheet Name", type: "string" },
+      { name: "rowNum", description: "Row index", type: "number" }
+    ];
+
+    const wb = read(raw, {type: "buffer", WTF:1});
+    for(let name of wb.SheetNames) {
+      const fields = {};
+      const ws = wb.Sheets[name];
+      if(!ws) return;
+
+      const aoo = utils.sheet_to_json(ws);
+      aoo.forEach((row, idx) => {
+        result.push({
+          pageContent: "Row " + (idx + 1) + " has the following content: \n" + Object.entries(row).map(kv => `- ${kv[0]}: ${kv[1]}`).join("\n") + "\n",
+          metadata: {
+            worksheet: name,
+            rowNum: row["__rowNum__"],
+            ...metadata,
+            ...row
+          }
+        });
+        Object.entries(row).forEach(([k,v]) => { if(v != null) (fields[k] || (fields[k] = {}))[v instanceof Date ? "date" : typeof v] = true } );
+      });
+      Object.entries(fields).forEach(([k,v]) => this.attributes.push({
+        name: k, description: k, type: Object.keys(v).join(" or ")
+      }));
+    }
+
+    return result;
+  }
+};

docz/static/loadofsheet/query.mjs

@@ -0,0 +1,39 @@
+import { existsSync } from 'fs';
+import { ChatOllama } from "@langchain/community/chat_models/ollama";
+import { OllamaEmbeddings } from "@langchain/community/embeddings/ollama"
+import { HNSWLib } from "@langchain/community/vectorstores/hnswlib";
+import { SelfQueryRetriever } from "langchain/retrievers/self_query";
+import { FunctionalTranslator } from "@langchain/core/structured_query";
+
+import LoadOfSheet from "./loadofsheet.mjs";
+
+const modelName = "llama3-chatqa:8b-v1.5-q8_0";
+
+const model = new ChatOllama({ baseUrl: "http://localhost:11434", model: modelName });
+const embeddings = new OllamaEmbeddings({model: modelName});
+
+const loader = new LoadOfSheet("./cd.xls");
+const docs = await loader.load();
+
+const vectorstore = await (async() => {
+  if(!existsSync("store/hnswlib.index")) {
+    const vectorstore = await HNSWLib.fromDocuments(docs, embeddings);
+    await vectorstore.save("store");
+    return vectorstore;
+  }
+  return await HNSWLib.load("store", embeddings);
+})();
+
+const selfQueryRetriever = SelfQueryRetriever.fromLLM({
+  llm: model,
+  vectorStore: vectorstore,
+  documentContents: "Data rows from a worksheet",
+  attributeInfo: loader.attributes,
+  structuredQueryTranslator: new FunctionalTranslator(),
+  searchParams: { k: 1024 } // default is 4
+});
+
+const res = await selfQueryRetriever.invoke(
+  "Which rows have over 40 miles per gallon?"
+);
+res.forEach(({metadata}) => { console.log({ Name: metadata.Name, MPG: metadata.Miles_per_Gallon }); });