forked from sheetjs/docs.sheetjs.com
Loader tutorial
This commit is contained in:
parent
a768b7ce8c
commit
30827f4b7f
@ -222,11 +222,13 @@ const prez = raw_data.filter(row => row.terms.some(term => term.type == "prez"))
|
||||
### Sorting by First Term
|
||||
|
||||
The dataset is sorted in chronological order by the first presidential or vice
|
||||
presidential term. The Vice President and President in a given term are sorted
|
||||
alphabetically. Joe Biden and Barack Obama were Vice President and President
|
||||
respectively in 2009. Since "Biden" is alphabetically before "Obama", Biden's
|
||||
data point appears first. The goal is to sort the presidents in order of their
|
||||
presidential term.
|
||||
presidential term. The Vice President and President in a given term are sorted
|
||||
alphabetically.
|
||||
|
||||
Barack Obama became President and Joseph Biden became Vice President in 2009.
|
||||
Since "Biden" is alphabetically before "Obama", Biden's data appears first.
|
||||
|
||||
The goal is to sort the presidents in order of their initial presidential term.
|
||||
|
||||
The first step is adding the first presidential term start date to the dataset.
|
||||
The following code looks at each president and creates a `start` property that
|
||||
|
707
docz/docs/02-getting-started/02-examples/06-loader.md
Normal file
707
docz/docs/02-getting-started/02-examples/06-loader.md
Normal file
@ -0,0 +1,707 @@
|
||||
---
|
||||
title: Loader Tutorial
|
||||
pagination_prev: getting-started/installation/index
|
||||
pagination_next: getting-started/roadmap
|
||||
sidebar_position: 6
|
||||
---
|
||||
|
||||
import current from '/version.js';
|
||||
import Tabs from '@theme/Tabs';
|
||||
import TabItem from '@theme/TabItem';
|
||||
import CodeBlock from '@theme/CodeBlock';
|
||||
|
||||
Many existing systems and platforms include support for loading data from CSV
|
||||
files. Many users prefer to work in spreadsheet software and multi-sheet file
|
||||
formats including XLSX. SheetJS libraries help bridge the gap by translating
|
||||
complex workbooks to simple CSV data.
|
||||
|
||||
The goal of this example is to load spreadsheet data into a vector store and use
|
||||
a large language model to generate queries based on English language input. The
|
||||
existing tooling supports CSV but does not support real spreadsheets.
|
||||
|
||||
In ["SheetJS Conversion"](#sheetjs-conversion), we will use SheetJS libraries to
|
||||
generate CSV files for the LangChain CSV loader. These conversions can be run in
|
||||
a preprocessing step without disrupting existing CSV workflows.
|
||||
|
||||
In ["SheetJS Loader"](#sheetjs-loader), we will use SheetJS libraries in a custom
|
||||
loader to directly generate documents and metadata.
|
||||
|
||||
:::note Tested Deployments
|
||||
|
||||
This demo was tested in the following configurations:
|
||||
|
||||
| Date | Platform |
|
||||
|:-----------|:--------------------------------------------------------------|
|
||||
| 2024-06-19 | Apple M2 Max 12-Core CPU + 30-Core GPU (32 GB unified memory) |
|
||||
| 2024-06-19 | NVIDIA RTX 4080 SUPER (16 GB VRAM) + i9-10910 (128 GB RAM) |
|
||||
|
||||
This explanation was verified against LangChain 0.2.
|
||||
|
||||
:::
|
||||
|
||||
## CSV Loader
|
||||
|
||||
Document loaders generate data objects ("documents") and associated metadata
|
||||
from data sources.
|
||||
|
||||
LangChain offers a `CSVLoader`[^1] component for loading CSV data from a file:
|
||||
|
||||
```js title="Generating Documents from a CSV file"
|
||||
import { CSVLoader } from "@langchain/community/document_loaders/fs/csv";
|
||||
|
||||
const loader = new CSVLoader("pres.csv");
|
||||
const docs = await loader.load();
|
||||
|
||||
console.log(docs);
|
||||
```
|
||||
|
||||
The CSV loader uses the first row to determine column headers and generates one
|
||||
document per data row. For example, the following CSV holds Presidential data:
|
||||
|
||||
```csv
|
||||
Name,Index
|
||||
Bill Clinton,42
|
||||
GeorgeW Bush,43
|
||||
Barack Obama,44
|
||||
Donald Trump,45
|
||||
Joseph Biden,46
|
||||
```
|
||||
|
||||
Each data row is translated to a document whose content is a list of attributes
|
||||
and values. For example, the third data row is shown below:
|
||||
|
||||
<table>
|
||||
<thead><tr><th>CSV Row</th><th>Document Content</th></tr></thead>
|
||||
<tbody><tr><td>
|
||||
|
||||
```
|
||||
Name,Index
|
||||
Barack Obama,44
|
||||
```
|
||||
|
||||
</td><td>
|
||||
|
||||
```
|
||||
Name: Barack Obama
|
||||
Index: 44
|
||||
```
|
||||
|
||||
</td></tr></tbody>
|
||||
</table>
|
||||
|
||||
The LangChain CSV loader will include source metadata in the document:
|
||||
|
||||
```js title="Document generated by the CSV loader"
|
||||
Document {
|
||||
pageContent: 'Name: Barack Obama\nIndex: 44',
|
||||
metadata: { source: 'pres.csv', line: 3 }
|
||||
}
|
||||
```
|
||||
|
||||
## SheetJS Conversion
|
||||
|
||||
The [SheetJS NodeJS module](/docs/getting-started/installation/nodejs) can be
|
||||
imported in NodeJS scripts that use LangChain and other JavaScript libraries.
|
||||
|
||||
A simple pre-processing step can convert workbooks to spreadsheets
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
file[(Workbook\nXLSX/XLS)]
|
||||
subgraph SheetJS Structures
|
||||
wb(((SheetJS\nWorkbook)))
|
||||
ws((SheetJS\nWorksheet))
|
||||
end
|
||||
csv(CSV\nstring)
|
||||
docs[[Documents\nArray]]
|
||||
file --> |readFile\n\n| wb
|
||||
wb --> |wb.Sheets\nselect sheet| ws
|
||||
ws --> |sheet_to_csv\n\n| csv
|
||||
csv --> |CSVLoader\n\n| docs
|
||||
linkStyle 0,1,2 color:blue,stroke:blue;
|
||||
```
|
||||
|
||||
The SheetJS `readFile` method[^2] can read general workbooks. The method returns
|
||||
a workbook object that conforms to the SheetJS data model[^3].
|
||||
|
||||
Workbook objects represent multi-sheet workbook files. They store individual
|
||||
worksheet objects and other metadata.
|
||||
|
||||
Each worksheet in the workbook can be written to CSV text using the SheetJS
|
||||
`sheet_to_csv`[^4] method.
|
||||
|
||||
For example, the following NodeJS script reads `pres.xlsx` and displays CSV rows
|
||||
from the first worksheet:
|
||||
|
||||
```js title="Print CSV data from the first worksheet"
|
||||
/* Load SheetJS Libraries */
|
||||
import { readFile, set_fs, utils } from 'xlsx';
|
||||
|
||||
/* Load 'fs' for readFile support */
|
||||
import * as fs from 'fs';
|
||||
set_fs(fs);
|
||||
|
||||
/* Parse `pres.xlsx` */
|
||||
const wb = readFile("pres.xlsx");
|
||||
|
||||
/* Print CSV rows from first worksheet */
|
||||
const first_ws = wb.Sheets[wb.SheetNames[0]];
|
||||
const csv = utils.sheet_to_csv(first_ws);
|
||||
console.log(csv);
|
||||
```
|
||||
|
||||
### Single Worksheet
|
||||
|
||||
For a single worksheet, a SheetJS pre-processing step can write the CSV rows to
|
||||
file and the `CSVLoader` can load the newly written file.
|
||||
|
||||
<details open>
|
||||
<summary><b>Code example</b> (click to hide)</summary>
|
||||
|
||||
```js title="Pulling data from the first worksheet of a workbook"
|
||||
import { CSVLoader } from "@langchain/community/document_loaders/fs/csv";
|
||||
import { readFile, set_fs, utils } from 'xlsx';
|
||||
|
||||
/* Load 'fs' for readFile support */
|
||||
import * as fs from 'fs';
|
||||
set_fs(fs);
|
||||
|
||||
/* Parse `pres.xlsx`` */
|
||||
const wb = readFile("pres.xlsx");
|
||||
|
||||
/* Generate CSV and write to `pres.xlsx.csv` */
|
||||
const first_ws = wb.Sheets[wb.SheetNames[0]];
|
||||
const csv = utils.sheet_to_csv(first_ws);
|
||||
fs.writeFileSync("pres.xlsx.csv", csv);
|
||||
|
||||
/* Create documents with CSVLoader */
|
||||
const loader = new CSVLoader("pres.xlsx.csv");
|
||||
const docs = await loader.load();
|
||||
|
||||
console.log(docs);
|
||||
// ...
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### Workbook
|
||||
|
||||
A workbook is a collection of worksheets. Each worksheet can be exported to a
|
||||
separate CSV. If the CSVs are written to a subfolder, a `DirectoryLoader`[^5]
|
||||
can process the files in one step.
|
||||
|
||||
<details open>
|
||||
<summary><b>Code example</b> (click to hide)</summary>
|
||||
|
||||
In this example, the script creates a subfolder named `csv`. Each worksheet in
|
||||
the workbook will be processed and the generated CSV will be stored to numbered
|
||||
files. The first worksheet will be stored to `csv/0.csv`.
|
||||
|
||||
```js title="Pulling data from the each worksheet of a workbook"
|
||||
import { CSVLoader } from "@langchain/community/document_loaders/fs/csv";
|
||||
import { DirectoryLoader } from "langchain/document_loaders/fs/directory";
|
||||
import { readFile, set_fs, utils } from 'xlsx';
|
||||
|
||||
/* Load 'fs' for readFile support */
|
||||
import * as fs from 'fs';
|
||||
set_fs(fs);
|
||||
|
||||
/* Parse `pres.xlsx`` */
|
||||
const wb = readFile("pres.xlsx");
|
||||
|
||||
/* Create a folder `csv` */
|
||||
try { fs.mkdirSync("csv"); } catch(e) {}
|
||||
|
||||
/* Generate CSV data for each worksheet */
|
||||
wb.SheetNames.forEach((name, idx) => {
|
||||
const ws = wb.Sheets[name];
|
||||
const csv = utils.sheet_to_csv(ws);
|
||||
fs.writeFileSync(`csv/${idx}.csv`, csv);
|
||||
});
|
||||
|
||||
/* Create documents with DirectoryLoader */
|
||||
const loader = new DirectoryLoader("csv", {
|
||||
".csv": (path) => new CSVLoader(path)
|
||||
});
|
||||
const docs = await loader.load();
|
||||
|
||||
console.log(docs);
|
||||
// ...
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## SheetJS Loader
|
||||
|
||||
The `CSVLoader` that ships with LangChain does not add any Document metadata and
|
||||
does not generate any attributes. A custom loader can work around limitations in
|
||||
the CSV tooling and potentially include metadata that has no CSV equivalent.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
file[(Workbook\nXLSX/XLS)]
|
||||
subgraph SheetJS Structures
|
||||
wb(((SheetJS\nWorkbook)))
|
||||
ws((SheetJS\nWorksheet))
|
||||
end
|
||||
aoo[(Array of\nObjects)]
|
||||
docs[[Documents\nArray]]
|
||||
file --> |readFile\n\n| wb
|
||||
wb --> |wb.Sheets\nEach worksheet| ws
|
||||
ws --> |sheet_to_json\n\n| aoo
|
||||
aoo --> |new Document\nEach Row| docs
|
||||
linkStyle 0,1,2 color:blue,stroke:blue;
|
||||
```
|
||||
|
||||
The demo [`LoadOfSheet` loader](pathname:///loadofsheet/loadofsheet.mjs) will
|
||||
generate one Document per data row across all worksheets. It will also attempt
|
||||
to build metadata and attributes for use in self-querying retrievers.
|
||||
|
||||
<details>
|
||||
<summary><b>Sample SheetJS Loader</b> (click to show)</summary>
|
||||
|
||||
This example loader pulls data from each worksheet. It assumes each worksheet
|
||||
includes one header row and a number of data rows.
|
||||
|
||||
```js title="loadofsheet.mjs"
|
||||
import { Document } from "@langchain/core/documents";
|
||||
import { BufferLoader } from "langchain/document_loaders/fs/buffer";
|
||||
import { read, utils } from "xlsx";
|
||||
|
||||
/**
|
||||
* Document loader that uses SheetJS to load documents.
|
||||
*
|
||||
* Each worksheet is parsed into an array of row objects using the SheetJS
|
||||
* `sheet_to_json` method and projected to a `Document`. Metadata includes
|
||||
* original sheet name, row data, and row index
|
||||
*/
|
||||
export default class LoadOfSheet extends BufferLoader {
|
||||
/** @type {import("langchain/chains/query_constructor").AttributeInfo[]} */
|
||||
attributes = [];
|
||||
|
||||
/**
|
||||
* Document loader that uses SheetJS to load documents.
|
||||
*
|
||||
* @param {string|Blob} filePathOrBlob Source Data
|
||||
*/
|
||||
constructor(filePathOrBlob) {
|
||||
super(filePathOrBlob);
|
||||
this.attributes = [];
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse document
|
||||
*
|
||||
* NOTE: column labels in multiple sheets are not disambiguated!
|
||||
*
|
||||
* @param {Buffer} raw Raw data Buffer
|
||||
* @param {Document["metadata"]} metadata Document metadata
|
||||
* @returns {Promise<Document[]>} Array of Documents
|
||||
*/
|
||||
async parse(raw, metadata) {
|
||||
/** @type {Document[]} */
|
||||
const result = [];
|
||||
|
||||
this.attributes = [
|
||||
{ name: "worksheet", description: "Sheet or Worksheet Name", type: "string" },
|
||||
{ name: "rowNum", description: "Row index", type: "number" }
|
||||
];
|
||||
|
||||
const wb = read(raw, {type: "buffer", WTF:1});
|
||||
for(let name of wb.SheetNames) {
|
||||
const fields = {};
|
||||
const ws = wb.Sheets[name];
|
||||
if(!ws) return;
|
||||
|
||||
const aoo = utils.sheet_to_json(ws);
|
||||
aoo.forEach((row, idx) => {
|
||||
result.push({
|
||||
pageContent: "Row " + (idx + 1) + " has the following content: \n" + Object.entries(row).map(kv => `- ${kv[0]}: ${kv[1]}`).join("\n") + "\n",
|
||||
metadata: {
|
||||
worksheet: name,
|
||||
rowNum: row["__rowNum__"],
|
||||
...metadata,
|
||||
...row
|
||||
}
|
||||
});
|
||||
Object.entries(row).forEach(([k,v]) => { if(v != null) (fields[k] || (fields[k] = {}))[v instanceof Date ? "date" : typeof v] = true } );
|
||||
});
|
||||
Object.entries(fields).forEach(([k,v]) => this.attributes.push({
|
||||
name: k, description: k, type: Object.keys(v).join(" or ")
|
||||
}));
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### From Text to Binary
|
||||
|
||||
Many libraries and platforms offer generic "text" loaders that process files
|
||||
assuming the UTF8 encoding. This corrupts many spreadsheet formats including
|
||||
XLSX, XLSB, XLSM and XLS.
|
||||
|
||||
:::note pass
|
||||
|
||||
This issue affects many JavaScript tools. Various demos cover workarounds:
|
||||
|
||||
- [ViteJS plugins](/docs/demos/static/vitejs#plugins) receive the relative path
|
||||
to the workbook file and can read the file directly.
|
||||
|
||||
- [Webpack Plugins](/docs/demos/static/webpack#sheetjs-loader) have a special
|
||||
option to instruct the library to pass raw binary data rather than text.
|
||||
|
||||
:::
|
||||
|
||||
The `CSVLoader` extends a special `TextLoader` that forces UTF8 text parsing.
|
||||
|
||||
There is a separate `BufferLoader` class, used by the PDF loader, that passes
|
||||
the raw data using NodeJS `Buffer` objects.
|
||||
|
||||
<table>
|
||||
<thead><tr><th>Binary</th><th>Text</th></tr></thead>
|
||||
<tbody><tr><td>
|
||||
|
||||
```ts title="pdf.ts (structure)"
|
||||
export class PDFLoader extends BufferLoader {
|
||||
// ...
|
||||
public async parse(
|
||||
raw: Buffer,
|
||||
metadata: Document["metadata"]
|
||||
): Promise<Document[]> {
|
||||
// ...
|
||||
}
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
</td><td>
|
||||
|
||||
```ts title="csv.ts (structure)"
|
||||
export class CSVLoader extends TextLoader {
|
||||
// ...
|
||||
protected async parse(
|
||||
raw: string
|
||||
|
||||
): Promise<string[]> {
|
||||
// ...
|
||||
}
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
</td></tr></tbody>
|
||||
</table>
|
||||
|
||||
### NodeJS Buffers
|
||||
|
||||
The SheetJS `read` method supports NodeJS Buffer objects directly[^6]:
|
||||
|
||||
```js title="Parsing a workbook in a BufferLoader"
|
||||
import { BufferLoader } from "langchain/document_loaders/fs/buffer";
|
||||
import { read, utils } from "xlsx";
|
||||
|
||||
export default class LoadOfSheet extends BufferLoader {
|
||||
// ...
|
||||
async parse(raw, metadata) {
|
||||
// highlight-next-line
|
||||
const wb = read(raw, {type: "buffer"});
|
||||
// At this point, `wb` is a SheetJS workbook object
|
||||
// ...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `read` method returns a SheetJS workbook object[^7].
|
||||
|
||||
### Generating Content
|
||||
|
||||
The SheetJS `sheet_to_json` method[^8] returns an array of data objects whose
|
||||
keys are drawn from the first row of the worksheet.
|
||||
|
||||
<table>
|
||||
<thead><tr><th>Spreadsheet</th><th>Array of Objects</th></tr></thead>
|
||||
<tbody><tr><td>
|
||||
|
||||
![`pres.xlsx` data](pathname:///pres.png)
|
||||
|
||||
</td><td>
|
||||
|
||||
```js
|
||||
[
|
||||
{ Name: "Bill Clinton", Index: 42 },
|
||||
{ Name: "GeorgeW Bush", Index: 43 },
|
||||
{ Name: "Barack Obama", Index: 44 },
|
||||
{ Name: "Donald Trump", Index: 45 },
|
||||
{ Name: "Joseph Biden", Index: 46 }
|
||||
]
|
||||
```
|
||||
|
||||
</td></tr></tbody></table>
|
||||
|
||||
The original `CSVLoader` wrote one row for each key-value pair. This text can be
|
||||
generated by looping over the keys and values of the data row object. The
|
||||
`Object.entries` helper function simplifies the conversion:
|
||||
|
||||
```js
|
||||
function make_csvloader_doc_from_row_object(row) {
|
||||
return Object.entries(row).map(([k,v]) => `${k}: ${v}`).join("\n");
|
||||
}
|
||||
```
|
||||
|
||||
### Generating Documents
|
||||
|
||||
The loader must generate row objects for each worksheet in the workbook.
|
||||
|
||||
In the SheetJS data model, the workbook object has two relevant fields:
|
||||
|
||||
- `SheetNames` is an array of sheet names
|
||||
- `Sheets` is an object whose keys are sheet names and values are sheet objects.
|
||||
|
||||
A `for..of` loop can iterate across the worksheets:
|
||||
|
||||
```js title="Looping over a workbook (skeleton)"
|
||||
const wb = read(raw, {type: "buffer", WTF:1});
|
||||
for(let name of wb.SheetNames) {
|
||||
const ws = wb.Sheets[name];
|
||||
const aoa = utils.sheet_to_json(ws);
|
||||
// at this point, `aoa` is an array of objects
|
||||
}
|
||||
```
|
||||
|
||||
This simplified `parse` function uses the snippet from the previous section:
|
||||
|
||||
```js title="BufferLoader parse function (skeleton)"
|
||||
async parse(raw, metadata) {
|
||||
/* array to hold generated documents */
|
||||
const result = [];
|
||||
|
||||
/* read workbook */
|
||||
const wb = read(raw, {type: "buffer", WTF:1});
|
||||
|
||||
/* loop over worksheets */
|
||||
for(let name of wb.SheetNames) {
|
||||
const ws = wb.Sheets[name];
|
||||
const aoa = utils.sheet_to_json(ws);
|
||||
|
||||
/* loop over data rows */
|
||||
aoa.forEach((row, idx) => {
|
||||
/* generate a new document and add to the result array */
|
||||
result.push({
|
||||
pageContent: Object.entries(row).map(([k,v]) => `${k}: ${v}`).join("\n")
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
```
|
||||
|
||||
### Metadata and Attributes
|
||||
|
||||
It is strongly recommended to generate additional metadata and attributes for
|
||||
self-query retrieval applications.
|
||||
|
||||
<details>
|
||||
<summary><b>Implementation Details</b> (click to show)</summary>
|
||||
|
||||
**Metadata**
|
||||
|
||||
Metadata is attached to each document object. The following example appends the
|
||||
raw row data to the document metadata:
|
||||
|
||||
```js title="Document with metadata (snippet)"
|
||||
/* generate a new document and add to the result array */
|
||||
result.push({
|
||||
pageContent: Object.entries(row).map(([k,v]) => `${k}: ${v}`).join("\n"),
|
||||
metadata: {
|
||||
worksheet: name, // name of the worksheet
|
||||
rowNum: idx, // data row index
|
||||
...row // raw row data
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
**Attributes**
|
||||
|
||||
Each attribute object specifies three properties:
|
||||
|
||||
- `name` corresponds to the field in the document metadata
|
||||
- `description` is a description of the field
|
||||
- `type` is a description of the data type.
|
||||
|
||||
While looping through data rows, a simple type check can keep track of the data
|
||||
type for each column:
|
||||
|
||||
```js title="Tracking column types (sketch)"
|
||||
for(let name of wb.SheetNames) {
|
||||
/* track column types */
|
||||
const fields = {};
|
||||
// ...
|
||||
|
||||
aoo.forEach((row, idx) => {
|
||||
result.push({/* ... */});
|
||||
/* Check each property */
|
||||
Object.entries(row).forEach(([k,v]) => {
|
||||
/* Update fields entry to reflect the new data point */
|
||||
if(v != null) (fields[k] || (fields[k] = {}))[v instanceof Date ? "date" : typeof v] = true
|
||||
});
|
||||
});
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
Attributes can be generated after writing the worksheet data. Storing attributes
|
||||
in a loader property will make it accessible to scripts that use the loader.
|
||||
|
||||
```js title="Adding Attributes to a Loader (sketch)"
|
||||
export default class LoadOfSheet extends BufferLoader {
|
||||
// highlight-next-line
|
||||
attributes = [];
|
||||
// ...
|
||||
|
||||
async parse(raw, metadata) {
|
||||
// Add the worksheet name and row index attributes
|
||||
// highlight-start
|
||||
this.attributes = [
|
||||
{ name: "worksheet", description: "Sheet or Worksheet Name", type: "string" },
|
||||
{ name: "rowNum", description: "Row index", type: "number" }
|
||||
];
|
||||
// highlight-end
|
||||
const wb = read(raw, {type: "buffer", WTF:1});
|
||||
for(let name of wb.SheetNames) {
|
||||
// highlight-next-line
|
||||
const fields = {};
|
||||
// ...
|
||||
const aoo = utils.sheet_to_json(ws);
|
||||
aoo.forEach((row, idx) => {
|
||||
result.push({/* ... */});
|
||||
/* Check each property */
|
||||
Object.entries(row).forEach(([k,v]) => {
|
||||
/* Update fields entry to reflect the new data point */
|
||||
if(v != null) (fields[k] || (fields[k] = {}))[v instanceof Date ? "date" : typeof v] = true
|
||||
});
|
||||
});
|
||||
/* Add one attribute per metadata field */
|
||||
// highlight-start
|
||||
Object.entries(fields).forEach(([k,v]) => this.attributes.push({
|
||||
name: k, description: k,
|
||||
/* { number: true, string: true } -> "number or string" */
|
||||
type: Object.keys(v).join(" or ")
|
||||
}));
|
||||
// highlight-end
|
||||
}
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## SheetJS Loader Demo
|
||||
|
||||
The demo performs the query "Which rows have over 40 miles per gallon?" against
|
||||
a [sample cars dataset](pathname:///cd.xls) and displays the results.
|
||||
|
||||
:::caution pass
|
||||
|
||||
This demo was tested using the ChatQA-1.5 model[^9] in Ollama[^10].
|
||||
|
||||
The tested model requires 9.2GB VRAM. It is strongly recommended to run the demo
|
||||
on a newer Apple Silicon Mac or a PC with an Nvidia GPU with at least 12GB VRAM.
|
||||
|
||||
:::
|
||||
|
||||
0) Create a new project:
|
||||
|
||||
```bash
|
||||
mkdir sheetjs-loader
|
||||
cd sheetjs-loader
|
||||
npm init -y
|
||||
```
|
||||
|
||||
1) Download the demo scripts:
|
||||
|
||||
- [`loadofsheet.mjs`](pathname:///loadofsheet/loadofsheet.mjs)
|
||||
- [`query.mjs`](pathname:///loadofsheet/query.mjs)
|
||||
|
||||
```bash
|
||||
curl -LO https://docs.sheetjs.com/loadofsheet/query.mjs
|
||||
curl -LO https://docs.sheetjs.com/loadofsheet/loadofsheet.mjs
|
||||
```
|
||||
|
||||
2) Install the SheetJS NodeJS module:
|
||||
|
||||
<CodeBlock language="bash">{`\
|
||||
npm i --save https://cdn.sheetjs.com/xlsx-${current}/xlsx-${current}.tgz`}
|
||||
</CodeBlock>
|
||||
|
||||
3) Install LangChain and HNSWLib dependencies:
|
||||
|
||||
```bash
|
||||
npm i --save @langchain/community@0.2.0 @langchain/core@0.2.6 langchain@0.2.5 hnswlib-node@3.0.0 peggy@3.0.2
|
||||
```
|
||||
|
||||
4) Download the [cars dataset](pathname:///cd.xls):
|
||||
|
||||
```bash
|
||||
curl -LO https://docs.sheetjs.com/cd.xls
|
||||
```
|
||||
|
||||
5) Install the `llama3-chatqa:8b-v1.5-q8_0` model using Ollama:
|
||||
|
||||
```bash
|
||||
ollama pull llama3-chatqa:8b-v1.5-q8_0
|
||||
```
|
||||
|
||||
:::note pass
|
||||
|
||||
If the command cannot be found, install Ollama[^10] and run the command in a new
|
||||
terminal window.
|
||||
|
||||
:::
|
||||
|
||||
6) Run the demo script
|
||||
|
||||
```bash
|
||||
node query.mjs
|
||||
```
|
||||
|
||||
The demo performs the query "Which rows have over 40 miles per gallon?". It will
|
||||
print the following nine results:
|
||||
|
||||
```js title="Expected output"
|
||||
{ Name: 'volkswagen rabbit custom diesel', MPG: 43.1 }
|
||||
{ Name: 'vw rabbit c (diesel)', MPG: 44.3 }
|
||||
{ Name: 'renault lecar deluxe', MPG: 40.9 }
|
||||
{ Name: 'honda civic 1500 gl', MPG: 44.6 }
|
||||
{ Name: 'datsun 210', MPG: 40.8 }
|
||||
{ Name: 'vw pickup', MPG: 44 }
|
||||
{ Name: 'mazda glc', MPG: 46.6 }
|
||||
{ Name: 'vw dasher (diesel)', MPG: 43.4 }
|
||||
{ Name: 'vw rabbit', MPG: 41.5 }
|
||||
```
|
||||
|
||||
To find the expected results:
|
||||
|
||||
- Open the `cd.xls` spreadsheet in Excel
|
||||
- Select Home > Sort & Filter > Filter in the Ribbon
|
||||
- Select the filter option for column B (`Miles_per_Gallon`)
|
||||
- In the popup, select "Greater Than" in the Filter dropdown and type 40
|
||||
|
||||
The filtered results should match the following screenshot:
|
||||
|
||||
![Expected Results](pathname:///loadofsheet/expected.png)
|
||||
|
||||
[^1]: See ["How to load CSV data"](https://js.langchain.com/v0.2/docs/how_to/document_loader_csv) in the LangChain documentation
|
||||
[^2]: See [`readFile` in "Reading Files"](/docs/api/parse-options)
|
||||
[^3]: See ["SheetJS Data Model"](/docs/csf/)
|
||||
[^4]: See [`sheet_to_csv` in "CSV and Text"](/docs/api/utilities/csv#delimiter-separated-output)
|
||||
[^5]: See ["Folders with multiple files"](https://js.langchain.com/v0.2/docs/integrations/document_loaders/file_loaders/directory/) in the LangChain documentation
|
||||
[^6]: See ["Supported Output Formats" type in "Writing Files"](/docs/api/write-options#supported-output-formats)
|
||||
[^7]: See ["Workbook Object"](/docs/csf/book)
|
||||
[^8]: See [`sheet_to_json` in "Utilities"](/docs/api/utilities/array#array-output)
|
||||
[^9]: See [the official ChatQA website](https://chatqa-project.github.io/) for the ChatQA paper and other model details.
|
||||
[^10]: See [the official Ollama website](https://ollama.com/download) for installation instructions.
|
@ -23,3 +23,8 @@ The ["Import Tutorial"](/docs/getting-started/examples/import) examines the data
|
||||
import process. A legacy file is downloaded and parsed. The underlying data is
|
||||
ultimately displayed to the user in a HTML table.
|
||||
|
||||
## Loading Sheets
|
||||
|
||||
["Loading Sheets"](/docs/getting-started/examples/loader) explores deep SheetJS
|
||||
integrations. Based on the existing CSV and binary loaders, a spreadsheet loader
|
||||
for LangChain is developed and tested.
|
||||
|
@ -258,23 +258,23 @@ saving file 0 |RedRockA.xls| to file0.xls
|
||||
```
|
||||
|
||||
Lines starting with `####` show the attachment file name and the worksheet name.
|
||||
The following line explains that there is a worksheet named `"Oct 26, 2001"` in
|
||||
The following line explains that there is a worksheet named `"Oneok at 2500"` in
|
||||
the file `RedRockA.xls`:
|
||||
|
||||
```
|
||||
#### RedRockA.xls ! Oct 26, 2001
|
||||
#### RedRockA.xls ! Oneok at 2500
|
||||
```
|
||||
|
||||
Every other line is a CSV row from the named worksheet. For example, the first
|
||||
four lines of worksheet `"Oct 26, 2001"` in `RedRockA.xls` are shown below:
|
||||
four lines of worksheet `"Oneok at 2500"` in `RedRockA.xls` are shown below:
|
||||
|
||||
```text
|
||||
#### RedRockA.xls ! Oct 26, 2001
|
||||
#### RedRockA.xls ! Oneok at 2500
|
||||
// highlight-start
|
||||
RED ROCK EXPANSION PROJECT,,,,,,,,,,,,,,,,,,
|
||||
,,,,,,,,,,,,,,,,,,
|
||||
,,,, , , ,,,,,,,,,,,,
|
||||
SHIPPER,CONTRACT #,Term,MMBtu/d,RECEIPT POINT,DELIVERY POINT,MMBtu/d,,,,,,,,,,,,
|
||||
RED ROCK EXPANSION PROJECT,,,,,,,,
|
||||
,,,,,,,,
|
||||
,,REQUESTED,REQUESTED,,,,,
|
||||
,,RECEIPT,DELIVERY,,,Allocation,,
|
||||
// highlight-end
|
||||
```
|
||||
|
||||
|
@ -491,6 +491,29 @@ quasar dev -m android
|
||||
|
||||
If prompted to select an external IP, press <kbd>Enter</kbd>.
|
||||
|
||||
:::caution pass
|
||||
|
||||
If the app is blank or not refreshing, delete the app and close the simulator,
|
||||
then restart the development process.
|
||||
|
||||
:::
|
||||
|
||||
:::warning pass
|
||||
|
||||
On Windows, the command failed with a Gradle error
|
||||
|
||||
```
|
||||
Could not find an installed version of Gradle either in Android Studio,
|
||||
or on your system to install the gradle wrapper. Please include gradle
|
||||
in your path, or install Android Studio
|
||||
```
|
||||
|
||||
[Gradle](https://gradle.org/) (the complete version) must be extracted and the
|
||||
`bin` folder must be added to the user PATH variable. After adding to PATH,
|
||||
launch a new PowerShell or CMD window and run the command.
|
||||
|
||||
:::
|
||||
|
||||
To test that reading works:
|
||||
|
||||
- Click and drag `pres.numbers` from a Finder window into the simulator.
|
||||
|
@ -514,14 +514,13 @@ export default defineConfig(async () => ({
|
||||
// highlight-end
|
||||
```
|
||||
|
||||
- Add the highlighted line to `tsconfig.json`:
|
||||
- Edit `tsconfig.json`. In `compilerOptions` add the option `"jsx": "preserve"`:
|
||||
|
||||
```js title="tsconfig.json (add highlighted line)"
|
||||
{
|
||||
"compilerOptions": {
|
||||
// highlight-next-line
|
||||
"jsx": "preserve",
|
||||
"target": "ES2020",
|
||||
```
|
||||
|
||||
- Replace `index.html` with the following codeblock:
|
||||
|
@ -61,7 +61,7 @@ NodeJS push streams were introduced in 2012. The text streaming methods `to_csv`
|
||||
and `to_html` are supported in NodeJS v0.10 and later while the object streaming
|
||||
method `to_json` is supported in NodeJS v0.12 and later.
|
||||
|
||||
The first streaming write function, `to_csv`, was introduced in early 2017. It
|
||||
The first SheetJS streaming write function, `to_csv`, was introduced in 2017. It
|
||||
used and still uses the same NodeJS streaming API.
|
||||
|
||||
Years later, browser vendors are settling on a different stream API.
|
||||
|
@ -585,8 +585,8 @@ string rather than a proper Date object.
|
||||
```mermaid
|
||||
flowchart LR
|
||||
ws(("new Date#40;#41;"\nObject))
|
||||
jstr[["'#quot;2001-01-01T00:00:00.000Z#quot;'"\nEncoded String]]
|
||||
js[["#quot;2001-01-01T00:00:00.000Z#quot;"\nJavaScript String]]
|
||||
jstr[["'#quot;2048-10-06T00:00:00.000Z#quot;'"\nEncoded String]]
|
||||
js[["#quot;2048-10-06T00:00:00.000Z#quot;"\nJavaScript String]]
|
||||
ws --> |JSON\nstringify| jstr
|
||||
jstr --> |JSON.parse\n\n| js
|
||||
js --> |\nJSON.stringify| jstr
|
||||
@ -601,8 +601,8 @@ while the "Date Obj" data will be converted to a spreadsheet date.
|
||||
```jsx live
|
||||
function SheetJSONDates() { return ( <button onClick={() => {
|
||||
const aoa = [
|
||||
["ISO Text", "2001-01-01T00:00:00.000Z"], // B1 will be text
|
||||
["Date Obj", new Date("2001-01-01T00:00:00.000Z")] // B2 will be a date
|
||||
["ISO Text", "2048-10-06T00:00:00.000Z"], // B1 will be text
|
||||
["Date Obj", new Date("2048-10-06T00:00:00.000Z")] // B2 will be a date
|
||||
];
|
||||
const ws = XLSX.utils.aoa_to_sheet(aoa);
|
||||
const wb = XLSX.utils.book_new(ws, "Data");
|
||||
|
407
docz/static/cd.csv
Normal file
407
docz/static/cd.csv
Normal file
@ -0,0 +1,407 @@
|
||||
Name,Miles_per_Gallon,Cylinders,Displacement,Horsepower,Weight_in_lbs,Acceleration,Year,Origin
|
||||
chevrolet chevelle malibu,18,8,307,130,3504,12,1970-01-01,USA
|
||||
buick skylark 320,15,8,350,165,3693,11.5,1970-01-01,USA
|
||||
plymouth satellite,18,8,318,150,3436,11,1970-01-01,USA
|
||||
amc rebel sst,16,8,304,150,3433,12,1970-01-01,USA
|
||||
ford torino,17,8,302,140,3449,10.5,1970-01-01,USA
|
||||
ford galaxie 500,15,8,429,198,4341,10,1970-01-01,USA
|
||||
chevrolet impala,14,8,454,220,4354,9,1970-01-01,USA
|
||||
plymouth fury iii,14,8,440,215,4312,8.5,1970-01-01,USA
|
||||
pontiac catalina,14,8,455,225,4425,10,1970-01-01,USA
|
||||
amc ambassador dpl,15,8,390,190,3850,8.5,1970-01-01,USA
|
||||
citroen ds-21 pallas,,4,133,115,3090,17.5,1970-01-01,Europe
|
||||
chevrolet chevelle concours (sw),,8,350,165,4142,11.5,1970-01-01,USA
|
||||
ford torino (sw),,8,351,153,4034,11,1970-01-01,USA
|
||||
plymouth satellite (sw),,8,383,175,4166,10.5,1970-01-01,USA
|
||||
amc rebel sst (sw),,8,360,175,3850,11,1970-01-01,USA
|
||||
dodge challenger se,15,8,383,170,3563,10,1970-01-01,USA
|
||||
plymouth 'cuda 340,14,8,340,160,3609,8,1970-01-01,USA
|
||||
ford mustang boss 302,,8,302,140,3353,8,1970-01-01,USA
|
||||
chevrolet monte carlo,15,8,400,150,3761,9.5,1970-01-01,USA
|
||||
buick estate wagon (sw),14,8,455,225,3086,10,1970-01-01,USA
|
||||
toyota corona mark ii,24,4,113,95,2372,15,1970-01-01,Japan
|
||||
plymouth duster,22,6,198,95,2833,15.5,1970-01-01,USA
|
||||
amc hornet,18,6,199,97,2774,15.5,1970-01-01,USA
|
||||
ford maverick,21,6,200,85,2587,16,1970-01-01,USA
|
||||
datsun pl510,27,4,97,88,2130,14.5,1970-01-01,Japan
|
||||
volkswagen 1131 deluxe sedan,26,4,97,46,1835,20.5,1970-01-01,Europe
|
||||
peugeot 504,25,4,110,87,2672,17.5,1970-01-01,Europe
|
||||
audi 100 ls,24,4,107,90,2430,14.5,1970-01-01,Europe
|
||||
saab 99e,25,4,104,95,2375,17.5,1970-01-01,Europe
|
||||
bmw 2002,26,4,121,113,2234,12.5,1970-01-01,Europe
|
||||
amc gremlin,21,6,199,90,2648,15,1970-01-01,USA
|
||||
ford f250,10,8,360,215,4615,14,1970-01-01,USA
|
||||
chevy c20,10,8,307,200,4376,15,1970-01-01,USA
|
||||
dodge d200,11,8,318,210,4382,13.5,1970-01-01,USA
|
||||
hi 1200d,9,8,304,193,4732,18.5,1970-01-01,USA
|
||||
datsun pl510,27,4,97,88,2130,14.5,1971-01-01,Japan
|
||||
chevrolet vega 2300,28,4,140,90,2264,15.5,1971-01-01,USA
|
||||
toyota corona,25,4,113,95,2228,14,1971-01-01,Japan
|
||||
ford pinto,25,4,98,,2046,19,1971-01-01,USA
|
||||
volkswagen super beetle 117,,4,97,48,1978,20,1971-01-01,Europe
|
||||
amc gremlin,19,6,232,100,2634,13,1971-01-01,USA
|
||||
plymouth satellite custom,16,6,225,105,3439,15.5,1971-01-01,USA
|
||||
chevrolet chevelle malibu,17,6,250,100,3329,15.5,1971-01-01,USA
|
||||
ford torino 500,19,6,250,88,3302,15.5,1971-01-01,USA
|
||||
amc matador,18,6,232,100,3288,15.5,1971-01-01,USA
|
||||
chevrolet impala,14,8,350,165,4209,12,1971-01-01,USA
|
||||
pontiac catalina brougham,14,8,400,175,4464,11.5,1971-01-01,USA
|
||||
ford galaxie 500,14,8,351,153,4154,13.5,1971-01-01,USA
|
||||
plymouth fury iii,14,8,318,150,4096,13,1971-01-01,USA
|
||||
dodge monaco (sw),12,8,383,180,4955,11.5,1971-01-01,USA
|
||||
ford country squire (sw),13,8,400,170,4746,12,1971-01-01,USA
|
||||
pontiac safari (sw),13,8,400,175,5140,12,1971-01-01,USA
|
||||
amc hornet sportabout (sw),18,6,258,110,2962,13.5,1971-01-01,USA
|
||||
chevrolet vega (sw),22,4,140,72,2408,19,1971-01-01,USA
|
||||
pontiac firebird,19,6,250,100,3282,15,1971-01-01,USA
|
||||
ford mustang,18,6,250,88,3139,14.5,1971-01-01,USA
|
||||
mercury capri 2000,23,4,122,86,2220,14,1971-01-01,USA
|
||||
opel 1900,28,4,116,90,2123,14,1971-01-01,Europe
|
||||
peugeot 304,30,4,79,70,2074,19.5,1971-01-01,Europe
|
||||
fiat 124b,30,4,88,76,2065,14.5,1971-01-01,Europe
|
||||
toyota corolla 1200,31,4,71,65,1773,19,1971-01-01,Japan
|
||||
datsun 1200,35,4,72,69,1613,18,1971-01-01,Japan
|
||||
volkswagen model 111,27,4,97,60,1834,19,1971-01-01,Europe
|
||||
plymouth cricket,26,4,91,70,1955,20.5,1971-01-01,USA
|
||||
toyota corona hardtop,24,4,113,95,2278,15.5,1972-01-01,Japan
|
||||
dodge colt hardtop,25,4,97.5,80,2126,17,1972-01-01,USA
|
||||
volkswagen type 3,23,4,97,54,2254,23.5,1972-01-01,Europe
|
||||
chevrolet vega,20,4,140,90,2408,19.5,1972-01-01,USA
|
||||
ford pinto runabout,21,4,122,86,2226,16.5,1972-01-01,USA
|
||||
chevrolet impala,13,8,350,165,4274,12,1972-01-01,USA
|
||||
pontiac catalina,14,8,400,175,4385,12,1972-01-01,USA
|
||||
plymouth fury iii,15,8,318,150,4135,13.5,1972-01-01,USA
|
||||
ford galaxie 500,14,8,351,153,4129,13,1972-01-01,USA
|
||||
amc ambassador sst,17,8,304,150,3672,11.5,1972-01-01,USA
|
||||
mercury marquis,11,8,429,208,4633,11,1972-01-01,USA
|
||||
buick lesabre custom,13,8,350,155,4502,13.5,1972-01-01,USA
|
||||
oldsmobile delta 88 royale,12,8,350,160,4456,13.5,1972-01-01,USA
|
||||
chrysler newport royal,13,8,400,190,4422,12.5,1972-01-01,USA
|
||||
mazda rx2 coupe,19,3,70,97,2330,13.5,1972-01-01,Japan
|
||||
amc matador (sw),15,8,304,150,3892,12.5,1972-01-01,USA
|
||||
chevrolet chevelle concours (sw),13,8,307,130,4098,14,1972-01-01,USA
|
||||
ford gran torino (sw),13,8,302,140,4294,16,1972-01-01,USA
|
||||
plymouth satellite custom (sw),14,8,318,150,4077,14,1972-01-01,USA
|
||||
volvo 145e (sw),18,4,121,112,2933,14.5,1972-01-01,Europe
|
||||
volkswagen 411 (sw),22,4,121,76,2511,18,1972-01-01,Europe
|
||||
peugeot 504 (sw),21,4,120,87,2979,19.5,1972-01-01,Europe
|
||||
renault 12 (sw),26,4,96,69,2189,18,1972-01-01,Europe
|
||||
ford pinto (sw),22,4,122,86,2395,16,1972-01-01,USA
|
||||
datsun 510 (sw),28,4,97,92,2288,17,1972-01-01,Japan
|
||||
toyouta corona mark ii (sw),23,4,120,97,2506,14.5,1972-01-01,Japan
|
||||
dodge colt (sw),28,4,98,80,2164,15,1972-01-01,USA
|
||||
toyota corolla 1600 (sw),27,4,97,88,2100,16.5,1972-01-01,Japan
|
||||
buick century 350,13,8,350,175,4100,13,1973-01-01,USA
|
||||
amc matador,14,8,304,150,3672,11.5,1973-01-01,USA
|
||||
chevrolet malibu,13,8,350,145,3988,13,1973-01-01,USA
|
||||
ford gran torino,14,8,302,137,4042,14.5,1973-01-01,USA
|
||||
dodge coronet custom,15,8,318,150,3777,12.5,1973-01-01,USA
|
||||
mercury marquis brougham,12,8,429,198,4952,11.5,1973-01-01,USA
|
||||
chevrolet caprice classic,13,8,400,150,4464,12,1973-01-01,USA
|
||||
ford ltd,13,8,351,158,4363,13,1973-01-01,USA
|
||||
plymouth fury gran sedan,14,8,318,150,4237,14.5,1973-01-01,USA
|
||||
chrysler new yorker brougham,13,8,440,215,4735,11,1973-01-01,USA
|
||||
buick electra 225 custom,12,8,455,225,4951,11,1973-01-01,USA
|
||||
amc ambassador brougham,13,8,360,175,3821,11,1973-01-01,USA
|
||||
plymouth valiant,18,6,225,105,3121,16.5,1973-01-01,USA
|
||||
chevrolet nova custom,16,6,250,100,3278,18,1973-01-01,USA
|
||||
amc hornet,18,6,232,100,2945,16,1973-01-01,USA
|
||||
ford maverick,18,6,250,88,3021,16.5,1973-01-01,USA
|
||||
plymouth duster,23,6,198,95,2904,16,1973-01-01,USA
|
||||
volkswagen super beetle,26,4,97,46,1950,21,1973-01-01,Europe
|
||||
chevrolet impala,11,8,400,150,4997,14,1973-01-01,USA
|
||||
ford country,12,8,400,167,4906,12.5,1973-01-01,USA
|
||||
plymouth custom suburb,13,8,360,170,4654,13,1973-01-01,USA
|
||||
oldsmobile vista cruiser,12,8,350,180,4499,12.5,1973-01-01,USA
|
||||
amc gremlin,18,6,232,100,2789,15,1973-01-01,USA
|
||||
toyota carina,20,4,97,88,2279,19,1973-01-01,Japan
|
||||
chevrolet vega,21,4,140,72,2401,19.5,1973-01-01,USA
|
||||
datsun 610,22,4,108,94,2379,16.5,1973-01-01,Japan
|
||||
maxda rx3,18,3,70,90,2124,13.5,1973-01-01,Japan
|
||||
ford pinto,19,4,122,85,2310,18.5,1973-01-01,USA
|
||||
mercury capri v6,21,6,155,107,2472,14,1973-01-01,USA
|
||||
fiat 124 sport coupe,26,4,98,90,2265,15.5,1973-01-01,Europe
|
||||
chevrolet monte carlo s,15,8,350,145,4082,13,1973-01-01,USA
|
||||
pontiac grand prix,16,8,400,230,4278,9.5,1973-01-01,USA
|
||||
fiat 128,29,4,68,49,1867,19.5,1973-01-01,Europe
|
||||
opel manta,24,4,116,75,2158,15.5,1973-01-01,Europe
|
||||
audi 100ls,20,4,114,91,2582,14,1973-01-01,Europe
|
||||
volvo 144ea,19,4,121,112,2868,15.5,1973-01-01,Europe
|
||||
dodge dart custom,15,8,318,150,3399,11,1973-01-01,USA
|
||||
saab 99le,24,4,121,110,2660,14,1973-01-01,Europe
|
||||
toyota mark ii,20,6,156,122,2807,13.5,1973-01-01,Japan
|
||||
oldsmobile omega,11,8,350,180,3664,11,1973-01-01,USA
|
||||
plymouth duster,20,6,198,95,3102,16.5,1974-01-01,USA
|
||||
ford maverick,21,6,200,,2875,17,1974-01-01,USA
|
||||
amc hornet,19,6,232,100,2901,16,1974-01-01,USA
|
||||
chevrolet nova,15,6,250,100,3336,17,1974-01-01,USA
|
||||
datsun b210,31,4,79,67,1950,19,1974-01-01,Japan
|
||||
ford pinto,26,4,122,80,2451,16.5,1974-01-01,USA
|
||||
toyota corolla 1200,32,4,71,65,1836,21,1974-01-01,Japan
|
||||
chevrolet vega,25,4,140,75,2542,17,1974-01-01,USA
|
||||
chevrolet chevelle malibu classic,16,6,250,100,3781,17,1974-01-01,USA
|
||||
amc matador,16,6,258,110,3632,18,1974-01-01,USA
|
||||
plymouth satellite sebring,18,6,225,105,3613,16.5,1974-01-01,USA
|
||||
ford gran torino,16,8,302,140,4141,14,1974-01-01,USA
|
||||
buick century luxus (sw),13,8,350,150,4699,14.5,1974-01-01,USA
|
||||
dodge coronet custom (sw),14,8,318,150,4457,13.5,1974-01-01,USA
|
||||
ford gran torino (sw),14,8,302,140,4638,16,1974-01-01,USA
|
||||
amc matador (sw),14,8,304,150,4257,15.5,1974-01-01,USA
|
||||
audi fox,29,4,98,83,2219,16.5,1974-01-01,Europe
|
||||
volkswagen dasher,26,4,79,67,1963,15.5,1974-01-01,Europe
|
||||
opel manta,26,4,97,78,2300,14.5,1974-01-01,Europe
|
||||
toyota corona,31,4,76,52,1649,16.5,1974-01-01,Japan
|
||||
datsun 710,32,4,83,61,2003,19,1974-01-01,Japan
|
||||
dodge colt,28,4,90,75,2125,14.5,1974-01-01,USA
|
||||
fiat 128,24,4,90,75,2108,15.5,1974-01-01,Europe
|
||||
fiat 124 tc,26,4,116,75,2246,14,1974-01-01,Europe
|
||||
honda civic,24,4,120,97,2489,15,1974-01-01,Japan
|
||||
subaru,26,4,108,93,2391,15.5,1974-01-01,Japan
|
||||
fiat x1.9,31,4,79,67,2000,16,1974-01-01,Europe
|
||||
plymouth valiant custom,19,6,225,95,3264,16,1975-01-01,USA
|
||||
chevrolet nova,18,6,250,105,3459,16,1975-01-01,USA
|
||||
mercury monarch,15,6,250,72,3432,21,1975-01-01,USA
|
||||
ford maverick,15,6,250,72,3158,19.5,1975-01-01,USA
|
||||
pontiac catalina,16,8,400,170,4668,11.5,1975-01-01,USA
|
||||
chevrolet bel air,15,8,350,145,4440,14,1975-01-01,USA
|
||||
plymouth grand fury,16,8,318,150,4498,14.5,1975-01-01,USA
|
||||
ford ltd,14,8,351,148,4657,13.5,1975-01-01,USA
|
||||
buick century,17,6,231,110,3907,21,1975-01-01,USA
|
||||
chevroelt chevelle malibu,16,6,250,105,3897,18.5,1975-01-01,USA
|
||||
amc matador,15,6,258,110,3730,19,1975-01-01,USA
|
||||
plymouth fury,18,6,225,95,3785,19,1975-01-01,USA
|
||||
buick skyhawk,21,6,231,110,3039,15,1975-01-01,USA
|
||||
chevrolet monza 2+2,20,8,262,110,3221,13.5,1975-01-01,USA
|
||||
ford mustang ii,13,8,302,129,3169,12,1975-01-01,USA
|
||||
toyota corolla,29,4,97,75,2171,16,1975-01-01,Japan
|
||||
ford pinto,23,4,140,83,2639,17,1975-01-01,USA
|
||||
amc gremlin,20,6,232,100,2914,16,1975-01-01,USA
|
||||
pontiac astro,23,4,140,78,2592,18.5,1975-01-01,USA
|
||||
toyota corona,24,4,134,96,2702,13.5,1975-01-01,Japan
|
||||
volkswagen dasher,25,4,90,71,2223,16.5,1975-01-01,Europe
|
||||
datsun 710,24,4,119,97,2545,17,1975-01-01,Japan
|
||||
ford pinto,18,6,171,97,2984,14.5,1975-01-01,USA
|
||||
volkswagen rabbit,29,4,90,70,1937,14,1975-01-01,Europe
|
||||
amc pacer,19,6,232,90,3211,17,1975-01-01,USA
|
||||
audi 100ls,23,4,115,95,2694,15,1975-01-01,Europe
|
||||
peugeot 504,23,4,120,88,2957,17,1975-01-01,Europe
|
||||
volvo 244dl,22,4,121,98,2945,14.5,1975-01-01,Europe
|
||||
saab 99le,25,4,121,115,2671,13.5,1975-01-01,Europe
|
||||
honda civic cvcc,33,4,91,53,1795,17.5,1975-01-01,Japan
|
||||
fiat 131,28,4,107,86,2464,15.5,1976-01-01,Europe
|
||||
opel 1900,25,4,116,81,2220,16.9,1976-01-01,Europe
|
||||
capri ii,25,4,140,92,2572,14.9,1976-01-01,USA
|
||||
dodge colt,26,4,98,79,2255,17.7,1976-01-01,USA
|
||||
renault 12tl,27,4,101,83,2202,15.3,1976-01-01,Europe
|
||||
chevrolet chevelle malibu classic,17.5,8,305,140,4215,13,1976-01-01,USA
|
||||
dodge coronet brougham,16,8,318,150,4190,13,1976-01-01,USA
|
||||
amc matador,15.5,8,304,120,3962,13.9,1976-01-01,USA
|
||||
ford gran torino,14.5,8,351,152,4215,12.8,1976-01-01,USA
|
||||
plymouth valiant,22,6,225,100,3233,15.4,1976-01-01,USA
|
||||
chevrolet nova,22,6,250,105,3353,14.5,1976-01-01,USA
|
||||
ford maverick,24,6,200,81,3012,17.6,1976-01-01,USA
|
||||
amc hornet,22.5,6,232,90,3085,17.6,1976-01-01,USA
|
||||
chevrolet chevette,29,4,85,52,2035,22.2,1976-01-01,USA
|
||||
chevrolet woody,24.5,4,98,60,2164,22.1,1976-01-01,USA
|
||||
vw rabbit,29,4,90,70,1937,14.2,1976-01-01,Europe
|
||||
honda civic,33,4,91,53,1795,17.4,1976-01-01,Japan
|
||||
dodge aspen se,20,6,225,100,3651,17.7,1976-01-01,USA
|
||||
ford granada ghia,18,6,250,78,3574,21,1976-01-01,USA
|
||||
pontiac ventura sj,18.5,6,250,110,3645,16.2,1976-01-01,USA
|
||||
amc pacer d/l,17.5,6,258,95,3193,17.8,1976-01-01,USA
|
||||
volkswagen rabbit,29.5,4,97,71,1825,12.2,1976-01-01,Europe
|
||||
datsun b-210,32,4,85,70,1990,17,1976-01-01,Japan
|
||||
toyota corolla,28,4,97,75,2155,16.4,1976-01-01,Japan
|
||||
ford pinto,26.5,4,140,72,2565,13.6,1976-01-01,USA
|
||||
volvo 245,20,4,130,102,3150,15.7,1976-01-01,Europe
|
||||
plymouth volare premier v8,13,8,318,150,3940,13.2,1976-01-01,USA
|
||||
peugeot 504,19,4,120,88,3270,21.9,1976-01-01,Europe
|
||||
toyota mark ii,19,6,156,108,2930,15.5,1976-01-01,Japan
|
||||
mercedes-benz 280s,16.5,6,168,120,3820,16.7,1976-01-01,Europe
|
||||
cadillac seville,16.5,8,350,180,4380,12.1,1976-01-01,USA
|
||||
chevy c10,13,8,350,145,4055,12,1976-01-01,USA
|
||||
ford f108,13,8,302,130,3870,15,1976-01-01,USA
|
||||
dodge d100,13,8,318,150,3755,14,1976-01-01,USA
|
||||
honda Accelerationord cvcc,31.5,4,98,68,2045,18.5,1977-01-01,Japan
|
||||
buick opel isuzu deluxe,30,4,111,80,2155,14.8,1977-01-01,USA
|
||||
renault 5 gtl,36,4,79,58,1825,18.6,1977-01-01,Europe
|
||||
plymouth arrow gs,25.5,4,122,96,2300,15.5,1977-01-01,USA
|
||||
datsun f-10 hatchback,33.5,4,85,70,1945,16.8,1977-01-01,Japan
|
||||
chevrolet caprice classic,17.5,8,305,145,3880,12.5,1977-01-01,USA
|
||||
oldsmobile cutlass supreme,17,8,260,110,4060,19,1977-01-01,USA
|
||||
dodge monaco brougham,15.5,8,318,145,4140,13.7,1977-01-01,USA
|
||||
mercury cougar brougham,15,8,302,130,4295,14.9,1977-01-01,USA
|
||||
chevrolet concours,17.5,6,250,110,3520,16.4,1977-01-01,USA
|
||||
buick skylark,20.5,6,231,105,3425,16.9,1977-01-01,USA
|
||||
plymouth volare custom,19,6,225,100,3630,17.7,1977-01-01,USA
|
||||
ford granada,18.5,6,250,98,3525,19,1977-01-01,USA
|
||||
pontiac grand prix lj,16,8,400,180,4220,11.1,1977-01-01,USA
|
||||
chevrolet monte carlo landau,15.5,8,350,170,4165,11.4,1977-01-01,USA
|
||||
chrysler cordoba,15.5,8,400,190,4325,12.2,1977-01-01,USA
|
||||
ford thunderbird,16,8,351,149,4335,14.5,1977-01-01,USA
|
||||
volkswagen rabbit custom,29,4,97,78,1940,14.5,1977-01-01,Europe
|
||||
pontiac sunbird coupe,24.5,4,151,88,2740,16,1977-01-01,USA
|
||||
toyota corolla liftback,26,4,97,75,2265,18.2,1977-01-01,Japan
|
||||
ford mustang ii 2+2,25.5,4,140,89,2755,15.8,1977-01-01,USA
|
||||
chevrolet chevette,30.5,4,98,63,2051,17,1977-01-01,USA
|
||||
dodge colt m/m,33.5,4,98,83,2075,15.9,1977-01-01,USA
|
||||
subaru dl,30,4,97,67,1985,16.4,1977-01-01,Japan
|
||||
volkswagen dasher,30.5,4,97,78,2190,14.1,1977-01-01,Europe
|
||||
datsun 810,22,6,146,97,2815,14.5,1977-01-01,Japan
|
||||
bmw 320i,21.5,4,121,110,2600,12.8,1977-01-01,Europe
|
||||
mazda rx-4,21.5,3,80,110,2720,13.5,1977-01-01,Japan
|
||||
volkswagen rabbit custom diesel,43.1,4,90,48,1985,21.5,1978-01-01,Europe
|
||||
ford fiesta,36.1,4,98,66,1800,14.4,1978-01-01,USA
|
||||
mazda glc deluxe,32.8,4,78,52,1985,19.4,1978-01-01,Japan
|
||||
datsun b210 gx,39.4,4,85,70,2070,18.6,1978-01-01,Japan
|
||||
honda civic cvcc,36.1,4,91,60,1800,16.4,1978-01-01,Japan
|
||||
oldsmobile cutlass salon brougham,19.9,8,260,110,3365,15.5,1978-01-01,USA
|
||||
dodge diplomat,19.4,8,318,140,3735,13.2,1978-01-01,USA
|
||||
mercury monarch ghia,20.2,8,302,139,3570,12.8,1978-01-01,USA
|
||||
pontiac phoenix lj,19.2,6,231,105,3535,19.2,1978-01-01,USA
|
||||
chevrolet malibu,20.5,6,200,95,3155,18.2,1978-01-01,USA
|
||||
ford fairmont (auto),20.2,6,200,85,2965,15.8,1978-01-01,USA
|
||||
ford fairmont (man),25.1,4,140,88,2720,15.4,1978-01-01,USA
|
||||
plymouth volare,20.5,6,225,100,3430,17.2,1978-01-01,USA
|
||||
amc concord,19.4,6,232,90,3210,17.2,1978-01-01,USA
|
||||
buick century special,20.6,6,231,105,3380,15.8,1978-01-01,USA
|
||||
mercury zephyr,20.8,6,200,85,3070,16.7,1978-01-01,USA
|
||||
dodge aspen,18.6,6,225,110,3620,18.7,1978-01-01,USA
|
||||
amc concord d/l,18.1,6,258,120,3410,15.1,1978-01-01,USA
|
||||
chevrolet monte carlo landau,19.2,8,305,145,3425,13.2,1978-01-01,USA
|
||||
buick regal sport coupe (turbo),17.7,6,231,165,3445,13.4,1978-01-01,USA
|
||||
ford futura,18.1,8,302,139,3205,11.2,1978-01-01,USA
|
||||
dodge magnum xe,17.5,8,318,140,4080,13.7,1978-01-01,USA
|
||||
chevrolet chevette,30,4,98,68,2155,16.5,1978-01-01,USA
|
||||
toyota corona,27.5,4,134,95,2560,14.2,1978-01-01,Japan
|
||||
datsun 510,27.2,4,119,97,2300,14.7,1978-01-01,Japan
|
||||
dodge omni,30.9,4,105,75,2230,14.5,1978-01-01,USA
|
||||
toyota celica gt liftback,21.1,4,134,95,2515,14.8,1978-01-01,Japan
|
||||
plymouth sapporo,23.2,4,156,105,2745,16.7,1978-01-01,USA
|
||||
oldsmobile starfire sx,23.8,4,151,85,2855,17.6,1978-01-01,USA
|
||||
datsun 200-sx,23.9,4,119,97,2405,14.9,1978-01-01,Japan
|
||||
audi 5000,20.3,5,131,103,2830,15.9,1978-01-01,Europe
|
||||
volvo 264gl,17,6,163,125,3140,13.6,1978-01-01,Europe
|
||||
saab 99gle,21.6,4,121,115,2795,15.7,1978-01-01,Europe
|
||||
peugeot 604sl,16.2,6,163,133,3410,15.8,1978-01-01,Europe
|
||||
volkswagen scirocco,31.5,4,89,71,1990,14.9,1978-01-01,Europe
|
||||
honda Accelerationord lx,29.5,4,98,68,2135,16.6,1978-01-01,Japan
|
||||
pontiac lemans v6,21.5,6,231,115,3245,15.4,1979-01-01,USA
|
||||
mercury zephyr 6,19.8,6,200,85,2990,18.2,1979-01-01,USA
|
||||
ford fairmont 4,22.3,4,140,88,2890,17.3,1979-01-01,USA
|
||||
amc concord dl 6,20.2,6,232,90,3265,18.2,1979-01-01,USA
|
||||
dodge aspen 6,20.6,6,225,110,3360,16.6,1979-01-01,USA
|
||||
chevrolet caprice classic,17,8,305,130,3840,15.4,1979-01-01,USA
|
||||
ford ltd landau,17.6,8,302,129,3725,13.4,1979-01-01,USA
|
||||
mercury grand marquis,16.5,8,351,138,3955,13.2,1979-01-01,USA
|
||||
dodge st. regis,18.2,8,318,135,3830,15.2,1979-01-01,USA
|
||||
buick estate wagon (sw),16.9,8,350,155,4360,14.9,1979-01-01,USA
|
||||
ford country squire (sw),15.5,8,351,142,4054,14.3,1979-01-01,USA
|
||||
chevrolet malibu classic (sw),19.2,8,267,125,3605,15,1979-01-01,USA
|
||||
chrysler lebaron town @ country (sw),18.5,8,360,150,3940,13,1979-01-01,USA
|
||||
vw rabbit custom,31.9,4,89,71,1925,14,1979-01-01,Europe
|
||||
maxda glc deluxe,34.1,4,86,65,1975,15.2,1979-01-01,Japan
|
||||
dodge colt hatchback custom,35.7,4,98,80,1915,14.4,1979-01-01,USA
|
||||
amc spirit dl,27.4,4,121,80,2670,15,1979-01-01,USA
|
||||
mercedes benz 300d,25.4,5,183,77,3530,20.1,1979-01-01,Europe
|
||||
cadillac eldorado,23,8,350,125,3900,17.4,1979-01-01,USA
|
||||
peugeot 504,27.2,4,141,71,3190,24.8,1979-01-01,Europe
|
||||
oldsmobile cutlass salon brougham,23.9,8,260,90,3420,22.2,1979-01-01,USA
|
||||
plymouth horizon,34.2,4,105,70,2200,13.2,1979-01-01,USA
|
||||
plymouth horizon tc3,34.5,4,105,70,2150,14.9,1979-01-01,USA
|
||||
datsun 210,31.8,4,85,65,2020,19.2,1979-01-01,Japan
|
||||
fiat strada custom,37.3,4,91,69,2130,14.7,1979-01-01,Europe
|
||||
buick skylark limited,28.4,4,151,90,2670,16,1979-01-01,USA
|
||||
chevrolet citation,28.8,6,173,115,2595,11.3,1979-01-01,USA
|
||||
oldsmobile omega brougham,26.8,6,173,115,2700,12.9,1979-01-01,USA
|
||||
pontiac phoenix,33.5,4,151,90,2556,13.2,1979-01-01,USA
|
||||
vw rabbit,41.5,4,98,76,2144,14.7,1980-01-01,Europe
|
||||
toyota corolla tercel,38.1,4,89,60,1968,18.8,1980-01-01,Japan
|
||||
chevrolet chevette,32.1,4,98,70,2120,15.5,1980-01-01,USA
|
||||
datsun 310,37.2,4,86,65,2019,16.4,1980-01-01,Japan
|
||||
chevrolet citation,28,4,151,90,2678,16.5,1980-01-01,USA
|
||||
ford fairmont,26.4,4,140,88,2870,18.1,1980-01-01,USA
|
||||
amc concord,24.3,4,151,90,3003,20.1,1980-01-01,USA
|
||||
dodge aspen,19.1,6,225,90,3381,18.7,1980-01-01,USA
|
||||
audi 4000,34.3,4,97,78,2188,15.8,1980-01-01,Europe
|
||||
toyota corona liftback,29.8,4,134,90,2711,15.5,1980-01-01,Japan
|
||||
mazda 626,31.3,4,120,75,2542,17.5,1980-01-01,Japan
|
||||
datsun 510 hatchback,37,4,119,92,2434,15,1980-01-01,Japan
|
||||
toyota corolla,32.2,4,108,75,2265,15.2,1980-01-01,Japan
|
||||
mazda glc,46.6,4,86,65,2110,17.9,1980-01-01,Japan
|
||||
dodge colt,27.9,4,156,105,2800,14.4,1980-01-01,USA
|
||||
datsun 210,40.8,4,85,65,2110,19.2,1980-01-01,Japan
|
||||
vw rabbit c (diesel),44.3,4,90,48,2085,21.7,1980-01-01,Europe
|
||||
vw dasher (diesel),43.4,4,90,48,2335,23.7,1980-01-01,Europe
|
||||
audi 5000s (diesel),36.4,5,121,67,2950,19.9,1980-01-01,Europe
|
||||
mercedes-benz 240d,30,4,146,67,3250,21.8,1980-01-01,Europe
|
||||
honda civic 1500 gl,44.6,4,91,67,1850,13.8,1980-01-01,Japan
|
||||
renault lecar deluxe,40.9,4,85,,1835,17.3,1980-01-01,Europe
|
||||
subaru dl,33.8,4,97,67,2145,18,1980-01-01,Japan
|
||||
vokswagen rabbit,29.8,4,89,62,1845,15.3,1980-01-01,Europe
|
||||
datsun 280-zx,32.7,6,168,132,2910,11.4,1980-01-01,Japan
|
||||
mazda rx-7 gs,23.7,3,70,100,2420,12.5,1980-01-01,Japan
|
||||
triumph tr7 coupe,35,4,122,88,2500,15.1,1980-01-01,Europe
|
||||
ford mustang cobra,23.6,4,140,,2905,14.3,1980-01-01,USA
|
||||
honda Accelerationord,32.4,4,107,72,2290,17,1980-01-01,Japan
|
||||
plymouth reliant,27.2,4,135,84,2490,15.7,1982-01-01,USA
|
||||
buick skylark,26.6,4,151,84,2635,16.4,1982-01-01,USA
|
||||
dodge aries wagon (sw),25.8,4,156,92,2620,14.4,1982-01-01,USA
|
||||
chevrolet citation,23.5,6,173,110,2725,12.6,1982-01-01,USA
|
||||
plymouth reliant,30,4,135,84,2385,12.9,1982-01-01,USA
|
||||
toyota starlet,39.1,4,79,58,1755,16.9,1982-01-01,Japan
|
||||
plymouth champ,39,4,86,64,1875,16.4,1982-01-01,USA
|
||||
honda civic 1300,35.1,4,81,60,1760,16.1,1982-01-01,Japan
|
||||
subaru,32.3,4,97,67,2065,17.8,1982-01-01,Japan
|
||||
datsun 210,37,4,85,65,1975,19.4,1982-01-01,Japan
|
||||
toyota tercel,37.7,4,89,62,2050,17.3,1982-01-01,Japan
|
||||
mazda glc 4,34.1,4,91,68,1985,16,1982-01-01,Japan
|
||||
plymouth horizon 4,34.7,4,105,63,2215,14.9,1982-01-01,USA
|
||||
ford escort 4w,34.4,4,98,65,2045,16.2,1982-01-01,USA
|
||||
ford escort 2h,29.9,4,98,65,2380,20.7,1982-01-01,USA
|
||||
volkswagen jetta,33,4,105,74,2190,14.2,1982-01-01,Europe
|
||||
renault 18i,34.5,4,100,,2320,15.8,1982-01-01,Europe
|
||||
honda prelude,33.7,4,107,75,2210,14.4,1982-01-01,Japan
|
||||
toyota corolla,32.4,4,108,75,2350,16.8,1982-01-01,Japan
|
||||
datsun 200sx,32.9,4,119,100,2615,14.8,1982-01-01,Japan
|
||||
mazda 626,31.6,4,120,74,2635,18.3,1982-01-01,Japan
|
||||
peugeot 505s turbo diesel,28.1,4,141,80,3230,20.4,1982-01-01,Europe
|
||||
saab 900s,,4,121,110,2800,15.4,1982-01-01,Europe
|
||||
volvo diesel,30.7,6,145,76,3160,19.6,1982-01-01,Europe
|
||||
toyota cressida,25.4,6,168,116,2900,12.6,1982-01-01,Japan
|
||||
datsun 810 maxima,24.2,6,146,120,2930,13.8,1982-01-01,Japan
|
||||
buick century,22.4,6,231,110,3415,15.8,1982-01-01,USA
|
||||
oldsmobile cutlass ls,26.6,8,350,105,3725,19,1982-01-01,USA
|
||||
ford granada gl,20.2,6,200,88,3060,17.1,1982-01-01,USA
|
||||
chrysler lebaron salon,17.6,6,225,85,3465,16.6,1982-01-01,USA
|
||||
chevrolet cavalier,28,4,112,88,2605,19.6,1982-01-01,USA
|
||||
chevrolet cavalier wagon,27,4,112,88,2640,18.6,1982-01-01,USA
|
||||
chevrolet cavalier 2-door,34,4,112,88,2395,18,1982-01-01,USA
|
||||
pontiac j2000 se hatchback,31,4,112,85,2575,16.2,1982-01-01,USA
|
||||
dodge aries se,29,4,135,84,2525,16,1982-01-01,USA
|
||||
pontiac phoenix,27,4,151,90,2735,18,1982-01-01,USA
|
||||
ford fairmont futura,24,4,140,92,2865,16.4,1982-01-01,USA
|
||||
amc concord dl,23,4,151,,3035,20.5,1982-01-01,USA
|
||||
volkswagen rabbit l,36,4,105,74,1980,15.3,1982-01-01,Europe
|
||||
mazda glc custom l,37,4,91,68,2025,18.2,1982-01-01,Japan
|
||||
mazda glc custom,31,4,91,68,1970,17.6,1982-01-01,Japan
|
||||
plymouth horizon miser,38,4,105,63,2125,14.7,1982-01-01,USA
|
||||
mercury lynx l,36,4,98,70,2125,17.3,1982-01-01,USA
|
||||
nissan stanza xe,36,4,120,88,2160,14.5,1982-01-01,Japan
|
||||
honda Accelerationord,36,4,107,75,2205,14.5,1982-01-01,Japan
|
||||
toyota corolla,34,4,108,70,2245,16.9,1982-01-01,Japan
|
||||
honda civic,38,4,91,67,1965,15,1982-01-01,Japan
|
||||
honda civic (auto),32,4,91,67,1965,15.7,1982-01-01,Japan
|
||||
datsun 310 gx,38,4,91,67,1995,16.2,1982-01-01,Japan
|
||||
buick century limited,25,6,181,110,2945,16.4,1982-01-01,USA
|
||||
oldsmobile cutlass ciera (diesel),38,6,262,85,3015,17,1982-01-01,USA
|
||||
chrysler lebaron medallion,26,4,156,92,2585,14.5,1982-01-01,USA
|
||||
ford granada l,22,6,232,112,2835,14.7,1982-01-01,USA
|
||||
toyota celica gt,32,4,144,96,2665,13.9,1982-01-01,Japan
|
||||
dodge charger 2.2,36,4,135,84,2370,13,1982-01-01,USA
|
||||
chevrolet camaro,27,4,151,90,2950,17.3,1982-01-01,USA
|
||||
ford mustang gl,27,4,140,86,2790,15.6,1982-01-01,USA
|
||||
vw pickup,44,4,97,52,2130,24.6,1982-01-01,Europe
|
||||
dodge rampage,32,4,135,84,2295,11.6,1982-01-01,USA
|
||||
ford ranger,28,4,120,79,2625,18.6,1982-01-01,USA
|
||||
chevy s-10,31,4,119,82,2720,19.4,1982-01-01,USA
|
|
BIN
docz/static/loadofsheet/expected.png
Normal file
BIN
docz/static/loadofsheet/expected.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 140 KiB |
70
docz/static/loadofsheet/loadofsheet.mjs
Normal file
70
docz/static/loadofsheet/loadofsheet.mjs
Normal file
@ -0,0 +1,70 @@
|
||||
import { Document } from "@langchain/core/documents";
|
||||
import { BufferLoader } from "langchain/document_loaders/fs/buffer";
|
||||
import { read, utils } from "xlsx";
|
||||
|
||||
/**
|
||||
* Document loader that uses SheetJS to load documents.
|
||||
*
|
||||
* Each worksheet is parsed into an array of row objects using the SheetJS
|
||||
* `sheet_to_json` method and projected to a `Document`. Metadata includes
|
||||
* original sheet name, row data, and row index
|
||||
*/
|
||||
export default class LoadOfSheet extends BufferLoader {
|
||||
/** @type {import("langchain/chains/query_constructor").AttributeInfo[]} */
|
||||
attributes = [];
|
||||
|
||||
/**
|
||||
* Document loader that uses SheetJS to load documents.
|
||||
*
|
||||
* @param {string|Blob} filePathOrBlob Source Data
|
||||
*/
|
||||
constructor(filePathOrBlob) {
|
||||
super(filePathOrBlob);
|
||||
this.attributes = [];
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse document
|
||||
*
|
||||
* NOTE: column labels in multiple sheets are not disambiguated!
|
||||
*
|
||||
* @param {Buffer} raw Raw data Buffer
|
||||
* @param {Document["metadata"]} metadata Document metadata
|
||||
* @returns {Promise<Document[]>} Array of Documents
|
||||
*/
|
||||
async parse(raw, metadata) {
|
||||
/** @type {Document[]} */
|
||||
const result = [];
|
||||
|
||||
this.attributes = [
|
||||
{ name: "worksheet", description: "Sheet or Worksheet Name", type: "string" },
|
||||
{ name: "rowNum", description: "Row index", type: "number" }
|
||||
];
|
||||
|
||||
const wb = read(raw, {type: "buffer", WTF:1});
|
||||
for(let name of wb.SheetNames) {
|
||||
const fields = {};
|
||||
const ws = wb.Sheets[name];
|
||||
if(!ws) return;
|
||||
|
||||
const aoo = utils.sheet_to_json(ws);
|
||||
aoo.forEach((row, idx) => {
|
||||
result.push({
|
||||
pageContent: "Row " + (idx + 1) + " has the following content: \n" + Object.entries(row).map(kv => `- ${kv[0]}: ${kv[1]}`).join("\n") + "\n",
|
||||
metadata: {
|
||||
worksheet: name,
|
||||
rowNum: row["__rowNum__"],
|
||||
...metadata,
|
||||
...row
|
||||
}
|
||||
});
|
||||
Object.entries(row).forEach(([k,v]) => { if(v != null) (fields[k] || (fields[k] = {}))[v instanceof Date ? "date" : typeof v] = true } );
|
||||
});
|
||||
Object.entries(fields).forEach(([k,v]) => this.attributes.push({
|
||||
name: k, description: k, type: Object.keys(v).join(" or ")
|
||||
}));
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
};
|
39
docz/static/loadofsheet/query.mjs
Normal file
39
docz/static/loadofsheet/query.mjs
Normal file
@ -0,0 +1,39 @@
|
||||
import { existsSync } from 'fs';
|
||||
import { ChatOllama } from "@langchain/community/chat_models/ollama";
|
||||
import { OllamaEmbeddings } from "@langchain/community/embeddings/ollama"
|
||||
import { HNSWLib } from "@langchain/community/vectorstores/hnswlib";
|
||||
import { SelfQueryRetriever } from "langchain/retrievers/self_query";
|
||||
import { FunctionalTranslator } from "@langchain/core/structured_query";
|
||||
|
||||
import LoadOfSheet from "./loadofsheet.mjs";
|
||||
|
||||
const modelName = "llama3-chatqa:8b-v1.5-q8_0";
|
||||
|
||||
const model = new ChatOllama({ baseUrl: "http://localhost:11434", model: modelName });
|
||||
const embeddings = new OllamaEmbeddings({model: modelName});
|
||||
|
||||
const loader = new LoadOfSheet("./cd.xls");
|
||||
const docs = await loader.load();
|
||||
|
||||
const vectorstore = await (async() => {
|
||||
if(!existsSync("store/hnswlib.index")) {
|
||||
const vectorstore = await HNSWLib.fromDocuments(docs, embeddings);
|
||||
await vectorstore.save("store");
|
||||
return vectorstore;
|
||||
}
|
||||
return await HNSWLib.load("store", embeddings);
|
||||
})();
|
||||
|
||||
const selfQueryRetriever = SelfQueryRetriever.fromLLM({
|
||||
llm: model,
|
||||
vectorStore: vectorstore,
|
||||
documentContents: "Data rows from a worksheet",
|
||||
attributeInfo: loader.attributes,
|
||||
structuredQueryTranslator: new FunctionalTranslator(),
|
||||
searchParams: { k: 1024 } // default is 4
|
||||
});
|
||||
|
||||
const res = await selfQueryRetriever.invoke(
|
||||
"Which rows have over 40 miles per gallon?"
|
||||
);
|
||||
res.forEach(({metadata}) => { console.log({ Name: metadata.Name, MPG: metadata.Miles_per_Gallon }); });
|
Loading…
Reference in New Issue
Block a user