docs.sheetjs.com/docz/docs/03-demos/23-data/17-mariadb.md

403 lines
11 KiB
Markdown
Raw Normal View History

2023-10-30 23:28:40 +00:00
---
2023-12-05 03:46:54 +00:00
title: Sheets with MariaDB and MySQL
sidebar_label: MariaDB / MySQL
2024-03-18 08:24:41 +00:00
pagination_prev: demos/cli/index
2023-10-30 23:28:40 +00:00
pagination_next: demos/local/index
sidebar_custom_props:
sql: true
---
import current from '/version.js';
import CodeBlock from '@theme/CodeBlock';
2023-12-05 03:46:54 +00:00
[MariaDB](https://mariadb.com/) is an open source object-relational database
system compatible with MySQL.
2023-10-30 23:28:40 +00:00
[SheetJS](https://sheetjs.com) is a JavaScript library for reading and writing
data from spreadsheets.
2023-12-05 03:46:54 +00:00
This demo uses SheetJS to exchange data between spreadsheets and MariaDB
2023-11-16 04:20:57 +00:00
databases. We'll explore how to save tables from a database to spreadsheets and
how to add data from spreadsheets into a database.
2023-10-30 23:28:40 +00:00
:::caution pass
2023-12-05 03:46:54 +00:00
**It is strongly recommended to use MariaDB with a query builder or ORM.**
2023-10-30 23:28:40 +00:00
While it is possible to generate SQL statements directly, there are many subtle
details and pitfalls. Battle-tested solutions generally provide mitigations
against SQL injection and other vulnerabilities.
:::
2023-12-05 03:46:54 +00:00
:::note Tested Deployments
2023-10-30 23:28:40 +00:00
2023-12-05 03:46:54 +00:00
This demo was tested in the following environments:
| MariaDB | Connector Library | Date |
|:---------|:-------------------|:-----------|
| `11.2.2` | `mysql2` (`3.6.5`) | 2023-12-04 |
2023-10-30 23:28:40 +00:00
:::
## Integration Details
The [SheetJS NodeJS module](/docs/getting-started/installation/nodejs) can be
2023-12-05 03:46:54 +00:00
loaded in NodeJS scripts that connect to MariaDB and MySQL databases.
2023-10-30 23:28:40 +00:00
2023-12-05 03:46:54 +00:00
This demo uses the `mysql2` connector module[^1], but the same mechanics apply
to other MariaDB and MySQL libraries.
2023-10-30 23:28:40 +00:00
### Exporting Data
2023-12-05 03:46:54 +00:00
`Connection#execute` returns a Promise that resolves to a result array. The
first entry of the result is an array of objects.
2023-10-30 23:28:40 +00:00
The SheetJS `json_to_sheet` method[^2] can generate a worksheet object[^3] from
the array of objects:
```js
2023-12-05 03:46:54 +00:00
const mysql = require("mysql2/promise"), XLSX = require("xlsx");
const conn = await mysql.createConnection({
database: "SheetJSMariaDB",
/* ... other options ... */
});
2023-10-30 23:28:40 +00:00
const table_name = "Tabeller1"; // name of table
/* fetch all data from specified table */
2023-12-05 03:46:54 +00:00
const [rows, fields] = await conn.execute(`SELECT * FROM ${mysql.escapeId(table_name)}`);
2023-10-30 23:28:40 +00:00
/* generate a SheetJS worksheet object from the data */
2023-12-05 03:46:54 +00:00
const worksheet = XLSX.utils.json_to_sheet(rows);
2023-10-30 23:28:40 +00:00
```
A workbook object can be built from the worksheet using utility functions[^4].
The workbook can be exported using the SheetJS `writeFile` method[^5]:
```js
/* create a new workbook and add the worksheet */
const wb = XLSX.utils.book_new();
XLSX.utils.book_append_sheet(wb, worksheet, "Sheet1");
/* export workbook to XLSX */
2023-12-05 03:46:54 +00:00
XLSX.writeFile(wb, "SheetJSMariaDBExport.xlsx");
2023-10-30 23:28:40 +00:00
```
### Importing Data
The SheetJS `sheet_to_json` function[^6] takes a worksheet object and generates
an array of objects.
Queries must be manually generated from the objects. Assuming the field names
in the object match the column headers, a loop can generate `INSERT` queries.
:::danger pass
2023-10-30 23:28:40 +00:00
2023-12-05 03:46:54 +00:00
**MariaDB does not allow parameterized queries with variable column names**
2023-10-30 23:28:40 +00:00
```sql
INSERT INTO table_name (?) VALUES (?);
-- ---------------------^ variable column names are not valid
```
Queries are generated manually. To help prevent SQL injection vulnerabilities,
2023-12-05 03:46:54 +00:00
the undocumented `escapeId` method [^7] escapes identifiers and fields.
2023-10-30 23:28:40 +00:00
:::
```js
/* generate an array of arrays from the worksheet */
const aoo = XLSX.utils.sheet_to_json(ws);
const table_name = "Blatte1"; // name of table
/* loop through the data rows */
for(let row of aoo) {
2023-12-05 03:46:54 +00:00
/* generate INSERT column names and values */
2023-10-30 23:28:40 +00:00
const ent = Object.entries(row);
2023-12-05 03:46:54 +00:00
const Istr = ent.map(e => I(e[0])).join(", ");
const Vstr = ent.map(e => E(e[1])).join(", ");
2023-10-30 23:28:40 +00:00
/* execute INSERT statement */
2023-12-05 03:46:54 +00:00
await conn.execute(`INSERT INTO ${I(table_name)} (${Istr}) VALUES (${Vstr})`);
2023-10-30 23:28:40 +00:00
}
```
### Creating a Table
The array of objects can be scanned to determine column names and types. With
the names and types, a `CREATE TABLE` query can be written.
<details>
<summary><b>Implementation Details</b> (click to show)</summary>
2023-10-30 23:28:40 +00:00
2023-12-05 03:46:54 +00:00
The `aoo_to_mariadb_table` function:
2023-10-30 23:28:40 +00:00
- scans each row object to determine column names and types
- drops and creates a new table with the determined column names and types
- loads the entire dataset into the new table
```js
2023-12-05 03:46:54 +00:00
/* create table and load data given an array of objects and a mysql2 connection */
async function aoo_to_mariadb_table(conn, aoo, table_name) {
2023-10-30 23:28:40 +00:00
/* define types that can be converted (e.g. boolean can be stored in float) */
2023-12-05 03:46:54 +00:00
const T_FLOAT = ["DOUBLE", "BOOLEAN"];
const T_BOOL = ["BOOLEAN"];
2023-10-30 23:28:40 +00:00
/* types is a map from column headers to Knex schema column type */
const types = {};
/* names is an ordered list of the column header names */
const names = [];
/* loop across each row object */
aoo.forEach(row =>
/* Object.entries returns a row of [key, value] pairs */
Object.entries(row).forEach(([k,v]) => {
/* If this is first occurrence, mark unknown and append header to names */
if(!types[k]) { types[k] = ""; names.push(k); }
/* skip null and undefined values */
if(v == null) return;
/* check and resolve type */
switch(typeof v) {
/* change type if it is empty or can be stored in a float */
2023-12-05 03:46:54 +00:00
case "number": if(!types[k] || T_FLOAT.includes(types[k])) types[k] = "DOUBLE"; break;
2023-10-30 23:28:40 +00:00
/* change type if it is empty or can be stored in a boolean */
2023-12-05 03:46:54 +00:00
case "boolean": if(!types[k] || T_BOOL.includes(types[k])) types[k] = "BOOLEAN"; break;
2023-10-30 23:28:40 +00:00
/* no other type can hold strings */
2023-12-05 03:46:54 +00:00
case "string": types[k] = "TEXT"; break;
default: types[k] = "TEXT"; break;
2023-10-30 23:28:40 +00:00
}
})
);
2023-12-05 03:46:54 +00:00
const I = (id) => mysql.escapeId(id), E = (d) => mysql.escape(d);
2023-10-30 23:28:40 +00:00
/* Delete table if it exists in the DB */
2023-12-05 03:46:54 +00:00
await conn.execute(`DROP TABLE IF EXISTS ${I(table_name)};`);
2023-10-30 23:28:40 +00:00
/* Create table */
{
2023-12-05 03:46:54 +00:00
const Istr = Object.entries(types).map(e => `${I(e[0])} ${e[1]}`).join(", ");
await conn.execute(`CREATE TABLE ${I(table_name)} (${Istr});`);
2023-10-30 23:28:40 +00:00
}
/* Insert each row */
for(let row of aoo) {
const ent = Object.entries(row);
2023-12-05 03:46:54 +00:00
const Istr = ent.map(e => I(e[0])).join(", ");
const Vstr = ent.map(e => E(e[1])).join(", ");
await conn.execute(`INSERT INTO ${I(table_name)} (${Istr}) VALUES (${Vstr})`);
2023-10-30 23:28:40 +00:00
}
2023-12-05 03:46:54 +00:00
return conn;
2023-10-30 23:28:40 +00:00
}
```
</details>
## Complete Example
2023-12-05 03:46:54 +00:00
0) Install and start the MariaDB server.
2023-10-30 23:28:40 +00:00
<details>
<summary><b>Installation Notes</b> (click to show)</summary>
2023-10-30 23:28:40 +00:00
2023-12-05 03:46:54 +00:00
On macOS, install the `mariadb` formula with Homebrew:
2023-10-30 23:28:40 +00:00
```bash
2023-12-05 03:46:54 +00:00
brew install mariadb
2023-10-30 23:28:40 +00:00
```
2023-11-16 04:20:57 +00:00
The last few lines of the installer explain how to start the database:
2023-10-30 23:28:40 +00:00
```text
Or, if you don't want/need a background service you can just run:
// highlight-next-line
2023-12-05 03:46:54 +00:00
/usr/local/opt/mariadb/bin/mysqld_safe --datadir\=/usr/local/var/mysql
2023-10-30 23:28:40 +00:00
```
Run the command to start a local database instance.
</details>
2023-12-05 03:46:54 +00:00
1) Drop any existing database with the name `SheetJSMariaDB`:
2023-10-30 23:28:40 +00:00
```bash
2023-12-05 03:46:54 +00:00
mysql -e 'drop database if exists SheetJSMariaDB;'
2023-10-30 23:28:40 +00:00
```
:::info pass
If the server is running elsewhere, or if the username is different from the
current user, command-line flags can override the defaults.
| Option | Explanation
|:--------------|:--------------------------|
| `-h HOSTNAME` | Name of the server |
2023-12-05 03:46:54 +00:00
| `-P PORT` | specifies the port number |
2023-10-30 23:28:40 +00:00
| `-U USERNAME` | specifies the username |
2023-12-05 03:46:54 +00:00
| `-p PASSWORD` | specifies the password |
2023-10-30 23:28:40 +00:00
:::
2023-12-05 03:46:54 +00:00
2) Create an empty `SheetJSMariaDB` database:
2023-10-30 23:28:40 +00:00
```bash
2023-12-05 03:46:54 +00:00
mysql -e 'create database SheetJSMariaDB;'
2023-10-30 23:28:40 +00:00
```
### Connector Test
3) Create a project folder:
```bash
2023-12-05 03:46:54 +00:00
mkdir sheetjs-mariadb
cd sheetjs-mariadb
2023-10-30 23:28:40 +00:00
npm init -y
```
2023-12-05 03:46:54 +00:00
4) Install the `mysql2` connector module:
2023-10-30 23:28:40 +00:00
```bash
2023-12-05 03:46:54 +00:00
npm i --save mysql2@3.6.5
2023-10-30 23:28:40 +00:00
```
2023-12-05 03:46:54 +00:00
5) Save the following example codeblock to `MariaDBTest.js`:
2023-10-30 23:28:40 +00:00
2023-12-05 03:46:54 +00:00
```js title="MariaDBTest.js"
const mysql = require("mysql2/promise");
(async() => {
const conn = await mysql.createConnection({
database:"SheetJSMariaDB",
2023-10-30 23:28:40 +00:00
// highlight-start
host: "127.0.0.1", // localhost
2023-12-05 03:46:54 +00:00
port: 3306,
user: "sheetjs",
2023-10-30 23:28:40 +00:00
//password: ""
// highlight-end
});
2023-12-05 03:46:54 +00:00
const [rows, fields] = await conn.execute('SELECT ? as message', ['Hello world!']);
console.log(rows[0].message); // Hello world!
await conn.end();
2023-10-30 23:28:40 +00:00
})();
```
2023-12-05 03:46:54 +00:00
6) Edit the new `MariaDBTest.js` script and modify the highlighted lines from
the codeblock to reflect the database deployment settings.
2023-10-30 23:28:40 +00:00
2023-12-05 03:46:54 +00:00
- Set `user` to the username (it is almost certainly not `"sheetjs"`)
2023-10-30 23:28:40 +00:00
- If the server is not running on your computer, set `host` and `port` to the
correct host name and port number.
2023-12-05 03:46:54 +00:00
- If the server expects a password, uncomment the `password` line and replace
the value with the password.
2023-10-30 23:28:40 +00:00
7) Run the script:
```bash
2023-12-05 03:46:54 +00:00
node MariaDBTest.js
2023-10-30 23:28:40 +00:00
```
It should print `Hello world!`
:::caution pass
If the output is not `Hello world!` or if there is an error, please report the
2024-01-17 20:22:38 +00:00
issue to the `mysql2` connector project for further diagnosis.
2023-10-30 23:28:40 +00:00
:::
### Add SheetJS
8) Install dependencies:
<CodeBlock language="bash">{`\
2023-12-05 03:46:54 +00:00
npm i --save https://cdn.sheetjs.com/xlsx-${current}/xlsx-${current}.tgz`}
2023-10-30 23:28:40 +00:00
</CodeBlock>
2023-12-05 03:46:54 +00:00
9) Download [`SheetJSMariaDB.js`](pathname:///mariadb/SheetJSMariaDB.js):
2023-10-30 23:28:40 +00:00
```bash
2023-12-05 03:46:54 +00:00
curl -LO https://docs.sheetjs.com/mariadb/SheetJSMariaDB.js
2023-10-30 23:28:40 +00:00
```
This script will:
- read and parse the test file `pres.numbers`
2023-12-05 03:46:54 +00:00
- create a connection to the `SheetJSMariaDB` database on a local MariaDB server
2023-10-30 23:28:40 +00:00
- load data from the first worksheet into a table with name `Presidents`
- disconnect and reconnect to the database
- dump data from the table `Presidents`
2023-12-05 03:46:54 +00:00
- export the dataset to `SheetJSMariaDB.xlsx`
2023-10-30 23:28:40 +00:00
2023-12-05 03:46:54 +00:00
10) Edit the `SheetJSMariaDB.js` script.
2023-10-30 23:28:40 +00:00
The script defines an `opts` object:
2023-12-05 03:46:54 +00:00
```js title="SheetJSMariaDB.js (configuration lines)"
2023-10-30 23:28:40 +00:00
const XLSX = require("xlsx");
const opts = {
2023-12-05 03:46:54 +00:00
database:"SheetJSMariaDB",
2023-10-30 23:28:40 +00:00
// highlight-start
host: "127.0.0.1", // localhost
2023-12-05 03:46:54 +00:00
port: 3306,
user: "sheetjs",
2023-10-30 23:28:40 +00:00
//password: ""
// highlight-end
};
```
2023-12-05 03:46:54 +00:00
Modify the highlighted lines to reflect the database deployment settings.
- Set `user` to the username (it is almost certainly not `"sheetjs"`)
2023-10-30 23:28:40 +00:00
- If the server is not running on your computer, set `host` and `port` to the
correct host name and port number.
2023-12-05 03:46:54 +00:00
- If the server expects a password, uncomment the `password` line and replace
the value with the password.
2023-10-30 23:28:40 +00:00
2023-12-05 03:46:54 +00:00
11) Fetch the example file [`pres.numbers`](https://sheetjs.com/pres.numbers):
2023-10-30 23:28:40 +00:00
```bash
2023-12-05 03:46:54 +00:00
curl -L -O https://sheetjs.com/pres.numbers
2023-10-30 23:28:40 +00:00
```
2023-12-05 03:46:54 +00:00
12) Run the script:
2023-10-30 23:28:40 +00:00
```bash
2023-12-05 03:46:54 +00:00
node SheetJSMariaDB.js
2023-10-30 23:28:40 +00:00
```
2023-12-05 03:46:54 +00:00
13) Verify the result:
2023-10-30 23:28:40 +00:00
2023-12-05 03:46:54 +00:00
- `SheetJSMariaDBExport.xlsx` can be opened in a spreadsheet app or tested in the terminal
2023-10-30 23:28:40 +00:00
```bash
2023-12-05 03:46:54 +00:00
npx xlsx-cli SheetJSMariaDBExport.xlsx
2023-10-30 23:28:40 +00:00
```
2023-12-05 03:46:54 +00:00
- The database server can be queried using the `mysql` command line tool.
2023-10-30 23:28:40 +00:00
2023-12-05 03:46:54 +00:00
If the server is running locally, the command will be:
2023-10-30 23:28:40 +00:00
2023-12-05 03:46:54 +00:00
```bash
mysql -D SheetJSMariaDB -e 'SELECT * FROM `Presidents`;'
```
2023-10-30 23:28:40 +00:00
2024-01-17 20:22:38 +00:00
[^1]: See [the official `mysql2` website](https://sidorares.github.io/node-mysql2/docs) for more info.
2023-10-30 23:28:40 +00:00
[^2]: See [`json_to_sheet` in "Utilities"](/docs/api/utilities/array#array-of-objects-input)
[^3]: See ["Sheet Objects"](/docs/csf/sheet) in "SheetJS Data Model" for more details.
[^4]: See ["Workbook Helpers" in "Utilities"](/docs/api/utilities/wb) for details on `book_new` and `book_append_sheet`.
[^5]: See [`writeFile` in "Writing Files"](/docs/api/write-options)
[^6]: See [`sheet_to_json` in "Utilities"](/docs/api/utilities/array#array-output)
2023-12-05 03:46:54 +00:00
[^7]: The `mysql2` connector library `escapeId` method is not mentioned in the documentation but is present in the TypeScript definitions.