docs.sheetjs.com/docz/docs/03-demos/07-data/26-redis.md

284 lines
8.5 KiB
Markdown
Raw Normal View History

2023-02-24 07:46:48 +00:00
---
2023-08-23 08:05:30 +00:00
title: Redis Databases Wrapped in Sheets
sidebar_label: Redis
description: Store complex datasets in Redis. Seamlessly save data to spreadsheets and read data from sheets using SheetJS. Enable Excel spreadsheet experts to update content.
2023-02-28 11:40:44 +00:00
pagination_prev: demos/desktop/index
pagination_next: demos/local/index
2023-02-24 07:46:48 +00:00
sidebar_custom_props:
type: nosql
---
2023-04-27 09:12:19 +00:00
import current from '/version.js';
2023-05-07 13:58:36 +00:00
import CodeBlock from '@theme/CodeBlock';
2023-04-27 09:12:19 +00:00
2023-08-23 08:05:30 +00:00
[Redis](https://redis.io/) is a open source in-memory data store. It is capable
of storing sets, lists and other simple data structures.
[SheetJS](https://sheetjs.com) is a JavaScript library for reading and writing
data from spreadsheets.
This demo defines a schema for storing Redis databases in spreadsheets. We'll
explore how to use SheetJS and Redis NodeJS connector modules to pull data
from XLSX files to a Redis database and to serialize a database to a workbook.
#### Overview
2023-02-24 07:46:48 +00:00
Redis has 5 core data types: "String", List", "Set", "Sorted Set", and "Hash".
Since the keys and values are limited to simple strings (and numbers), it is
possible to store complete databases in a single worksheet.
![SheetJSRedis.xlsx](pathname:///nosql/sheetjsredis.png)
## Integration Details
2023-08-23 08:05:30 +00:00
:::note pass
2023-02-24 07:46:48 +00:00
[`SheetJSRedis.mjs`](pathname:///nosql/SheetJSRedis.mjs) exports the methods:
- `redis_to_ws` creates a SheetJS worksheet by querying a redis client
- `ws_to_redis` creates an array of query objects from the SheetJS worksheet
:::
2023-08-23 08:05:30 +00:00
### Array of Arrays
The shared data representation is an "array of arrays"[^1]. Each array within
the structure corresponds to one row.
2023-02-24 07:46:48 +00:00
2023-08-23 08:05:30 +00:00
The Redis to SheetJS converter generates an array of arrays of the data by
running queries to fetch data from the database. The SheetJS `aoa_to_sheet` and
`sheet_add_aoa`[^2] methods build up worksheets from arrays of arrays. Once the
worksheet is created, it can be added to a SheetJS workbook object[^3] and
exported using `writeFile`[^4].
The SheetJS to Redis converter works in reverse. Workbook files are parsed with
the SheetJS `readFile` method[^5] and the desired worksheet is pulled from the
workbook object. An array of arrays can be created with the `sheet_to_json`[^6]
utility function. The data structure can be scanned to generate Redis queries.
### Appending Columns
Since the data is column-oriented, the goal is to add the data starting on the
first row of the column after the data.
To calculate the starting point for writing data, SheetJS provides `encode_cell`
and `decode_range` utility functions for working with addresses and ranges[^7].
The following snippet takes an array of arrays of values and writes the values
to a worksheet starting from the column after the worksheet range:
2023-02-24 07:46:48 +00:00
```js
function add_aoa_to_next_column(worksheet, aoa) {
/* get range of worksheet */
const range = XLSX.utils.decode_range(worksheet["!ref"])
/* the origin to write new data will start in the column after the range */
const origin = XLSX.utils.encode_cell({
r: 0, // start on first row
c: range.e.c + 1 // column after end
});
/* add data */
XLSX.utils.sheet_add_aoa(worksheet, aoa, { origin });
}
```
2023-08-23 08:05:30 +00:00
### Strings
2023-02-24 07:46:48 +00:00
Strings can be stored in a unified String table. The first column holds keys
and the second column holds values:
```
XXX| A | B |
---+---------+-------+
1 | Strings | |
2 | | |
3 | Hello | World |
4 | Sheet | JS |
```
The SheetJS array-of-arrays representation of the string table is an array of
2023-08-23 08:05:30 +00:00
key/value pairs.
The pairs can be generated from Redis by querying for all of the keys using the
`KEYS`[^8] method, testing if the corresponding value is a string using the
`TYPE`[^9] method, and fetching string values using the `GET`[^10] method:
2023-02-24 07:46:48 +00:00
```js
const aoa = ["Strings"]; aoa.length = 2; // [ "Strings", empty ]
const keys = await client.KEYS("*");
for(let key of keys) {
const type = await client.TYPE(key);
// highlight-start
if(type == "string") aoa.push([key, await client.GET(key)]);
// highlight-end
}
```
2023-08-23 08:05:30 +00:00
### Lists
2023-02-24 07:46:48 +00:00
Lists are unidimensional and can be stored in their own columns.
```
XXX| C |
---+---------+
1 | List |
2 | List1 |
3 | List1V1 |
4 | List1V2 |
```
The SheetJS array-of-arrays representation of lists is a column of values.
2023-08-23 08:05:30 +00:00
Redis `LRANGE`[^11] returns a simple array of values. `sheet_add_aoa` interprets
the result as one row, so the data should be transposed.
The code transposes the result with `values.map(v => [v])`.
2023-02-24 07:46:48 +00:00
```js
const values = await client.LRANGE(key, 0, -1);
const aoa = [ ["List"], [key] ].concat(values.map(v => [v]));
```
2023-08-23 08:05:30 +00:00
### Sets
2023-02-24 07:46:48 +00:00
Sets are unidimensional and can be stored in their own columns.
```
XXX| D |
---+-------+
1 | Set |
2 | Set1 |
3 | Set1A |
4 | Set1B |
```
The SheetJS array-of-arrays representation of sets is a column of values.
2023-08-23 08:05:30 +00:00
Redis `SMEMBERS`[^12] returns an array of values. `sheet_add_aoa` interprets the
result as one row, so the data should be transposed.
The code transposes the result with `values.map(v => [v])`.
2023-02-24 07:46:48 +00:00
```js
const values = await client.SMEMBERS(key);
const aoa = [ ["Set"], [key] ].concat(values.map(v => [v]));
```
2023-08-23 08:05:30 +00:00
### Sorted Sets
2023-02-24 07:46:48 +00:00
Sorted Sets have an associated score which can be stored in the second column.
```
XXX| E | F |
---+---------+---+
1 | Sorted | |
2 | ZSet1 | |
3 | Key1 | 1 |
4 | Key2 | 2 |
```
The SheetJS array-of-arrays representation is an array of key/score pairs.
2023-08-23 08:05:30 +00:00
`ZRANGE_WITHSCORES`[^13] returns an array of objects which can be reshaped.
2023-02-24 07:46:48 +00:00
```js
const values = await client.ZRANGE_WITHSCORES(key, 0, -1);
const aoa = [ ["Sorted"], [key] ].concat(values.map(v => [v.value, v.score]));
```
2023-08-23 08:05:30 +00:00
### Hashes
2023-02-24 07:46:48 +00:00
Hashes are stored like the string table, with key and value columns in order.
```
XXX| G | H |
---+-------+-------+
1 | Hash | |
2 | Hash1 | |
3 | Key1 | Val1 |
4 | Key2 | Val2 |
```
The SheetJS array-of-arrays representation is an array of key/value pairs.
2023-08-23 08:05:30 +00:00
`HGETALL`[^14] returns an object which can be converted using `Object.entries`:
2023-02-24 07:46:48 +00:00
```js
const values = await client.HGETALL(key);
const aoa = [ ["Hash"], [key] ].concat(Object.entries(values));
```
## Complete Example
:::note
2023-08-23 08:05:30 +00:00
This demo was last tested on 2023 August 22 with Redis 7.2.0, Redis connector
module 4.6.7 and NodeJS 20.5.1.
2023-02-24 07:46:48 +00:00
:::
2023-09-19 19:08:29 +00:00
:::warning pass
2023-02-24 07:46:48 +00:00
The most recent version of the `redis` node module does not work with most
versions of NodeJS. It is "ESM-only", requiring NodeJS 18 or later. As a result,
2023-05-11 06:17:10 +00:00
this demo also requires NodeJS version 18 or later.
2023-02-24 07:46:48 +00:00
:::
2023-05-11 06:17:10 +00:00
0) Set up and start a local Redis server.
:::note
2023-08-23 08:05:30 +00:00
This demo was last tested on macOS. Redis was installed with:
2023-05-11 06:17:10 +00:00
```bash
2023-08-23 08:05:30 +00:00
brew install redis
2023-05-11 06:17:10 +00:00
```
The following command started the server process:
2023-02-24 07:46:48 +00:00
```bash
2023-05-11 06:17:10 +00:00
/usr/local/opt/redis/bin/redis-server /usr/local/etc/redis.conf
2023-02-24 07:46:48 +00:00
```
2023-05-11 06:17:10 +00:00
:::
2023-02-24 07:46:48 +00:00
1) Download the following scripts:
- [`SheetJSRedis.mjs`](pathname:///nosql/SheetJSRedis.mjs)
- [`SheetJSRedisTest.mjs`](pathname:///nosql/SheetJSRedisTest.mjs)
```bash
curl -LO https://docs.sheetjs.com/nosql/SheetJSRedis.mjs
curl -LO https://docs.sheetjs.com/nosql/SheetJSRedisTest.mjs
```
2023-05-11 06:17:10 +00:00
2) Install dependencies:
2023-02-24 07:46:48 +00:00
2023-05-07 13:58:36 +00:00
<CodeBlock language="bash">{`\
2023-08-23 08:05:30 +00:00
npm i --save https://cdn.sheetjs.com/xlsx-${current}/xlsx-${current}.tgz redis@4.6.7`}
2023-05-07 13:58:36 +00:00
</CodeBlock>
2023-02-24 07:46:48 +00:00
2023-05-11 06:17:10 +00:00
3) Run the test script:
```bash
node SheetJSRedisTest.mjs
```
2023-02-24 07:46:48 +00:00
Inspect the output and compare with the data in `SheetJSRedisTest.mjs`.
Open `SheetJSRedis.xlsx` and verify the columns have the correct data
2023-08-23 08:05:30 +00:00
[^1]: See ["Array of Arrays" in "Utility Functions"](/docs/api/utilities/array#array-of-arrays)
[^2]: See ["Array of Arrays Input" in "Utility Functions"](/docs/api/utilities/array#array-of-arrays-input).
[^3]: See ["Workbook Helpers" in "Utility Functions"](/docs/api/utilities/wb)
[^4]: See [`writeFile` in "Writing Files"](/docs/api/write-options)
[^5]: See [`readFile` in "Reading Files"](/docs/api/parse-options)
[^6]: See [`sheet_to_json` in "Utilities"](/docs/api/utilities/array#array-output)
[^7]: See ["Utilities" in "Addresses and Ranges"](/docs/csf/general#utilities)
[^8]: See [`KEYS`](https://redis.io/commands/keys/) in the Redis documentation.
[^9]: See [`TYPE`](https://redis.io/commands/type/) in the Redis documentation.
[^10]: See [`GET`](https://redis.io/commands/get/) in the Redis documentation.
[^11]: See [`LRANGE`](https://redis.io/commands/lrange/) in the Redis documentation.
[^12]: See [`SMEMBERS`](https://redis.io/commands/smembers/) in the Redis documentation.
[^13]: The official command is [`ZRANGE`](https://redis.io/commands/zrange/). `ZRANGE_WITHSCORES` is a special command supported by the NodeJS wrapper.
[^14]: See [`HGETALL`](https://redis.io/commands/hgetall/) in the Redis documentation.