docs.sheetjs.com/21-pandas.md at 1480a3e0213b0efb92c760c363726de065a52ad7

14 KiB

Raw Blame History

title	sidebar_label	description	pagination_prev	pagination_next
Spreadsheet Data in Python	Python DataFrames	Process structured data in Python DataFrames. Seamlessly integrate spreadsheets into your workflow with SheetJS. Analyze complex Excel spreadsheets with confidence.	demos/index	demos/frontend/index

import current from '/version.js'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock';

Pandas is a Python library for data analysis.

SheetJS is a JavaScript library for reading and writing data from spreadsheets.

This demo uses SheetJS to process data from a spreadsheet and translate to the Pandas DataFrame format. We'll explore how to load SheetJS from Python scripts, generate DataFrames from workbooks, and write DataFrames back to workbooks.

The "Complete Example" includes a wrapper library that simplifies importing and exporting spreadsheets.

:::info pass

Pandas includes limited support for reading spreadsheets (pandas.from_excel) and writing XLSX spreadsheets (pandas.DataFrame.to_excel).

SheetJS supports many common spreadsheet formats that Pandas cannot process.

SheetJS operations also offer more flexibility in processing complex worksheets.

:::

:::note Tested Environments

This demo was tested in the following deployments:

Architecture	JS Engine	Pandas	Python	Date
`darwin-x64`	Duktape `2.7.0`	2.2.1	3.12.2	2024-03-15
`darwin-arm`	Duktape `2.7.0`	2.0.3	3.11.7	2024-02-13
`linux-x64`	Duktape `2.7.0`	1.5.3	3.11.3	2024-01-29

:::

Integration Details

sheetjs.py is a wrapper script that provides helper methods for reading and writing spreadsheets. Installation notes are included in the "Complete Example" section.

JS in Python

JS code cannot be directly evaluated in Python implementations.

To run JS code from Python, JavaScript engines¹ can be embedded in Python modules or dynamically loaded using the ctypes foreign function library². This demo uses ctypes with the Duktape engine.

Wrapper

The script exports a class named SheetJSWrapper. It is a context manager that initializes the Duktape engine and executes SheetJS scripts on entrance. All work should be performed in the context:

#!/usr/bin/env python3
from sheetjs import SheetJSWrapper

with SheetJSWrapper() as sheetjs:

  # Parse file
  wb = sheetjs.read_file("pres.numbers")
  print("Loaded file pres.numbers")

  # Get first worksheet name
  first_ws_name = wb.get_sheet_names()[0]
  print(f"Reading from sheet {first_ws_name}")

  # Generate DataFrame from first worksheet
  df = wb.get_df(first_ws_name)
  print(df.info())

  # Export DataFrame to XLSB
  sheetjs.write_df(df, "SheetJSPandas.xlsb", sheet_name="DataFrame")

Reading Files

sheetjs.read_file accepts a path to a spreadsheet file. It will parse the file and return an object representing the workbook.

The get_sheet_names method of the workbook returns a list of sheet names.

The get_df method of the workbook generates a DataFrame from the workbook. The specific sheet can be selected by passing the name.

For example, the following code reads pres.numbers and generates a DataFrame from the second worksheet:

with SheetJSWrapper() as sheetjs:
  # Parse file
  wb = sheetjs.read_file(path)

  # Generate DataFrame from second worksheet
  ws_name = wb.get_sheet_names()[1]
  df = wb.get_df(ws_name)

  # Print metadata
  print(df.info())

Under the hood, sheetjs.py performs the following steps:

flowchart LR
  file[(workbook\nfile)]
  subgraph SheetJS operations
    bytes(Byte\nstring)
    wb((SheetJS\nWorkbook))
    csv(CSV\nstring)
  end
  subgraph Pandas operations
    stream(CSV\nStream)
    df[(Pandas\nDataFrame)]
  end
  file --> |`open`/`read`\nPython ops| bytes
  bytes --> |`XLSX.read`\nParse Bytes| wb
  wb --> |`sheet_to_csv`\nExtract Data| csv
  csv --> |`StringIO`\nPython ops| stream
  stream --> |`read_csv`\nParse CSV| df

Pure Python operations read the spreadsheet file and generate a byte string.
SheetJS libraries parse the string and generate a clean CSV.

The read method³ parses file bytes into a SheetJS workbook object⁴
After selecting a worksheet, sheet_to_csv⁵ generates a CSV string

Python operations convert the CSV string to a stream object.⁶
The Pandas read_csv method⁷ ingests the stream and generate a DataFrame.

Writing Files

sheetjs.write_df accepts a DataFrame and a path. It will attempt to export the data to a spreadsheet file.

For example, the following code exports a DataFrame to SheetJSPandas.xlsb:

with SheetJSWrapper() as sheetjs:
  # Export DataFrame to XLSB
  sheetjs.write_df(df, "SheetJSPandas.xlsb", sheet_name="DataFrame")

Under the hood, sheetjs.py performs the following steps:

flowchart LR
  subgraph Pandas operations
    df[(Pandas\nDataFrame)]
    json(JSON\nString)
  end
  subgraph SheetJS operations
    aoo(array of\nobjects)
    wb((SheetJS\nWorkbook))
    u8a(File\nbytes)
  end
  file[(workbook\nfile)]
  df --> |`to_json`\nPandas ops| json
  json --> |`JSON.parse`\nJS Engine| aoo
  aoo --> |`json_to_sheet`\nSheetJS Ops| wb
  wb --> |`XLSX.write`\nUint8Array| u8a
  u8a --> |`open`/`write`\nPython ops| file

The Pandas DataFrame to_json method⁸ generates a JSON string.
JS engine operations translate the JSON string to an array of objects.
SheetJS libraries process the data array and generate file bytes.

The json_to_sheet method⁹ creates a SheetJS sheet object from the data.
The book_new method¹⁰ creates a SheetJS workbook that includes the sheet.
The write method¹¹ generates the spreadsheet file bytes.

Pure Python operations write the bytes to file.

Complete Example

This example will extract data from an Apple Numbers spreadsheet and generate a DataFrame. The DataFrame will be exported to the binary XLSB spreadsheet format.

Install Pandas:

sudo python3 -m pip install pandas

:::caution pass

On Arch Linux-based platforms including the Steam Deck, the install may fail:

error: externally-managed-environment

In these situations, Pandas must be installed through the package manager:

sudo pacman -Syu python-pandas

:::

:::caution pass

On macOS systems with a Python version from Homebrew, the install will fail:

error: externally-managed-environment

When the demo was last tested, Homebrew did not have a pre-built Pandas formula. Instead, Pandas should be installed with the --break-system-packages option:

sudo python3 -m pip install pandas --break-system-packages

:::

Build the Duktape shared library:

curl -LO https://duktape.org/duktape-2.7.0.tar.xz
tar -xJf duktape-2.7.0.tar.xz
cd duktape-2.7.0
make -f Makefile.sharedlibrary
cd ..

Copy the shared library to the current folder. When the demo was last tested, the shared library file name differed by platform:

OS	name
Darwin	`libduktape.207.20700.so`
Linux	`libduktape.so.207.20700`

cp duktape-*/libduktape.* .

Download the SheetJS Standalone script and move to the project directory:

shim.min.js
xlsx.full.min.js

{\ curl -LO https://cdn.sheetjs.com/xlsx-${current}/package/dist/shim.min.js curl -LO https://cdn.sheetjs.com/xlsx-${current}/package/dist/xlsx.full.min.js}

Download the following test scripts and files:

curl -LO https://sheetjs.com/pres.numbers
curl -LO https://docs.sheetjs.com/pandas/sheetjs.py
curl -LO https://docs.sheetjs.com/pandas/SheetJSPandas.py

Edit the sheetjs.py script.

The lib variable declares the path to the library:

# highlight-next-line
lib = "libduktape.207.20700.so"

The name of the library is libduktape.207.20700.so:

# highlight-next-line
lib = "libduktape.207.20700.so"

The name of the library is libduktape.so.207.20700:

# highlight-next-line
lib = "libduktape.so.207.20700"

Run the script:

python3 SheetJSPandas.py pres.numbers

If successful, the script will display DataFrame metadata:

RangeIndex: 5 entries, 0 to 4
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   Name    5 non-null      object
 1   Index   5 non-null      int64
dtypes: int64(1), object(1)

It will also export the DataFrame to SheetJSPandas.xlsb. The file can be inspected with a spreadsheet editor that supports XLSB files.

Other Libraries

Other Python DataFrame libraries mirror the Pandas DataFrame API.

Polars

Polars is a similar DataFrame library that offers many features from Pandas DataFrames.

:::info pass

Polars includes limited support for reading and writing spreadsheets by wrapping third-party libraries. In practice, Polars communicates with the third-party libraries using intermediate CSV files.¹²

SheetJS supports many common spreadsheet formats that Polars cannot process.

SheetJS operations also offer more flexibility in processing complex worksheets.

:::

The Pandas example requires a few slight changes to work with Polars:

Polars DataFrames expose write_json instead of to_json:

-  json = df.to_json(orient="records")
+  json = df.write_json(row_oriented=True)

Polars DataFrames do not expose info

Polars Demo

:::note Tested Environments

This demo was tested in the following deployments:

Architecture	JS Engine	Polars	Python	Date
`darwin-x64`	Duktape `2.7.0`	0.20.15	3.12.2	2024-03-15
`darwin-arm`	Duktape `2.7.0`	0.20.7	3.11.7	2024-02-13
`linux-x64`	Duktape `2.7.0`	0.20.6	3.11.3	2024-01-30

:::

Follow the Pandas "Complete Example" through the end.
Edit sheetjs.py.

Near the top of the script, change the import from pandas to polars:

-from pandas import read_csv
+from polars import read_csv

:::note pass

The red lines starting with - should be removed from the file and the green lines starting with + should be added to the file. Black lines show the source context and should not be changed.

:::

Within the export_df_to_wb function, change the df.to_json line:

 def export_df_to_wb(ctx, df, path, sheet_name="Sheet1", book_type=None):
-  json = df.to_json(orient="records")
+  json = df.write_json(row_oriented=True)

Edit SheetJSPandas.py.

In the script, change df.info() to df:

 def export_df_to_wb(ctx, df, path, sheet_name="Sheet1", book_type=None):
-    print(df.info())
+    print(df)

Change the export filename from SheetJSPandas.xlsb to SheetJSPolars.xlsb:

     # Export DataFrame to XLSB
-    sheetjs.write_df(df, "SheetJSPandas.xlsb", sheet_name="DataFrame")
+    sheetjs.write_df(df, "SheetJSPolars.xlsb", sheet_name="DataFrame")

Install Polars:

sudo python3 -m pip install polars

:::caution pass

On Arch Linux-based platforms including the Steam Deck, the install may fail:

error: externally-managed-environment

It is recommended to use a virtual environment for Polars:

mkdir sheetjs-polars
cd sheetjs-polars
python3 -m venv .
./bin/pip install polars

:::

Run the script:

python3 SheetJSPandas.py pres.numbers

:::note pass

If the virtual environment was configured in the previous step, run:

./bin/python3 SheetJSPandas.py pres.numbers

:::

If successful, the script will display DataFrame data:

shape: (5, 2)
┌──────────────┬───────┐
│ Name         ┆ Index │
│ ---          ┆ ---   │
│ str          ┆ i64   │
╞══════════════╪═══════╡
│ Bill Clinton ┆ 42    │
│ GeorgeW Bush ┆ 43    │
│ Barack Obama ┆ 44    │
│ Donald Trump ┆ 45    │
│ Joseph Biden ┆ 46    │
└──────────────┴───────┘

It will also export the DataFrame to SheetJSPolars.xlsb. The file can be inspected with a spreadsheet editor that supports XLSB files.

See "Other Languages" for more examples. ↩︎
See ctypes in the Python documentation. ↩︎
See read in "Reading Files" ↩︎
See "Workbook Object" ↩︎
See sheet_to_csv in "Utilities" ↩︎
See the examples in "IO tools" in the Pandas documentation. ↩︎
See pandas.read_csv in the Pandas documentation. ↩︎
See pandas.DataFrame.to_json in the Pandas documentation. ↩︎
See json_to_sheet in "Utilities" ↩︎
See book_new in "Utilities" ↩︎
See write in "Writing Files" ↩︎
As explained in the Polars documentation, "... the target Excel sheet is first converted to CSV ... and then parsed with Polars’ read_csv() function." ↩︎

14 KiB Raw Blame History Unescape Escape

Integration Details

JS in Python

Wrapper

Reading Files

Writing Files

Complete Example

Other Libraries

Polars

Polars Demo

14 KiB

Raw Blame History