docs.sheetjs.com/docz/docs/09-miscellany/05-contributing.md
2025-01-15 13:43:08 -05:00

18 KiB

title sidebar_position
Contributing 5

Due to the precarious nature of the Open Specifications Promise, it is very important to ensure code is cleanroom.

The Contribution Notes should be perused before contributing code.

File organization (click to show)

Folders:

folder contents
bin server-side bin scripts (xlsx.njs)
bits raw source files that make up the final script
dist dist files for web browsers and nonstandard JS environments
misc miscellaneous supporting scripts
modules TypeScript source files that generate some of the bits
packages Support libraries and tools
test_files test files (pulled from the test artifacts distribution)
tests browser tests (run make ctest to rebuild)
types TypeScript definitions and tests

After cloning the repo, running make help will display a list of commands.

Setup

These instructions will cover system configuration, cloning the source repo, building, reproducing official releases, and running NodeJS and browser tests.

:::note Tested Deployments

These instructions were tested on the following platforms:

Platform Architecture Test Date
Linux (Steam Deck Holo x64) linux-x64 2025-01-10
Linux (Debian Linux AArch64) linux-arm 2025-01-14
MacOS 14.4 (x64) darwin-x64 2024-07-12
MacOS 15.1 (ARM64) darwin-arm 2024-11-10
Windows 10 (x64) + WSL Ubuntu win10-x64 2024-07-12
Windows 11 (x64) + WSL Ubuntu win11-x64 2025-01-14
Windows 11 (ARM) + WSL Ubuntu win11-arm 2024-05-23

With some additional dependencies, the unminified scripts are reproducible and tests will pass in Windows XP with NodeJS 5.10.0.

:::

Install dependencies

OS-Specific Setup

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';

A) Ensure WSL ("WSL 2" in Windows 10) and the Ubuntu distribution are installed.

Installation Notes (click to hide)

In "Turn Windows features on or off", all available features from the following list should be enabled:

  • "Hyper-V" (if the option is available)
  • "Virtual Machine Platform"
  • "Windows Hypervisor Platform"
  • "Windows Subsystem for Linux"

:::note pass

The WSL requirements have changed over the years. To be safe, it is recommended to install every listed feature that is available for the Windows version.

:::

The following PowerShell command updates WSL:

wsl --update

The following PowerShell command installs Ubuntu within WSL:

wsl --install Ubuntu

:::note pass

In some versions of wsl, the -d flag must be specified:

wsl --install -d Ubuntu

:::

:::info pass

In some tests, the install failed with a WSL_E_DEFAULT_DISTRO_NOT_FOUND error.

The resolution is to switch to WSL1, install, and switch back to WSL2:

wsl --set-default-version 1
wsl --install Ubuntu
wsl --set-default-version 2
wsl --install Ubuntu

:::

:::caution pass

In Windows 11 24H2, the wsl commands may fail with the following message:

Class not registered
Error code: Wsl/CallMsi/Install/REGDB_E_CLASSNOTREG

WSL should be manually installed. Once installed, WSL2 should be activated:

wsl --set-default-version 2
wsl --install -d Ubuntu

:::

:::danger pass

WSL will not run in a Windows on ARM VM on computers with the M1 CPU

Apple Silicon M1 processors do not support nested virtualization.

M2 processors do support nested virtualization. SheetJS users have reported success with Windows on ARM running on computers with the M2 Max CPU.

:::

B) Install NVM within WSL:

sudo apt-get install curl
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/master/install.sh | bash

C) Exit the WSL session and start a new session

D) Install NodeJS 16 using NVM:

nvm install 16
nvm use 16

E) Clone the js-crc32 repo

git clone https://git.sheetjs.com/sheetjs/js-crc32

:::note pass

On Windows 10, this clone may fail due to issues with core.filemode:

fatal: could not set 'core.filemode' to 'false'

The main drive must be remounted with the metadata option:

cd /
sudo umount /mnt/c
sudo mount -t drvfs C: /mnt/c -o metadata
cd -

:::

:::note pass

If this clone fails with an error message that mentions SSL or secure connection or certificates, build and install a version of Git with proper SSL support:

: # Git does not support OpenSSL out of the box, must do this
curl -LO https://github.com/niko-dunixi/git-openssl-shellscript/raw/main/compile-git-with-openssl.sh
chmod +x compile-git-with-openssl.sh
./compile-git-with-openssl.sh

:::

F) Set git config core.autocrlf setting to false. The following commands should be run twice, once within PowerShell (if Git for Windows is installed) and once within WSL bash:

git config --global --add core.autocrlf false
git config --global --unset core.autocrlf true

G) Run unzip. If the program is missing, install manually:

sudo apt-get install -y unzip

H) Run make. If the program is missing, install manually:

sudo apt-get install -y make

A) Open a terminal window and run git:

git --version

If Xcode or the command-line tools are not installed, you will be asked to install. Click "Install" and run through the steps.

B) Open a terminal window and install the Homebrew package manager:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Follow the on-screen instructions.

After the installation finishes, add Homebrew to your PATH. The instructions are displayed in the Next steps section:

: # zsh
echo >> $HOME/.zprofile
echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> $HOME/.zprofile

: # bash
echo >> $HOME/.bash_profile
echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> $HOME/.bash_profile

: # other
echo >> $HOME/.profile
echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> $HOME/.profile

C) Close the window, open a new terminal window, and disable analytics:

brew analytics off

To confirm analytics are disabled, run

brew analytics state

The message should state that analytics are disabled or destroyed.

D) Install NodeJS.

Open the official NodeJS site with a web browser and select "Prebuilt Installer" from the Categories dropdown.

Adjust the next three dropdowns as follows:

  • "I want the": Select the first LTS option ("v22.11.0 (LTS)" when last tested)
  • "version of Node.js for": Select "macOS"
  • "running": Select "x64" for Intel Macs or "ARM64" for Apple Silicon Macs

Click the green Download button to download the package. After the download finishes, open the package and follow the steps to install NodeJS

:::note pass

Older versions of macOS are not compatible with newer versions of NodeJS.

In local testing, macOS 10.13 required NodeJS version 12.22.12:

curl -LO https://nodejs.org/download/release/v12.22.12/node-v12.22.12.pkg
open node-v12.22.12.pkg

:::

A) Install curl, git, and build tools using the system package manager.

On Debian and Ubuntu systems, build-essential, curl and git are required:

sudo apt update
sudo apt-get install build-essential curl git

On Arch Linux, base-devel, curl and git are required:

sudo pacman -Syu base-devel curl git

Other Linux distributions may use other package managers.

Steam Deck (click to hide)

Desktop Mode on the Steam Deck uses pacman. It also requires a few steps.

  1. Switch to Desktop mode and open Konsole

:::tip pass

At this point, it is strongly recommended to install the ungoogled-chromium browser from the "Discover" app and open this page on the Steam Deck. Running the browser on the device makes it easy to copy and paste commands.

:::

  1. Set a password for the user by running passwd and following instructions.

  2. Disable read-only mode:

sudo steamos-readonly disable

(When prompted, enter the password assigned in step 1)

  1. Configure keyring:
sudo sh -c 'echo "keyserver hkps://keyserver.ubuntu.com" >> /etc/pacman.d/gnupg/gpg.conf'
sudo pacman-key --init
sudo pacman-key --populate
sudo pacman-key --refresh-keys

:::caution pass

When this demo was last tested, keys were refreshed after 25 minutes.

Do not switch to gaming mode or let the device sleep during the process!

To ensure the device does not sleep, click the battery icon in the lower right corner of the screen and enable "Manually block sleep and screen locking".

:::

  1. Install dependencies:
sudo pacman -S base-devel git curl

:::note pass

In local testing on the Steam Deck, some of the C / C++ demos failed to build. This issue was resolved by manually installing glibc and linux-api-headers:

sudo pacman -S glibc linux-api-headers

These packages are not required for building or testing the library.

:::

B) Install NodeJS.

The official NodeJS site provides Linux binaries, but it is strongly recommended to use nvm to install NodeJS:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh | bash

After installing nvm, start a new terminal session and install NodeJS "LTS":

nvm install --lts
node --version

:::note pass

If the nvm command cannot be found, close the current terminal session and start a new session.

:::

:::caution pass

If the node command fails with a glibc error, an older version of NodeJS must be installed. For example, Ubuntu 18.04 does not support Node 18 but does support Node 16.20.0:

nvm install 16
nvm use 16

:::

C) Run unzip. If the program is missing, install manually:

sudo apt-get install -y unzip

D) Run make. If the program is missing, install manually:

sudo apt-get install -y make

Build from source tree

  1. Clone the project:
git clone https://git.sheetjs.com/sheetjs/sheetjs
cd sheetjs

:::note pass

On older platforms, the clone may fail due to SSL certificate problems:

fatal: unable to access 'https://git.sheetjs.com/sheetjs/sheetjs/': SSL certificate problem: certificate has expired

The simplest workaround is to disable SSL verification:

git config --global http.sslVerify false

It is strongly recommended to re-enable SSL verification after cloning:

git config --global http.sslVerify true

:::

  1. Install NodeJS modules for building the scripts:
npm i
npm i -g mocha@2.5.3 voc @sheetjs/uglify-js

:::caution pass

If npm i -g fails with a permissions issue, run the command with sudo:

npm i
sudo npm i -g mocha@2.5.3 voc @sheetjs/uglify-js

:::

:::note Older Versions of Dependencies

Some of the dependencies are wildly out of date. While SheetJS libraries aim to run in older versions of NodeJS and browsers, some libraries have opted to break backwards compatibility. The specific versions are used because they are known to work and known to produce consistent and reproducible results.

:::

  1. Initialize the test files:
rmdir test_files
curl -LO https://test-files.sheetjs.com/test_files.zip
unzip test_files.zip
mkdir -p tmp

:::note pass

The rmdir command may fail if the folder is missing. The error can be ignored.

:::

This step may take a few minutes as the current test snapshot is large.

  1. Run the esbuild tool once:
npx -y esbuild@0.14.14 --version

The command will print the version number 0.14.14.

  1. Run a build and verify with a short test:
: # Full Build
cd modules; make clean; make; cd ..
make
make dist

: # Short test
make test_misc

: # Reset repo
git checkout -- .

:::info pass

In some tests on older releases of macOS, the build failed with an error:

ReferenceError: n is not defined

The first error in the call stack points to dist/xlsx.zahl.js.

Older versions of macOS sed are known to misinterpret newline characters. The workaround is to upgrade to a newer version of sed. On macOS:

brew install gnu-sed
echo 'export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"' >> ~/.profile
. ~/.profile

:::

Reproduce official builds

  1. Run git log and search for the commit that matches a particular release version. For example, version 0.20.3 can be found with:
git log | grep -B4 "version bump 0.20.3"

The output should look like:

$ git log | grep -B4 "version bump 0.20.3"
# highlight-next-line
commit 8a7cfd47bde8258c0d91df6a737bf0136699cdf8 <-- this is the commit hash
Author: SheetJS <dev@sheetjs.com>
Date:   Fri Jul 12 11:47:14 2024 -0400

    version bump 0.20.3
  1. Switch to the commit identified by the hash from the previous step:
git checkout 8a7cfd47bde8258c0d91df6a737bf0136699cdf8
  1. Run the full build sequence:
make clean; make
cd modules; make clean; make; cd ..
make
make dist
  1. Verify the scripts by computing the MD5 checksum.

The checksum for the generated xlsx.full.min.js script can be computed using the md5 command on macOS or the md5sum command on WSL and Linux.

md5sum dist/xlsx.full.min.js
md5 dist/xlsx.full.min.js
md5sum dist/xlsx.full.min.js

The checksum for the equivalent script on the SheetJS CDN can be computed with:

curl -L https://cdn.sheetjs.com/xlsx-0.20.3/package/dist/xlsx.full.min.js | md5sum -

The following output was captured in win11-arm for SheetJS version 0.20.3:

$ md5sum dist/xlsx.full.min.js
# highlight-next-line
6b3130af1ceadf07caa0ec08af7addff  dist/xlsx.full.min.js
$ curl -L https://cdn.sheetjs.com/xlsx-0.20.3/package/dist/xlsx.full.min.js | md5sum -
# highlight-next-line
6b3130af1ceadf07caa0ec08af7addff  -
curl -k -L https://cdn.sheetjs.com/xlsx-0.20.3/package/dist/xlsx.full.min.js | md5

The following output was captured in darwin-arm for SheetJS version 0.20.3:

$ md5 dist/xlsx.full.min.js
# highlight-next-line
MD5 (dist/xlsx.full.min.js) = 6b3130af1ceadf07caa0ec08af7addff
$ curl -k -L https://cdn.sheetjs.com/xlsx-0.20.3/package/dist/xlsx.full.min.js | md5
# highlight-next-line
6b3130af1ceadf07caa0ec08af7addff
curl -L https://cdn.sheetjs.com/xlsx-0.20.3/package/dist/xlsx.full.min.js | md5sum -

The following output was captured in linux-arm for SheetJS version 0.20.3:

$ md5sum dist/xlsx.full.min.js
# highlight-next-line
6b3130af1ceadf07caa0ec08af7addff  dist/xlsx.full.min.js
$ curl -L https://cdn.sheetjs.com/xlsx-0.20.3/package/dist/xlsx.full.min.js | md5sum -
# highlight-next-line
6b3130af1ceadf07caa0ec08af7addff  -

The two hashes should match.

  1. Return to the HEAD commit:
git checkout master

Test in web browsers

  1. Start local server:
make ctestserv

The terminal will display URL. For example:

Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/)
                                   ^^^^^^^^^^^^^^^^^^^^ URL
  1. Open a browser window and access the displayed URL.

Development

The xlsx.js and xlsx.mjs files are constructed from the files in the bits subfolder. The build script (run make) will concatenate the individual bits to produce the scripts.

When changing the .js scripts in bits, the following sequence rebuilds the xlsx.js and xlsx.mjs scripts:

make

When changing the .ts scripts in modules, the following sequence rebuilds the xlsx.js and xlsx.mjs scripts:

cd modules; make clean; make; cd ..

To produce the dist files, run make dist.

:::info pass

The various xlsx.* scripts in the base folder and the files in the dist folder are updated on each version release.

They should not be committed between versions!

:::

Tests

The test_misc target runs the targeted feature tests. It should take 5-10 seconds to perform feature tests without testing against the full test battery. New features should be accompanied with tests for the relevant file formats.

For tests involving the read side, an appropriate feature test would involve reading an existing file and checking the resulting workbook object. If a parameter is involved, files should be read with different values to verify that the feature is working as expected.

For tests involving a new write feature which can already be parsed, appropriate feature tests would involve writing a workbook with the feature and then opening and verifying that the feature is preserved.

For tests involving a new write feature without an existing read ability, please add a feature test to the kitchen sink tests/write.js.