mirror of
https://github.com/opencultureconsulting/openrefine-client.git
synced 2025-03-30 00:00:46 +01:00
Compare commits
24 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
02cf1192c4 | ||
|
|
e4d52818fc | ||
|
|
16541c522e | ||
|
|
965c4e97fd | ||
|
|
0563b54fc6 | ||
|
|
fa3e352879 | ||
|
|
1dd0cafd4e | ||
|
|
a368147bdf | ||
|
|
f66c88ee35 | ||
|
|
2735db3f3f | ||
|
|
0df6050deb | ||
|
|
65b0500d5e | ||
|
|
a0274f6166 | ||
|
|
9eeebce47c | ||
|
|
21bf1f2adc | ||
|
|
8a726d9a30 | ||
|
|
bc89a98776 | ||
|
|
82da3f7b4e | ||
|
|
356315bc9e | ||
|
|
41b90c38b3 | ||
|
|
1c812d1253 | ||
|
|
f0d76b6acd | ||
|
|
ecf253ca44 | ||
|
|
c47ce10eba |
4
.gitignore
vendored
4
.gitignore
vendored
@ -4,4 +4,6 @@ dist
|
||||
.*
|
||||
openrefine_client.egg-info
|
||||
refine.spec
|
||||
README.html
|
||||
openrefine-*
|
||||
openrefine-client_*
|
||||
tests-cli.log
|
||||
|
||||
160
README.md
160
README.md
@ -1,18 +1,20 @@
|
||||
# OpenRefine Python Client with extended command line interface
|
||||
# OpenRefine Python Client with extended command line interface (⌨️ for 💎)
|
||||
|
||||
[](https://www.codacy.com/app/felixlohmeier/openrefine-client?utm_source=github.com&utm_medium=referral&utm_content=opencultureconsulting/openrefine-client&utm_campaign=Badge_Grade) [](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [](https://pypi.org/project/openrefine-client/) [](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master)
|
||||
[](https://www.codacy.com/gh/opencultureconsulting/openrefine-client/dashboard) [](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [](https://pypi.org/project/openrefine-client/) [](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master)
|
||||
|
||||
The [OpenRefine Python Client from PaulMakepeace](https://github.com/PaulMakepeace/refine-client-py) provides a library for communicating with an [OpenRefine](http://openrefine.org) server.
|
||||
This fork extends the command line interface (CLI) and is distributed as a convenient one-file-executable (Windows, Linux, macOS).
|
||||
It is also available via Docker Hub, PyPI and Binder.
|
||||
|
||||
works with OpenRefine 2.7, 2.8, 3.0, 3.1, 3.2, 3.3, 3.4, 3.4.1, 3.5.0
|
||||
|
||||
## Download
|
||||
|
||||
One-file-executables:
|
||||
|
||||
- Windows: [openrefine-client_0-3-8_windows.exe](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.8/openrefine-client_0-3-8_windows.exe) (~5 MB)
|
||||
- macOS: [openrefine-client_0-3-8_macos](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.8/openrefine-client_0-3-8_macos) (~5 MB)
|
||||
- Linux: [openrefine-client_0-3-8_linux](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.8/openrefine-client_0-3-8_linux) (~5 MB)
|
||||
- Windows: [openrefine-client_0-3-10_windows.exe](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_windows.exe) (~5 MB)
|
||||
- macOS: [openrefine-client_0-3-10_macos](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_macos) (~5 MB)
|
||||
- Linux: [openrefine-client_0-3-10_linux](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_linux) (~5 MB)
|
||||
|
||||
For [Docker](#docker) containers, native [Python](#python) installation and free [Binder](#binder) on-demand server see the corresponding chapters below.
|
||||
|
||||
@ -55,13 +57,13 @@ To use the client:
|
||||
- macOS:
|
||||
|
||||
```sh
|
||||
chmod +x openrefine-client_0-3-8_macos
|
||||
chmod +x openrefine-client_0-3-10_macos
|
||||
```
|
||||
|
||||
- Linux:
|
||||
|
||||
```sh
|
||||
chmod +x openrefine-client_0-3-8_linux
|
||||
chmod +x openrefine-client_0-3-10_linux
|
||||
```
|
||||
|
||||
3. Execute the file.
|
||||
@ -69,19 +71,19 @@ To use the client:
|
||||
- Windows:
|
||||
|
||||
```sh
|
||||
.\openrefine-client_0-3-8_windows.exe
|
||||
.\openrefine-client_0-3-10_windows.exe
|
||||
```
|
||||
|
||||
- macOS:
|
||||
|
||||
```sh
|
||||
./openrefine-client_0-3-8_macos
|
||||
./openrefine-client_0-3-10_macos
|
||||
```
|
||||
|
||||
- Linux:
|
||||
|
||||
```sh
|
||||
./openrefine-client_0-3-8_linux
|
||||
./openrefine-client_0-3-10_linux
|
||||
```
|
||||
|
||||
Using tab completion and command history is highly recommended:
|
||||
@ -100,25 +102,25 @@ Download example data (`--download`) and create project from file (`--create`):
|
||||
- Windows:
|
||||
|
||||
```sh
|
||||
.\openrefine-client_0-3-8_windows.exe --download "https://git.io/fj5hF" --output=duplicates.csv
|
||||
.\openrefine-client_0-3-8_windows.exe --download "https://git.io/fj5ju" --output=duplicates-deletion.json
|
||||
.\openrefine-client_0-3-8_windows.exe --create duplicates.csv
|
||||
.\openrefine-client_0-3-10_windows.exe --download "https://git.io/fj5hF" --output=duplicates.csv
|
||||
.\openrefine-client_0-3-10_windows.exe --download "https://git.io/fj5ju" --output=duplicates-deletion.json
|
||||
.\openrefine-client_0-3-10_windows.exe --create duplicates.csv
|
||||
```
|
||||
|
||||
- macOS:
|
||||
|
||||
```sh
|
||||
./openrefine-client_0-3-8_macos --download "https://git.io/fj5hF" --output=duplicates.csv
|
||||
./openrefine-client_0-3-8_macos --download "https://git.io/fj5ju" --output=duplicates-deletion.json
|
||||
./openrefine-client_0-3-8_macos --create duplicates.csv
|
||||
./openrefine-client_0-3-10_macos --download "https://git.io/fj5hF" --output=duplicates.csv
|
||||
./openrefine-client_0-3-10_macos --download "https://git.io/fj5ju" --output=duplicates-deletion.json
|
||||
./openrefine-client_0-3-10_macos --create duplicates.csv
|
||||
```
|
||||
|
||||
- Linux:
|
||||
|
||||
```sh
|
||||
./openrefine-client_0-3-8_linux --download "https://git.io/fj5hF" --output=duplicates.csv
|
||||
./openrefine-client_0-3-8_linux --download "https://git.io/fj5ju" --output=duplicates-deletion.json
|
||||
./openrefine-client_0-3-8_linux --create duplicates.csv
|
||||
./openrefine-client_0-3-10_linux --download "https://git.io/fj5hF" --output=duplicates.csv
|
||||
./openrefine-client_0-3-10_linux --download "https://git.io/fj5ju" --output=duplicates-deletion.json
|
||||
./openrefine-client_0-3-10_linux --create duplicates.csv
|
||||
```
|
||||
|
||||
Other commands:
|
||||
@ -145,7 +147,7 @@ If your OpenRefine server is running somewhere else then you may set hostname an
|
||||
- set host: `-H example.com`
|
||||
- set port: `-P 80`
|
||||
|
||||
### Advanced Templating
|
||||
### Templating
|
||||
|
||||
The OpenRefine [Templating](https://github.com/OpenRefine/OpenRefine/wiki/Export-As-YAML) supports exporting data in any text format (i.e. to construct JSON or XML).
|
||||
The graphical user interface offers four input fields:
|
||||
@ -218,10 +220,35 @@ There is another option to use the value in the first column instead:
|
||||
Because our project "advanced" contains duplicates in the first column "email" this command will overwrite files (e.g. `advanced_melanie.white@example2.edu.json`).
|
||||
When using this option, the first column should contain unique identifiers.
|
||||
|
||||
### Append data to an existing project
|
||||
|
||||
OpenRefine does not support appending rows to an existing project.
|
||||
As long as the [feature request](https://github.com/OpenRefine/OpenRefine/issues/715) is not yet implemented, you can use the openrefine-client to script a workaround:
|
||||
|
||||
1. export existing project as csv
|
||||
2. put old and new data into a zip archive
|
||||
3. create new project by importing the zip archive
|
||||
|
||||
Here is an example that replaces the existing project:
|
||||
|
||||
```
|
||||
openrefine-client --export myproject --output old.csv
|
||||
openrefine-client --delete myproject
|
||||
zip combined.zip old.csv new.csv
|
||||
openrefine-client --create combined.zip --format csv --projectName myproject
|
||||
```
|
||||
|
||||
Note that the project id will change.
|
||||
If you want to distinguish between old and new data, you can use the additional flag includeFileSources:
|
||||
|
||||
```
|
||||
openrefine-client --create combined.zip --format csv --projectName myproject --includeFileSources true
|
||||
```
|
||||
|
||||
### See also
|
||||
|
||||
- Linux Bash script to run OpenRefine in batch mode (import, transform, export): [openrefine-batch](https://github.com/opencultureconsulting/openrefine-batch)
|
||||
- [Jupyter notebook demonstrating usage in Linux Bash](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/openrefine-client-bash.ipynb)
|
||||
- [Jupyter notebook demonstrating usage in Linux Bash](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/notebooks/openrefine-client-bash.ipynb)
|
||||
- Use case [HOS-MetadataTransformations](https://github.com/subhh/HOS-MetadataTransformations): Automated workflow for harvesting, transforming and indexing of metadata using metha, OpenRefine and Solr. Part of the Hamburg Open Science "Schaufenster" software stack.
|
||||
- Use case [Data processing of ILS data to facilitate a new discovery layer for the German Literature Archive (DLA)](https://doi.org/10.5281/zenodo.2678113): Custom data processing pipeline based on Pandas (a Python library) and OpenRefine.
|
||||
|
||||
@ -230,7 +257,7 @@ When using this option, the first column should contain unique identifiers.
|
||||
[felixlohmeier/openrefine-client](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [](https://hub.docker.com/r/felixlohmeier/openrefine-client/)
|
||||
|
||||
```sh
|
||||
docker pull felixlohmeier/openrefine-client:v0.3.8
|
||||
docker pull felixlohmeier/openrefine-client:v0.3.10
|
||||
```
|
||||
|
||||
### Option 1: Dockerized client
|
||||
@ -238,7 +265,7 @@ docker pull felixlohmeier/openrefine-client:v0.3.8
|
||||
Run client and mount current directory as workspace:
|
||||
|
||||
```sh
|
||||
docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8
|
||||
docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10
|
||||
```
|
||||
|
||||
The docker option `--network=host` allows you to connect to a local or remote OpenRefine via the host network:
|
||||
@ -246,13 +273,13 @@ The docker option `--network=host` allows you to connect to a local or remote Op
|
||||
- list projects on default URL (http://localhost:3333)
|
||||
|
||||
```sh
|
||||
docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 --list
|
||||
docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --list
|
||||
```
|
||||
|
||||
- list projects on a remote server (http://example.com)
|
||||
|
||||
```sh
|
||||
docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H example.com -P 80 --list
|
||||
docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H example.com -P 80 --list
|
||||
```
|
||||
|
||||
Usage: same commands as explained above (see [Basic Commands](#basic-commands) and [Advanced Templating](#advanced-templating))
|
||||
@ -270,22 +297,22 @@ Run openrefine-client linked to a dockerized OpenRefine ([felixlohmeier/openrefi
|
||||
2. Run server (will be available at http://localhost:3333)
|
||||
|
||||
```sh
|
||||
docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.2
|
||||
docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.5.0
|
||||
```
|
||||
|
||||
3. Run client with some [basic commands](#basic-commands): 1. download example files, 2. create project from file, 3. list projects, 4. show metadata, 5. export to terminal, 6. apply transformation rules (deduplication), 7. export again to terminal, 8. export to xls file and 9. delete project
|
||||
|
||||
```sh
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 --download "https://git.io/fj5hF" --output=duplicates.csv
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 --download "https://git.io/fj5ju" --output=duplicates-deletion.json
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --create duplicates.csv
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --list
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --info "duplicates"
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --export "duplicates"
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --apply duplicates-deletion.json "duplicates"
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --export "duplicates"
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --export --output=deduped.xls "duplicates"
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --delete "duplicates"
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --download "https://git.io/fj5hF" --output=duplicates.csv
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --download "https://git.io/fj5ju" --output=duplicates-deletion.json
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --create duplicates.csv
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --list
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --info "duplicates"
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export "duplicates"
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --apply duplicates-deletion.json "duplicates"
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export "duplicates"
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export --output=deduped.xls "duplicates"
|
||||
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --delete "duplicates"
|
||||
```
|
||||
|
||||
4. Stop and delete server:
|
||||
@ -310,7 +337,7 @@ Customize OpenRefine server:
|
||||
- Example for [allocating more memory](https://github.com/OpenRefine/OpenRefine/wiki/FAQ#out-of-memory-errors---feels-slow---could-not-reserve-enough-space-for-object-heap) to OpenRefine with additional option `-m 4G`
|
||||
|
||||
```sh
|
||||
docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.2 -i 0.0.0.0 -d /data -m 4G
|
||||
docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.5.0 -i 0.0.0.0 -d /data -m 4G
|
||||
```
|
||||
|
||||
- The OpenRefine version is defined by the docker tag.
|
||||
@ -597,8 +624,8 @@ See also:
|
||||
- free to use on-demand server with Jupyter notebook, OpenRefine and Bash
|
||||
- no registration needed, will start within a few minutes
|
||||
- [restricted](https://mybinder.readthedocs.io/en/latest/faq.html#how-much-memory-am-i-given-when-using-binder) to 2 GB RAM and server will be deleted after 10 minutes of inactivity
|
||||
- [bash_kernel demo notebook](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/openrefine-client-bash.ipynb) for using the openrefine-client in a Linux Bash environment [](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master?urlpath=/tree/openrefine-client-bash.ipynb)
|
||||
- [python2 demo notebook](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/openrefine-client-python.ipynb) for using the openrefine-client in a Python 2 environment [](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master?urlpath=/tree/openrefine-client-python.ipynb)
|
||||
- [bash_kernel demo notebook](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/notebooks/openrefine-client-bash.ipynb) for using the openrefine-client in a Linux Bash environment [](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master?urlpath=/tree/notebooks/openrefine-client-bash.ipynb)
|
||||
- [python2 demo notebook](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/notebooks/openrefine-client-python.ipynb) for using the openrefine-client in a Python 2 environment [](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master?urlpath=/tree/notebooks/openrefine-client-python.ipynb)
|
||||
|
||||
## Development
|
||||
|
||||
@ -624,39 +651,54 @@ The Python client library includes several unit tests.
|
||||
|
||||
There is also a script that uses docker images to run the unit tests with different versions of OpenRefine.
|
||||
|
||||
- run tests on all OpenRefine versions (from 2.0 up to 3.2)
|
||||
- run tests on all OpenRefine versions (from 2.0 up to 3.5.0)
|
||||
|
||||
```sh
|
||||
./tests.sh -a
|
||||
```
|
||||
|
||||
- run tests on tag 3.2
|
||||
- run tests on tag 3.5.0
|
||||
|
||||
```sh
|
||||
./tests.sh -t 3.2
|
||||
./tests.sh -t 3.5.0
|
||||
```
|
||||
|
||||
- run tests on tag 3.2 interactively (pause before and after tests)
|
||||
- run tests on tag 3.5.0 interactively (pause before and after tests)
|
||||
|
||||
```sh
|
||||
./tests.sh -t 3.2 -i
|
||||
./tests.sh -t 3.5.0 -i
|
||||
```
|
||||
|
||||
- run tests on tags 3.2 and 2.7
|
||||
- run tests on tags 3.5.0 and 2.7
|
||||
|
||||
```sh
|
||||
./tests.sh -t 3.2 -t 2.7
|
||||
./tests.sh -t 3.5.0 -t 2.7
|
||||
```
|
||||
|
||||
For Linux there are also functional tests for all command line options.
|
||||
|
||||
- run all functional tests on OpenRefine 3.5.0
|
||||
|
||||
```sh
|
||||
./tests-cli.sh 3.5.0
|
||||
```
|
||||
|
||||
- run all functional tests on OpenRefine 3.5.0 with one-file-executable
|
||||
|
||||
```sh
|
||||
./tests-cli.sh 3.5.0 openrefine-client_0-3-7_linux
|
||||
```
|
||||
|
||||
### Distributing
|
||||
|
||||
Note to myself: When releasing a new version...
|
||||
|
||||
1. Run tests
|
||||
1. Run functional tests
|
||||
|
||||
```sh
|
||||
./tests.sh -a
|
||||
jupyter notebook tests/cli_python2.ipynb
|
||||
for v in 2.7 2.8 3.0 3.1 3.2 3.3 3.4 3.4.1 3.5.0; do
|
||||
./tests-cli.sh $v
|
||||
done
|
||||
```
|
||||
|
||||
2. Make final changes in Git
|
||||
@ -670,24 +712,25 @@ Note to myself: When releasing a new version...
|
||||
- Run PyInstaller in Python 2 environments on native Windows, macOS and Linux. Should be "the oldest version of the OS you need to support"! Current release is built with:
|
||||
|
||||
- Ubuntu 16.04 LTS (64-bit)
|
||||
- macOS Sierra 10.12
|
||||
- Windows 10
|
||||
- macOS Sierra 10.12 (64-bit)
|
||||
- Windows 7 (32-bit)
|
||||
|
||||
- One-file-executables will be available in `dist/`.
|
||||
|
||||
```sh
|
||||
git clone https://github.com/opencultureconsulting/openrefine-client.git
|
||||
cd openrefine-client
|
||||
python -m pip install . --user
|
||||
python -m pip install pyinstaller --user
|
||||
pyinstaller --onefile refine.py --hidden-import google.refine.__main__
|
||||
python2 -m pip install pyinstaller --user
|
||||
python2 -m pip install urllib2_file --user
|
||||
python2 -m PyInstaller --onefile refine.py --hidden-import google.refine.__main__
|
||||
```
|
||||
|
||||
4. Run test with Linux executable
|
||||
4. Run functional tests with Linux executable
|
||||
|
||||
```sh
|
||||
./tests.sh -a
|
||||
jupyter notebook tests/cli_bash.ipynb
|
||||
for v in 2.7 2.8 3.0 3.1 3.2 3.3 3.4 3.4.1 3.5.0; do
|
||||
./tests-cli.sh $v openrefine-client_0-3-7_linux
|
||||
done
|
||||
```
|
||||
|
||||
5. Create release in GitHub
|
||||
@ -709,8 +752,7 @@ Note to myself: When releasing a new version...
|
||||
8. Bump openrefine-client version in related projects
|
||||
|
||||
- openrefine-batch: [openrefine-batch.sh](https://github.com/opencultureconsulting/openrefine-batch/blob/master/openrefine-batch.sh#L7) and [openrefine-batch-docker.sh](https://github.com/opencultureconsulting/openrefine-batch/blob/master/openrefine-batch-docker.sh)
|
||||
|
||||
- openrefineder: [postBuild](https://github.com/felixlohmeier/openrefineder/blob/master/postBuild)
|
||||
- openrefineder: [postBuild](https://github.com/felixlohmeier/openrefineder/blob/master/binder/postBuild)
|
||||
|
||||
## Credits
|
||||
|
||||
|
||||
@ -1,4 +1,4 @@
|
||||
FROM alpine:latest
|
||||
FROM alpine:3.11
|
||||
LABEL maintainer="felixlohmeier@opencultureconsulting.com"
|
||||
# The OpenRefine Python Client Library from PaulMakepeace provides an interface to communicating with an OpenRefine server. This fork extends the command line interface (CLI) and supports communication between docker containers.
|
||||
# Source: https://github.com/opencultureconsulting/openrefine-client
|
||||
|
||||
@ -41,7 +41,6 @@ def apply(project_id, history_file):
|
||||
print('File %s has been successfully applied to project %s' %
|
||||
(history_file, project_id))
|
||||
|
||||
|
||||
def create(project_file,
|
||||
project_format=None,
|
||||
columnWidths=None,
|
||||
@ -165,17 +164,19 @@ def download(url, output_file=None):
|
||||
def export(project_id, encoding=None, output_file=None, export_format=None):
|
||||
"""Dump a project to stdout or file."""
|
||||
project = refine.RefineProject(project_id)
|
||||
if not export_format:
|
||||
export_format = 'tsv'
|
||||
if not output_file:
|
||||
if not export_format:
|
||||
export_format = 'tsv'
|
||||
if export_format in ['csv', 'tsv', 'txt']:
|
||||
encoding = 'UTF-8'
|
||||
sys.stdout.write(project.export(
|
||||
export_format=export_format, encoding=encoding).read())
|
||||
else:
|
||||
ext = os.path.splitext(output_file)[1][1:]
|
||||
if ext:
|
||||
if ext and not export_format:
|
||||
export_format = ext.lower()
|
||||
if not export_format:
|
||||
export_format = 'tsv'
|
||||
if export_format in ['csv', 'tsv', 'txt']:
|
||||
encoding = 'UTF-8'
|
||||
with open(output_file, 'wb') as f:
|
||||
@ -199,7 +200,7 @@ def info(project_id):
|
||||
project_model = refine.RefineProject(project_id).get_models()
|
||||
columns = [c['name'] for c in project_model['columnModel']['columns']]
|
||||
for (i, v) in enumerate(columns, start=1):
|
||||
print(u'{0:>20}: {1}'.format(u'column ' + str(i).zfill(3), v))
|
||||
print(u'{0:>20}: {1}'.format(u'column ' + str(i).zfill(3), v).encode('utf-8'))
|
||||
else:
|
||||
print('Error: No project found with id %s.\n'
|
||||
'Check existing projects with command --list' % (project_id))
|
||||
@ -215,7 +216,7 @@ def ls():
|
||||
projects.sort(key=lambda v: date_to_epoch(v[1]['modified']), reverse=True)
|
||||
if projects:
|
||||
for project_id, project_info in projects:
|
||||
print(u'{0:>14}: {1}'.format(project_id, project_info['name']))
|
||||
print(u'{0:>14}: {1}'.format(project_id, project_info['name']).encode('utf-8'))
|
||||
else:
|
||||
print('Error: No projects found')
|
||||
|
||||
@ -301,7 +302,7 @@ def templating(project_id,
|
||||
# if key column is not blank (=record)
|
||||
template = ('{{forNonBlank(' +
|
||||
'with(row.columnNames[0],cn,cells[cn].value),' +
|
||||
'v,"' + split + '")}}' +
|
||||
'v,"' + split + '", "")}}' +
|
||||
templateconfig['template'])
|
||||
templateconfig.update({'prefix': '',
|
||||
'suffix': '',
|
||||
|
||||
@ -53,6 +53,18 @@ class RefineServer(object):
|
||||
server = self.url()
|
||||
self.server = server[:-1] if server.endswith('/') else server
|
||||
self.__version = None # see version @property below
|
||||
self.token = None # CSRF token introduced in OpenRefine 3.3
|
||||
self.get_csrf_token()
|
||||
|
||||
def get_csrf_token(self):
|
||||
"""Return csrf token."""
|
||||
try:
|
||||
url = self.server + '/command/core/get-csrf-token'
|
||||
response = json.loads(urllib2.urlopen(url).read())
|
||||
self.token = response['token']
|
||||
return self.token
|
||||
except:
|
||||
pass # fail silently to not disturb usage of OpenRefine <3.3
|
||||
|
||||
def urlopen(self, command, data=None, params=None, project_id=None):
|
||||
"""Open a Refine URL and with optional query params and POST data.
|
||||
@ -73,6 +85,9 @@ class RefineServer(object):
|
||||
data['project'] = project_id
|
||||
else:
|
||||
params['project'] = project_id
|
||||
# be lazy and send the token for each API call (even when not needed)
|
||||
if self.token:
|
||||
params['csrf_token'] = self.token
|
||||
if params:
|
||||
url += '?' + urllib.urlencode(params)
|
||||
req = urllib2.Request(url)
|
||||
@ -82,7 +97,8 @@ class RefineServer(object):
|
||||
try:
|
||||
response = urllib2.urlopen(req)
|
||||
except urllib2.HTTPError as e:
|
||||
raise Exception('HTTP %d "%s" for %s\n\t%s' % (e.code, e.msg, e.geturl(), data))
|
||||
raise Exception('HTTP %d "%s" for %s\n\t%s' %
|
||||
(e.code, e.msg, e.geturl(), data))
|
||||
except urllib2.URLError as e:
|
||||
raise urllib2.URLError(
|
||||
'%s for %s. No Refine server reachable/running; ENV set?' %
|
||||
@ -98,6 +114,10 @@ class RefineServer(object):
|
||||
"""Open a Refine URL, optionally POST data, and return parsed JSON."""
|
||||
response = json.loads(self.urlopen(*args, **kwargs).read())
|
||||
if 'code' in response and response['code'] not in ('ok', 'pending'):
|
||||
if 'Missing or invalid csrf_token parameter' == response['message']:
|
||||
self.get_csrf_token()
|
||||
response = json.loads(self.urlopen(*args, **kwargs).read())
|
||||
return response
|
||||
error_message = ('server ' + response['code'] + ': ' +
|
||||
response.get('message', response.get('stack', response)))
|
||||
raise Exception(error_message)
|
||||
@ -236,6 +256,7 @@ class Refine:
|
||||
# POST is broken at the moment, so we send it in the URL
|
||||
new_style_options = dict(opts, **{
|
||||
'encoding': s(encoding),
|
||||
'separator': s(separator)
|
||||
})
|
||||
params = {
|
||||
'options': json.dumps(new_style_options),
|
||||
@ -244,7 +265,6 @@ class Refine:
|
||||
# old style options
|
||||
options = {
|
||||
'format': project_format,
|
||||
'separator': s(separator),
|
||||
'ignore-lines': s(ignore_lines),
|
||||
'header-lines': s(header_lines),
|
||||
'skip-data-lines': s(skip_data_lines),
|
||||
@ -253,7 +273,7 @@ class Refine:
|
||||
'process-quotes': s(process_quotes),
|
||||
'store-blank-rows': s(store_blank_rows),
|
||||
'store-blank-cells-as-nulls': s(store_blank_cells_as_nulls),
|
||||
'include-file-sources': s(include_file_sources),
|
||||
'include-file-sources': s(include_file_sources)
|
||||
}
|
||||
|
||||
if project_url is not None:
|
||||
@ -398,7 +418,10 @@ class RefineProject:
|
||||
for i, column in enumerate(column_model['columns']):
|
||||
name = column['name']
|
||||
self.column_order[name] = i
|
||||
column_index[name] = column['cellIndex']
|
||||
try:
|
||||
column_index[name] = column['cellIndex']
|
||||
except KeyError:
|
||||
column_index[name] = i
|
||||
self.key_column = column_model['keyColumnName']
|
||||
self.has_records = response['recordModel'].get('hasRecords', False)
|
||||
self.rows_response_factory = RowsResponseFactory(column_index)
|
||||
|
||||
2
setup.py
2
setup.py
@ -25,7 +25,7 @@ def read(filename):
|
||||
return open(os.path.join(os.path.dirname(__file__), filename)).read()
|
||||
|
||||
setup(name='openrefine-client',
|
||||
version='0.3.8',
|
||||
version='0.3.10',
|
||||
description=('The OpenRefine Python Client Library provides an '
|
||||
'interface to communicating with an OpenRefine server. '
|
||||
'This fork extends the command line interface (CLI).'),
|
||||
|
||||
123
tests-cli.sh
Executable file
123
tests-cli.sh
Executable file
@ -0,0 +1,123 @@
|
||||
#!/bin/bash
|
||||
# Script for running functional tests against the CLI
|
||||
|
||||
# Copyright (c) 2011 Paul Makepeace, Real Programmers. All rights reserved.
|
||||
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program. If not, see <http://www.gnu.org/licenses/>
|
||||
|
||||
# ================================== CONFIG ================================== #
|
||||
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
|
||||
port=3334
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
version="${1}"
|
||||
else
|
||||
version="3.2"
|
||||
fi
|
||||
refine="openrefine-${version}/refine"
|
||||
|
||||
if [[ ${2} ]]; then
|
||||
client="$(readlink -e "${2}")"
|
||||
else
|
||||
client="python2 $(readlink -e refine.py)"
|
||||
fi
|
||||
cmd="${client} -H localhost -P ${port}"
|
||||
|
||||
if [[ ${3} ]]; then
|
||||
filename="${3%%.*}"
|
||||
else
|
||||
filename=""
|
||||
fi
|
||||
cmd="${client} -H localhost -P ${port}"
|
||||
|
||||
# =============================== REQUIREMENTS =============================== #
|
||||
|
||||
# check existence of java and cURL
|
||||
if [[ -z "$(command -v java 2> /dev/null)" ]] ; then
|
||||
echo 1>&2 "ERROR: OpenRefine requires JAVA runtime environment (jre)" \
|
||||
"https://openjdk.java.net/install/"
|
||||
exit 1
|
||||
fi
|
||||
if [[ -z "$(command -v curl 2> /dev/null)" ]] ; then
|
||||
echo 1>&2 "ERROR: This shell script requires cURL" \
|
||||
"https://curl.haxx.se/download.html"
|
||||
exit 1
|
||||
fi
|
||||
# download OpenRefine
|
||||
if [[ -z "$(readlink -e "${refine}")" ]]; then
|
||||
echo "Download OpenRefine ${version}..."
|
||||
mkdir -p "$(dirname "${refine}")"
|
||||
curl -L --output openrefine.tar.gz \
|
||||
"https://github.com/OpenRefine/OpenRefine/releases/download/${version}/openrefine-linux-${version}.tar.gz"
|
||||
echo "Install OpenRefine ${version} in subdirectory $(dirname "${refine}")..."
|
||||
tar -xzf openrefine.tar.gz -C "$(dirname "${refine}")" --strip 1 --totals
|
||||
rm -f openrefine.tar.gz
|
||||
# do not try to open OpenRefine in browser
|
||||
sed -i '$ a JAVA_OPTIONS=-Drefine.headless=true' \
|
||||
"$(dirname "${refine}")"/refine.ini
|
||||
# set autosave period from 5 minutes to 25 hours
|
||||
sed -i 's/#REFINE_AUTOSAVE_PERIOD=60/REFINE_AUTOSAVE_PERIOD=1500/' \
|
||||
"$(dirname "${refine}")"/refine.ini
|
||||
echo
|
||||
fi
|
||||
|
||||
# ================================== SETUP =================================== #
|
||||
|
||||
dir="$(readlink -f "tests/tmp")"
|
||||
mkdir -p "${dir}"
|
||||
rm -f tests-cli.log
|
||||
|
||||
echo "start OpenRefine ${version}..."
|
||||
${refine} -v warn -p ${port} -d "${dir}" &>> tests-cli.log &
|
||||
pid_server=${!}
|
||||
timeout 30s bash -c "until curl -s 'http://localhost:3334' \
|
||||
| cat | grep -q -o 'OpenRefine' ; do sleep 1; done" \
|
||||
|| error "starting OpenRefine server failed!"
|
||||
echo
|
||||
|
||||
# ================================== TESTS =================================== #
|
||||
|
||||
echo "running tests, please wait..."
|
||||
tests=()
|
||||
results=()
|
||||
for t in tests/*${filename}*.sh; do
|
||||
tests+=("${t}")
|
||||
echo "======================= ${t} =======================" &>> tests-cli.log
|
||||
bash "${t}" "${cmd}" "${version}" &>> tests-cli.log
|
||||
results+=(${?})
|
||||
done
|
||||
echo
|
||||
|
||||
# ================================= TEARDOWN ================================= #
|
||||
|
||||
echo "cleanup..."
|
||||
{ kill -9 "${pid_server}" && wait "${pid_server}"; } 2>/dev/null
|
||||
rm -rf "${dir}"
|
||||
echo
|
||||
|
||||
# ================================= SUMMARY ================================== #
|
||||
|
||||
printf "%s\t%s\n" "code" "test"
|
||||
printf "%s\t%s\n" "----" "----------------"
|
||||
for i in "${!tests[@]}"; do
|
||||
printf "%s\t%s\n" "${results[$i]}" "${tests[$i]}"
|
||||
done
|
||||
echo
|
||||
if [[ " ${results[*]} " =~ [1-9] ]]; then
|
||||
echo "failed tests! check tests-cli.log for debugging"; echo
|
||||
else
|
||||
echo "all tests passed!"; echo
|
||||
fi
|
||||
12
tests.sh
12
tests.sh
@ -17,8 +17,8 @@
|
||||
# along with this program. If not, see <http://www.gnu.org/licenses/>
|
||||
|
||||
# defaults:
|
||||
all=(3.2-java12 3.2-java11 3.2-java10 3.2-java9 3.2 3.1-java9 3.1 3.0-java9 3.0 2.8-java9 2.8 2.8-java7 2.7 2.7-java7 2.5-java7 2.5-java6 2.1-java6 2.0-java6)
|
||||
main=(3.2 3.1 3.0 2.8 2.7 2.5-java6 2.1-java6 2.0-java6)
|
||||
all=(3.5.0 3.4.1 3.4 3.3 3.2-java12 3.2-java11 3.2-java10 3.2-java9 3.2 3.1-java9 3.1 3.0-java9 3.0 2.8-java9 2.8 2.8-java7 2.7 2.7-java7 2.5-java7 2.5-java6 2.1-java6 2.0-java6)
|
||||
main=(3.5.0 3.4.1 3.4 3.3 3.2 3.1 3.0 2.8 2.7 2.5-java6 2.1-java6 2.0-java6)
|
||||
interactively=false
|
||||
port="3333"
|
||||
|
||||
@ -31,10 +31,10 @@ Script for running tests with different OpenRefine and Java versions.
|
||||
It uses docker images from https://hub.docker.com/r/felixlohmeier/openrefine.
|
||||
|
||||
Examples:
|
||||
./tests.sh -a # run tests on all OpenRefine versions (from 2.0 up to 3.2)
|
||||
./tests.sh -t 3.2 # run tests on tag 3.2
|
||||
./tests.sh -t 3.2 -i # run tests on tag 3.2 interactively (pause before and after tests)
|
||||
./tests.sh -t 3.2 -t 2.7 # run tests on tags 3.2 and 2.7
|
||||
./tests.sh -a # run tests on all OpenRefine versions (from 2.0 up to 3.5.0)
|
||||
./tests.sh -t 3.5.0 # run tests on tag 3.5.0
|
||||
./tests.sh -t 3.5.0 -i # run tests on tag 3.5.0 interactively (pause before and after tests)
|
||||
./tests.sh -t 3.5.0 -t 2.7 # run tests on tags 3.5.0 and 2.7
|
||||
|
||||
Advanced:
|
||||
./tests.sh -j # run tests on all OpenRefine versions and each with all supported Java versions (requires a lot of docker images to be downloaded!)
|
||||
|
||||
57
tests/apply-utf8.sh
Normal file
57
tests/apply-utf8.sh
Normal file
@ -0,0 +1,57 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.transform"
|
||||
[
|
||||
{
|
||||
"op": "core/column-addition",
|
||||
"engineConfig": {
|
||||
"mode": "row-based"
|
||||
},
|
||||
"newColumnName": "apply",
|
||||
"columnInsertIndex": 2,
|
||||
"baseColumnName": "b",
|
||||
"expression": "grel:value.replace('2','⛲')",
|
||||
"onError": "set-to-blank"
|
||||
}
|
||||
]
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b apply c
|
||||
1 2 ⛲ 3
|
||||
0 0 0 0
|
||||
$ \ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
57
tests/apply.sh
Normal file
57
tests/apply.sh
Normal file
@ -0,0 +1,57 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.transform"
|
||||
[
|
||||
{
|
||||
"op": "core/column-addition",
|
||||
"engineConfig": {
|
||||
"mode": "row-based"
|
||||
},
|
||||
"newColumnName": "apply",
|
||||
"columnInsertIndex": 2,
|
||||
"baseColumnName": "b",
|
||||
"expression": "grel:value.replace('2','TEST')",
|
||||
"onError": "set-to-blank"
|
||||
}
|
||||
]
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b apply c
|
||||
1 2 TEST 3
|
||||
0 0 0 0
|
||||
$ \ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
@ -1,819 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Test executable in a Linux Bash environment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Install\n",
|
||||
"\n",
|
||||
"This notebook requires a [Bash kernel](https://github.com/takluyver/bash_kernel) environment and an OpenRefine server running at http://127.0.0.1:3333."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 49,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/tmp/20190822_013937\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"workspace=$(date +%Y%m%d_%H%M%S)\n",
|
||||
"mkdir -p /tmp/$workspace\n",
|
||||
"cp -r data /tmp/$workspace\n",
|
||||
"cd /tmp/$workspace && pwd"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 50,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.8/openrefine-client_0-3-8_linux:\n",
|
||||
"2019-08-22 01:39:40 ERROR 404: Not Found.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"wget -nv https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.8/openrefine-client_0-3-8_linux -O openrefine-client\n",
|
||||
"chmod +x openrefine-client"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## README.MD"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Download"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 51,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Download to file duplicates.csv complete\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --download \"https://git.io/fj5hF\" --output=duplicates.csv"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Create"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 52,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"id: 2334935475634\n",
|
||||
"rows: 10\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --create duplicates.csv"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### List"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 53,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 2334935475634: duplicates\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Info"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 54,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" id: 2334935475634\n",
|
||||
" url: http://127.0.0.1:3333/project?project=2334935475634\n",
|
||||
" name: duplicates\n",
|
||||
" modified: 2019-08-21T23:40:30Z\n",
|
||||
" created: 2019-08-21T23:40:30Z\n",
|
||||
" rowCount: 10\n",
|
||||
"importOptionMetadata: [{u'storeEmptyStrings': True, u'fileSource': u'duplicates.csv', u'storeBlankRows': True, u'encoding': u'', u'projectName': u'duplicates', u'processQuotes': True, u'separator': u',', u'trimStrings': False, u'limit': -1, u'storeBlankCellsAsNulls': True, u'guessCellValueTypes': False, u'includeFileSources': False}]\n",
|
||||
" column 001: email\n",
|
||||
" column 002: name\n",
|
||||
" column 003: state\n",
|
||||
" column 004: gender\n",
|
||||
" column 005: purchase\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --info \"duplicates\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Export"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 55,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"email\tname\tstate\tgender\tpurchase\n",
|
||||
"danny.baron@example1.com\tDanny Baron\tCA\tM\tTV\n",
|
||||
"melanie.white@example2.edu\tMelanie White\tNC\tF\tiPhone\n",
|
||||
"danny.baron@example1.com\tD. Baron\tCA\tM\tWinter jacket\n",
|
||||
"ben.tyler@example3.org\tBen Tyler\tNV\tM\tFlashlight\n",
|
||||
"arthur.duff@example4.com\tArthur Duff\tOR\tM\tDining table\n",
|
||||
"danny.baron@example1.com\tDaniel Baron\tCA\tM\tBike\n",
|
||||
"jean.griffith@example5.org\tJean Griffith\tWA\tF\tPower drill\n",
|
||||
"melanie.white@example2.edu\tMelanie White\tNC\tF\tiPad\n",
|
||||
"ben.morisson@example6.org\tBen Morisson\tFL\tM\tAmplifier\n",
|
||||
"arthur.duff@example4.com\tArthur Duff\tOR\tM\tNight table\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --export \"duplicates\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Apply"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 56,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Download to file duplicates-deletion.json complete\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --download \"https://git.io/fj5ju\" --output=duplicates-deletion.json"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 57,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"File duplicates-deletion.json has been successfully applied to project 2334935475634\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --apply duplicates-deletion.json \"duplicates\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 58,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"email\tcount\tname\tstate\tgender\tpurchase\n",
|
||||
"arthur.duff@example4.com\t2\tArthur Duff\tOR\tM\tDining table\n",
|
||||
"ben.morisson@example6.org\t1\tBen Morisson\tFL\tM\tAmplifier\n",
|
||||
"ben.tyler@example3.org\t1\tBen Tyler\tNV\tM\tFlashlight\n",
|
||||
"danny.baron@example1.com\t3\tDanny Baron\tCA\tM\tTV\n",
|
||||
"jean.griffith@example5.org\t1\tJean Griffith\tWA\tF\tPower drill\n",
|
||||
"melanie.white@example2.edu\t2\tMelanie White\tNC\tF\tiPhone\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --export \"duplicates\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Export XLS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 59,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Export to file deduped.xls complete\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --export \"duplicates\" --output deduped.xls"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Delete"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 60,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Project 2334935475634 has been successfully deleted\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --delete \"duplicates\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Templating"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 61,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"id: 1633409429491\n",
|
||||
"rows: 10\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --create duplicates.csv --projectName=advanced"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 62,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"{ \"events\" : [\n",
|
||||
" { \"name\" : \"Melanie White\", \"purchase\" : \"iPhone\" },\n",
|
||||
" { \"name\" : \"Jean Griffith\", \"purchase\" : \"Power drill\" },\n",
|
||||
" { \"name\" : \"Melanie White\", \"purchase\" : \"iPad\" }\n",
|
||||
"] }"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --export \"advanced\" \\\n",
|
||||
"--prefix='{ \"events\" : [\n",
|
||||
"' \\\n",
|
||||
"--template=' { \"name\" : {{jsonize(cells[\"name\"].value)}}, \"purchase\" : {{jsonize(cells[\"purchase\"].value)}} }' \\\n",
|
||||
"--rowSeparator=',\n",
|
||||
"' \\\n",
|
||||
"--suffix='\n",
|
||||
"] }' \\\n",
|
||||
"--filterQuery='^F$' \\\n",
|
||||
"--filterColumn='gender'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 63,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Export to files complete. Last file: advanced_3.json\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --export \"advanced\" \\\n",
|
||||
"--prefix='{ \"events\" : [\n",
|
||||
"' \\\n",
|
||||
"--template=' { \"name\" : {{jsonize(cells[\"name\"].value)}}, \"purchase\" : {{jsonize(cells[\"purchase\"].value)}} }' \\\n",
|
||||
"--rowSeparator=',\n",
|
||||
"' \\\n",
|
||||
"--suffix='\n",
|
||||
"] }' \\\n",
|
||||
"--filterQuery='^F$' \\\n",
|
||||
"--filterColumn='gender' \\\n",
|
||||
"--output=advanced.json \\\n",
|
||||
"--splitToFiles=true"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 64,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Export to files complete. Last file: advanced_melanie.white@example2.edu.json\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --export \"advanced\" \\\n",
|
||||
"--prefix='{ \"events\" : [\n",
|
||||
"' \\\n",
|
||||
"--template=' { \"name\" : {{jsonize(cells[\"name\"].value)}}, \"purchase\" : {{jsonize(cells[\"purchase\"].value)}} }' \\\n",
|
||||
"--rowSeparator=',\n",
|
||||
"' \\\n",
|
||||
"--suffix='\n",
|
||||
"] }' \\\n",
|
||||
"--filterQuery='^F$' \\\n",
|
||||
"--filterColumn='gender' \\\n",
|
||||
"--output=advanced.json \\\n",
|
||||
"--splitToFiles=true \\\n",
|
||||
"--suffixById=true"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 65,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"advanced_1.json \u001b[0m\u001b[38;5;33mdata\u001b[0m\n",
|
||||
"advanced_2.json deduped.xls\n",
|
||||
"advanced_3.json duplicates.csv\n",
|
||||
"advanced_jean.griffith@example5.org.json duplicates-deletion.json\n",
|
||||
"advanced_melanie.white@example2.edu.json \u001b[38;5;40mopenrefine-client\u001b[0m\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"ls"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Delete"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 66,
|
||||
"metadata": {
|
||||
"scrolled": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Project 1633409429491 has been successfully deleted\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --delete \"advanced\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Unicode"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### fruits"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 67,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"id: 2280962953279\n",
|
||||
"rows: 5\n",
|
||||
" id: 2280962953279\n",
|
||||
" url: http://127.0.0.1:3333/project?project=2280962953279\n",
|
||||
" name: evil-fruits\n",
|
||||
" modified: 2019-08-21T23:40:43Z\n",
|
||||
" created: 2019-08-21T23:40:43Z\n",
|
||||
" rowCount: 5\n",
|
||||
"importOptionMetadata: [{u'storeEmptyStrings': True, u'fileSource': u'data/cli/evil-fruits.tsv', u'storeBlankRows': True, u'encoding': u'', u'projectName': u'evil-fruits', u'processQuotes': True, u'limit': -1, u'trimStrings': False, u'storeBlankCellsAsNulls': True, u'guessCellValueTypes': False, u'includeFileSources': False}]\n",
|
||||
" column 001: 🔣\n",
|
||||
" column 002: code\n",
|
||||
" column 003: meaning\n",
|
||||
"🔣\tcode\tmeaning\n",
|
||||
"🍇\t1F347\tGRAPES\n",
|
||||
"🍉\t1F349\tWATERMELON\n",
|
||||
"🍒\t1F352\tCHERRIES\n",
|
||||
"🍓\t1F353\tSTRAWBERRY\n",
|
||||
"🍍\t1F34D\tPINEAPPLE\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --create data/cli/evil-fruits.tsv\n",
|
||||
"./openrefine-client --info \"evil-fruits\"\n",
|
||||
"./openrefine-client --export \"evil-fruits\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 68,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Export to file emojis.csv complete\n",
|
||||
"🔣,code,meaning\n",
|
||||
"🍇,1F347,GRAPES\n",
|
||||
"🍉,1F349,WATERMELON\n",
|
||||
"🍒,1F352,CHERRIES\n",
|
||||
"🍓,1F353,STRAWBERRY\n",
|
||||
"🍍,1F34D,PINEAPPLE\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --export \"evil-fruits\" --output emojis.csv\n",
|
||||
"cat emojis.csv"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 69,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"{ \"emojis\" : [\n",
|
||||
" { \"symbol\" : \"🍇\", \"meaning\" : \"GRAPES\" },\n",
|
||||
" { \"symbol\" : \"🍉\", \"meaning\" : \"WATERMELON\" },\n",
|
||||
" { \"symbol\" : \"🍍\", \"meaning\" : \"PINEAPPLE\" }\n",
|
||||
"] }"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --export \"evil-fruits\" \\\n",
|
||||
"--prefix='{ \"emojis\" : [\n",
|
||||
"' \\\n",
|
||||
"--template=' { \"symbol\" : {{jsonize(with(row.columnNames[0],cn,cells[cn].value))}}, \"meaning\" : {{jsonize(cells[\"meaning\"].value)}} }' \\\n",
|
||||
"--rowSeparator=',\n",
|
||||
"' \\\n",
|
||||
"--suffix='\n",
|
||||
"] }' \\\n",
|
||||
"--filterQuery='^1F34' \\\n",
|
||||
"--filterColumn='code'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 70,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Export to files complete. Last file: trái cây_3.json\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --export \"evil-fruits\" \\\n",
|
||||
"--prefix='{ \"emojis\" : [\n",
|
||||
"' \\\n",
|
||||
"--template=' { \"symbol\" : {{jsonize(with(row.columnNames[0],cn,cells[cn].value))}}, \"meaning\" : {{jsonize(cells[\"meaning\"].value)}} }' \\\n",
|
||||
"--rowSeparator=',\n",
|
||||
"' \\\n",
|
||||
"--suffix='\n",
|
||||
"] }' \\\n",
|
||||
"--filterQuery='^1F34' \\\n",
|
||||
"--filterColumn='code' \\\n",
|
||||
"--output='trái cây.json' \\\n",
|
||||
"--splitToFiles=true"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 71,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Export to files complete. Last file: trái cây_🍍.json\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --export \"evil-fruits\" \\\n",
|
||||
"--prefix='{ \"emojis\" : [\n",
|
||||
"' \\\n",
|
||||
"--template=' { \"symbol\" : {{jsonize(with(row.columnNames[0],cn,cells[cn].value))}}, \"meaning\" : {{jsonize(cells[\"meaning\"].value)}} }' \\\n",
|
||||
"--rowSeparator=',\n",
|
||||
"' \\\n",
|
||||
"--suffix='\n",
|
||||
"] }' \\\n",
|
||||
"--filterQuery='^1F34' \\\n",
|
||||
"--filterColumn='code' \\\n",
|
||||
"--output='trái cây.json' \\\n",
|
||||
"--splitToFiles=true \\\n",
|
||||
"--suffixById=true"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 72,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" advanced_1.json emojis.csv\n",
|
||||
" advanced_2.json \u001b[0m\u001b[38;5;40mopenrefine-client\u001b[0m\n",
|
||||
" advanced_3.json 'trái cây_1.json'\n",
|
||||
" advanced_jean.griffith@example5.org.json 'trái cây_2.json'\n",
|
||||
" advanced_melanie.white@example2.edu.json 'trái cây_3.json'\n",
|
||||
" \u001b[38;5;33mdata\u001b[0m 'trái cây_🍇.json'\n",
|
||||
" deduped.xls 'trái cây_🍉.json'\n",
|
||||
" duplicates.csv 'trái cây_🍍.json'\n",
|
||||
" duplicates-deletion.json\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"ls"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 73,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Project 2280962953279 has been successfully deleted\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --delete \"evil-fruits\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### emoji-data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 74,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"id: 2019865211741\n",
|
||||
"rows: 20\n",
|
||||
" id: 2019865211741\n",
|
||||
" url: http://127.0.0.1:3333/project?project=2019865211741\n",
|
||||
" name: dữ liệu biểu tượng cảm xúc\n",
|
||||
" modified: 2019-08-21T23:41:06Z\n",
|
||||
" created: 2019-08-21T23:41:06Z\n",
|
||||
" rowCount: 20\n",
|
||||
"importOptionMetadata: [{u'storeEmptyStrings': True, u'fileSource': u'data/cli/d\\u1eef li\\u1ec7u bi\\u1ec3u t\\u01b0\\u1ee3ng c\\u1ea3m x\\xfac.txt', u'storeBlankRows': True, u'encoding': u'', u'projectName': u'd\\u1eef li\\u1ec7u bi\\u1ec3u t\\u01b0\\u1ee3ng c\\u1ea3m x\\xfac', u'processQuotes': True, u'skipDataLines': 34, u'limit': 20, u'trimStrings': False, u'storeBlankCellsAsNulls': True, u'guessCellValueTypes': False, u'includeFileSources': False, u'headerLines': 0}]\n",
|
||||
" column 001: Column 1\n",
|
||||
" column 002: Column 2\n",
|
||||
" column 003: Column 3\n",
|
||||
" column 004: Column 4\n",
|
||||
" column 005: Column 5\n",
|
||||
" column 006: Column 6\n",
|
||||
"Column 1\tColumn 2\tColumn 3\tColumn 4\tColumn 5\tColumn 6\n",
|
||||
"00A9 ;\ttext ;\tL1 ;\tnone ;\tj\t# V1.1 (©) COPYRIGHT SIGN\n",
|
||||
"00AE ;\ttext ;\tL1 ;\tnone ;\tj\t# V1.1 (®) REGISTERED SIGN\n",
|
||||
"203C ;\ttext ;\tL1 ;\tnone ;\ta j\t# V1.1 (‼) DOUBLE EXCLAMATION MARK\n",
|
||||
"2049 ;\ttext ;\tL1 ;\tnone ;\ta j\t# V3.0 (⁉) EXCLAMATION QUESTION MARK\n",
|
||||
"2122 ;\ttext ;\tL1 ;\tnone ;\tj\t# V1.1 (™) TRADE MARK SIGN\n",
|
||||
"2139 ;\ttext ;\tL1 ;\tnone ;\tj\t# V3.0 (ℹ) INFORMATION SOURCE\n",
|
||||
"2194 ;\ttext ;\tL1 ;\tnone ;\tz j\t# V1.1 (↔) LEFT RIGHT ARROW\n",
|
||||
"2195 ;\ttext ;\tL1 ;\tnone ;\tz j\t# V1.1 (↕) UP DOWN ARROW\n",
|
||||
"2196 ;\ttext ;\tL1 ;\tnone ;\tj\t# V1.1 (↖) NORTH WEST ARROW\n",
|
||||
"2197 ;\ttext ;\tL1 ;\tnone ;\tj\t# V1.1 (↗) NORTH EAST ARROW\n",
|
||||
"2198 ;\ttext ;\tL1 ;\tnone ;\tj\t# V1.1 (↘) SOUTH EAST ARROW\n",
|
||||
"2199 ;\ttext ;\tL1 ;\tnone ;\tj\t# V1.1 (↙) SOUTH WEST ARROW\n",
|
||||
"21A9 ;\ttext ;\tL1 ;\tnone ;\tj\t# V1.1 (↩) LEFTWARDS ARROW WITH HOOK\n",
|
||||
"21AA ;\ttext ;\tL1 ;\tnone ;\tj\t# V1.1 (↪) RIGHTWARDS ARROW WITH HOOK\n",
|
||||
"231A ;\temoji ;\tL1 ;\tnone ;\tj\t# V1.1 (⌚) WATCH\n",
|
||||
"231B ;\temoji ;\tL1 ;\tnone ;\tj\t# V1.1 (⌛) HOURGLASS\n",
|
||||
"2328 ;\ttext ;\tL2 ;\tnone ;\tx\t# V1.1 (⌨) KEYBOARD\n",
|
||||
"23CF ;\ttext ;\tL2 ;\tnone ;\tx\t# V4.0 (⏏) EJECT SYMBOL\n",
|
||||
"23E9 ;\temoji ;\tL1 ;\tnone ;\tj w\t# V6.0 (⏩) BLACK RIGHT-POINTING DOUBLE TRIANGLE\n",
|
||||
"23EA ;\temoji ;\tL1 ;\tnone ;\tj w\t# V6.0 (⏪) BLACK LEFT-POINTING DOUBLE TRIANGLE\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --create \"data/cli/dữ liệu biểu tượng cảm xúc.txt\" \\\n",
|
||||
"--format=tsv \\\n",
|
||||
"--headerLines=0 \\\n",
|
||||
"--skipDataLines=34 \\\n",
|
||||
"--limit=20\n",
|
||||
"./openrefine-client --info \"dữ liệu biểu tượng cảm xúc\"\n",
|
||||
"./openrefine-client --export \"dữ liệu biểu tượng cảm xúc\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 75,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
" 2019865211741: dữ liệu biểu tượng cảm xúc\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --list"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Delete"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 76,
|
||||
"metadata": {
|
||||
"scrolled": false
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Project 2019865211741 has been successfully deleted\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"./openrefine-client --delete \"dữ liệu biểu tượng cảm xúc\""
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Bash",
|
||||
"language": "bash",
|
||||
"name": "bash"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": "shell",
|
||||
"file_extension": ".sh",
|
||||
"mimetype": "text/x-sh",
|
||||
"name": "bash"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
41
tests/create-csv-encoding.sh
Normal file
41
tests/create-csv-encoding.sh
Normal file
@ -0,0 +1,41 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}-utf8.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
ä,é,ß
|
||||
$,\,'
|
||||
DATA
|
||||
iconv -f UTF-8 -t ISO-8859-1 "tmp/${t}/${t}-utf8.csv" > "tmp/${t}/${t}.csv"
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
1 2 3
|
||||
ä é ß
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --encoding "ISO-8859-1"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
40
tests/create-csv-guessCellValueTypes.sh
Normal file
40
tests/create-csv-guessCellValueTypes.sh
Normal file
@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
01,02,03
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
1 2 3
|
||||
1 2 3
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --guessCellValueTypes "true"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
41
tests/create-csv-headerLines.sh
Normal file
41
tests/create-csv-headerLines.sh
Normal file
@ -0,0 +1,41 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
Column 1 Column 2 Column 3
|
||||
a b c
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --headerLines "0"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
39
tests/create-csv-ignoreLines.sh
Normal file
39
tests/create-csv-ignoreLines.sh
Normal file
@ -0,0 +1,39 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --ignoreLines "1"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
39
tests/create-csv-limit.sh
Normal file
39
tests/create-csv-limit.sh
Normal file
@ -0,0 +1,39 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
1 2 3
|
||||
0 0 0
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --limit "2"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
41
tests/create-csv-processQuotes.sh
Normal file
41
tests/create-csv-processQuotes.sh
Normal file
@ -0,0 +1,41 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,"2,0",3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c Column 4
|
||||
1 2 0 3
|
||||
0 0 0
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
# OpenRefine 4.x fails without manually set headerLines
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --processQuotes "false" --headerLines 1
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
45
tests/create-csv-projectTags.sh
Normal file
45
tests/create-csv-projectTags.sh
Normal file
@ -0,0 +1,45 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
if [[ ${2} ]]; then
|
||||
version="${2}"
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
if [[ "${version:0:1}" = "2" ]]; then
|
||||
echo "projectTags were introduced in OpenRefine 3.0"
|
||||
exit 200
|
||||
else
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
tags: [u'beta', u'client1']
|
||||
DATA
|
||||
fi
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --projectTags "beta" --projectTags "client1"
|
||||
${cmd} --info "${t}" | grep ' tags: ' > "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
40
tests/create-csv-separator.sh
Normal file
40
tests/create-csv-separator.sh
Normal file
@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a;b;c
|
||||
1;2;3
|
||||
0;0;0
|
||||
$;\;'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --separator ";"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
39
tests/create-csv-skipDataLines.sh
Normal file
39
tests/create-csv-skipDataLines.sh
Normal file
@ -0,0 +1,39 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
0 0 0
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --skipDataLines "1"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
58
tests/create-csv-storeBlankCellsAsNulls.sh
Normal file
58
tests/create-csv-storeBlankCellsAsNulls.sh
Normal file
@ -0,0 +1,58 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.transform"
|
||||
[
|
||||
{
|
||||
"op": "core/text-transform",
|
||||
"engineConfig": {
|
||||
"facets": [],
|
||||
"mode": "row-based"
|
||||
},
|
||||
"columnName": "b",
|
||||
"expression": "grel:isNull(value)",
|
||||
"onError": "keep-original",
|
||||
"repeat": false,
|
||||
"repeatCount": 10
|
||||
}
|
||||
]
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
1 false 3
|
||||
0 false 0
|
||||
$ false '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --storeBlankCellsAsNulls "false"
|
||||
${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
39
tests/create-csv-storeBlankRows.sh
Normal file
39
tests/create-csv-storeBlankRows.sh
Normal file
@ -0,0 +1,39 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
,,
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
1 2 3
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --storeBlankRows "false"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
40
tests/create-csv-utf8.sh
Normal file
40
tests/create-csv-utf8.sh
Normal file
@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
⌨,code,meaning
|
||||
⛲,1F347,FOUNTAIN
|
||||
⛳,1F349,FLAG IN HOLE
|
||||
⛵,1F352,SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
⌨ code meaning
|
||||
⛲ 1F347 FOUNTAIN
|
||||
⛳ 1F349 FLAG IN HOLE
|
||||
⛵ 1F352 SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --projectName "${t} biểu tượng cảm xúc ⛲"
|
||||
${cmd} --export "${t} biểu tượng cảm xúc ⛲" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
40
tests/create-csv.sh
Normal file
40
tests/create-csv.sh
Normal file
@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
55
tests/create-json-recordPath.sh
Normal file
55
tests/create-json-recordPath.sh
Normal file
@ -0,0 +1,55 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.json"
|
||||
{
|
||||
"rows":[
|
||||
{
|
||||
"a":1,
|
||||
"b":2,
|
||||
"c":3
|
||||
},
|
||||
{
|
||||
"a":0,
|
||||
"b":0,
|
||||
"c":0
|
||||
},
|
||||
{
|
||||
"a":"$",
|
||||
"b":"\\",
|
||||
"c":"\""
|
||||
}
|
||||
]
|
||||
}
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
_ - a _ - b _ - c
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ """"
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.json" --recordPath "_" --recordPath "rows" --recordPath "_"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
52
tests/create-json-storeEmptyStrings.sh
Normal file
52
tests/create-json-storeEmptyStrings.sh
Normal file
@ -0,0 +1,52 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.json"
|
||||
[
|
||||
{
|
||||
"a": 1,
|
||||
"b": 2,
|
||||
"c": 3
|
||||
},
|
||||
{
|
||||
"a": "",
|
||||
"b": "",
|
||||
"c": ""
|
||||
},
|
||||
{
|
||||
"a": "$",
|
||||
"b": "\\",
|
||||
"c": "\""
|
||||
}
|
||||
]
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
_ - a _ - b _ - c
|
||||
1 2 3
|
||||
$ \ """"
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.json" --storeEmptyStrings "false"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
62
tests/create-json-trimStrings.sh
Normal file
62
tests/create-json-trimStrings.sh
Normal file
@ -0,0 +1,62 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
if [[ ${2} ]]; then
|
||||
version="${2}"
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.json"
|
||||
[
|
||||
{
|
||||
"a": 1,
|
||||
"b": 2,
|
||||
"c": 3
|
||||
},
|
||||
{
|
||||
"a": "0",
|
||||
"b": " 0",
|
||||
"c": "0 "
|
||||
},
|
||||
{
|
||||
"a": "$",
|
||||
"b": "\\",
|
||||
"c": "\""
|
||||
}
|
||||
]
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
if [[ "${version:0:1}" = "2" || "${version}" = "3.0" || "${version}" = "3.1" || "${version}" = "3.2" || "${version}" = "3.3" ]]; then
|
||||
echo "trimStrings option does not work in OpenRefine <=3.3"
|
||||
echo "https://github.com/OpenRefine/OpenRefine/issues/2409"
|
||||
exit 200
|
||||
else
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
_ - a _ - b _ - c
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ """"
|
||||
DATA
|
||||
fi
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.json" --trimStrings "true"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
53
tests/create-json-utf8.sh
Normal file
53
tests/create-json-utf8.sh
Normal file
@ -0,0 +1,53 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.json"
|
||||
[
|
||||
{
|
||||
"⌨": "⛲",
|
||||
"code": "1F347",
|
||||
"meaning": "FOUNTAIN"
|
||||
},
|
||||
{
|
||||
"⌨": "⛳",
|
||||
"code": "1F349",
|
||||
"meaning": "FLAG IN HOLE"
|
||||
},
|
||||
{
|
||||
"⌨": "⛵",
|
||||
"code": "1F352",
|
||||
"meaning": "SAILBOAT"
|
||||
}
|
||||
]
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
_ - ⌨ _ - code _ - meaning
|
||||
⛲ 1F347 FOUNTAIN
|
||||
⛳ 1F349 FLAG IN HOLE
|
||||
⛵ 1F352 SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.json"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
53
tests/create-json.sh
Normal file
53
tests/create-json.sh
Normal file
@ -0,0 +1,53 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.json"
|
||||
[
|
||||
{
|
||||
"a": 1,
|
||||
"b": 2,
|
||||
"c": 3
|
||||
},
|
||||
{
|
||||
"a": 0,
|
||||
"b": 0,
|
||||
"c": 0
|
||||
},
|
||||
{
|
||||
"a": "$",
|
||||
"b": "\\",
|
||||
"c": "\""
|
||||
}
|
||||
]
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
_ - a _ - b _ - c
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ """"
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.json"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
44
tests/create-ods-sheets-utf8.sh
Normal file
44
tests/create-ods-sheets-utf8.sh
Normal file
@ -0,0 +1,44 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
if [[ ${2} ]]; then
|
||||
version="${2}"
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cp "data/example.ods" "tmp/${t}/${t}.ods"
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
if [[ "${version}" = "2.7" ]]; then
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
⌨ code meaning Column Column 5 Column 6 Column 7 Column 8
|
||||
⛲ 1F347 FOUNTAIN
|
||||
⛳ 1F349 FLAG IN HOLE
|
||||
⛵ 1F352 SAILBOAT
|
||||
DATA
|
||||
else
|
||||
#TODO
|
||||
echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
|
||||
exit 200
|
||||
fi
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.ods" --sheets 1
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
48
tests/create-ods.sh
Normal file
48
tests/create-ods.sh
Normal file
@ -0,0 +1,48 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
if [[ ${2} ]]; then
|
||||
version="${2}"
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cp "data/example.ods" "tmp/${t}/${t}.ods"
|
||||
#a b c
|
||||
#1 2 3
|
||||
#0 0 0
|
||||
#$ \ '
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
if [[ "${version}" = "2.7" ]]; then
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c Column Column 5 Column 6 Column 7 Column 8
|
||||
1.0 2.0 3.0
|
||||
0.0 0.0 0.0
|
||||
$ \ '
|
||||
DATA
|
||||
else
|
||||
#TODO
|
||||
echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
|
||||
exit 200
|
||||
fi
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.ods"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
40
tests/create-tsv-utf8.sh
Normal file
40
tests/create-tsv-utf8.sh
Normal file
@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.tsv"
|
||||
⌨ code meaning
|
||||
⛲ 1F347 FOUNTAIN
|
||||
⛳ 1F349 FLAG IN HOLE
|
||||
⛵ 1F352 SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
⌨,code,meaning
|
||||
⛲,1F347,FOUNTAIN
|
||||
⛳,1F349,FLAG IN HOLE
|
||||
⛵,1F352,SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.tsv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.csv"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.csv"
|
||||
40
tests/create-tsv.sh
Normal file
40
tests/create-tsv.sh
Normal file
@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.tsv"
|
||||
a b c
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.tsv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.csv"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.csv"
|
||||
80
tests/create-txt-fixed-width-headerLines.sh
Normal file
80
tests/create-txt-fixed-width-headerLines.sh
Normal file
@ -0,0 +1,80 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.txt"
|
||||
1 2 3
|
||||
mon tue wed
|
||||
$2 $300 $1
|
||||
DATA
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.transform"
|
||||
[
|
||||
{
|
||||
"op": "core/text-transform",
|
||||
"engineConfig": {
|
||||
"facets": [],
|
||||
"mode": "row-based"
|
||||
},
|
||||
"columnName": "1",
|
||||
"expression": "grel:value.trim()",
|
||||
"onError": "keep-original",
|
||||
"repeat": false,
|
||||
"repeatCount": 10
|
||||
},
|
||||
{
|
||||
"op": "core/text-transform",
|
||||
"engineConfig": {
|
||||
"facets": [],
|
||||
"mode": "row-based"
|
||||
},
|
||||
"columnName": "2",
|
||||
"expression": "grel:value.trim()",
|
||||
"onError": "keep-original",
|
||||
"repeat": false,
|
||||
"repeatCount": 10
|
||||
},
|
||||
{
|
||||
"op": "core/text-transform",
|
||||
"engineConfig": {
|
||||
"facets": [],
|
||||
"mode": "row-based"
|
||||
},
|
||||
"columnName": "3",
|
||||
"expression": "grel:value.trim()",
|
||||
"onError": "keep-original",
|
||||
"repeat": false,
|
||||
"repeatCount": 10
|
||||
}
|
||||
]
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
1 2 3
|
||||
mon tue wed
|
||||
$2 $300 $1
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.txt" --columnWidths "6" --columnWidths "6" --columnWidths "6" --headerLines "1"
|
||||
${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
81
tests/create-txt-fixed-width-utf8.sh
Normal file
81
tests/create-txt-fixed-width-utf8.sh
Normal file
@ -0,0 +1,81 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.txt"
|
||||
⛲ 1F347 FOUNTAIN
|
||||
⛳ 1F349 FLAG IN HOLE
|
||||
⛵ 1F352 SAILBOAT
|
||||
DATA
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.transform"
|
||||
[
|
||||
{
|
||||
"op": "core/text-transform",
|
||||
"engineConfig": {
|
||||
"facets": [],
|
||||
"mode": "row-based"
|
||||
},
|
||||
"columnName": "Column 1",
|
||||
"expression": "grel:value.trim()",
|
||||
"onError": "keep-original",
|
||||
"repeat": false,
|
||||
"repeatCount": 10
|
||||
},
|
||||
{
|
||||
"op": "core/text-transform",
|
||||
"engineConfig": {
|
||||
"facets": [],
|
||||
"mode": "row-based"
|
||||
},
|
||||
"columnName": "Column 2",
|
||||
"expression": "grel:value.trim()",
|
||||
"onError": "keep-original",
|
||||
"repeat": false,
|
||||
"repeatCount": 10
|
||||
},
|
||||
{
|
||||
"op": "core/text-transform",
|
||||
"engineConfig": {
|
||||
"facets": [],
|
||||
"mode": "row-based"
|
||||
},
|
||||
"columnName": "Column 3",
|
||||
"expression": "grel:value.trim()",
|
||||
"onError": "keep-original",
|
||||
"repeat": false,
|
||||
"repeatCount": 10
|
||||
}
|
||||
]
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
Column 1 Column 2 Column 3
|
||||
⛲ 1F347 FOUNTAIN
|
||||
⛳ 1F349 FLAG IN HOLE
|
||||
⛵ 1F352 SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.txt" --columnWidths "6" --columnWidths "6" --columnWidths "60"
|
||||
${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
81
tests/create-txt-fixed-width.sh
Normal file
81
tests/create-txt-fixed-width.sh
Normal file
@ -0,0 +1,81 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.txt"
|
||||
1 2 3
|
||||
mon tue wed
|
||||
$2 $300 $1
|
||||
DATA
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.transform"
|
||||
[
|
||||
{
|
||||
"op": "core/text-transform",
|
||||
"engineConfig": {
|
||||
"facets": [],
|
||||
"mode": "row-based"
|
||||
},
|
||||
"columnName": "Column 1",
|
||||
"expression": "grel:value.trim()",
|
||||
"onError": "keep-original",
|
||||
"repeat": false,
|
||||
"repeatCount": 10
|
||||
},
|
||||
{
|
||||
"op": "core/text-transform",
|
||||
"engineConfig": {
|
||||
"facets": [],
|
||||
"mode": "row-based"
|
||||
},
|
||||
"columnName": "Column 2",
|
||||
"expression": "grel:value.trim()",
|
||||
"onError": "keep-original",
|
||||
"repeat": false,
|
||||
"repeatCount": 10
|
||||
},
|
||||
{
|
||||
"op": "core/text-transform",
|
||||
"engineConfig": {
|
||||
"facets": [],
|
||||
"mode": "row-based"
|
||||
},
|
||||
"columnName": "Column 3",
|
||||
"expression": "grel:value.trim()",
|
||||
"onError": "keep-original",
|
||||
"repeat": false,
|
||||
"repeatCount": 10
|
||||
}
|
||||
]
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
Column 1 Column 2 Column 3
|
||||
1 2 3
|
||||
mon tue wed
|
||||
$2 $300 $1
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.txt" --columnWidths "6" --columnWidths "6" --columnWidths "6"
|
||||
${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
39
tests/create-txt-linesPerRow.sh
Normal file
39
tests/create-txt-linesPerRow.sh
Normal file
@ -0,0 +1,39 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.txt"
|
||||
mon tue wed
|
||||
$2 $300 $1
|
||||
thu fri sat
|
||||
$70 $20 $50
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
Column 1 Column 2
|
||||
mon tue wed $2 $300 $1
|
||||
thu fri sat $70 $20 $50
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.txt" --linesPerRow "2"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
39
tests/create-txt.sh
Normal file
39
tests/create-txt.sh
Normal file
@ -0,0 +1,39 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.txt"
|
||||
1 2 3
|
||||
mon tue wed
|
||||
$2 $300 $1
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
Column 1
|
||||
1 2 3
|
||||
mon tue wed
|
||||
$2 $300 $1
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.txt"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
44
tests/create-xls-sheets-utf8.sh
Normal file
44
tests/create-xls-sheets-utf8.sh
Normal file
@ -0,0 +1,44 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
if [[ ${2} ]]; then
|
||||
version="${2}"
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cp "data/example.xls" "tmp/${t}/${t}.xls"
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
if [[ "${version}" = "2.7" ]]; then
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
⌨ code meaning
|
||||
⛲ 1F347 FOUNTAIN
|
||||
⛳ 1F349 FLAG IN HOLE
|
||||
⛵ 1F352 SAILBOAT
|
||||
DATA
|
||||
else
|
||||
#TODO
|
||||
echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
|
||||
exit 200
|
||||
fi
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.xls" --sheets 1
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
48
tests/create-xls.sh
Normal file
48
tests/create-xls.sh
Normal file
@ -0,0 +1,48 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
if [[ ${2} ]]; then
|
||||
version="${2}"
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cp "data/example.xls" "tmp/${t}/${t}.xls"
|
||||
#a b c
|
||||
#1 2 3
|
||||
#0 0 0
|
||||
#$ \ '
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
if [[ "${version}" = "2.7" ]]; then
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
1.0 2.0 3.0
|
||||
0.0 0.0 0.0
|
||||
$ \ '
|
||||
DATA
|
||||
else
|
||||
#TODO
|
||||
echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
|
||||
exit 200
|
||||
fi
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.xls"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
44
tests/create-xlsx-sheets-utf8.sh
Normal file
44
tests/create-xlsx-sheets-utf8.sh
Normal file
@ -0,0 +1,44 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
if [[ ${2} ]]; then
|
||||
version="${2}"
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cp "data/example.xlsx" "tmp/${t}/${t}.xlsx"
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
if [[ "${version}" = "2.7" ]]; then
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
⌨ code meaning
|
||||
⛲ 1F347 FOUNTAIN
|
||||
⛳ 1F349 FLAG IN HOLE
|
||||
⛵ 1F352 SAILBOAT
|
||||
DATA
|
||||
else
|
||||
#TODO
|
||||
echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
|
||||
exit 200
|
||||
fi
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.xlsx" --sheets 1
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
48
tests/create-xlsx.sh
Normal file
48
tests/create-xlsx.sh
Normal file
@ -0,0 +1,48 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
if [[ ${2} ]]; then
|
||||
version="${2}"
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cp "data/example.xlsx" "tmp/${t}/${t}.xlsx"
|
||||
#a b c
|
||||
#1 2 3
|
||||
#0 0 0
|
||||
#$ \ '
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
if [[ "${version}" = "2.7" ]]; then
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
1.0 2.0 3.0
|
||||
0.0 0.0 0.0
|
||||
$ \ '
|
||||
DATA
|
||||
else
|
||||
#TODO
|
||||
echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
|
||||
exit 200
|
||||
fi
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.xlsx"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
96
tests/create-xml-recordPath.sh
Normal file
96
tests/create-xml-recordPath.sh
Normal file
@ -0,0 +1,96 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.xml"
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<root>
|
||||
<record>
|
||||
<a>1</a>
|
||||
<b>2</b>
|
||||
<c>3</c>
|
||||
</record>
|
||||
<record>
|
||||
<a>0</a>
|
||||
<b>0</b>
|
||||
<c>0</c>
|
||||
</record>
|
||||
<record>
|
||||
<a>$</a>
|
||||
<b>\</b>
|
||||
<c>'</c>
|
||||
</record>
|
||||
</root>
|
||||
DATA
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.transform"
|
||||
[
|
||||
{
|
||||
"op": "core/column-reorder",
|
||||
"columnNames": [
|
||||
"record - a",
|
||||
"record - b",
|
||||
"record - c"
|
||||
],
|
||||
"description": "Reorder columns"
|
||||
},
|
||||
{
|
||||
"op": "core/row-removal",
|
||||
"engineConfig": {
|
||||
"facets": [
|
||||
{
|
||||
"type": "list",
|
||||
"name": "Blank Rows",
|
||||
"expression": "(filter(row.columnNames,cn,isNonBlank(cells[cn].value)).length()==0).toString()",
|
||||
"columnName": "",
|
||||
"invert": false,
|
||||
"omitBlank": false,
|
||||
"omitError": false,
|
||||
"selection": [
|
||||
{
|
||||
"v": {
|
||||
"v": "true",
|
||||
"l": "true"
|
||||
}
|
||||
}
|
||||
],
|
||||
"selectBlank": false,
|
||||
"selectError": false
|
||||
}
|
||||
],
|
||||
"mode": "record-based"
|
||||
}
|
||||
}
|
||||
]
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
record - a record - b record - c
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.xml" --recordPath "root" --recordPath "record"
|
||||
${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
96
tests/create-xml-utf8.sh
Normal file
96
tests/create-xml-utf8.sh
Normal file
@ -0,0 +1,96 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.xml"
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<root>
|
||||
<record>
|
||||
<icon>⛲</icon>
|
||||
<code>1F347</code>
|
||||
<meaning>FOUNTAIN</meaning>
|
||||
</record>
|
||||
<record>
|
||||
<icon>⛳</icon>
|
||||
<code>1F349</code>
|
||||
<meaning>FLAG IN HOLE</meaning>
|
||||
</record>
|
||||
<record>
|
||||
<icon>⛵</icon>
|
||||
<code>1F352</code>
|
||||
<meaning>SAILBOAT</meaning>
|
||||
</record>
|
||||
</root>
|
||||
DATA
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.transform"
|
||||
[
|
||||
{
|
||||
"op": "core/column-reorder",
|
||||
"columnNames": [
|
||||
"root - record - icon",
|
||||
"root - record - code",
|
||||
"root - record - meaning"
|
||||
],
|
||||
"description": "Reorder columns"
|
||||
},
|
||||
{
|
||||
"op": "core/row-removal",
|
||||
"engineConfig": {
|
||||
"facets": [
|
||||
{
|
||||
"type": "list",
|
||||
"name": "Blank Rows",
|
||||
"expression": "(filter(row.columnNames,cn,isNonBlank(cells[cn].value)).length()==0).toString()",
|
||||
"columnName": "",
|
||||
"invert": false,
|
||||
"omitBlank": false,
|
||||
"omitError": false,
|
||||
"selection": [
|
||||
{
|
||||
"v": {
|
||||
"v": "true",
|
||||
"l": "true"
|
||||
}
|
||||
}
|
||||
],
|
||||
"selectBlank": false,
|
||||
"selectError": false
|
||||
}
|
||||
],
|
||||
"mode": "record-based"
|
||||
}
|
||||
}
|
||||
]
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
root - record - icon root - record - code root - record - meaning
|
||||
⛲ 1F347 FOUNTAIN
|
||||
⛳ 1F349 FLAG IN HOLE
|
||||
⛵ 1F352 SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.xml"
|
||||
${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
96
tests/create-xml.sh
Normal file
96
tests/create-xml.sh
Normal file
@ -0,0 +1,96 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.xml"
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<root>
|
||||
<record>
|
||||
<a>1</a>
|
||||
<b>2</b>
|
||||
<c>3</c>
|
||||
</record>
|
||||
<record>
|
||||
<a>0</a>
|
||||
<b>0</b>
|
||||
<c>0</c>
|
||||
</record>
|
||||
<record>
|
||||
<a>$</a>
|
||||
<b>\</b>
|
||||
<c>'</c>
|
||||
</record>
|
||||
</root>
|
||||
DATA
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.transform"
|
||||
[
|
||||
{
|
||||
"op": "core/column-reorder",
|
||||
"columnNames": [
|
||||
"root - record - a",
|
||||
"root - record - b",
|
||||
"root - record - c"
|
||||
],
|
||||
"description": "Reorder columns"
|
||||
},
|
||||
{
|
||||
"op": "core/row-removal",
|
||||
"engineConfig": {
|
||||
"facets": [
|
||||
{
|
||||
"type": "list",
|
||||
"name": "Blank Rows",
|
||||
"expression": "(filter(row.columnNames,cn,isNonBlank(cells[cn].value)).length()==0).toString()",
|
||||
"columnName": "",
|
||||
"invert": false,
|
||||
"omitBlank": false,
|
||||
"omitError": false,
|
||||
"selection": [
|
||||
{
|
||||
"v": {
|
||||
"v": "true",
|
||||
"l": "true"
|
||||
}
|
||||
}
|
||||
],
|
||||
"selectBlank": false,
|
||||
"selectError": false
|
||||
}
|
||||
],
|
||||
"mode": "record-based"
|
||||
}
|
||||
}
|
||||
]
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
root - record - a root - record - b root - record - c
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.xml"
|
||||
${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
44
tests/create-zip-includeFileSources.sh
Normal file
44
tests/create-zip-includeFileSources.sh
Normal file
@ -0,0 +1,44 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}-1.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
DATA
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}-2.csv"
|
||||
a,b,c
|
||||
4,5,6
|
||||
DATA
|
||||
|
||||
zip "tmp/${t}/${t}.zip" "tmp/${t}/${t}-1.csv" "tmp/${t}/${t}-2.csv"
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << DATA > "tmp/${t}/${t}.assert"
|
||||
File a b c
|
||||
tmp/${t}/${t}-1.csv 1 2 3
|
||||
tmp/${t}/${t}-2.csv 4 5 6
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.zip" --includeFileSources "true"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
44
tests/create-zip.sh
Normal file
44
tests/create-zip.sh
Normal file
@ -0,0 +1,44 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}-1.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
DATA
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}-2.csv"
|
||||
a,b,c
|
||||
4,5,6
|
||||
DATA
|
||||
|
||||
zip "tmp/${t}/${t}.zip" "tmp/${t}/${t}-1.csv" "tmp/${t}/${t}-2.csv"
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
1 2 3
|
||||
4 5 6
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.zip"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
BIN
tests/data/example.ods
Normal file
BIN
tests/data/example.ods
Normal file
Binary file not shown.
BIN
tests/data/example.xls
Normal file
BIN
tests/data/example.xls
Normal file
Binary file not shown.
BIN
tests/data/example.xlsx
Normal file
BIN
tests/data/example.xlsx
Normal file
Binary file not shown.
36
tests/delete-utf8.sh
Normal file
36
tests/delete-utf8.sh
Normal file
@ -0,0 +1,36 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << DATA > "tmp/${t}/${t}.assert"
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --projectName "${t} biểu tượng cảm xúc ⛲"
|
||||
${cmd} --list | grep "${t}" || exit 1
|
||||
${cmd} --delete "${t} biểu tượng cảm xúc ⛲"
|
||||
${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
36
tests/delete.sh
Normal file
36
tests/delete.sh
Normal file
@ -0,0 +1,36 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << DATA > "tmp/${t}/${t}.assert"
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --list | grep "${t}" || exit 1
|
||||
${cmd} --delete "${t}"
|
||||
${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
21
tests/download.sh
Normal file
21
tests/download.sh
Normal file
@ -0,0 +1,21 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --download "https://git.io/fj5ju" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "data/duplicates-deletion.json" "tmp/${t}/${t}.output"
|
||||
44
tests/export-csv-utf8.sh
Normal file
44
tests/export-csv-utf8.sh
Normal file
@ -0,0 +1,44 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.tsv"
|
||||
🔣 code meaning
|
||||
🍇 1F347 GRAPES
|
||||
🍉 1F349 WATERMELON
|
||||
🍒 1F352 CHERRIES
|
||||
🍓 1F353 STRAWBERRY
|
||||
🍍 1F34D PINEAPPLE
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
🔣,code,meaning
|
||||
🍇,1F347,GRAPES
|
||||
🍉,1F349,WATERMELON
|
||||
🍒,1F352,CHERRIES
|
||||
🍓,1F353,STRAWBERRY
|
||||
🍍,1F34D,PINEAPPLE
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.tsv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t} biểu tượng cảm xúc 🍉.csv"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t} biểu tượng cảm xúc 🍉.csv"
|
||||
40
tests/export-csv.sh
Normal file
40
tests/export-csv.sh
Normal file
@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.tsv"
|
||||
a b c
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.tsv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.csv"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.csv"
|
||||
72
tests/export-html-utf8.sh
Normal file
72
tests/export-html-utf8.sh
Normal file
@ -0,0 +1,72 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
if [[ ${2} ]]; then
|
||||
majorversion="${2%%.*}"
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
⌨,code,meaning
|
||||
⛲,1F347,FOUNTAIN
|
||||
⛳,1F349,FLAG IN HOLE
|
||||
⛵,1F352,SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
if [[ "$majorversion" = 2 ]]; then
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
<html>
|
||||
<head>
|
||||
<title>export-html-utf8</title>
|
||||
<meta charset="utf-8" />
|
||||
</head>
|
||||
<body>
|
||||
<table>
|
||||
<tr><th>⌨</th><th>code</th><th>meaning</th></tr>
|
||||
<tr><td>⛲</td><td>1F347</td><td>FOUNTAIN</td></tr>
|
||||
<tr><td>⛳</td><td>1F349</td><td>FLAG IN HOLE</td></tr>
|
||||
<tr><td>⛵</td><td>1F352</td><td>SAILBOAT</td></tr>
|
||||
</table>
|
||||
</body>
|
||||
</html>
|
||||
DATA
|
||||
else
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
<html>
|
||||
<head>
|
||||
<title>export-html-utf8</title>
|
||||
<meta charset="utf-8" />
|
||||
</head>
|
||||
<body>
|
||||
<table>
|
||||
<tr><th>⌨</th><th>code</th><th>meaning</th></tr>
|
||||
<tr><td>⛲</td><td>1F347</td><td>FOUNTAIN</td></tr>
|
||||
<tr><td>⛳</td><td>1F349</td><td>FLAG IN HOLE</td></tr>
|
||||
<tr><td>⛵</td><td>1F352</td><td>SAILBOAT</td></tr>
|
||||
</table>
|
||||
</body>
|
||||
</html>
|
||||
DATA
|
||||
fi
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.html"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.html"
|
||||
50
tests/export-html.sh
Normal file
50
tests/export-html.sh
Normal file
@ -0,0 +1,50 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
<html>
|
||||
<head>
|
||||
<title>export-html</title>
|
||||
<meta charset="utf-8" />
|
||||
</head>
|
||||
<body>
|
||||
<table>
|
||||
<tr><th>a</th><th>b</th><th>c</th></tr>
|
||||
<tr><td>1</td><td>2</td><td>3</td></tr>
|
||||
<tr><td>0</td><td>0</td><td>0</td></tr>
|
||||
<tr><td>$</td><td>\</td><td>'</td></tr>
|
||||
</table>
|
||||
</body>
|
||||
</html>
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.html"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.html"
|
||||
43
tests/export-ods-utf8.sh
Normal file
43
tests/export-ods-utf8.sh
Normal file
@ -0,0 +1,43 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
⌨,code,meaning
|
||||
⛲,1F347,FOUNTAIN
|
||||
⛳,1F349,FLAG IN HOLE
|
||||
⛵,1F352,SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
⌨,code,meaning
|
||||
⛲,1F347,FOUNTAIN
|
||||
⛳,1F349,"FLAG IN HOLE"
|
||||
⛵,1F352,SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.ods"
|
||||
(cd tmp/"${t}" &&
|
||||
ssconvert -S "${t}.ods" "${t}.csv" &&
|
||||
mv "${t}.csv.1" "${t}.output")
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
47
tests/export-ods.sh
Normal file
47
tests/export-ods.sh
Normal file
@ -0,0 +1,47 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
if [[ -z "$(command -v ssconvert 2> /dev/null)" ]] ; then
|
||||
echo 1>&2 "ERROR: This test requires ssconvert (gnumeric)"
|
||||
exit 127
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.ods"
|
||||
(cd tmp/"${t}" &&
|
||||
ssconvert -S "${t}.ods" "${t}.csv" &&
|
||||
mv "${t}.csv.1" "${t}.output")
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
44
tests/export-tsv-utf8.sh
Normal file
44
tests/export-tsv-utf8.sh
Normal file
@ -0,0 +1,44 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
🔣,code,meaning
|
||||
🍇,1F347,GRAPES
|
||||
🍉,1F349,WATERMELON
|
||||
🍒,1F352,CHERRIES
|
||||
🍓,1F353,STRAWBERRY
|
||||
🍍,1F34D,PINEAPPLE
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
🔣 code meaning
|
||||
🍇 1F347 GRAPES
|
||||
🍉 1F349 WATERMELON
|
||||
🍒 1F352 CHERRIES
|
||||
🍓 1F353 STRAWBERRY
|
||||
🍍 1F34D PINEAPPLE
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.tsv"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.tsv"
|
||||
40
tests/export-tsv.sh
Normal file
40
tests/export-tsv.sh
Normal file
@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.tsv"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.tsv"
|
||||
44
tests/export-utf8.sh
Normal file
44
tests/export-utf8.sh
Normal file
@ -0,0 +1,44 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
🔣,code,meaning
|
||||
🍇,1F347,GRAPES
|
||||
🍉,1F349,WATERMELON
|
||||
🍒,1F352,CHERRIES
|
||||
🍓,1F353,STRAWBERRY
|
||||
🍍,1F34D,PINEAPPLE
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
🔣 code meaning
|
||||
🍇 1F347 GRAPES
|
||||
🍉 1F349 WATERMELON
|
||||
🍒 1F352 CHERRIES
|
||||
🍓 1F353 STRAWBERRY
|
||||
🍍 1F34D PINEAPPLE
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" > "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
43
tests/export-xls-utf8.sh
Normal file
43
tests/export-xls-utf8.sh
Normal file
@ -0,0 +1,43 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
⌨,code,meaning
|
||||
⛲,1F347,FOUNTAIN
|
||||
⛳,1F349,FLAG IN HOLE
|
||||
⛵,1F352,SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
⌨,code,meaning
|
||||
⛲,1F347,FOUNTAIN
|
||||
⛳,1F349,FLAG IN HOLE
|
||||
⛵,1F352,SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.xls"
|
||||
(cd tmp/"${t}" &&
|
||||
ssconvert -S "${t}.xls" "${t}.csv" &&
|
||||
mv "${t}.csv" "${t}.output")
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
43
tests/export-xls.sh
Normal file
43
tests/export-xls.sh
Normal file
@ -0,0 +1,43 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.xls"
|
||||
(cd tmp/"${t}" &&
|
||||
ssconvert -S "${t}.xls" "${t}.csv" &&
|
||||
mv "${t}.csv" "${t}.output")
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
45
tests/export-xlsx-utf8.sh
Normal file
45
tests/export-xlsx-utf8.sh
Normal file
@ -0,0 +1,45 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
⌨,code,meaning
|
||||
⛲,1F347,FOUNTAIN
|
||||
⛳,1F349,FLAG IN HOLE
|
||||
⛵,1F352,SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
⌨,code,meaning
|
||||
⛲,1F347,FOUNTAIN
|
||||
⛳,1F349,FLAG IN HOLE
|
||||
⛵,1F352,SAILBOAT
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.xlsx"
|
||||
(cd tmp/"${t}" &&
|
||||
ssconvert -S "${t}.xlsx" "${t}.csv" &&
|
||||
mv "${t}.csv" "${t}.output")
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
|
||||
|
||||
43
tests/export-xlsx.sh
Normal file
43
tests/export-xlsx.sh
Normal file
@ -0,0 +1,43 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.xlsx"
|
||||
(cd tmp/"${t}" &&
|
||||
ssconvert -S "${t}.xlsx" "${t}.csv" &&
|
||||
mv "${t}.csv" "${t}.output")
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
40
tests/export.sh
Normal file
40
tests/export.sh
Normal file
@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" > "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
40
tests/format-create-separator.sh
Normal file
40
tests/format-create-separator.sh
Normal file
@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.txt"
|
||||
a;b;c
|
||||
1;2;3
|
||||
0;0;0
|
||||
$;\;'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a b c
|
||||
1 2 3
|
||||
0 0 0
|
||||
$ \ '
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.txt" --format "csv" --separator ";"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
41
tests/format-create.sh
Normal file
41
tests/format-create.sh
Normal file
@ -0,0 +1,41 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
Column 1
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --format "line-based"
|
||||
${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
40
tests/format-export-output.sh
Normal file
40
tests/format-export-output.sh
Normal file
@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" --format "csv" --output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
40
tests/format-export.sh
Normal file
40
tests/format-export.sh
Normal file
@ -0,0 +1,40 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
a,b,c
|
||||
1,2,3
|
||||
0,0,0
|
||||
$,\,'
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" --format "csv" > "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
27
tests/help.sh
Normal file
27
tests/help.sh
Normal file
@ -0,0 +1,27 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
Script to provide a command line interface to an OpenRefine server.
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --help | sed '3q;d' > "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
39
tests/info-utf8.sh
Normal file
39
tests/info-utf8.sh
Normal file
@ -0,0 +1,39 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.tsv"
|
||||
🔣 code meaning
|
||||
🍇 1F347 GRAPES
|
||||
🍉 1F349 WATERMELON
|
||||
🍒 1F352 CHERRIES
|
||||
🍓 1F353 STRAWBERRY
|
||||
🍍 1F34D PINEAPPLE
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << DATA > "tmp/${t}/${t}.assert"
|
||||
column 001: 🔣
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.tsv"
|
||||
${cmd} --info "${t}" | grep 'column 001' > "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
35
tests/info.sh
Normal file
35
tests/info.sh
Normal file
@ -0,0 +1,35 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << DATA > "tmp/${t}/${t}.assert"
|
||||
column 002: b
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --info "${t}" | grep 'column 002' > "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
35
tests/list-utf8.sh
Normal file
35
tests/list-utf8.sh
Normal file
@ -0,0 +1,35 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << DATA > "tmp/${t}/${t}.assert"
|
||||
${t} biểu tượng cảm xúc ⛲
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv" --projectName "${t} biểu tượng cảm xúc ⛲"
|
||||
${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
35
tests/list.sh
Normal file
35
tests/list.sh
Normal file
@ -0,0 +1,35 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
a,b,c
|
||||
1,2,3
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << DATA > "tmp/${t}/${t}.assert"
|
||||
${t}
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
58
tests/template-facets.sh
Normal file
58
tests/template-facets.sh
Normal file
@ -0,0 +1,58 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
email,name,state,gender,purchase
|
||||
danny.baron@example1.com,Danny Baron,CA,M,TV
|
||||
melanie.white@example2.edu,Melanie White,NC,F,iPhone
|
||||
danny.baron@example1.com,D. Baron,CA,M,Winter jacket
|
||||
ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
|
||||
arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
|
||||
danny.baron@example1.com,Daniel Baron,CA,M,Bike
|
||||
jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
|
||||
melanie.white@example2.edu,Melanie White,NC,F,iPad
|
||||
ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
|
||||
arthur.duff@example4.com,Arthur Duff,OR,M,Night table
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
{ "events" : [
|
||||
{ "name" : "Melanie White", "purchase" : "iPhone" },
|
||||
{ "name" : "Jean Griffith", "purchase" : "Power drill" },
|
||||
{ "name" : "Melanie White", "purchase" : "iPad" }
|
||||
] }
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" \
|
||||
--prefix '{ "events" : [
|
||||
' \
|
||||
--template ' { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
|
||||
--rowSeparator ',
|
||||
' \
|
||||
--suffix '
|
||||
] }
|
||||
' \
|
||||
--facets '{"type":"list","name":"gender","columnName":"gender","expression":"value","omitBlank":false,"omitError":false,"selection":[{"v":{"v":"F","l":"F"}}],"selectBlank":false,"selectError":false,"invert":false}' \
|
||||
--output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
54
tests/template-filterQuery-utf8.sh
Normal file
54
tests/template-filterQuery-utf8.sh
Normal file
@ -0,0 +1,54 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
🔣,code,meaning
|
||||
🍇,1F347,GRAPES
|
||||
🍉,1F349,WATERMELON
|
||||
🍒,1F352,CHERRIES
|
||||
🍓,1F353,STRAWBERRY
|
||||
🍍,1F34D,PINEAPPLE
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
{ "emojis" : [
|
||||
{ "symbol" : "🍇", "meaning" : "GRAPES" },
|
||||
{ "symbol" : "🍉", "meaning" : "WATERMELON" },
|
||||
{ "symbol" : "🍍", "meaning" : "PINEAPPLE" }
|
||||
] }
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" \
|
||||
--prefix '{ "emojis" : [
|
||||
' \
|
||||
--template ' { "symbol" : {{jsonize(with(row.columnNames[0],cn,cells[cn].value))}}, "meaning" : {{jsonize(cells["meaning"].value)}} }' \
|
||||
--rowSeparator ',
|
||||
' \
|
||||
--suffix '
|
||||
] }
|
||||
' \
|
||||
--filterQuery '^1F34' \
|
||||
--filterColumn 'code' \
|
||||
--output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
59
tests/template-filterQuery.sh
Normal file
59
tests/template-filterQuery.sh
Normal file
@ -0,0 +1,59 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
email,name,state,gender,purchase
|
||||
danny.baron@example1.com,Danny Baron,CA,M,TV
|
||||
melanie.white@example2.edu,Melanie White,NC,F,iPhone
|
||||
danny.baron@example1.com,D. Baron,CA,M,Winter jacket
|
||||
ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
|
||||
arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
|
||||
danny.baron@example1.com,Daniel Baron,CA,M,Bike
|
||||
jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
|
||||
melanie.white@example2.edu,Melanie White,NC,F,iPad
|
||||
ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
|
||||
arthur.duff@example4.com,Arthur Duff,OR,M,Night table
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
{ "events" : [
|
||||
{ "name" : "Melanie White", "purchase" : "iPhone" },
|
||||
{ "name" : "Jean Griffith", "purchase" : "Power drill" },
|
||||
{ "name" : "Melanie White", "purchase" : "iPad" }
|
||||
] }
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" \
|
||||
--prefix '{ "events" : [
|
||||
' \
|
||||
--template ' { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
|
||||
--rowSeparator ',
|
||||
' \
|
||||
--suffix '
|
||||
] }
|
||||
' \
|
||||
--filterQuery '^F$' \
|
||||
--filterColumn 'gender' \
|
||||
--output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
58
tests/template-splitToFiles-mode.sh
Normal file
58
tests/template-splitToFiles-mode.sh
Normal file
@ -0,0 +1,58 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
email,name,state,gender,purchase
|
||||
arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
|
||||
,Arthur Duff,OR,M,Night table
|
||||
ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
|
||||
ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
|
||||
danny.baron@example1.com,Daniel Baron,CA,M,Bike
|
||||
,Danny Baron,CA,M,TV
|
||||
,D. Baron,CA,M,Winter jacket
|
||||
jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
|
||||
melanie.white@example2.edu,Melanie White,NC,F,iPad
|
||||
,Melanie White,NC,F,iPhone
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
{ "events" : [
|
||||
{ "name" : "Melanie White", "purchase" : "iPad" } { "name" : "Melanie White", "purchase" : "iPhone" }
|
||||
] }
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" \
|
||||
--prefix '{ "events" : [
|
||||
' \
|
||||
--template ' { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
|
||||
--rowSeparator ',
|
||||
' \
|
||||
--suffix '
|
||||
] }
|
||||
' \
|
||||
--mode "record-based" \
|
||||
--splitToFiles true \
|
||||
--output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
ls "tmp/${t}"
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}_6.output"
|
||||
52
tests/template-splitToFiles-suffixById-utf8.sh
Normal file
52
tests/template-splitToFiles-suffixById-utf8.sh
Normal file
@ -0,0 +1,52 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
🔣,code,meaning
|
||||
🍇,1F347,GRAPES
|
||||
🍉,1F349,WATERMELON
|
||||
🍒,1F352,CHERRIES
|
||||
🍓,1F353,STRAWBERRY
|
||||
🍍,1F34D,PINEAPPLE
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
{ "emojis" : [
|
||||
{ "symbol" : "🍍", "meaning" : "PINEAPPLE" }
|
||||
] }
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" \
|
||||
--prefix '{ "emojis" : [
|
||||
' \
|
||||
--template ' { "symbol" : {{jsonize(with(row.columnNames[0],cn,cells[cn].value))}}, "meaning" : {{jsonize(cells["meaning"].value)}} }' \
|
||||
--rowSeparator ',
|
||||
' \
|
||||
--suffix '
|
||||
] }
|
||||
' \
|
||||
--splitToFiles true \
|
||||
--suffixById true \
|
||||
--output "tmp/${t}/trái cây.json"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/trái cây_🍍.json"
|
||||
58
tests/template-splitToFiles-suffixById.sh
Normal file
58
tests/template-splitToFiles-suffixById.sh
Normal file
@ -0,0 +1,58 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
email,name,state,gender,purchase
|
||||
danny.baron@example1.com,Danny Baron,CA,M,TV
|
||||
melanie.white@example2.edu,Melanie White,NC,F,iPhone
|
||||
danny.baron@example1.com,D. Baron,CA,M,Winter jacket
|
||||
ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
|
||||
arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
|
||||
danny.baron@example1.com,Daniel Baron,CA,M,Bike
|
||||
jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
|
||||
melanie.white@example2.edu,Melanie White,NC,F,iPad
|
||||
ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
|
||||
arthur.duff@example4.com,Arthur Duff,OR,M,Night table
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
{ "events" : [
|
||||
{ "name" : "Arthur Duff", "purchase" : "Night table" }
|
||||
] }
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" \
|
||||
--prefix '{ "events" : [
|
||||
' \
|
||||
--template ' { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
|
||||
--rowSeparator ',
|
||||
' \
|
||||
--suffix '
|
||||
] }
|
||||
' \
|
||||
--splitToFiles true \
|
||||
--suffixById true \
|
||||
--output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
ls "tmp/${t}"
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}_arthur.duff@example4.com.output"
|
||||
51
tests/template-splitToFiles-utf8.sh
Normal file
51
tests/template-splitToFiles-utf8.sh
Normal file
@ -0,0 +1,51 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
🔣,code,meaning
|
||||
🍇,1F347,GRAPES
|
||||
🍉,1F349,WATERMELON
|
||||
🍒,1F352,CHERRIES
|
||||
🍓,1F353,STRAWBERRY
|
||||
🍍,1F34D,PINEAPPLE
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
{ "emojis" : [
|
||||
{ "symbol" : "🍍", "meaning" : "PINEAPPLE" }
|
||||
] }
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" \
|
||||
--prefix '{ "emojis" : [
|
||||
' \
|
||||
--template ' { "symbol" : {{jsonize(with(row.columnNames[0],cn,cells[cn].value))}}, "meaning" : {{jsonize(cells["meaning"].value)}} }' \
|
||||
--rowSeparator ',
|
||||
' \
|
||||
--suffix '
|
||||
] }
|
||||
' \
|
||||
--splitToFiles true \
|
||||
--output "tmp/${t}/trái cây.json"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/trái cây_5.json"
|
||||
57
tests/template-splitToFiles.sh
Normal file
57
tests/template-splitToFiles.sh
Normal file
@ -0,0 +1,57 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
email,name,state,gender,purchase
|
||||
danny.baron@example1.com,Danny Baron,CA,M,TV
|
||||
melanie.white@example2.edu,Melanie White,NC,F,iPhone
|
||||
danny.baron@example1.com,D. Baron,CA,M,Winter jacket
|
||||
ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
|
||||
arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
|
||||
danny.baron@example1.com,Daniel Baron,CA,M,Bike
|
||||
jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
|
||||
melanie.white@example2.edu,Melanie White,NC,F,iPad
|
||||
ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
|
||||
arthur.duff@example4.com,Arthur Duff,OR,M,Night table
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
{ "events" : [
|
||||
{ "name" : "Arthur Duff", "purchase" : "Night table" }
|
||||
] }
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" \
|
||||
--prefix '{ "events" : [
|
||||
' \
|
||||
--template ' { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
|
||||
--rowSeparator ',
|
||||
' \
|
||||
--suffix '
|
||||
] }
|
||||
' \
|
||||
--splitToFiles true \
|
||||
--output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
ls "tmp/${t}"
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}_10.output"
|
||||
54
tests/template-utf8.sh
Normal file
54
tests/template-utf8.sh
Normal file
@ -0,0 +1,54 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
🔣,code,meaning
|
||||
🍇,1F347,GRAPES
|
||||
🍉,1F349,WATERMELON
|
||||
🍒,1F352,CHERRIES
|
||||
🍓,1F353,STRAWBERRY
|
||||
🍍,1F34D,PINEAPPLE
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
{ "emojis" : [
|
||||
{ "symbol" : "🍇", "meaning" : "GRAPES" },
|
||||
{ "symbol" : "🍉", "meaning" : "WATERMELON" },
|
||||
{ "symbol" : "🍒", "meaning" : "CHERRIES" },
|
||||
{ "symbol" : "🍓", "meaning" : "STRAWBERRY" },
|
||||
{ "symbol" : "🍍", "meaning" : "PINEAPPLE" }
|
||||
] }
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" \
|
||||
--prefix '{ "emojis" : [
|
||||
' \
|
||||
--template ' { "symbol" : {{jsonize(with(row.columnNames[0],cn,cells[cn].value))}}, "meaning" : {{jsonize(cells["meaning"].value)}} }' \
|
||||
--rowSeparator ',
|
||||
' \
|
||||
--suffix '
|
||||
] }
|
||||
' \
|
||||
--output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
64
tests/template.sh
Normal file
64
tests/template.sh
Normal file
@ -0,0 +1,64 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# =================================== DATA =================================== #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.csv"
|
||||
email,name,state,gender,purchase
|
||||
danny.baron@example1.com,Danny Baron,CA,M,TV
|
||||
melanie.white@example2.edu,Melanie White,NC,F,iPhone
|
||||
danny.baron@example1.com,D. Baron,CA,M,Winter jacket
|
||||
ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
|
||||
arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
|
||||
danny.baron@example1.com,Daniel Baron,CA,M,Bike
|
||||
jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
|
||||
melanie.white@example2.edu,Melanie White,NC,F,iPad
|
||||
ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
|
||||
arthur.duff@example4.com,Arthur Duff,OR,M,Night table
|
||||
DATA
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
{ "events" : [
|
||||
{ "name" : "Danny Baron", "purchase" : "TV" },
|
||||
{ "name" : "Melanie White", "purchase" : "iPhone" },
|
||||
{ "name" : "D. Baron", "purchase" : "Winter jacket" },
|
||||
{ "name" : "Ben Tyler", "purchase" : "Flashlight" },
|
||||
{ "name" : "Arthur Duff", "purchase" : "Dining table" },
|
||||
{ "name" : "Daniel Baron", "purchase" : "Bike" },
|
||||
{ "name" : "Jean Griffith", "purchase" : "Power drill" },
|
||||
{ "name" : "Melanie White", "purchase" : "iPad" },
|
||||
{ "name" : "Ben Morisson", "purchase" : "Amplifier" },
|
||||
{ "name" : "Arthur Duff", "purchase" : "Night table" }
|
||||
] }
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} --create "tmp/${t}/${t}.csv"
|
||||
${cmd} --export "${t}" \
|
||||
--prefix '{ "events" : [
|
||||
' \
|
||||
--template ' { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
|
||||
--rowSeparator ',
|
||||
' \
|
||||
--suffix '
|
||||
] }
|
||||
' \
|
||||
--output "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
27
tests/usage.sh
Normal file
27
tests/usage.sh
Normal file
@ -0,0 +1,27 @@
|
||||
#!/bin/bash
|
||||
|
||||
# =============================== ENVIRONMENT ================================ #
|
||||
|
||||
if [[ ${1} ]]; then
|
||||
cmd="${1}"
|
||||
else
|
||||
echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
|
||||
fi
|
||||
|
||||
t="$(basename "${BASH_SOURCE[0]}" .sh)"
|
||||
cd "${BASH_SOURCE%/*}/" || exit 1
|
||||
mkdir -p "tmp/${t}"
|
||||
|
||||
# ================================= ASSERTION ================================ #
|
||||
|
||||
cat << "DATA" > "tmp/${t}/${t}.assert"
|
||||
Usage:
|
||||
DATA
|
||||
|
||||
# ================================== ACTION ================================== #
|
||||
|
||||
${cmd} | head -n 1 | cut -c 1-6 > "tmp/${t}/${t}.output"
|
||||
|
||||
# =================================== TEST =================================== #
|
||||
|
||||
diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
|
||||
Loading…
x
Reference in New Issue
Block a user