Merge pull request #21 from mohammed78620/master

moved csrf token check into urlopen_json

.gitignore

@@ -2,5 +2,8 @@ build
 dist
 *.pyc
 .*
-refine_client.egg-info
-README.html
+openrefine_client.egg-info
+refine.spec
+openrefine-*
+openrefine-client_*
+tests-cli.log

MANIFEST.in

@@ -1,4 +1,4 @@
-include README.rst
+include README.md
 include COPYING.txt
 recursive-include tests/data *.csv
 recursive-include tests      *.py

README.md

@@ -0,0 +1,773 @@
+# OpenRefine Python Client with extended command line interface (⌨️ for 💎)
+
+[![Codacy Badge](https://app.codacy.com/project/badge/Grade/43ad9bfd707b4627bd45e5c5f912a8e0)](https://www.codacy.com/gh/opencultureconsulting/openrefine-client/dashboard) [![Docker](https://img.shields.io/microbadger/image-size/felixlohmeier/openrefine-client?label=docker)](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [![PyPI](https://img.shields.io/pypi/v/openrefine-client)](https://pypi.org/project/openrefine-client/) [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master)
+
+The [OpenRefine Python Client from PaulMakepeace](https://github.com/PaulMakepeace/refine-client-py) provides a library for communicating with an [OpenRefine](http://openrefine.org) server.
+This fork extends the command line interface (CLI) and is distributed as a convenient one-file-executable (Windows, Linux, macOS).
+It is also available via Docker Hub, PyPI and Binder.
+
+works with OpenRefine 2.7, 2.8, 3.0, 3.1, 3.2, 3.3, 3.4, 3.4.1, 3.5.0
+
+## Download
+
+One-file-executables:
+
+- Windows: [openrefine-client_0-3-10_windows.exe](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_windows.exe) (~5 MB)
+- macOS: [openrefine-client_0-3-10_macos](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_macos) (~5 MB)
+- Linux: [openrefine-client_0-3-10_linux](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_linux) (~5 MB)
+
+For [Docker](#docker) containers, native [Python](#python) installation and free [Binder](#binder) on-demand server see the corresponding chapters below.
+
+## Peek
+
+A short video loop that demonstrates the basic features (list, create, apply, export):
+
+![video loop that demonstrates basic features](openrefine-client-peek.gif)
+
+## Usage
+
+Ensure you have [OpenRefine](http://openrefine.org) running (i.e. available at http://localhost:3333 or [another URL](#change-url)).
+
+To use the client:
+
+1. Open a terminal pointing to the folder where you have [downloaded](#download) the one-file-executable (e.g. Downloads in your home directory).
+
+   - Windows: Open PowerShell and enter following command
+
+     ```sh
+     cd ~\Downloads
+     ```
+
+   - macOS: Open Terminal (Finder > Applications > Utilities > Terminal) and enter following command
+
+     ```sh
+     cd ~/Downloads
+     ```
+
+   - Linux: Open terminal app (Terminal, Konsole, xterm, ...) and enter following command
+
+     ```sh
+     cd ~/Downloads
+     ```
+
+2. Make the file executable.
+
+   - Windows: not necessary
+
+   - macOS:
+
+     ```sh
+     chmod +x openrefine-client_0-3-10_macos
+     ```
+
+   - Linux:
+
+     ```sh
+     chmod +x openrefine-client_0-3-10_linux
+     ```
+
+3. Execute the file.
+
+   - Windows:
+
+     ```sh
+     .\openrefine-client_0-3-10_windows.exe
+     ```
+
+   - macOS:
+
+     ```sh
+     ./openrefine-client_0-3-10_macos
+     ```
+
+   - Linux:
+
+     ```sh
+     ./openrefine-client_0-3-10_linux
+     ```
+
+Using tab completion and command history is highly recommended:
+
+- autocomplete filenames: enter a few characters and press `↹`
+- recall previous command: press `↑`
+
+### Basic commands
+
+Execute the client by entering its filename followed by the desired command.
+
+The following example will download two small files ([duplicates.csv](https://raw.githubusercontent.com/opencultureconsulting/openrefine-client/master/tests/data/duplicates.csv) and [duplicates-deletion.json](https://raw.githubusercontent.com/opencultureconsulting/openrefine-client/master/tests/data/duplicates-deletion.json)) into the current directory and will create a new OpenRefine project from file duplicates.csv.
+
+Download example data (`--download`) and create project from file (`--create`):
+
+- Windows:
+
+  ```sh
+  .\openrefine-client_0-3-10_windows.exe --download "https://git.io/fj5hF" --output=duplicates.csv
+  .\openrefine-client_0-3-10_windows.exe --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+  .\openrefine-client_0-3-10_windows.exe --create duplicates.csv
+  ```
+
+- macOS:
+
+  ```sh
+  ./openrefine-client_0-3-10_macos --download "https://git.io/fj5hF" --output=duplicates.csv
+  ./openrefine-client_0-3-10_macos --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+  ./openrefine-client_0-3-10_macos --create duplicates.csv
+  ```
+
+- Linux:
+
+  ```sh
+  ./openrefine-client_0-3-10_linux --download "https://git.io/fj5hF" --output=duplicates.csv
+  ./openrefine-client_0-3-10_linux --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+  ./openrefine-client_0-3-10_linux --create duplicates.csv
+  ```
+
+Other commands:
+
+- list all projects: `--list`
+- show project metadata: `--info "duplicates"`
+- export project to terminal: `--export "duplicates"`
+- apply [rules from json file](http://kb.refinepro.com/2012/06/google-refine-json-and-my-notepad-or.html): `--apply duplicates-deletion.json "duplicates"`
+- export project to file: `--export --output=deduped.xls "duplicates"`
+- delete project: `--delete "duplicates"`
+
+### Getting help
+
+Check `--help` for further options.
+
+Please file an [issue](https://github.com/opencultureconsulting/openrefine-client/issues) if you miss some features in the command line interface or if you have tracked a bug.
+And you are welcome to ask any questions!
+
+### Change URL
+
+By default the client connects to the usual URL of OpenRefine [http://localhost:3333](http://localhost:3333).
+If your OpenRefine server is running somewhere else then you may set hostname and port with additional command line options (e.g. http://example.com):
+
+- set host: `-H example.com`
+- set port: `-P 80`
+
+### Templating
+
+The OpenRefine [Templating](https://github.com/OpenRefine/OpenRefine/wiki/Export-As-YAML) supports exporting data in any text format (i.e. to construct JSON or XML).
+The graphical user interface offers four input fields:
+
+1. prefix
+2. row template
+   - supports [GREL](https://github.com/OpenRefine/OpenRefine/wiki/General-Refine-Expression-Language) inside two curly brackets, e.g. `{{jsonize(cells["name"].value)}}`
+3. row separator
+4. suffix
+
+This templating functionality is available via the openrefine-client command line interface.
+It even provides an additional feature for splitting results into multiple files.
+
+To try out the functionality create another project from the example file above.
+
+```sh
+--create duplicates.csv --projectName=advanced
+```
+
+The following example code will export...
+
+- the columns "name" and "purchase" in JSON format
+- from the project "advanced"
+- for rows matching the regex text filter `^F$` in column "gender"
+
+macOS/Linux Terminal (multi-line input with `\` ):
+
+```sh
+"advanced" \
+--prefix='{ "events" : [
+' \
+--template='    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
+--rowSeparator=',
+' \
+--suffix='
+] }' \
+--filterQuery='^F$' \
+--filterColumn='gender'
+```
+
+Windows PowerShell (multi-line input with `` ` ``; quotes needs to be doubled):
+
+```sh
+"advanced" `
+--prefix='{ ""events"" : [
+' `
+--template='    { ""name"" : {{jsonize(cells[""name""].value)}}, ""purchase"" : {{jsonize(cells[""purchase""].value)}} }' `
+--rowSeparator=',
+' `
+--suffix='
+] }' `
+--filterQuery='^F$' `
+--filterColumn='gender'
+```
+
+Add the following options to the last command (recall with `↑`) to store the results in multiple files.
+Each file will contain the prefix, an processed row, and the suffix.
+
+```sh
+--output=advanced.json --splitToFiles=true
+```
+
+Filenames are suffixed with the row number by default (e.g. `advanced_1.json`, `advanced_2.json` etc.).
+There is another option to use the value in the first column instead:
+
+```sh
+--output=advanced.json --splitToFiles=true --suffixById=true
+```
+
+Because our project "advanced" contains duplicates in the first column "email" this command will overwrite files (e.g. `advanced_melanie.white@example2.edu.json`).
+When using this option, the first column should contain unique identifiers.
+
+### Append data to an existing project
+
+OpenRefine does not support appending rows to an existing project.
+As long as the [feature request](https://github.com/OpenRefine/OpenRefine/issues/715) is not yet implemented, you can use the openrefine-client to script a workaround:
+
+1. export existing project as csv
+2. put old and new data into a zip archive
+3. create new project by importing the zip archive
+
+Here is an example that replaces the existing project:
+
+```
+openrefine-client --export myproject --output old.csv
+openrefine-client --delete myproject
+zip combined.zip old.csv new.csv
+openrefine-client --create combined.zip --format csv --projectName myproject
+```
+
+Note that the project id will change.
+If you want to distinguish between old and new data, you can use the additional flag includeFileSources:
+
+```
+openrefine-client --create combined.zip --format csv --projectName myproject --includeFileSources true
+```
+
+### See also
+
+- Linux Bash script to run OpenRefine in batch mode (import, transform, export): [openrefine-batch](https://github.com/opencultureconsulting/openrefine-batch)
+- [Jupyter notebook demonstrating usage in Linux Bash](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/notebooks/openrefine-client-bash.ipynb)
+- Use case [HOS-MetadataTransformations](https://github.com/subhh/HOS-MetadataTransformations): Automated workflow for harvesting, transforming and indexing of metadata using metha, OpenRefine and Solr. Part of the Hamburg Open Science "Schaufenster" software stack.
+- Use case [Data processing of ILS data to facilitate a new discovery layer for the German Literature Archive (DLA)](https://doi.org/10.5281/zenodo.2678113): Custom data processing pipeline based on Pandas (a Python library) and OpenRefine.
+
+## Docker
+
+[felixlohmeier/openrefine-client](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [![Docker](https://img.shields.io/microbadger/image-size/felixlohmeier/openrefine-client?label=docker)](https://hub.docker.com/r/felixlohmeier/openrefine-client/)
+
+```sh
+docker pull felixlohmeier/openrefine-client:v0.3.10
+```
+
+### Option 1: Dockerized client
+
+Run client and mount current directory as workspace:
+
+```sh
+docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10
+```
+
+The docker option `--network=host` allows you to connect to a local or remote OpenRefine via the host network:
+
+- list projects on default URL (http://localhost:3333)
+
+  ```sh
+  docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --list
+  ```
+
+- list projects on a remote server (http://example.com)
+
+  ```sh
+  docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H example.com -P 80 --list
+  ```
+
+Usage: same commands as explained above (see [Basic Commands](#basic-commands) and [Advanced Templating](#advanced-templating))
+
+### Option 2: Dockerized client and dockerized OpenRefine
+
+Run openrefine-client linked to a dockerized OpenRefine ([felixlohmeier/openrefine](https://hub.docker.com/r/felixlohmeier/openrefine/) [![Docker](https://img.shields.io/microbadger/image-size/felixlohmeier/openrefine?label=docker)](https://hub.docker.com/r/felixlohmeier/openrefine)):
+
+1. Create docker network
+
+   ```sh
+   docker network create openrefine
+   ```
+
+2. Run server (will be available at http://localhost:3333)
+
+   ```sh
+   docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.5.0
+   ```
+
+3. Run client with some [basic commands](#basic-commands): 1. download example files, 2. create project from file, 3. list projects, 4. show metadata, 5. export to terminal, 6. apply transformation rules (deduplication), 7. export again to terminal, 8. export to xls file and 9. delete project
+
+   ```sh
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --download "https://git.io/fj5hF" --output=duplicates.csv
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --create duplicates.csv
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --list
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --info "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --apply duplicates-deletion.json "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export --output=deduped.xls "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --delete "duplicates"
+   ```
+
+4. Stop and delete server:
+
+   ```sh
+   docker stop openrefine-server
+   docker rm openrefine-server
+   ```
+
+5. Delete docker network:
+
+   ```sh
+   docker network rm openrefine
+   ```
+
+Customize OpenRefine server:
+
+- If you want to add an OpenRefine startup option you need to repeat the default commands (cf. [Dockerfile](https://hub.docker.com/r/felixlohmeier/openrefine/dockerfile))
+  - `-i 0.0.0.0` sets OpenRefine to be accessible from outside the container, i.e. from host
+  - `-d /data` sets OpenRefine workspace
+
+- Example for [allocating more memory](https://github.com/OpenRefine/OpenRefine/wiki/FAQ#out-of-memory-errors---feels-slow---could-not-reserve-enough-space-for-object-heap) to OpenRefine with additional option `-m 4G`
+
+  ```sh
+  docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.5.0 -i 0.0.0.0 -d /data -m 4G
+  ```
+
+- The OpenRefine version is defined by the docker tag.
+  Check the [DockerHub repository](https://hub.docker.com/r/felixlohmeier/openrefine) for available tags.
+  Example for OpenRefine `2.8` with same options as above:
+
+  ```sh
+  docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:2.8 -i 0.0.0.0 -d /data -m 4G
+  ```
+
+- If you want OpenRefine to read and write persistent data in host directory (i.e. store projects) you can mount the container path `/data`. Example for host directory `/home/felix/refine`:
+
+  ```sh
+  docker run -d -p 3333:3333 -v /home/felix/refine:/data:z --network=openrefine name=openrefine-server felixlohmeier/openrefine:2.8 -i 0.0.0.0 -d /data -m 4G
+  ```
+
+See also:
+
+- [GitHub Repository](https://github.com/opencultureconsulting/openrefine-docker) for docker container `felixlohmeier/openrefine`
+- Linux Bash script to run OpenRefine in batch mode (import, transform, export) with docker containers: [openrefine-batch-docker.sh](https://github.com/opencultureconsulting/openrefine-batch/#docker)
+
+## Python
+
+[openrefine-client](https://pypi.org/project/openrefine-client/) [![PyPI](https://img.shields.io/pypi/v/openrefine-client)](https://pypi.org/project/openrefine-client/) (requires Python 2.x)
+
+```sh
+python2 -m pip install openrefine-client --user
+```
+
+This will install the package `openrefine-client` containing modules in `google.refine`.
+
+A command line script `openrefine-client` will also be installed.
+
+### Option 1: command line script
+
+```sh
+openrefine-client --help
+```
+
+Usage: same commands as explained above (see [Basic Commands](#basic-commands) and [Advanced Templating](#advanced-templating))
+
+### Option 2: using cli functions in Python 2.x environment
+
+Import module cli:
+
+```python
+from google.refine import cli
+```
+
+Change URL (if necessary):
+
+```python
+cli.refine.REFINE_HOST = 'localhost'
+cli.refine.REFINE_PORT = '3333'
+```
+
+Help screen:
+
+```python
+help(cli)
+```
+
+Commands:
+
+* download (e.g. example data):
+
+  ```python
+  cli.download('https://git.io/fj5hF','duplicates.csv')
+  cli.download('https://git.io/fj5ju','duplicates-deletion.json')
+  ```
+
+* list projects:
+
+  ```python
+  cli.ls()
+  ```
+
+* create project:
+
+  ```python
+  p1 = cli.create('duplicates.csv')
+  ```
+
+* show metadata:
+
+  ```python
+  cli.info(p1.project_id)
+  ```
+
+* apply rules from file to project:
+
+  ```python
+  cli.apply(p1.project_id, 'duplicates-deletion.json')
+  ```
+
+* export project to terminal:
+
+  ```python
+  cli.export(p1.project_id)
+  ```
+
+* export project to file in xls format:
+
+  ```python
+  cli.export(p1.project_id, 'deduped.xls')
+  ```
+
+* export templating (see [Advanced Templating](#advanced-templating) above):
+
+  ```python
+  cli.templating(
+      p1.project_id,
+      prefix='''{ "events" : [
+  ''',template='''    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }''',
+      rowSeparator=''',
+  ''',suffix='''
+  ] }''')
+  ```
+
+* delete project:
+
+  ```python
+  cli.delete(p1.project_id)
+  ```
+
+### Option 3: the upstream way
+
+This fork can be used in the same way as the upstream [Python client library](https://github.com/PaulMakepeace/refine-client-py/).
+
+Some functions in the python client library are not yet compatible with OpenRefine >=3.0 (cf. [issue #19 in refine-client-py](https://github.com/paulmakepeace/refine-client-py/issues/19)).
+
+Import module refine:
+
+```python
+from google.refine import refine
+```
+
+Server Commands:
+
+* set up connection:
+
+  ```python
+  server1 = refine.Refine('http://localhost:3333')
+  ```
+
+- show version:
+
+  ```python
+  server1.server.get_version()
+  server1.server.version
+  ```
+
+- list projects:
+
+  ```python
+  server1.list_projects()
+  ```
+
+  - pretty print the returned dict with json.dumps:
+
+    ```python
+    import json
+    print(json.dumps(server1.list_projects(), indent=1))
+    ```
+
+- create project:
+
+  ```python
+  server1.new_project(project_file='duplicates.csv')
+  ```
+
+  * create and open the returned project in one step:
+
+    ```python
+    project1 = server1.new_project(project_file='duplicates.csv')
+    ```
+
+Project commands:
+
+* open project:
+
+  ```python
+  project1 = server1.open_project('1234567890123')
+  ```
+
+* print full URL to project:
+
+  ```python
+  project1.project_url()
+  ```
+
+* list columns:
+
+  ```python
+  project1.columns
+  ```
+
+* compute text facet on first column (**fails with OpenRefine >=3.2**):
+
+  ```python
+  project1.compute_facets(facet.TextFacet(project1.columns[0]))
+  ```
+
+  * print returned object
+
+    ```python
+    facets = project1.compute_facets(facet.TextFacet(project1.columns[0])).facets[0]
+    for k in sorted(facets.choices, key=lambda k: facets.choices[k].count, reverse=True):
+        print(facets.choices[k].count, k)
+    ```
+
+* compute clusters on first column:
+
+  ```python
+  project1.compute_clusters(project1.columns[0])
+  ```
+
+* apply rules from file to project:
+
+  ```python
+  project1.apply_operations('duplicates-deletion.json')
+  ```
+
+* export project:
+
+  ```python
+  project1.export(export_format='tsv')
+  ```
+
+  * print the returned fileobject:
+
+    ```python
+    print(project1.export(export_format='tsv').read())
+    ```
+
+  * save the returned fileobject to file:
+
+    ```python
+    with open('export.tsv', 'wb') as f:
+        f.write(project1.export(export_format='tsv').read())
+    ```
+
+* templating export (**function was added in this fork**, see [Advanced Templating](#advanced-templating) above):
+
+  ```python
+  data = project1.export_templating(
+      prefix='''{ "events" : [
+  ''',template='''    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }''',
+      rowSeparator=''',
+  ''',suffix='''
+  ] }''')
+  print(data.read())
+  ```
+
+* print help screen with available commands (many more!):
+
+  ```python
+  help(project1)
+  ```
+
+* example for custom commands:
+
+  ```python
+  project1.do_json('get-rows')['total']
+  ```
+
+* delete project:
+
+  ```python
+  project1.delete()
+  ```
+
+See also:
+
+- Jupyter notebook by Trevor Muñoz (2013-08-18): [Programmatic Use of Open Refine to Facet and Cluster Names of 'Dishes' from NYPL's What's on the menu?](https://nbviewer.jupyter.org/gist/trevormunoz/6265360)
+- Jupyter notebook by Tony Hirst (2019-01-09) [Notebook demonstrating how to control OpenRefine via a Python client.](https://nbviewer.jupyter.org/github/ouseful-PR/openrefineder/blob/4cef25a4ca6077536c5f49cafb531499fbcad96e/notebooks/OpenRefine%20Demos.ipynb)
+- Unittests [test_refine.py](tests/test_refine.py) and [test_tutorial.py](tests/test_tutorial.py) (both importing [refinetest.py](tests/refinetest.py))
+- [OpenRefine API](https://github.com/OpenRefine/OpenRefine/wiki/OpenRefine-API) in official OpenRefine wiki
+
+## Binder
+
+[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master)
+
+- free to use on-demand server with Jupyter notebook, OpenRefine and Bash
+- no registration needed, will start within a few minutes
+- [restricted](https://mybinder.readthedocs.io/en/latest/faq.html#how-much-memory-am-i-given-when-using-binder) to 2 GB RAM and server will be deleted after 10 minutes of inactivity
+- [bash_kernel demo notebook](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/notebooks/openrefine-client-bash.ipynb) for using the openrefine-client in a Linux Bash environment [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master?urlpath=/tree/notebooks/openrefine-client-bash.ipynb)
+- [python2 demo notebook](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/notebooks/openrefine-client-python.ipynb) for using the openrefine-client in a Python 2 environment [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master?urlpath=/tree/notebooks/openrefine-client-python.ipynb)
+
+## Development
+
+If you would like to contribute to the Python client library please consider a pull request to the upstream repository [refine-client-py](https://github.com/PaulMakepeace/refine-client-py/).
+
+### Tests
+
+Ensure you have OpenRefine running (i.e. available at http://localhost:3333). If necessary set the environment variables `OPENREFINE_HOST` and `OPENREFINE_PORT` to change the URL.
+
+The Python client library includes several unit tests.
+
+- run all tests
+
+  ```sh
+  python setup.py test
+  ```
+
+- run subset test_facet
+
+  ```sh
+  python setup.py --test-suite tests.test_facet
+  ```
+
+There is also a script that uses docker images to run the unit tests with different versions of OpenRefine.
+
+- run tests on all OpenRefine versions (from 2.0 up to 3.5.0)
+
+  ```sh
+  ./tests.sh -a
+  ```
+
+- run tests on tag 3.5.0
+
+  ```sh
+  ./tests.sh -t 3.5.0
+  ```
+
+- run tests on tag 3.5.0 interactively (pause before and after tests)
+
+  ```sh
+  ./tests.sh -t 3.5.0 -i
+  ```
+
+- run tests on tags 3.5.0 and 2.7
+
+  ```sh
+  ./tests.sh -t 3.5.0 -t 2.7
+  ```
+
+For Linux there are also functional tests for all command line options.
+
+- run all functional tests on OpenRefine 3.5.0
+
+  ```sh
+  ./tests-cli.sh 3.5.0
+  ```
+
+- run all functional tests on OpenRefine 3.5.0 with one-file-executable
+
+  ```sh
+  ./tests-cli.sh 3.5.0 openrefine-client_0-3-7_linux
+  ```
+
+### Distributing
+
+Note to myself: When releasing a new version...
+
+1. Run functional tests
+
+   ```sh
+   for v in 2.7 2.8 3.0 3.1 3.2 3.3 3.4 3.4.1 3.5.0; do
+      ./tests-cli.sh $v
+   done
+   ```
+
+2. Make final changes in Git
+
+   - update versions (e.g. 0.3.7 und 0-3-7) in [README.md](https://github.com/opencultureconsulting/openrefine-client/blob/master/README.md#download)
+   - update version in [setup.py](https://github.com/opencultureconsulting/openrefine-client/blob/master/setup.py)
+   - check if [Dockerfile](https://github.com/opencultureconsulting/openrefine-client/blob/master/docker/Dockerfile) needs to be changed
+
+3. Build executables with PyInstaller
+
+   - Run PyInstaller in Python 2 environments on native Windows, macOS and Linux. Should be "the oldest version of the OS you need to support"! Current release is built with: 
+
+     - Ubuntu 16.04 LTS (64-bit)
+     - macOS Sierra 10.12 (64-bit)
+     - Windows 7 (32-bit)
+
+   - One-file-executables will be available in `dist/`.
+
+     ```sh
+     git clone https://github.com/opencultureconsulting/openrefine-client.git
+     cd openrefine-client
+     python2 -m pip install pyinstaller --user
+     python2 -m pip install urllib2_file --user
+     python2 -m PyInstaller --onefile refine.py --hidden-import google.refine.__main__
+     ```
+
+4. Run functional tests with Linux executable
+
+   ```sh
+   for v in 2.7 2.8 3.0 3.1 3.2 3.3 3.4 3.4.1 3.5.0; do
+      ./tests-cli.sh $v openrefine-client_0-3-7_linux
+   done
+   ```
+
+5. Create release in GitHub
+
+   - draft [release notes](https://github.com/opencultureconsulting/openrefine-client/releases) and attach one-file-executables
+
+6. Build package and upload to PyPI
+
+   ```sh
+   python3 setup.py sdist bdist_wheel
+   python3 -m twine upload dist/*
+   ```
+
+7. Update Docker container
+
+   - add new autobuild for release version
+   - trigger latest build
+
+8. Bump openrefine-client version in related projects
+
+   - openrefine-batch: [openrefine-batch.sh](https://github.com/opencultureconsulting/openrefine-batch/blob/master/openrefine-batch.sh#L7) and [openrefine-batch-docker.sh](https://github.com/opencultureconsulting/openrefine-batch/blob/master/openrefine-batch-docker.sh)
+   - openrefineder: [postBuild](https://github.com/felixlohmeier/openrefineder/blob/master/binder/postBuild)
+
+## Credits
+
+[Paul Makepeace](http://paulm.com), author
+
+David Huynh, [initial cut](<http://markmail.org/message/jsxzlcu3gn6drtb7)
+
+[Artfinder](http://www.artfinder.com), inspiration
+
+[Felix Lohmeier](https://felixlohmeier.de), extended the CLI features
+
+Some data used in the test suite has been used from publicly available sources:
+
+- louisiana-elected-officials.csv: from http://www.sos.louisiana.gov/tabid/136/Default.aspx
+
+- us_economic_assistance.csv: ["The Green Book"](http://www.data.gov/raw/1554)
+
+- eli-lilly.csv: [ProPublica's "Docs for Dollars](http://projects.propublica.org/docdollars) leading to a [Lilly Faculty PDF](http://www.lillyfacultyregistry.com/documents/EliLillyFacultyRegistryQ22010.pdf) processed by [David Huynh's ScraperWiki script](http://scraperwiki.com/scrapers/eli-lilly-dollars-for-docs-scraper/edit/)

README.rst

@@ -1,144 +0,0 @@
-===================================
-OpenRefine Python Client Library
-===================================
-
-The OpenRefine Python Client Library provides an interface to
-communicating with an `OpenRefine <http://openrefine.org/>`_ server.
-
-If you are looking for a ready to use command line interface to OpenRefine then you might be interested in the docker variation of this library:
-`felixlohmeier/openrefine-client <https://hub.docker.com/r/felixlohmeier/openrefine-client/>`_. You will find examples for batch processing (e.g. for usage in shell scripts) there.
-
-If you are familiar with python and want to go into more depth, then read on!
-
-Features
-=============
-
-Command line interface:
-
-- list projects: refine.py --list
-- create project from file: refine.py --create [FILE]
-- apply `rules from json file <http://kb.refinepro.com/2012/06/google-refine-json-and-my-notepad-or.html>`_: refine.py --apply [FILE.json] [PROJECTID]
-- export project to file: refine.py --export [PROJECTID] --output=FILE.tsv
-
-Currently, the following API is supported:
-
-- project creation/import, deletion, export
-- facet computation
-
-  - text
-  - text filter
-  - numeric
-  - blank
-  - starred & flagged
-  - ... extensible class
-
-- 'engine': managing multiple facets and their computation results
-- sorting & reordering
-- clustering
-- transforms
-- transposes
-- single and mass edits
-- annotation (star/flag)
-- column
-
-  - move
-  - add
-  - split
-  - rename
-  - reorder
-  - remove
-
-- reconciliation
-
-  - reconciliation judgment facet
-  - guessing column type
-  - querying reconciliation services preferences
-  - perform reconciliation
-
-Configuration
-=============
-
-By default the OpenRefine server URL is http://127.0.0.1:3333
-The environment variables ``OPENREFINE_HOST`` and ``OPENREFINE_PORT``
-enable overriding the host & port.
-
-In order to run all tests, a live Refine server is needed. No existing projects
-are affected.
-
-Installation
-============
-
-(Someone with more familiarity with python's byzantine collection of installation
-frameworks is very welcome to improve/"best practice" all this.)
-
-#. Install dependencies, which currently is ``urllib2_file``:
-
-   ``sudo pip install -r requirements.txt``
-
-   (If you don't have ``pip`` visit `pip-installer.org <http://www.pip-installer.org/en/latest/installing.html#install-or-upgrade-pip>`_)
-
-#. Ensure you have a Refine server running somewhere and, if necessary, set
-   the environment vars as above.
-
-#. Run tests, build, and install:
-
-   ``python setup.py test # to do a subset, e.g., --test-suite tests.test_facet``
-
-   ``python setup.py build``
-
-   ``python setup.py install``
-
-There is a Makefile that will do this too, and more.
-
-TODO
-====
-
-The API so far has been filled out from building a test suite to carry out the
-actions in `David Huynh's Refine tutorial <http://davidhuynh.net/spaces/nicar2011/tutorial.pdf>`_ which while certainly showing off a
-wide range of Refine features doesn't cover the entire suite. Notable exceptions
-currently include:
-
-- reconciliation support is useful but not complete
-- undo/redo
-- Freebase
-- join columns
-- columns from URL
-
-Contribute
-============
-
-Pull requests with passing tests welcome! Source is at https://github.com/PaulMakepeace/refine-client-py
-
-Useful Tools
-------------
-
-One aspect of development is watching HTTP transactions. To that end, I found
-`Fiddler <http://www.fiddler2.com/>`_ on Windows and `HTTPScoop
-<http://www.tuffcode.com/>`_ invaluable. The latter won't URL-decode nor nicely
-format JSON but the `Online JavaScript Beautifier <http://jsbeautifier.org/>`_
-will.
-
-History
-=======
-
-OpenRefine used to be called Google Refine, and this library used to be called
-the Google Refine Python Client Library.
-
-Credits
-=======
-
-Paul Makepeace, author, <paulm@paulm.com>
-
-David Huynh, `initial cut <http://markmail.org/message/jsxzlcu3gn6drtb7>`_
-
-`Artfinder <http://www.artfinder.com/>`_, inspiration
-
-Some data used in the test suite has been used from publicly available sources,
-
-- louisiana-elected-officials.csv: from
-  http://www.sos.louisiana.gov/tabid/136/Default.aspx
-
-- us_economic_assistance.csv: `"The Green Book" <http://www.data.gov/raw/1554>`_
-
-- eli-lilly.csv: `ProPublica's "Docs for Dollars" <http://projects.propublica.org/docdollars/>`_ leading to a `Lilly Faculty PDF <http://www.lillyfacultyregistry.com/documents/EliLillyFacultyRegistryQ22010.pdf>`_ processed by `David Huynh's ScraperWiki script <http://scraperwiki.com/scrapers/eli-lilly-dollars-for-docs-scraper/edit/>`_
-

docker/Dockerfile

@@ -1,33 +1,28 @@
-FROM alpine:latest
-MAINTAINER felixlohmeier <felixlohmeier@opencultureconsulting.com>
-# The OpenRefine Python Client Library from PaulMakepeace provides an interface to communicating with an OpenRefine server. This fork extends the CLI with some options to create new OpenRefine projects from files.
-# Source: https://github.com/felixlohmeier/openrefine-client
+FROM alpine:3.11
+LABEL maintainer="felixlohmeier@opencultureconsulting.com"
+# The OpenRefine Python Client Library from PaulMakepeace provides an interface to communicating with an OpenRefine server. This fork extends the command line interface (CLI) and supports communication between docker containers.
+# Source: https://github.com/opencultureconsulting/openrefine-client
 
-# Install python, pip, wget, unzip and bash
+# Install python and pip
+# ... and curl for https://github.com/opencultureconsulting/openrefine-batch
 RUN apk add --no-cache \
-	bash \
-	curl \
-	grep \
 	python \
 	py-pip \
-	wget \
-	unzip
+	curl
 
 # Install dependency urllib2_file
 RUN pip install urllib2_file==0.2.1
 
-# Download and build openrefine-client-master
+# Copy python scripts
 WORKDIR /app
-RUN wget --no-check-certificate https://github.com/felixlohmeier/openrefine-client/archive/master.zip
-RUN unzip master.zip && rm master.zip
-RUN python openrefine-client-master/setup.py build
-RUN python openrefine-client-master/setup.py install
+COPY google google
+COPY refine.py .
 
-# Change docker WORKDIR (shall be mounted)
+# Change docker WORKDIR (shall be mounted by user)
 WORKDIR /data
 
 # Execute refine.py
-ENTRYPOINT ["/app/openrefine-client-master/refine.py"]
+ENTRYPOINT ["/app/refine.py"]
 
 # Default command: print help
 CMD ["-h"]

docker/README.md

@@ -1,83 +0,0 @@
-## batch processing with python-client
-
-There are some client libraries for OpenRefine that communicate with the [OpenRefine API](https://github.com/OpenRefine/OpenRefine/wiki/OpenRefine-API). I have prepared a docker container on top of the [Python Library from PaulMakepeace](https://github.com/PaulMakepeace/refine-client-py/) and extended the CLI with some options to create new OpenRefine projects from files.
-
-### basic usage
-
-**1) start server:**
-> docker run -d --name=openrefine-server felixlohmeier/openrefine
-
-**2) run client with one of the following commands:**
-
-list projects:
-> docker run --rm --link openrefine-server felixlohmeier/openrefine-client --list
-
-create project from file:
-> docker run --rm --link openrefine-server felixlohmeier/openrefine-client --create [FILE]
-
-apply [rules from json file](http://kb.refinepro.com/2012/06/google-refine-json-and-my-notepad-or.html):
-> docker run --rm --link openrefine-server felixlohmeier/openrefine-client --apply [FILE.json] [PROJECTID]
-
-export project to file:
-> docker run --rm --link openrefine-server felixlohmeier/openrefine-client --export [PROJECTID] --output=FILE.tsv
-
-check help screen for more options:
-> docker run --rm --link openrefine-server felixlohmeier/openrefine-client --help
-
-**3) cleanup:**
-> docker stop openrefine-server && docker rm openrefine-server
-
-### example for customized run commands in interactive mode (e.g. for usage in terminals)
-
-**1) start server in terminal A:**
-
-```docker run --rm --name=openrefine-server -p 80:3333 -v /home/felix/refine:/data:z felixlohmeier/openrefine -i 0.0.0.0 -m 4G -d /data```
-
-* automatically remove docker container when it exits
-* set name "openrefine" for docker container
-* publish internal port 3333 to host port 80
-* mount host directory /home/felix/refine as working directory
-* make openrefine available in the network
-* increase java heap size to 4 GB
-* set refine workspace to /data
-* OpenRefine should be available at http://localhost
-
-**2) start client in terminal B (prints help screen):**
-
-```docker run --rm --link openrefine-server -v /home/felix/refine:/data:z felixlohmeier/openrefine-client```
-
-* automatically remove docker container when it exits
-* build up network connection with docker container "openrefine"
-* mount host directory /home/felix/refine as working directory
-* apply history in file /home/felix/refine/history.json to project with id 1234567890123
-
-### example for customized run commands in detached mode (e.g. for usage in shell scripts)
-
-**1) define variables (bring your own example data)**
-> workingdir=/home/felix/refine
-> inputfile=example.csv
-> jsonfile=test.json
-
-**2) start server**
-
- ```docker run -d --name=openrefine-server -v ${workingdir}:/data:z felixlohmeier/openrefine -i 0.0.0.0 -m 4G -d /data```
-
-**3) create project (import file)**
-
-```docker run --rm --link openrefine-server -v ${workingdir}:/data:z felixlohmeier/openrefine-client --create $inputfile```
-
-**4) get project id**
-
-```project=($(docker run --rm --link openrefine-server -v ${workingdir}:/data felixlohmeier/openrefine-client --list | cut -c 2-14))```
-
-**5) apply transformations from json file**
-
-```docker run --rm --link openrefine-server -v ${workingdir}:/data felixlohmeier/openrefine-client --apply ${jsonfile} ${project}```
-
-**6) export project to file**
-
-```docker run --rm --link openrefine-server -v ${workingdir}:/data felixlohmeier/openrefine-client --export --output=${project}.tsv ${project}```
-
-**7) cleanup**
-
-```docker stop -t=500 openrefine-server && docker rm openrefine-server```

google/refine/__main__.py

@@ -0,0 +1,279 @@
+#! /usr/bin/env python
+"""
+Script to provide a command line interface to a Refine server.
+"""
+
+# Copyright (c) 2011 Paul Makepeace, Real Programmers. All rights reserved.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+
+import optparse
+
+from google.refine import refine
+from google.refine import cli
+
+
+class myParser(optparse.OptionParser):
+
+    def format_epilog(self, formatter):
+        return self.epilog
+
+
+PARSER = \
+    myParser(description=('Script to provide a command line interface to an '
+                          'OpenRefine server.'),
+             usage='usage: %prog [--help | OPTIONS]',
+             epilog="""
+Example data:
+  --download "https://git.io/fj5hF" --output=duplicates.csv
+  --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+
+Basic commands:
+  --list # list all projects
+  --list -H 127.0.0.1 -P 80 # specify hostname and port
+  --create duplicates.csv # create new project from file
+  --info "duplicates" # show project metadata
+  --apply duplicates-deletion.json "duplicates" # apply rules in file to project
+  --export "duplicates" # export project to terminal in tsv format
+  --export --output=deduped.xls "duplicates" # export project to file in xls format
+  --delete "duplicates" # delete project
+
+Some more examples:
+  --info 1234567890123 # specify project by id
+  --create example.tsv --encoding=UTF-8
+  --create example.xml --recordPath=collection --recordPath=record
+  --create example.json --recordPath=_ --recordPath=_
+  --create example.xlsx --sheets=0
+  --create example.ods --sheets=0
+
+Example for Templating Export:
+  Cf. https://github.com/opencultureconsulting/openrefine-client#advanced-templating
+""")
+
+group1 = optparse.OptionGroup(PARSER, 'Connection options')
+group1.add_option('-H', '--host', dest='host',
+                  metavar='127.0.0.1',
+                  help='OpenRefine hostname (default: 127.0.0.1)')
+group1.add_option('-P', '--port', dest='port',
+                  metavar='3333',
+                  help='OpenRefine port (default: 3333)')
+PARSER.add_option_group(group1)
+
+group2 = optparse.OptionGroup(PARSER, 'Commands')
+group2.add_option('-c', '--create', dest='create',
+                  metavar='[FILE]',
+                  help='Create project from file. The filename ending (e.g. .csv) defines the input format (csv,tsv,xml,json,txt,xls,xlsx,ods)')
+group2.add_option('-l', '--list', dest='list',
+                  action='store_true',
+                  help='List projects')
+group2.add_option('--download', dest='download',
+                  metavar='[URL]',
+                  help='Download file from URL (e.g. example data). Combine with --output to specify a filename.')
+PARSER.add_option_group(group2)
+
+group3 = optparse.OptionGroup(
+    PARSER, 'Commands with argument [PROJECTID/PROJECTNAME]')
+group3.add_option('-d', '--delete', dest='delete',
+                  action='store_true',
+                  help='Delete project')
+group3.add_option('-f', '--apply', dest='apply',
+                  metavar='[FILE]',
+                  help='Apply JSON rules to OpenRefine project')
+group3.add_option('-E', '--export', dest='export',
+                  action='store_true',
+                  help='Export project in tsv format to stdout.')
+group3.add_option('-o', '--output', dest='output',
+                  metavar='[FILE]',
+                  help='Export project to file. The filename ending (e.g. .tsv) defines the output format (csv,tsv,xls,xlsx,html)')
+group3.add_option('--template', dest='template',
+                  metavar='[STRING]',
+                  help='Export project with templating. Provide (big) text string that you enter in the *row template* textfield in the export/templating menu in the browser app)')
+group3.add_option('--info', dest='info',
+                  action='store_true',
+                  help='show project metadata')
+PARSER.add_option_group(group3)
+
+group4 = optparse.OptionGroup(PARSER, 'General options')
+group4.add_option('--format', dest='file_format',
+                  help='Override file detection (import: csv,tsv,xml,json,line-based,fixed-width,xls,xlsx,ods; export: csv,tsv,html,xls,xlsx,ods)')
+PARSER.add_option_group(group4)
+
+group5 = optparse.OptionGroup(PARSER, 'Create options')
+group5.add_option('--columnWidths', dest='columnWidths',
+                  action='append',
+                  type='int',
+                  help='(txt/fixed-width), please provide widths in multiple arguments, e.g. --columnWidths=7 --columnWidths=5')
+group5.add_option('--encoding', dest='encoding',
+                  help='(csv,tsv,txt), please provide short encoding name (e.g. UTF-8)')
+group5.add_option('--guessCellValueTypes', dest='guessCellValueTypes',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(xml,csv,tsv,txt,json, default: false)')
+group5.add_option('--headerLines', dest='headerLines',
+                  type="int",
+                  help='(csv,tsv,txt/fixed-width,xls,xlsx,ods), default: 1, default txt/fixed-width: 0')
+group5.add_option('--ignoreLines', dest='ignoreLines',
+                  type="int",
+                  help='(csv,tsv,txt,xls,xlsx,ods), default: -1')
+group5.add_option('--includeFileSources', dest='includeFileSources',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(all formats), default: false')
+group5.add_option('--limit', dest='limit',
+                  type="int",
+                  help='(all formats), default: -1')
+group5.add_option('--linesPerRow', dest='linesPerRow',
+                  type="int",
+                  help='(txt/line-based), default: 1')
+group5.add_option('--processQuotes', dest='processQuotes',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(csv,tsv), default: true')
+group5.add_option('--projectName', dest='projectName',
+                  help='(all formats), default: filename')
+group5.add_option('--projectTags', dest='projectTags',
+                  action='append',
+                  help='(all formats), please provide tags in multiple arguments, e.g. --projectTags=beta --projectTags=client1')
+group5.add_option('--recordPath', dest='recordPath',
+                  action='append',
+                  help='(xml,json), please provide path in multiple arguments, e.g. /collection/record/ should be entered: --recordPath=collection --recordPath=record, default xml: root element, default json: _ _')
+group5.add_option('--separator', dest='separator',
+                  help='(csv,tsv), default csv: , default tsv: \\t')
+group5.add_option('--sheets', dest='sheets',
+                  action='append',
+                  type="int",
+                  help='(xls,xlsx,ods), please provide sheets in multiple arguments, e.g. --sheets=0 --sheets=1, default: 0 (first sheet)')
+group5.add_option('--skipDataLines', dest='skipDataLines',
+                  type="int",
+                  help='(csv,tsv,txt,xls,xlsx,ods), default: 0, default line-based: -1')
+group5.add_option('--storeBlankCellsAsNulls', dest='storeBlankCellsAsNulls',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(csv,tsv,txt,xls,xlsx,ods), default: true')
+group5.add_option('--storeBlankRows', dest='storeBlankRows',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(csv,tsv,txt,xls,xlsx,ods), default: true')
+group5.add_option('--storeEmptyStrings', dest='storeEmptyStrings',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(xml,json), default: true')
+group5.add_option('--trimStrings', dest='trimStrings',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(xml,json), default: false')
+PARSER.add_option_group(group5)
+
+group6 = optparse.OptionGroup(PARSER, 'Templating options')
+group6.add_option('--mode', dest='mode',
+                  metavar='row-based/record-based',
+                  choices=('row-based', 'record-based'),
+                  help='engine mode (default: row-based)')
+group6.add_option('--prefix', dest='prefix',
+                  help='text string that you enter in the *prefix* textfield in the browser app')
+group6.add_option('--rowSeparator', dest='rowSeparator',
+                  help='text string that you enter in the *row separator* textfield in the browser app')
+group6.add_option('--suffix', dest='suffix',
+                  help='text string that you enter in the *suffix* textfield in the browser app')
+group6.add_option('--filterQuery', dest='filterQuery',
+                  metavar='REGEX',
+                  help='Simple RegEx text filter on filterColumn, e.g. ^12015$'),
+group6.add_option('--filterColumn', dest='filterColumn',
+                  metavar='COLUMNNAME',
+                  help='column name for filterQuery (default: name of first column)')
+group6.add_option('--facets', dest='facets',
+                  help='facets config in json format (may be extracted with browser dev tools in browser app)')
+group6.add_option('--splitToFiles', dest='splitToFiles',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='will split each row/record into a single file; it specifies a presumably unique character series for splitting; --prefix and --suffix will be applied to all files; filename-prefix can be specified with --output (default: %Y%m%d)')
+group6.add_option('--suffixById', dest='suffixById',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='enhancement option for --splitToFiles; will generate filename-suffix from values in key column')
+PARSER.add_option_group(group6)
+
+
+def main():
+    """Command line interface."""
+
+    options, args = PARSER.parse_args()
+
+    # set environment
+    if options.host:
+        refine.REFINE_HOST = options.host
+    if options.port:
+        refine.REFINE_PORT = options.port
+
+    # get project_id
+    if args and not str.isdigit(args[0]):
+        projects = refine.Refine(refine.RefineServer()).list_projects().items()
+        idlist = []
+        for project_id, project_info in projects:
+            if args[0].decode('UTF-8') == project_info['name']:
+                idlist.append(str(project_id))
+        if len(idlist) > 1:
+            print('Error: Found %s projects with name %s.\n'
+                  'Please specify project by id.' % (len(idlist), args[0]))
+            for i in idlist:
+                print('')
+                cli.info(i)
+            return
+        else:
+            try:
+                project_id = idlist[0]
+            except IndexError:
+                print('Error: No project found with name %s.\n'
+                      'Try command --list' % args[0])
+                return
+    elif args:
+        project_id = args[0]
+
+    # commands without args
+    if options.list:
+        cli.ls()
+    elif options.download:
+        cli.download(options.download, output_file=options.output)
+    elif options.create:
+        group5_dict = {group5_arg.dest: getattr(options, group5_arg.dest)
+                       for group5_arg in group5.option_list}
+        kwargs = {k: v for k, v in group5_dict.items()
+                  if v is not None and v not in ['true', 'false']}
+        kwargs.update({k: True for k, v in group5_dict.items()
+                       if v == 'true'})
+        kwargs.update({k: False for k, v in group5_dict.items()
+                       if v == 'false'})
+        if options.file_format:
+            kwargs.update({'project_format': options.file_format})
+        cli.create(options.create, **kwargs)
+    # commands with args
+    elif args and options.info:
+        cli.info(project_id)
+    elif args and options.delete:
+        cli.delete(project_id)
+    elif args and options.apply:
+        cli.apply(project_id, options.apply)
+    elif args and options.template:
+        group6_dict = {group6_arg.dest: getattr(options, group6_arg.dest)
+                       for group6_arg in group6.option_list}
+        kwargs = {k: v for k, v in group6_dict.items()
+                  if v is not None and v not in ['true', 'false']}
+        kwargs.update({k: True for k, v in group6_dict.items()
+                       if v == 'true'})
+        kwargs.update({k: False for k, v in group6_dict.items()
+                       if v == 'false'})
+        cli.templating(project_id, options.template,
+                       output_file=options.output, **kwargs)
+    elif args and (options.export or options.output):
+        cli.export(project_id, output_file=options.output,
+                   export_format=options.file_format)
+    else:
+        PARSER.print_usage()
+
+
+if __name__ == "__main__":
+    # execute only if run as a script
+    main()

google/refine/cli.py

@@ -0,0 +1,335 @@
+#! /usr/bin/env python
+"""
+Functions used by the command line interface (CLI)
+"""
+
+# Copyright (c) 2011 Paul Makepeace, Real Programmers. All rights reserved.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+
+import json
+import os
+import ssl
+import sys
+import time
+import urllib
+from xml.etree import ElementTree
+
+from google.refine import refine
+
+
+def apply(project_id, history_file):
+    """Apply OpenRefine history from json file to project."""
+    project = refine.RefineProject(project_id)
+    response = project.apply_operations(history_file)
+    if response != 'ok':
+        raise Exception('Failed to apply %s to %s: %s' %
+                        (history_file, project_id, response))
+    else:
+        print('File %s has been successfully applied to project %s' %
+              (history_file, project_id))
+
+def create(project_file,
+           project_format=None,
+           columnWidths=None,
+           encoding=None,
+           guessCellValueTypes=False,
+           headerLines=None,
+           ignoreLines=None,
+           includeFileSources=False,
+           limit=None,
+           linesPerRow=None,
+           processQuotes=True,
+           projectName=None,
+           projectTags=None,
+           recordPath=None,
+           separator=None,
+           sheets=None,
+           skipDataLines=None,
+           storeBlankCellsAsNulls=True,
+           storeBlankRows=True,
+           storeEmptyStrings=True,
+           trimStrings=False
+           ):
+    """Create a new project from file."""
+    # guess format from file extension
+    if not project_format:
+        project_format = os.path.splitext(project_file)[1][1:].lower()
+        if project_format == 'txt':
+            try:
+                columnWidths[0]
+                project_format = 'fixed-width'
+            except TypeError:
+                project_format = 'line-based'
+    # defaults for each file type
+    if project_format == 'xml':
+        project_format = 'text/xml'
+        if not recordPath:
+            recordPath = [ElementTree.parse(project_file).getroot().tag]
+    elif project_format == 'csv':
+        project_format = 'text/line-based/*sv'
+    elif project_format == 'tsv':
+        project_format = 'text/line-based/*sv'
+        if not separator:
+            separator = '\t'
+    elif project_format == 'line-based':
+        project_format = 'text/line-based'
+        if not skipDataLines:
+            skipDataLines = -1
+    elif project_format == 'fixed-width':
+        project_format = 'text/line-based/fixed-width'
+        if not headerLines:
+            headerLines = 0
+    elif project_format == 'json':
+        project_format = 'text/json'
+        if not recordPath:
+            recordPath = ['_', '_']
+    elif project_format == 'xls':
+        project_format = 'binary/text/xml/xls/xlsx'
+        if not sheets:
+            sheets = [0]
+            # TODO: new format for sheets option introduced in OpenRefine 2.8
+    elif project_format == 'xlsx':
+        project_format = 'binary/text/xml/xls/xlsx'
+        if not sheets:
+            sheets = [0]
+            # TODO: new format for sheets option introduced in OpenRefine 2.8
+    elif project_format == 'ods':
+        project_format = 'text/xml/ods'
+        if not sheets:
+            sheets = [0]
+            # TODO: new format for sheets option introduced in OpenRefine 2.8
+    # execute
+    kwargs = {k: v for k, v in vars().items() if v is not None}
+    project = refine.Refine(refine.RefineServer()).new_project(
+        guess_cell_value_types=guessCellValueTypes,
+        ignore_lines=ignoreLines,
+        header_lines=headerLines,
+        skip_data_lines=skipDataLines,
+        store_blank_rows=storeBlankRows,
+        process_quotes=processQuotes,
+        project_name=projectName,
+        store_blank_cells_as_nulls=storeBlankCellsAsNulls,
+        include_file_sources=includeFileSources,
+        **kwargs)
+    rows = project.do_json('get-rows')['total']
+    if rows > 0:
+        print('{0}: {1}'.format('id', project.project_id))
+        print('{0}: {1}'.format('rows', rows))
+        return project
+    else:
+        raise Exception(
+            'Project contains 0 rows. Please check --help for mandatory '
+            'arguments for xml, json, xlsx and ods')
+
+
+def delete(project_id):
+    """Delete project."""
+    project = refine.RefineProject(project_id)
+    response = project.delete()
+    if response != True:
+        raise Exception('Failed to delete %s: %s' %
+                        (project_id, response))
+    else:
+        print('Project %s has been successfully deleted' % project_id)
+
+
+def download(url, output_file=None):
+    """Integrated download function for your convenience."""
+    if not output_file:
+        output_file = os.path.basename(url)
+    if os.path.exists(output_file):
+        print('Error: File %s already exists.\n'
+              'Delete existing file or try command --output '
+              'to specify a different filename.' % output_file)
+        return
+    # Workaround for SSL verification problems in one-file-executables
+    context = ssl._create_unverified_context()
+    urllib.urlretrieve(url, output_file, context=context)
+    print('Download to file %s complete' % output_file)
+
+
+def export(project_id, encoding=None, output_file=None, export_format=None):
+    """Dump a project to stdout or file."""
+    project = refine.RefineProject(project_id)
+    if not output_file:
+        if not export_format:
+            export_format = 'tsv'
+        if export_format in ['csv', 'tsv', 'txt']:
+                encoding = 'UTF-8'
+        sys.stdout.write(project.export(
+            export_format=export_format, encoding=encoding).read())
+    else:
+        ext = os.path.splitext(output_file)[1][1:]
+        if ext and not export_format:
+            export_format = ext.lower()
+        if not export_format:
+            export_format = 'tsv'
+        if export_format in ['csv', 'tsv', 'txt']:
+            encoding = 'UTF-8'
+        with open(output_file, 'wb') as f:
+            f.write(project.export(
+                export_format=export_format, encoding=encoding).read())
+        print('Export to file %s complete' % output_file)
+
+
+def info(project_id):
+    """Show project metadata"""
+    projects = refine.Refine(refine.RefineServer()).list_projects()
+    if project_id in projects.keys():
+        print('{0:>20}: {1}'.format('id', project_id))
+        print('{0:>20}: {1}'.format('url', 'http://' +
+                                    refine.REFINE_HOST + ':' +
+                                    refine.REFINE_PORT +
+                                    '/project?project=' + project_id))
+        for k, v in projects[project_id].items():
+            if v:
+                    print(u'{0:>20}: {1}'.format(k, v))
+        project_model = refine.RefineProject(project_id).get_models()
+        columns = [c['name'] for c in project_model['columnModel']['columns']]
+        for (i, v) in enumerate(columns, start=1):
+            print(u'{0:>20}: {1}'.format(u'column ' + str(i).zfill(3), v).encode('utf-8'))
+    else:
+        print('Error: No project found with id %s.\n'
+              'Check existing projects with command --list' % (project_id))
+
+
+def ls():
+    """Query the server and list projects sorted by mtime."""
+    projects = refine.Refine(refine.RefineServer()).list_projects().items()
+
+    def date_to_epoch(json_dt):
+        """Convert a JSON date time into seconds-since-epoch."""
+        return time.mktime(time.strptime(json_dt, '%Y-%m-%dT%H:%M:%SZ'))
+    projects.sort(key=lambda v: date_to_epoch(v[1]['modified']), reverse=True)
+    if projects:
+        for project_id, project_info in projects:
+            print(u'{0:>14}: {1}'.format(project_id, project_info['name']).encode('utf-8'))
+    else:
+        print('Error: No projects found')
+
+
+def templating(project_id,
+               template,
+               encoding='UTF-8',
+               output_file=None,
+               mode=None,
+               prefix='',
+               rowSeparator='\n',
+               suffix='',
+               filterQuery=None,
+               filterColumn=None,
+               facets=None,
+               splitToFiles=False,
+               suffixById=None
+               ):
+    """Dump a project to stdout or file with templating."""
+    project = refine.RefineProject(project_id)
+
+    # basic config
+    templateconfig = {'prefix': prefix,
+                      'suffix': suffix,
+                      'template': template,
+                      'rowSeparator': rowSeparator,
+                      'encoding': encoding}
+
+    # construct the engine config
+    if mode == 'record-based':
+        engine = {'facets': [], 'mode': 'record-based'}
+    else:
+        engine = {'facets': [], 'mode': 'row-based'}
+    if facets:
+        engine['facets'].append(json.loads(facets))
+    if filterQuery:
+        if not filterColumn:
+            filterColumn = project.get_models()['columnModel']['keyColumnName']
+        textFilter = {'type': 'text',
+                      'name': filterColumn,
+                      'columnName': filterColumn,
+                      'mode': 'regex',
+                      'caseSensitive': False,
+                      'query': filterQuery}
+        engine['facets'].append(textFilter)
+    templateconfig.update({'engine': json.dumps(engine)})
+
+    if not splitToFiles:
+        # normal output
+        if not output_file:
+            sys.stdout.write(project.export_templating(
+                             **templateconfig).read())
+        else:
+            with open(output_file, 'wb') as f:
+                f.write(project.export_templating(**templateconfig).read())
+            print('Export to file %s complete' % output_file)
+    else:
+        # splitToFiles functionality
+        prefix = templateconfig['prefix']
+        suffix = templateconfig['suffix']
+        split = '===|||THISISTHEBEGINNINGOFANEWRECORD|||==='
+        if not output_file:
+            output_file = time.strftime('%Y%m%d')
+        else:
+            base = os.path.splitext(output_file)[0]
+            ext = os.path.splitext(output_file)[1][1:]
+        if not ext:
+            ext = 'txt'
+        # generate config for subfeature suffixById
+        if suffixById:
+            ids_template = ('{{forNonBlank(' +
+                            'with(row.columnNames[0],cn,cells[cn].value),' +
+                            'v,v,"")}}')
+            ids_templateconfig = {'engine': json.dumps(engine),
+                                  'template': ids_template,
+                                  'rowSeparator': '\n',
+                                  'encoding': encoding}
+            ids = [line.rstrip('\n') for line in project.export_templating(
+                   **ids_templateconfig) if line.rstrip('\n')]
+        # generate common config
+        if mode == 'record-based':
+            # record-based: split-character into template
+            #               if key column is not blank (=record)
+            template = ('{{forNonBlank(' +
+                         'with(row.columnNames[0],cn,cells[cn].value),' +
+                         'v,"' + split + '", "")}}' +
+                        templateconfig['template'])
+            templateconfig.update({'prefix': '',
+                                   'suffix': '',
+                                   'template': template,
+                                   'rowSeparator': ''})
+        else:
+            # row-based: split-character into template
+            template = split + templateconfig['template']
+            templateconfig.update({'prefix': '',
+                                   'suffix': '',
+                                   'template': template,
+                                   'rowSeparator': ''})
+        # execute
+        records = project.export_templating(
+            **templateconfig).read().split(split)
+        del records[0]  # skip first blank entry
+        if suffixById:
+            for index, record in enumerate(records):
+                output_file = base + '_' + ids[index] + '.' + ext
+                with open(output_file, 'wb') as f:
+                    f.writelines([prefix, record, suffix])
+            print('Export to files complete. Last file: %s' % output_file)
+        else:
+            zeros = len(str(len(records)))
+            for index, record in enumerate(records):
+                output_file = base + '_' + \
+                    str(index + 1).zfill(zeros) + '.' + ext
+                with open(output_file, 'wb') as f:
+                    f.writelines([prefix, record, suffix])
+            print('Export to files complete. Last file: %s' % output_file)

google/refine/refine.py

@@ -53,6 +53,18 @@ class RefineServer(object):
             server = self.url()
         self.server = server[:-1] if server.endswith('/') else server
         self.__version = None     # see version @property below
+        self.token = None         # CSRF token introduced in OpenRefine 3.3
+        self.get_csrf_token()
+
+    def get_csrf_token(self):
+        """Return csrf token."""
+        try:
+            url = self.server + '/command/core/get-csrf-token'
+            response = json.loads(urllib2.urlopen(url).read())
+            self.token = response['token']
+            return self.token
+        except:
+            pass # fail silently to not disturb usage of OpenRefine <3.3
 
     def urlopen(self, command, data=None, params=None, project_id=None):
         """Open a Refine URL and with optional query params and POST data.
@@ -73,6 +85,9 @@ class RefineServer(object):
                 data['project'] = project_id
             else:
                 params['project'] = project_id
+        # be lazy and send the token for each API call (even when not needed)
+        if self.token:
+            params['csrf_token'] = self.token
         if params:
             url += '?' + urllib.urlencode(params)
         req = urllib2.Request(url)
@@ -82,7 +97,8 @@ class RefineServer(object):
         try:
             response = urllib2.urlopen(req)
         except urllib2.HTTPError as e:
-            raise Exception('HTTP %d "%s" for %s\n\t%s' % (e.code, e.msg, e.geturl(), data))
+            raise Exception('HTTP %d "%s" for %s\n\t%s' %
+                            (e.code, e.msg, e.geturl(), data))
         except urllib2.URLError as e:
             raise urllib2.URLError(
                 '%s for %s. No Refine server reachable/running; ENV set?' %
@@ -98,6 +114,10 @@ class RefineServer(object):
         """Open a Refine URL, optionally POST data, and return parsed JSON."""
         response = json.loads(self.urlopen(*args, **kwargs).read())
         if 'code' in response and response['code'] not in ('ok', 'pending'):
+            if 'Missing or invalid csrf_token parameter' == response['message']:
+                self.get_csrf_token()
+                response = json.loads(self.urlopen(*args, **kwargs).read())
+                return response
             error_message = ('server ' + response['code'] + ': ' +
                              response.get('message', response.get('stack', response)))
             raise Exception(error_message)
@@ -216,7 +236,7 @@ class Refine:
                     skip_data_lines=0,
                     limit=-1,
                     store_blank_rows=True,
-                    guess_cell_value_types=True,
+                    guess_cell_value_types=False,
                     process_quotes=True,
                     store_blank_cells_as_nulls=True,
                     include_file_sources=False,
@@ -236,6 +256,7 @@ class Refine:
         # POST is broken at the moment, so we send it in the URL
         new_style_options = dict(opts, **{
             'encoding': s(encoding),
+            'separator': s(separator)
         })
         params = {
             'options': json.dumps(new_style_options),
@@ -244,7 +265,6 @@ class Refine:
         # old style options
         options = {
             'format': project_format,
-            'separator': s(separator),
             'ignore-lines': s(ignore_lines),
             'header-lines': s(header_lines),
             'skip-data-lines': s(skip_data_lines),
@@ -253,7 +273,7 @@ class Refine:
             'process-quotes': s(process_quotes),
             'store-blank-rows': s(store_blank_rows),
             'store-blank-cells-as-nulls': s(store_blank_cells_as_nulls),
-            'include-file-sources': s(include_file_sources),
+            'include-file-sources': s(include_file_sources)
         }
 
         if project_url is not None:
@@ -398,7 +418,10 @@ class RefineProject:
         for i, column in enumerate(column_model['columns']):
             name = column['name']
             self.column_order[name] = i
-            column_index[name] = column['cellIndex']
+            try:
+                column_index[name] = column['cellIndex']
+            except KeyError:
+                column_index[name] = i
         self.key_column = column_model['keyColumnName']
         self.has_records = response['recordModel'].get('hasRecords', False)
         self.rows_response_factory = RowsResponseFactory(column_index)
@@ -427,11 +450,31 @@ class RefineProject:
             return 'ok'
         return response_json['code']  # can be 'ok' or 'pending'
 
-    def export(self, export_format='tsv'):
+    def export(self, encoding=None, export_format='tsv'):
         """Return a fileobject of a project's data."""
-        url = ('export-rows/' + urllib.quote(self.project_name()) + '.' +
-               export_format)
-        return self.do_raw(url, data={'format': export_format})
+        url = ('export-rows/' +
+               urllib.quote(self.project_name().encode('utf8')) +
+               '.' + export_format)
+        data = {'format': export_format}
+        if encoding:
+            data['encoding'] = encoding
+        return self.do_raw(url, data)
+
+    def export_templating(self, encoding=None, engine='', prefix='',
+                          template='', rowSeparator='\n', suffix=''):
+        """Return a fileobject of a project's data in templating mode."""
+        url = ('export-rows/' +
+               urllib.quote(self.project_name().encode('utf8')) +
+               '.' + 'txt')
+        data = {'format': 'template',
+                'template': template,
+                'engine': engine,
+                'prefix': prefix,
+                'suffix': suffix,
+                'separator': rowSeparator}
+        if encoding:
+            data['encoding'] = encoding
+        return self.do_raw(url, data)
 
     def export_rows(self, **kwargs):
         """Return an iterable of parsed rows of a project's data."""

refine.py

@@ -18,553 +18,18 @@ Script to provide a command line interface to a Refine server.
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>
 
-
-import optparse
-import os
 import sys
-import time
 
-from google.refine import refine
+from google.refine import __main__, cli, refine
 
-import urllib2_file
-import urllib2
-import urlparse
-
-import json
-
-class myParser(optparse.OptionParser):
-
-    def format_epilog(self, formatter):
-        return self.epilog
-
-PARSER = \
-    myParser(description='Script to provide a command line interface to an OpenRefine server.',
-             usage='usage: %prog [--help | OPTIONS]',
-             epilog="""
-Examples:
-  ./refine.py --list # show list of Refine projects, ID: name
-  ./refine.py --list -H 127.0.0.1 -P 80 # specify hostname and port
-  ./refine.py --create example.csv # create new project from file example.csv
-  ./refine.py --create example.tsv --format=tsv --encoding=UTF-8
-  ./refine.py --create example.xml --format=xml --recordPath=collection --recordPath=record
-  ./refine.py --create example.json --format=json --recordPath=_ --recordPath=_
-  ./refine.py --create example.xlsx --format=xlsx --sheets=0
-  ./refine.py --create example.ods --format=ods --sheets=0
-  ./refine.py --export 1234... > project.tsv # export project 1234... in tsv format
-  ./refine.py --export --output=project.xls 1234... # export project in xls format
-  ./refine.py --apply trim.json 1234... # apply rules in trim.json to project 1234...
-
-""")
-
-group1 = optparse.OptionGroup(PARSER, 'Connection options')
-group1.add_option('-H', '--host', dest='host', metavar='127.0.0.1',
-                  help='OpenRefine hostname (default: localhost)')
-group1.add_option('-P', '--port', dest='port', metavar='3333',
-                  help='OpenRefine port (default: 3333)')
-PARSER.add_option_group(group1)
-
-group2 = optparse.OptionGroup(PARSER, 'Commands')
-group2.add_option('-l', '--list', dest='list', action='store_true',
-                  help='List projects: refine.py -l')
-group2.add_option('-c', '--create', dest='create', action='store_true',
-                  help='Create project from file: refine.py -c [FILE]')
-group2.add_option('-E', '--export', dest='export', action='store_true',
-                  help='Export project: refine.py -E [PROJECTID]')
-group2.add_option('-f', '--apply', dest='apply', metavar='file.json',
-                  help='Apply JSON rules: refine.py -f [FILE] [PROJECTID]')
-PARSER.add_option_group(group2)
-
-group3 = optparse.OptionGroup(PARSER, 'Export options (optional)')
-group3.add_option('-o', '--output', dest='output', metavar='file.csv',
-                  help='Specify output filename and filetype. The filename ending (e.g. .csv) defines the output format (csv,tsv,xls,html)')
-PARSER.add_option_group(group3)
-
-group4 = optparse.OptionGroup(PARSER, 'Create format (mandatory for xml, json, fixed-width, xlsx, ods)')
-group4.add_option('--format', dest='inputformat',
-                  help='Specify input format (csv, tsv, xml, json, line-based, fixed-width, xlsx, ods)')
-PARSER.add_option_group(group4)
-
-group5 = optparse.OptionGroup(PARSER, 'Create options (mandatory for xml, json, fixed-width, xslx, ods; only together with --format)')
-group5.add_option('--recordPath', dest='recordPath', action='append',
-                  help='(xml, json), please provide path in multiple arguments without slashes, e.g. /collection/record/ should be entered like this: --recordPath=collection --recordPath=record')
-group5.add_option('--columnWidths', dest='columnWidths',
-                  help='(fixed-width) please provide widths separated by comma (e.g. 7,5)')
-group5.add_option('--sheets', dest='sheets',
-                  help='(xlsx, ods), please provide sheets separated by comma (e.g. 0,1), default: 0 (first sheet)')
-PARSER.add_option_group(group5)
-
-group6 = optparse.OptionGroup(PARSER, 'More create options (optional, only together with --format)')
-group6.add_option('--projectName', dest='projectName',
-                  help='(all formats), default: (filename)')
-group6.add_option('--limit', dest='limit',
-                  help='(all formats), default: -1')
-group6.add_option('--includeFileSources', dest='includeFileSources',
-                  help='(all formats), default: false')
-group6.add_option('--trimStrings', dest='trimStrings',
-                  help='(xml, json), default: false')
-group6.add_option('--storeEmptyStrings', dest='storeEmptyStrings',
-                  help='(xml, json), default: true')
-group6.add_option('--guessCellValueTypes', dest='guessCellValueTypes',
-                  help='(xml, csv, tsv, fixed-width, json), default: false')
-group6.add_option('--encoding', dest='encoding',
-                  help='(csv, tsv, line-based, fixed-width), please provide short encoding name (e.g. UTF-8)')
-group6.add_option('--ignoreLines', dest='ignoreLines',
-                  help='(csv, tsv, line-based, fixed-width, xlsx, ods), default: -1')
-group6.add_option('--headerLines', dest='headerLines',
-                  help='(csv, tsv, fixed-width, xlsx, ods), default: 1')
-group6.add_option('--skipDataLines', dest='skipDataLines',
-                  help='(csv, tsv, line-based, fixed-width, xlsx, ods), default: 0')
-group6.add_option('--storeBlankRows', dest='storeBlankRows',
-                  help='(csv, tsv, line-based, fixed-width, xlsx, ods), default: true')
-group6.add_option('--separator', dest='separator',
-                  help='(csv, tsv), default: csv: , tsv: \\t')
-group6.add_option('--processQuotes', dest='processQuotes',
-                  help='(csv, tsv), default: true')
-group6.add_option('--storeBlankCellsAsNulls',
-                  dest='storeBlankCellsAsNulls',
-                  help='(csv, tsv, line-based, fixed-width, xlsx, ods), default: true')
-group6.add_option('--linesPerRow', dest='linesPerRow',
-                  help='(line-based), default: 1')
-PARSER.add_option_group(group6)
-
-def create_project(options, file_fullpath):
-    servernewproject = 'http://' + refine.REFINE_HOST
-    if refine.REFINE_PORT != '80':
-        servernewproject += ':' + refine.REFINE_PORT
-
-    input_format = ''
-    input_options = ''
-
-    # xml
-    if options.inputformat == 'xml':
-        input_format = 'text/xml'
-        recordPath = 'record'
-        if options.recordPath:
-            recordPath = options.recordPath
-        limit = '-1'
-        if options.limit:
-            limit = options.limit
-        trimStrings = 'false'
-        if options.trimStrings:
-            trimStrings = options.trimStrings
-        guessCellValueTypes = 'false'
-        if options.guessCellValueTypes:
-            guessCellValueTypes = options.guessCellValueTypes
-        storeEmptyStrings = 'true'
-        if options.storeEmptyStrings:
-            storeEmptyStrings = options.storeEmptyStrings
-        includeFileSources = 'false'
-        if options.includeFileSources:
-            includeFileSources = options.includeFileSources
-        input_options = '{"recordPath":["' + '","'.join(recordPath) + '"]' \
-            + ',"limit":' + limit \
-            + ',"trimStrings":' + trimStrings \
-            + ',"guessCellValueTypes":' + guessCellValueTypes \
-            + ',"storeEmptyStrings":' + storeEmptyStrings \
-            + ',"includeFileSources":' + includeFileSources \
-            + '}'
-
-    # csv
-    if options.inputformat == 'csv':
-        input_format = 'text/line-based/*sv'
-        encoding = ''
-        if options.encoding:
-            encoding = options.encoding
-        separator = ','
-        if options.separator:
-            separator = options.separator
-        ignoreLines = '-1'
-        if options.ignoreLines:
-            ignoreLines = options.ignoreLines
-        headerLines = '1'
-        if options.headerLines:
-            headerLines = options.headerLines
-        skipDataLines = '0'
-        if options.skipDataLines:
-            skipDataLines = options.skipDataLines
-        limit = '-1'
-        if options.limit:
-            limit = options.limit
-        storeBlankRows = 'true'
-        if options.storeBlankRows:
-            storeBlankRows = options.storeBlankRows
-        guessCellValueTypes = 'false'
-        if options.guessCellValueTypes:
-            guessCellValueTypes = options.guessCellValueTypes
-        processQuotes = 'true'
-        if options.processQuotes:
-            processQuotes = options.processQuotes
-        storeBlankCellsAsNulls = 'true'
-        if options.storeBlankCellsAsNulls:
-            storeBlankCellsAsNulls = options.storeBlankCellsAsNulls
-        includeFileSources = 'false'
-        if options.includeFileSources:
-            includeFileSources = options.includeFileSources
-        input_options = '{"encoding":"' + encoding + '"' \
-            + ',"separator":"' + separator + '"' \
-            + ',"ignoreLines":' + ignoreLines \
-            + ',"headerLines":' + headerLines \
-            + ',"skipDataLines":' + skipDataLines \
-            + ',"limit":' + limit \
-            + ',"storeBlankRows":' + storeBlankRows \
-            + ',"guessCellValueTypes":' + guessCellValueTypes \
-            + ',"processQuotes":' + processQuotes \
-            + ',"storeBlankCellsAsNulls":' + storeBlankCellsAsNulls \
-            + ',"includeFileSources":' + includeFileSources \
-            + '}'
-
-    # tsv
-    if options.inputformat == 'tsv':
-        input_format = 'text/line-based/*sv'
-        encoding = ''
-        if options.encoding:
-            encoding = options.encoding
-        separator = '\\t'
-        if options.separator:
-            separator = options.separator
-        ignoreLines = '-1'
-        if options.ignoreLines:
-            ignoreLines = options.ignoreLines
-        headerLines = '1'
-        if options.headerLines:
-            headerLines = options.headerLines
-        skipDataLines = '0'
-        if options.skipDataLines:
-            skipDataLines = options.skipDataLines
-        limit = '-1'
-        if options.limit:
-            limit = options.limit
-        storeBlankRows = 'true'
-        if options.storeBlankRows:
-            storeBlankRows = options.storeBlankRows
-        guessCellValueTypes = 'false'
-        if options.guessCellValueTypes:
-            guessCellValueTypes = options.guessCellValueTypes
-        processQuotes = 'true'
-        if options.processQuotes:
-            processQuotes = options.processQuotes
-        storeBlankCellsAsNulls = 'true'
-        if options.storeBlankCellsAsNulls:
-            storeBlankCellsAsNulls = options.storeBlankCellsAsNulls
-        includeFileSources = 'false'
-        if options.includeFileSources:
-            includeFileSources = options.includeFileSources
-        input_options = '{"encoding":"' + encoding + '"' \
-            + ',"separator":"' + separator + '"' \
-            + ',"ignoreLines":' + ignoreLines \
-            + ',"headerLines":' + headerLines \
-            + ',"skipDataLines":' + skipDataLines \
-            + ',"limit":' + limit \
-            + ',"storeBlankRows":' + storeBlankRows \
-            + ',"guessCellValueTypes":' + guessCellValueTypes \
-            + ',"processQuotes":' + processQuotes \
-            + ',"storeBlankCellsAsNulls":' + storeBlankCellsAsNulls \
-            + ',"includeFileSources":' + includeFileSources \
-            + '}'
-
-    # line-based
-    if options.inputformat == 'line-based':
-        input_format = 'text/line-based'
-        encoding = ''
-        if options.encoding:
-            encoding = options.encoding
-        linesPerRow = '1'
-        if options.linesPerRow:
-            linesPerRow = options.linesPerRow
-        ignoreLines = '-1'
-        if options.ignoreLines:
-            ignoreLines = options.ignoreLines
-        limit = '-1'
-        if options.limit:
-            limit = options.limit
-        skipDataLines = '-1'
-        if options.skipDataLines:
-            skipDataLines = options.skipDataLines
-        storeBlankRows = 'true'
-        if options.storeBlankRows:
-            storeBlankRows = options.storeBlankRows
-        storeBlankCellsAsNulls = 'true'
-        if options.storeBlankCellsAsNulls:
-            storeBlankCellsAsNulls = options.storeBlankCellsAsNulls
-        includeFileSources = 'false'
-        if options.includeFileSources:
-            includeFileSources = options.includeFileSources
-        input_options = '{"encoding":"' + encoding + '"' \
-            + ',"linesPerRow":' + linesPerRow \
-            + ',"ignoreLines":' + ignoreLines \
-            + ',"limit":' + limit \
-            + ',"skipDataLines":' + skipDataLines \
-            + ',"storeBlankRows":' + storeBlankRows \
-            + ',"storeBlankCellsAsNulls":' + storeBlankCellsAsNulls \
-            + ',"includeFileSources":' + includeFileSources \
-            + '}'
-
-    # fixed-width
-    if options.inputformat == 'fixed-width':
-        input_format = 'text/line-based/fixed-width'
-        encoding = ''
-        if options.encoding:
-            encoding = options.encoding
-        columnWidths = ''
-        if options.columnWidths:
-            columnWidths = options.columnWidths
-        ignoreLines = '-1'
-        if options.ignoreLines:
-            ignoreLines = options.ignoreLines
-        headerLines = '0'
-        if options.headerLines:
-            headerLines = options.headerLines
-        skipDataLines = '0'
-        if options.skipDataLines:
-            skipDataLines = options.skipDataLines
-        limit = '-1'
-        if options.limit:
-            limit = options.limit
-        guessCellValueTypes = 'false'
-        if options.guessCellValueTypes:
-            guessCellValueTypes = options.guessCellValueTypes
-        storeBlankRows = 'true'
-        if options.storeBlankRows:
-            storeBlankRows = options.storeBlankRows
-        storeBlankCellsAsNulls = 'true'
-        if options.storeBlankCellsAsNulls:
-            storeBlankCellsAsNulls = options.storeBlankCellsAsNulls
-        includeFileSources = 'false'
-        if options.includeFileSources:
-            includeFileSources = options.includeFileSources
-        input_options = '{"encoding":"' + encoding + '"' \
-            + ',"columnWidths":[' + columnWidths + ']' \
-            + ',"ignoreLines":' + ignoreLines \
-            + ',"headerLines":' + headerLines \
-            + ',"skipDataLines":' + skipDataLines \
-            + ',"limit":' + limit \
-            + ',"guessCellValueTypes":' + guessCellValueTypes \
-            + ',"storeBlankRows":' + storeBlankRows \
-            + ',"storeBlankCellsAsNulls":' + storeBlankCellsAsNulls \
-            + ',"includeFileSources":' + includeFileSources \
-            + '}'
-
-    # json
-    if options.inputformat == 'json':
-        input_format = 'text/json'
-        recordPath = ['_', '_']
-        if options.recordPath:
-            recordPath = options.recordPath
-        limit = '-1'
-        if options.limit:
-            limit = options.limit
-        trimStrings = 'false'
-        if options.trimStrings:
-            trimStrings = options.trimStrings
-        guessCellValueTypes = 'false'
-        if options.guessCellValueTypes:
-            guessCellValueTypes = options.guessCellValueTypes
-        storeEmptyStrings = 'true'
-        if options.storeEmptyStrings:
-            storeEmptyStrings = options.storeEmptyStrings
-        includeFileSources = 'false'
-        if options.includeFileSources:
-            includeFileSources = options.includeFileSources
-        input_options = '{"recordPath":["' + '","'.join(recordPath) + '"]' \
-            + ',"limit":' + limit \
-            + ',"trimStrings":' + trimStrings \
-            + ',"guessCellValueTypes":' + guessCellValueTypes \
-            + ',"storeEmptyStrings":' + storeEmptyStrings \
-            + ',"includeFileSources":' + includeFileSources \
-            + '}'
-
-    # xlsx
-    if options.inputformat == 'xlsx':
-        input_format = 'binary/text/xml/xls/xlsx'
-        sheets = '0'
-        if options.sheets:
-            sheets = options.sheets
-        ignoreLines = '-1'
-        if options.ignoreLines:
-            ignoreLines = options.ignoreLines
-        headerLines = '1'
-        if options.headerLines:
-            headerLines = options.headerLines
-        skipDataLines = '0'
-        if options.skipDataLines:
-            skipDataLines = options.skipDataLines
-        limit = '-1'
-        if options.limit:
-            limit = options.limit
-        storeBlankRows = 'true'
-        if options.storeBlankRows:
-            storeBlankRows = options.storeBlankRows
-        storeBlankCellsAsNulls = 'true'
-        if options.storeBlankCellsAsNulls:
-            storeBlankCellsAsNulls = options.storeBlankCellsAsNulls
-        includeFileSources = 'false'
-        if options.includeFileSources:
-            includeFileSources = options.includeFileSources
-        input_options = '{"sheets":[' + sheets + ']' \
-            + ',"ignoreLines":' + ignoreLines \
-            + ',"headerLines":' + headerLines \
-            + ',"skipDataLines":' + skipDataLines \
-            + ',"limit":' + limit \
-            + ',"storeBlankRows":' + storeBlankRows \
-            + ',"storeBlankCellsAsNulls":' + storeBlankCellsAsNulls \
-            + ',"includeFileSources":' + includeFileSources \
-            + '}'
-
-    # ods
-    if options.inputformat == 'ods':
-        input_format = 'text/xml/ods'
-        sheets = '0'
-        if options.sheets:
-            sheets = options.sheets
-        ignoreLines = '-1'
-        if options.ignoreLines:
-            ignoreLines = options.ignoreLines
-        headerLines = '1'
-        if options.headerLines:
-            headerLines = options.headerLines
-        skipDataLines = '0'
-        if options.skipDataLines:
-            skipDataLines = options.skipDataLines
-        limit = '-1'
-        if options.limit:
-            limit = options.limit
-        storeBlankRows = 'true'
-        if options.storeBlankRows:
-            storeBlankRows = options.storeBlankRows
-        storeBlankCellsAsNulls = 'true'
-        if options.storeBlankCellsAsNulls:
-            storeBlankCellsAsNulls = options.storeBlankCellsAsNulls
-        includeFileSources = 'false'
-        if options.includeFileSources:
-            includeFileSources = options.includeFileSources
-        input_options = '{"sheets":[' + sheets + ']' \
-            + ',"ignoreLines":' + ignoreLines \
-            + ',"headerLines":' + headerLines \
-            + ',"skipDataLines":' + skipDataLines \
-            + ',"limit":' + limit \
-            + ',"storeBlankRows":' + storeBlankRows \
-            + ',"storeBlankCellsAsNulls":' + storeBlankCellsAsNulls \
-            + ',"includeFileSources":' + includeFileSources \
-            + '}'
-
-    file_name = os.path.split(file_fullpath)[-1]
-    projectName = file_name
-    if options.projectName:
-        projectName = options.projectName
-    data = {}
-    data['project-file'] = {'fd': open(file_fullpath),
-                            'filename': file_name}
-    data['project-name'] = projectName
-
-    response = urllib2.urlopen(servernewproject
-                               + '/command/core/create-project-from-upload?format='
-                               + input_format + '&options='
-                               + input_options, data)
-    response_body = response.read()
-    url_params = \
-        urlparse.parse_qs(urlparse.urlparse(response.geturl()).query)
-
-    if 'project' in url_params:
-        project_id = url_params['project'][0]
-        print 'New project: ' + project_id
-    else:
-        raise Exception('Project not created')
-
-    # wait until project is created
-    def wait_until_idle(self, polling_delay=0.5):
-        while True:
-            response = urllib2.urlopen(servernewproject + '/command/core/get-processes?project=' + project_id)
-            response_body = response.read()
-            url_params = \
-                urlparse.parse_qs(urlparse.urlparse(response.geturl()).query)
-            if 'processes' in url_params and len(url_params['processes']) > 0:
-                time.sleep(polling_delay)
-            else:
-                print 'done'
-                return
-   
-    # check number of rows
-    response = urllib2.urlopen(servernewproject
-                               + '/command/core/get-rows?project='
-                               + project_id
-                               + '&start=0&limit=0')
-    response_body = response.read()
-    response_json = json.loads(response_body)
-    if 'total' in response_body and response_json['total'] > 0:
-        print 'Number of rows:', response_json['total']
-    else:
-        raise Exception('Project contains 0 rows. Please check --help for mandatory arguments for xml, json, xls and ods')
-
-def list_projects():
-    """Query the Refine server and list projects by ID: name."""
-    projects = refine.Refine(refine.RefineServer()).list_projects().items()
-
-    def date_to_epoch(json_dt):
-        """Convert a JSON date time into seconds-since-epoch."""
-        return time.mktime(time.strptime(json_dt, '%Y-%m-%dT%H:%M:%SZ'))
-    projects.sort(key=lambda v: date_to_epoch(v[1]['modified']), reverse=True)
-    for project_id, project_info in projects:
-        print('{0:>14}: {1}'.format(project_id, project_info['name']))
-
-
-def export_project(project, options):
-    """Dump a project to stdout or options.output file."""
-    export_format = 'tsv'
-    if options.output:
-        ext = os.path.splitext(options.output)[1][1:]     # 'xls'
-        if ext:
-            export_format = ext.lower()
-        output = open(options.output, 'wb')
-    else:
-        output = sys.stdout
-    output.writelines(project.export(export_format=export_format))
-    output.close()
-
-
-#noinspection PyPep8Naming
-def main():
-    """Main."""
-
-    # get environment variables in docker network
-    docker_host = os.environ.get('OPENREFINE_SERVER_PORT_3333_TCP_ADDR')
-    if docker_host:
-        os.environ["OPENREFINE_HOST"] = docker_host
-        refine.REFINE_HOST = docker_host
-    docker_port = os.environ.get('OPENREFINE_SERVER_PORT_3333_TCP_PORT')
-    if docker_port:
-        os.environ["OPENREFINE_PORT"] = docker_port
-        refine.REFINE_PORT = docker_port
-
-    options, args = PARSER.parse_args()
-
-    if options.host:
-        refine.REFINE_HOST = options.host
-    if options.port:
-        refine.REFINE_PORT = options.port
-
-    if not options.list and len(args) != 1:
-        PARSER.print_usage()
-
-    if options.list:
-        list_projects()
-
-    if options.create:
-        file_fullpath = args[0]
-        create_project(options, file_fullpath)
-
-    if options.apply:
-        project = refine.RefineProject(args[0])
-        response = project.apply_operations(options.apply)
-        if response != 'ok':
-            print >> sys.stderr, 'Failed to apply %s: %s' \
-                % (options.apply, response)
-
-    if options.export:
-        project = refine.RefineProject(args[0])
-        export_project(project, options)
-
-        return project
+# workaround for pyinstaller
+if getattr(sys, 'frozen', False) and hasattr(sys, '_MEIPASS'):
+    reload(sys)
+    sys.setdefaultencoding('utf-8')
+    if sys.platform == "win32":
+        import codecs
+        codecs.register(lambda name: codecs.lookup(
+            'utf-8') if name == 'cp65001' else None)
 
 if __name__ == '__main__':
-    # return project so that it's available interactively, python -i refine.py
-    refine_project = main()
+    __main__.main()

setup.py

@@ -24,23 +24,34 @@ from setuptools import find_packages
 def read(filename):
     return open(os.path.join(os.path.dirname(__file__), filename)).read()
 
-setup(name='refine-client',
-      version='0.2.1',
+setup(name='openrefine-client',
+      version='0.3.10',
       description=('The OpenRefine Python Client Library provides an '
-                   'interface to communicating with an OpenRefine server.'),
-      long_description=read('README.rst'),
-      author='Paul Makepeace',
-      author_email='paulm@paulm.com',
-      url='https://github.com/PaulMakepeace/refine-client-py',
+                   'interface to communicating with an OpenRefine server. '
+                   'This fork extends the command line interface (CLI).'),
+      long_description=read('README.md'),
+      long_description_content_type='text/markdown',
+      author='Felix Lohmeier',
+      author_email='felix.lohmeier@opencultureconsulting.com',
+      url='https://github.com/opencultureconsulting/openrefine-client',
       packages=find_packages(exclude=['tests']),
       install_requires=['urllib2_file'],
+      python_requires='>=2.7, !=3.*',
+      entry_points={
+          'console_scripts': [ 'openrefine-client = google.refine.__main__:main' ]
+      },
       platforms=['Any'],
+      keywords='openrefine client batch processing docker etl code4lib',
       classifiers = [
-          'Development Status :: 3 - Alpha',
+          'Development Status :: 4 - Beta',
           'Intended Audience :: Developers',
+          'Intended Audience :: System Administrators',
           'License :: OSI Approved :: GNU General Public License (GPL)',
+          'License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)',
           'Operating System :: OS Independent',
           'Programming Language :: Python',
+          'Programming Language :: Python :: 2',
+          'Programming Language :: Python :: 2.7',
           'Topic :: Software Development :: Libraries :: Python Modules',
           'Topic :: Text Processing',
       ],

tests-cli.sh

@@ -0,0 +1,123 @@
+#!/bin/bash
+# Script for running functional tests against the CLI
+
+# Copyright (c) 2011 Paul Makepeace, Real Programmers. All rights reserved.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+# ================================== CONFIG ================================== #
+
+cd "${BASH_SOURCE%/*}/" || exit 1
+
+port=3334
+
+if [[ ${1} ]]; then
+  version="${1}"
+else
+  version="3.2"
+fi
+refine="openrefine-${version}/refine"
+
+if [[ ${2} ]]; then
+  client="$(readlink -e "${2}")"
+else
+  client="python2 $(readlink -e refine.py)"
+fi
+cmd="${client} -H localhost -P ${port}"
+
+if [[ ${3} ]]; then
+  filename="${3%%.*}"
+else
+  filename=""
+fi
+cmd="${client} -H localhost -P ${port}"
+
+# =============================== REQUIREMENTS =============================== #
+
+# check existence of java and cURL
+if [[ -z "$(command -v java 2> /dev/null)" ]] ; then
+  echo 1>&2 "ERROR: OpenRefine requires JAVA runtime environment (jre)" \
+    "https://openjdk.java.net/install/"
+  exit 1
+fi
+if [[ -z "$(command -v curl 2> /dev/null)" ]] ; then
+  echo 1>&2 "ERROR: This shell script requires cURL" \
+    "https://curl.haxx.se/download.html"
+  exit 1
+fi
+# download OpenRefine
+if [[ -z "$(readlink -e "${refine}")" ]]; then
+  echo "Download OpenRefine ${version}..."
+  mkdir -p "$(dirname "${refine}")"
+  curl -L --output openrefine.tar.gz \
+    "https://github.com/OpenRefine/OpenRefine/releases/download/${version}/openrefine-linux-${version}.tar.gz"
+  echo "Install OpenRefine ${version} in subdirectory $(dirname "${refine}")..."
+  tar -xzf openrefine.tar.gz -C "$(dirname "${refine}")" --strip 1 --totals
+  rm -f openrefine.tar.gz
+  # do not try to open OpenRefine in browser
+  sed -i '$ a JAVA_OPTIONS=-Drefine.headless=true' \
+    "$(dirname "${refine}")"/refine.ini
+  # set autosave period from 5 minutes to 25 hours
+  sed -i 's/#REFINE_AUTOSAVE_PERIOD=60/REFINE_AUTOSAVE_PERIOD=1500/' \
+    "$(dirname "${refine}")"/refine.ini
+  echo
+fi
+
+# ================================== SETUP =================================== #
+
+dir="$(readlink -f "tests/tmp")"
+mkdir -p "${dir}"
+rm -f tests-cli.log
+
+echo "start OpenRefine ${version}..."
+${refine} -v warn -p ${port} -d "${dir}" &>> tests-cli.log &
+pid_server=${!}
+timeout 30s bash -c "until curl -s 'http://localhost:3334' \
+  | cat | grep -q -o 'OpenRefine' ; do sleep 1; done" \
+  || error "starting OpenRefine server failed!"
+echo
+
+# ================================== TESTS =================================== #
+
+echo "running tests, please wait..."
+tests=()
+results=()
+for t in tests/*${filename}*.sh; do
+  tests+=("${t}")
+  echo "======================= ${t} =======================" &>> tests-cli.log
+  bash "${t}" "${cmd}" "${version}" &>> tests-cli.log
+  results+=(${?})
+done
+echo
+
+# ================================= TEARDOWN ================================= #
+
+echo "cleanup..."
+{ kill -9 "${pid_server}" && wait "${pid_server}"; } 2>/dev/null
+rm -rf "${dir}"
+echo
+
+# ================================= SUMMARY ================================== #
+
+printf "%s\t%s\n" "code" "test"
+printf "%s\t%s\n" "----" "----------------"
+for i in "${!tests[@]}"; do
+  printf "%s\t%s\n" "${results[$i]}" "${tests[$i]}"
+done
+echo
+if [[ " ${results[*]} " =~ [1-9] ]]; then
+  echo "failed tests! check tests-cli.log for debugging"; echo
+else
+  echo "all tests passed!"; echo
+fi

tests.sh

@@ -0,0 +1,130 @@
+#!/bin/bash
+# Script for running tests with different OpenRefine and Java versions based on Docker images.
+
+# Copyright (c) 2011 Paul Makepeace, Real Programmers. All rights reserved.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+# defaults:
+all=(3.5.0 3.4.1 3.4 3.3 3.2-java12 3.2-java11 3.2-java10 3.2-java9 3.2 3.1-java9 3.1 3.0-java9 3.0 2.8-java9 2.8 2.8-java7 2.7 2.7-java7 2.5-java7 2.5-java6 2.1-java6 2.0-java6)
+main=(3.5.0 3.4.1 3.4 3.3 3.2 3.1 3.0 2.8 2.7 2.5-java6 2.1-java6 2.0-java6)
+interactively=false
+port="3333"
+
+# help screen
+function usage () {
+    cat <<EOF
+Usage: ./tests.sh [-t TAG] [-i] [-p] [-a] [-h]
+
+Script for running tests with different OpenRefine and Java versions.
+It uses docker images from https://hub.docker.com/r/felixlohmeier/openrefine.
+
+Examples:
+./tests.sh -a              # run tests on all OpenRefine versions (from 2.0 up to 3.5.0)
+./tests.sh -t 3.5.0        # run tests on tag 3.5.0
+./tests.sh -t 3.5.0 -i     # run tests on tag 3.5.0 interactively (pause before and after tests)
+./tests.sh -t 3.5.0 -t 2.7 # run tests on tags 3.5.0 and 2.7
+
+Advanced:
+./tests.sh -j                # run tests on all OpenRefine versions and each with all supported Java versions (requires a lot of docker images to be downloaded!)
+./tests.sh -t 3.1 -i -p 3334 # run tests on tag 3.1 interactively on port 3334
+
+Running tests interactively (-i) allows you to examine OpenRefine GUI at http://localhost:3333.
+Execute the script concurrently in another terminal on another port (-p 3334) to compare changes in the OpenRefine GUI at http://localhost:3333 and http://localhost:3334.
+
+Available tags (java 8 if java not mentioned in tag):
+EOF
+    for t in ${all[*]} ; do
+        echo "$t"
+    done
+    exit 1
+}
+
+# check input
+NUMARGS=$#
+if [ "$NUMARGS" -eq 0 ]; then
+  usage
+fi
+
+# check system requirements
+DOCKER="$(command -v docker 2> /dev/null)"
+if [ -z "$DOCKER" ] ; then
+    echo 1>&2 "This action requires you to have 'docker' installed and present in your PATH. You can download it for free at http://www.docker.com/"
+    exit 1
+fi
+DOCKERINFO="$(docker info 2>/dev/null | grep 'Server Version')"
+if [ -z "$DOCKERINFO" ]
+then
+    echo "command 'docker info' failed, trying again with sudo..."
+    DOCKERINFO="$(sudo docker info 2>/dev/null | grep 'Server Version')"
+    echo "OK"
+    docker=(sudo docker)
+    if [ -z "$DOCKERINFO" ] ; then
+        echo 1>&2 "This action requires you to start the docker daemon. Try 'sudo systemctl start docker' or 'sudo start docker'. If the docker daemon is already running then maybe some security privileges are missing to run docker commands.'"
+        exit 1
+    fi
+else
+    docker=(docker)
+fi
+CURLINFO="$(command -v curl 2>/dev/null)"
+if [ -z "$CURLINFO" ] ; then
+    echo 1>&2 "This action requires you to have 'curl' installed and present in your PATH."
+    exit 1
+fi
+
+# get user input
+options="t:p:iajh"
+while getopts $options opt; do
+   case $opt in
+   t )  tags+=("${OPTARG}");;
+   p )  port="${OPTARG}";export OPENREFINE_PORT="$port";;
+   i )  interactively=true;;
+   a )  tags=("${main[*]}");;
+   j )  tags=("${all[*]}");;
+   h )  usage ;;
+   \? ) echo 1>&2 "Unknown option: -$OPTARG"; usage; exit 1;;
+   :  ) echo 1>&2 "Missing option argument for -$OPTARG"; usage; exit 1;;
+   *  ) echo 1>&2 "Unimplemented option: -$OPTARG"; usage; exit 1;;
+   esac
+done
+shift $((OPTIND - 1))
+
+# print config
+echo "Tags: ${tags[*]}"
+echo "Port: $port"
+echo ""
+
+# safe cleanup handler
+cleanup()
+{
+    echo "cleanup..."
+    ${docker[*]} stop "$t"
+}
+trap "cleanup;exit" SIGHUP SIGINT SIGQUIT SIGTERM
+
+# run setup.py tests for each docker tag
+for t in ${tags[*]} ; do
+    echo "=== Tests for $t ==="
+    echo ""
+    echo "Begin: $(date)"
+    ${docker[*]} run -d -p "$port":3333 --rm --name "$t" felixlohmeier/openrefine:"$t"
+    until curl --silent -N http://localhost:"$port" | cat | grep -q -o "Refine" ; do sleep 1; done
+    echo "Refine running at http://localhost:${port}"
+    if [ $interactively = true ]; then read -r -p "Press [Enter] key to start tests..."; fi
+    python2 setup.py test
+    if [ $interactively = true ]; then read -r -p "Press [Enter] key to stop OpenRefine..."; fi
+    ${docker[*]} stop "$t"
+    echo "End: $(date)"
+    echo ""
+done

tests/apply-utf8.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-addition",
+    "engineConfig": {
+      "mode": "row-based"
+    },
+    "newColumnName": "apply",
+    "columnInsertIndex": 2,
+    "baseColumnName": "b",
+    "expression": "grel:value.replace('2','⛲')",
+    "onError": "set-to-blank"
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	apply	c
+1	2	⛲	3
+0	0	0	0
+$	\	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/apply.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-addition",
+    "engineConfig": {
+      "mode": "row-based"
+    },
+    "newColumnName": "apply",
+    "columnInsertIndex": 2,
+    "baseColumnName": "b",
+    "expression": "grel:value.replace('2','TEST')",
+    "onError": "set-to-blank"
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	apply	c
+1	2	TEST	3
+0	0	0	0
+$	\	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-encoding.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}-utf8.csv"
+a,b,c
+1,2,3
+ä,é,ß
+$,\,'
+DATA
+iconv -f UTF-8 -t ISO-8859-1 "tmp/${t}/${t}-utf8.csv" > "tmp/${t}/${t}.csv"
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+ä	é	ß
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --encoding "ISO-8859-1"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-guessCellValueTypes.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+01,02,03
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+1	2	3
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --guessCellValueTypes "true"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-headerLines.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1	Column 2	Column 3
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --headerLines "0"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-ignoreLines.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --ignoreLines "1"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-limit.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --limit "2"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-processQuotes.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,"2,0",3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c	Column 4
+1	2	0	3
+0	0	0	
+$	\	'	
+DATA
+
+# ================================== ACTION ================================== #
+
+# OpenRefine 4.x fails without manually set headerLines
+${cmd} --create "tmp/${t}/${t}.csv" --processQuotes "false" --headerLines 1
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-projectTags.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version:0:1}" = "2" ]]; then
+  echo "projectTags were introduced in OpenRefine 3.0"
+  exit 200
+else
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+                tags: [u'beta', u'client1']
+DATA
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --projectTags "beta" --projectTags "client1"
+${cmd} --info "${t}" | grep ' tags: ' > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-separator.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a;b;c
+1;2;3
+0;0;0
+$;\;'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --separator ";"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-skipDataLines.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --skipDataLines "1"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-storeBlankCellsAsNulls.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,,0
+$,\,'
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "b",
+    "expression": "grel:isNull(value)",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	false	3
+0	false	0
+$	false	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --storeBlankCellsAsNulls "false"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-storeBlankRows.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+,,
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --storeBlankRows "false"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-utf8.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨	code	meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --projectName "${t} biểu tượng cảm xúc ⛲"
+${cmd} --export "${t} biểu tượng cảm xúc ⛲" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json-recordPath.sh

@@ -0,0 +1,55 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+{
+   "rows":[
+      {
+         "a":1,
+         "b":2,
+         "c":3
+      },
+      {
+         "a":0,
+         "b":0,
+         "c":0
+      },
+      {
+         "a":"$",
+         "b":"\\",
+         "c":"\""
+      }
+   ]
+}
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - a	_ - b	_ - c
+1	2	3
+0	0	0
+$	\	""""
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json" --recordPath "_" --recordPath "rows" --recordPath "_"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json-storeEmptyStrings.sh

@@ -0,0 +1,52 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+[
+  {
+    "a": 1,
+    "b": 2,
+    "c": 3
+  },
+  {
+    "a": "",
+    "b": "",
+    "c": ""
+  },
+  {
+    "a": "$",
+    "b": "\\",
+    "c": "\""
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - a	_ - b	_ - c
+1	2	3
+$	\	""""
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json" --storeEmptyStrings "false"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json-trimStrings.sh

@@ -0,0 +1,62 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+[
+  {
+    "a": 1,
+    "b": 2,
+    "c": 3
+  },
+  {
+    "a": "0",
+    "b": " 0",
+    "c": "0 "
+  },
+  {
+    "a": "$",
+    "b": "\\",
+    "c": "\""
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version:0:1}" = "2" || "${version}" = "3.0" || "${version}" = "3.1" || "${version}" = "3.2" || "${version}" = "3.3" ]]; then
+  echo "trimStrings option does not work in OpenRefine <=3.3"
+  echo "https://github.com/OpenRefine/OpenRefine/issues/2409"
+  exit 200
+else
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - a	_ - b	_ - c
+1	2	3
+0	0	0
+$	\	""""
+DATA
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json" --trimStrings "true"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json-utf8.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+[
+  {
+    "⌨": "⛲",
+    "code": "1F347",
+    "meaning": "FOUNTAIN"
+  },
+  {
+    "⌨": "⛳",
+    "code": "1F349",
+    "meaning": "FLAG IN HOLE"
+  },
+  {
+    "⌨": "⛵",
+    "code": "1F352",
+    "meaning": "SAILBOAT"
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - ⌨	_ - code	_ - meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+[
+  {
+    "a": 1,
+    "b": 2,
+    "c": 3
+  },
+  {
+    "a": 0,
+    "b": 0,
+    "c": 0
+  },
+  {
+    "a": "$",
+    "b": "\\",
+    "c": "\""
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - a	_ - b	_ - c
+1	2	3
+0	0	0
+$	\	""""
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-ods-sheets-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.ods" "tmp/${t}/${t}.ods"
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨	code	meaning	Column	Column 5	Column 6	Column 7	Column 8
+⛲	1F347	FOUNTAIN					
+⛳	1F349	FLAG IN HOLE					
+⛵	1F352	SAILBOAT					
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.ods" --sheets 1
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-ods.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.ods" "tmp/${t}/${t}.ods"
+#a	b	c
+#1	2	3
+#0	0	0
+#$	\	'
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c	Column	Column 5	Column 6	Column 7	Column 8
+1.0	2.0	3.0					
+0.0	0.0	0.0					
+$	\	'					
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.ods"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-tsv-utf8.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+⌨	code	meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.csv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.csv"

tests/create-tsv.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.csv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.csv"

tests/create-txt-fixed-width-headerLines.sh

@@ -0,0 +1,80 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+1     2     3
+mon   tue   wed
+$2    $300  $1
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "1",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "2",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "3",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+1	2	3
+mon	tue	wed
+$2	$300	$1
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --columnWidths "6" --columnWidths "6" --columnWidths "6" --headerLines "1"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-txt-fixed-width-utf8.sh

@@ -0,0 +1,81 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+⛲     1F347 FOUNTAIN
+⛳     1F349 FLAG IN HOLE
+⛵     1F352 SAILBOAT
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 1",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 2",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 3",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1	Column 2	Column 3
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --columnWidths "6" --columnWidths "6" --columnWidths "60"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-txt-fixed-width.sh

@@ -0,0 +1,81 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+1     2     3
+mon   tue   wed
+$2    $300  $1
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 1",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 2",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 3",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1	Column 2	Column 3
+1	2	3
+mon	tue	wed
+$2	$300	$1
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --columnWidths "6" --columnWidths "6" --columnWidths "6"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-txt-linesPerRow.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+mon tue wed
+$2 $300 $1
+thu fri sat
+$70 $20 $50
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1	Column 2
+mon tue wed	$2 $300 $1
+thu fri sat	$70 $20 $50
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --linesPerRow "2"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-txt.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+1 2 3
+mon tue wed
+$2 $300 $1
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1
+1 2 3
+mon tue wed
+$2 $300 $1
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xls-sheets-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.xls" "tmp/${t}/${t}.xls"
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨	code	meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xls" --sheets 1
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xls.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.xls" "tmp/${t}/${t}.xls"
+#a	b	c
+#1	2	3
+#0	0	0
+#$	\	'
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1.0	2.0	3.0
+0.0	0.0	0.0
+$	\	'
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xls"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xlsx-sheets-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.xlsx" "tmp/${t}/${t}.xlsx"
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨	code	meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xlsx" --sheets 1
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xlsx.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.xlsx" "tmp/${t}/${t}.xlsx"
+#a	b	c
+#1	2	3
+#0	0	0
+#$	\	'
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1.0	2.0	3.0
+0.0	0.0	0.0
+$	\	'
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xlsx"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xml-recordPath.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.xml"
+<?xml version="1.0" encoding="UTF-8"?>
+<root>
+  <record>
+    <a>1</a>
+    <b>2</b>
+    <c>3</c>
+  </record>
+  <record>
+    <a>0</a>
+    <b>0</b>
+    <c>0</c>
+  </record>
+  <record>
+    <a>$</a>
+    <b>\</b>
+    <c>'</c>
+  </record>
+</root>
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-reorder",
+    "columnNames": [
+      "record - a",
+      "record - b",
+      "record - c"
+    ],
+    "description": "Reorder columns"
+  },
+  {
+    "op": "core/row-removal",
+    "engineConfig": {
+      "facets": [
+        {
+          "type": "list",
+          "name": "Blank Rows",
+          "expression": "(filter(row.columnNames,cn,isNonBlank(cells[cn].value)).length()==0).toString()",
+          "columnName": "",
+          "invert": false,
+          "omitBlank": false,
+          "omitError": false,
+          "selection": [
+            {
+              "v": {
+                "v": "true",
+                "l": "true"
+              }
+            }
+          ],
+          "selectBlank": false,
+          "selectError": false
+        }
+      ],
+      "mode": "record-based"
+    }
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+record - a	record - b	record - c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xml" --recordPath "root" --recordPath "record"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xml-utf8.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.xml"
+<?xml version="1.0" encoding="UTF-8"?>
+<root>
+  <record>
+    <icon>⛲</icon>
+    <code>1F347</code>
+    <meaning>FOUNTAIN</meaning>
+  </record>
+  <record>
+    <icon>⛳</icon>
+    <code>1F349</code>
+    <meaning>FLAG IN HOLE</meaning>
+  </record>
+  <record>
+    <icon>⛵</icon>
+    <code>1F352</code>
+    <meaning>SAILBOAT</meaning>
+  </record>
+</root>
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-reorder",
+    "columnNames": [
+      "root - record - icon",
+      "root - record - code",
+      "root - record - meaning"
+    ],
+    "description": "Reorder columns"
+  },
+  {
+    "op": "core/row-removal",
+    "engineConfig": {
+      "facets": [
+        {
+          "type": "list",
+          "name": "Blank Rows",
+          "expression": "(filter(row.columnNames,cn,isNonBlank(cells[cn].value)).length()==0).toString()",
+          "columnName": "",
+          "invert": false,
+          "omitBlank": false,
+          "omitError": false,
+          "selection": [
+            {
+              "v": {
+                "v": "true",
+                "l": "true"
+              }
+            }
+          ],
+          "selectBlank": false,
+          "selectError": false
+        }
+      ],
+      "mode": "record-based"
+    }
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+root - record - icon	root - record - code	root - record - meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xml"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xml.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.xml"
+<?xml version="1.0" encoding="UTF-8"?>
+<root>
+  <record>
+    <a>1</a>
+    <b>2</b>
+    <c>3</c>
+  </record>
+  <record>
+    <a>0</a>
+    <b>0</b>
+    <c>0</c>
+  </record>
+  <record>
+    <a>$</a>
+    <b>\</b>
+    <c>'</c>
+  </record>
+</root>
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-reorder",
+    "columnNames": [
+      "root - record - a",
+      "root - record - b",
+      "root - record - c"
+    ],
+    "description": "Reorder columns"
+  },
+  {
+    "op": "core/row-removal",
+    "engineConfig": {
+      "facets": [
+        {
+          "type": "list",
+          "name": "Blank Rows",
+          "expression": "(filter(row.columnNames,cn,isNonBlank(cells[cn].value)).length()==0).toString()",
+          "columnName": "",
+          "invert": false,
+          "omitBlank": false,
+          "omitError": false,
+          "selection": [
+            {
+              "v": {
+                "v": "true",
+                "l": "true"
+              }
+            }
+          ],
+          "selectBlank": false,
+          "selectError": false
+        }
+      ],
+      "mode": "record-based"
+    }
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+root - record - a	root - record - b	root - record - c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xml"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-zip-includeFileSources.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}-1.csv"
+a,b,c
+1,2,3
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}-2.csv"
+a,b,c
+4,5,6
+DATA
+
+zip "tmp/${t}/${t}.zip" "tmp/${t}/${t}-1.csv" "tmp/${t}/${t}-2.csv"
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+File	a	b	c
+tmp/${t}/${t}-1.csv	1	2	3
+tmp/${t}/${t}-2.csv	4	5	6
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.zip" --includeFileSources "true"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-zip.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}-1.csv"
+a,b,c
+1,2,3
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}-2.csv"
+a,b,c
+4,5,6
+DATA
+
+zip "tmp/${t}/${t}.zip" "tmp/${t}/${t}-1.csv" "tmp/${t}/${t}-2.csv"
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+4	5	6
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.zip"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/data/cli/duplicates.csv

@@ -0,0 +1,11 @@
+email,name,state,gender,purchase,count,date
+danny.baron@example1.com,Danny Baron,CA,M,TV (UTF-8: 📺),1,"Wed, 4 Jul 2001"
+melanie.white@example2.edu,Melanie White,NC,F,<iPhone>,1,2001-07-04T12:08:56
+danny.baron@example1.com,   D.	("Tab") Baron,CA,M,Winter jacket,1,2001-07-04
+ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight,1,2001/07/04
+arthur.duff@example4.com,Arthur Duff,OR,M,Dining table,1,2001-07
+danny.baron@example1.com,Daniel Baron,,,Bike,1,2001
+jean.griffith@example5.org,Jean Griffith,WA,F,Power drill,1,2000
+melanie.white@example2.edu,Melanie White,NC,F,'iPad',1,1999
+ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier,1,1998
+arthur.duff@example4.com,Arthur Duff,OR,M,Night table,1,1997

tests/data/cli/duplicates.json

@@ -0,0 +1,92 @@
+[
+  {
+    "email": "danny.baron@example1.com",
+    "name": "Danny Baron",
+    "state": "CA",
+    "gender": "M",
+    "purchase": "TV (UTF-8: 📺)",
+    "count": 1,
+    "date": "Wed, 4 Jul 2001"
+  },
+  {
+    "email": "melanie.white@example2.edu",
+    "name": "Melanie White",
+    "state": "NC",
+    "gender": "F",
+    "purchase": "<iPhone>",
+    "count": 1,
+    "date": "2001-07-04T12:08:56"
+  },
+  {
+    "email": "danny.baron@example1.com",
+    "name": "   D.\t(\"Tab\") Baron",
+    "state": "CA",
+    "gender": "M",
+    "purchase": "Winter jacket",
+    "count": 1,
+    "date": "2001-07-04"
+  },
+  {
+    "email": "ben.tyler@example3.org",
+    "name": "Ben Tyler",
+    "state": "NV",
+    "gender": "M",
+    "purchase": "Flashlight",
+    "count": 1,
+    "date": "2001/07/04"
+  },
+  {
+    "email": "arthur.duff@example4.com",
+    "name": "Arthur Duff",
+    "state": "OR",
+    "gender": "M",
+    "purchase": "Dining table",
+    "count": 1,
+    "date": "2001-07"
+  },
+  {
+    "email": "danny.baron@example1.com",
+    "name": "Daniel Baron",
+    "state": "",
+    "gender": "",
+    "purchase": "Bike",
+    "count": 1,
+    "date": 2001
+  },
+  {
+    "email": "jean.griffith@example5.org",
+    "name": "Jean Griffith",
+    "state": "WA",
+    "gender": "F",
+    "purchase": "Power drill",
+    "count": 1,
+    "date": 2000
+  },
+  {
+    "email": "melanie.white@example2.edu",
+    "name": "Melanie White",
+    "state": "NC",
+    "gender": "F",
+    "purchase": "'iPad'",
+    "count": 1,
+    "date": 1999
+  },
+  {
+    "email": "ben.morisson@example6.org",
+    "name": "Ben Morisson",
+    "state": "FL",
+    "gender": "M",
+    "purchase": "Amplifier",
+    "count": 1,
+    "date": 1998
+  },
+  {
+    "email": "arthur.duff@example4.com",
+    "name": "Arthur Duff",
+    "state": "OR",
+    "gender": "M",
+    "purchase": "Night table",
+    "count": 1,
+    "date": 1997
+  }
+]

tests/data/cli/duplicates.tsv

@@ -0,0 +1,11 @@
+email	name	state	gender	purchase	count	date
+danny.baron@example1.com	Danny Baron	CA	M	TV (UTF-8: 📺)	1	Wed, 4 Jul 2001
+melanie.white@example2.edu	Melanie White	NC	F	<iPhone>	1	2001-07-04T12:08:56
+danny.baron@example1.com	"D.	(""Tab"") Baron"	CA	M	Winter jacket	1	2001-07-04
+ben.tyler@example3.org	Ben Tyler	NV	M	Flashlight	1	2001/07/04
+arthur.duff@example4.com	Arthur Duff	OR	M	Dining table	1	2001-07
+danny.baron@example1.com	Daniel Baron			Bike	1	2001
+jean.griffith@example5.org	Jean Griffith	WA	F	Power drill	1	2000
+melanie.white@example2.edu	Melanie White	NC	F	'iPad'	1	1999
+ben.morisson@example6.org	Ben Morisson	FL	M	Amplifier	1	1998
+arthur.duff@example4.com	Arthur Duff	OR	M	Night table	1	1997

tests/data/cli/duplicates.txt

@@ -0,0 +1,11 @@
+email                      name                 state gender purchase       count date              
+danny.baron@example1.com   Danny Baron          CA    M      TV (UTF-8: 📺) 1     Wed, 4 Jul 2001   
+melanie.white@example2.edu Melanie White        NC    F      <iPhone>       1     2001-07-04T12:08:5
+danny.baron@example1.com      D.	("Tab") Baron  CA    M      Winter jacket  1     2001-07-04        
+ben.tyler@example3.org     Ben Tyler            NV    M      Flashlight     1     2001/07/04        
+arthur.duff@example4.com   Arthur Duff          OR    M      Dining table   1     2001-07           
+danny.baron@example1.com   Daniel Baron                      Bike           1     2001              
+jean.griffith@example5.org Jean Griffith        WA    F      Power drill    1     2000              
+melanie.white@example2.edu Melanie White        NC    F      'iPad'         1     1999              
+ben.morisson@example6.org  Ben Morisson         FL    M      Amplifier      1     1998              
+arthur.duff@example4.com   Arthur Duff          OR    M      Night table    1     1997              

tests/data/cli/duplicates.xml

@@ -0,0 +1,93 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<root>
+  <record>
+    <email>danny.baron@example1.com</email>
+    <name>Danny Baron</name>
+    <state>CA</state>
+    <gender>M</gender>
+    <purchase>TV (UTF-8: 📺)</purchase>
+    <count>1</count>
+    <date>Wed, 4 Jul 2001</date>
+  </record>
+  <record>
+    <email>melanie.white@example2.edu</email>
+    <name>Melanie White</name>
+    <state>NC</state>
+    <gender>F</gender>
+    <purchase>&lt;iPhone&gt;</purchase>
+    <count>1</count>
+    <date>2001-07-04T12:08:56</date>
+  </record>
+  <record>
+    <email>danny.baron@example1.com</email>
+    <name>   D.	(&quot;Tab&quot;) Baron</name>
+    <state>CA</state>
+    <gender>M</gender>
+    <purchase>Winter jacket</purchase>
+    <count>1</count>
+    <date>2001-07-04</date>
+  </record>
+  <record>
+    <email>ben.tyler@example3.org</email>
+    <name>Ben Tyler</name>
+    <state>NV</state>
+    <gender>M</gender>
+    <purchase>Flashlight</purchase>
+    <count>1</count>
+    <date>2001/07/04</date>
+  </record>
+  <record>
+    <email>arthur.duff@example4.com</email>
+    <name>Arthur Duff</name>
+    <state>OR</state>
+    <gender>M</gender>
+    <purchase>Dining table</purchase>
+    <count>1</count>
+    <date>2001-07</date>
+  </record>
+  <record>
+    <email>danny.baron@example1.com</email>
+    <name>Daniel Baron</name>
+    <state></state>
+    <gender></gender>
+    <purchase>Bike</purchase>
+    <count>1</count>
+    <date>2001</date>
+  </record>
+  <record>
+    <email>jean.griffith@example5.org</email>
+    <name>Jean Griffith</name>
+    <state>WA</state>
+    <gender>F</gender>
+    <purchase>Power drill</purchase>
+    <count>1</count>
+    <date>2000</date>
+  </record>
+  <record>
+    <email>melanie.white@example2.edu</email>
+    <name>Melanie White</name>
+    <state>NC</state>
+    <gender>F</gender>
+    <purchase>&apos;iPad&apos;</purchase>
+    <count>1</count>
+    <date>1999</date>
+  </record>
+  <record>
+    <email>ben.morisson@example6.org</email>
+    <name>Ben Morisson</name>
+    <state>FL</state>
+    <gender>M</gender>
+    <purchase>Amplifier</purchase>
+    <count>1</count>
+    <date>1998</date>
+  </record>
+  <record>
+    <email>arthur.duff@example4.com</email>
+    <name>Arthur Duff</name>
+    <state>OR</state>
+    <gender>M</gender>
+    <purchase>Night table</purchase>
+    <count>1</count>
+    <date>1997</date>
+  </record>
+</root>

tests/data/cli/duplicates1.xml

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<record>
+  <email>danny.baron@example1.com</email>
+  <name>Danny Baron</name>
+  <state>CA</state>
+  <gender>M</gender>
+  <purchase>TV (UTF-8: 📺)</purchase>
+  <count>1</count>
+  <date>Wed, 4 Jul 2001</date>
+</record>

tests/data/cli/evil-fruits.tsv

@@ -0,0 +1,6 @@
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE

tests/data/duplicates-deletion.json

@@ -0,0 +1,69 @@
+[
+  {
+    "op": "core/row-reorder",
+    "description": "Reorder rows",
+    "mode": "record-based",
+    "sorting": {
+      "criteria": [
+        {
+          "errorPosition": 1,
+          "caseSensitive": false,
+          "valueType": "string",
+          "column": "email",
+          "blankPosition": 2,
+          "reverse": false
+        }
+      ]
+    }
+  },
+  {
+    "op": "core/column-addition",
+    "description": "Create column count at index 1 based on column email using expression grel:facetCount(value, \"value\", \"email\")",
+    "engineConfig": {
+      "mode": "row-based",
+      "facets": []
+    },
+    "newColumnName": "count",
+    "columnInsertIndex": 1,
+    "baseColumnName": "email",
+    "expression": "grel:facetCount(value, \"value\", \"email\")",
+    "onError": "set-to-blank"
+  },
+  {
+    "op": "core/blank-down",
+    "description": "Blank down cells in column email",
+    "engineConfig": {
+      "mode": "row-based",
+      "facets": []
+    },
+    "columnName": "email"
+  },
+  {
+    "op": "core/row-removal",
+    "description": "Remove rows",
+    "engineConfig": {
+      "mode": "row-based",
+      "facets": [
+        {
+          "omitError": false,
+          "expression": "isBlank(value)",
+          "selectBlank": false,
+          "selection": [
+            {
+              "v": {
+                "v": true,
+                "l": "true"
+              }
+            }
+          ],
+          "selectError": false,
+          "invert": false,
+          "name": "email",
+          "omitBlank": false,
+          "type": "list",
+          "columnName": "email"
+        }
+      ]
+    }
+  }
+]

tests/delete-utf8.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --projectName "${t} biểu tượng cảm xúc ⛲"
+${cmd} --list | grep "${t}" || exit 1
+${cmd} --delete "${t} biểu tượng cảm xúc ⛲"
+${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/delete.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --list | grep "${t}" || exit 1
+${cmd} --delete "${t}"
+${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/download.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# ================================== ACTION ================================== #
+
+${cmd} --download "https://git.io/fj5ju" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "data/duplicates-deletion.json" "tmp/${t}/${t}.output"

tests/export-csv-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --export "${t}" --output "tmp/${t}/${t} biểu tượng cảm xúc 🍉.csv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t} biểu tượng cảm xúc 🍉.csv"

tests/export-csv.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.csv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.csv"

tests/export-html-utf8.sh

@@ -0,0 +1,72 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  majorversion="${2%%.*}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+if [[ "$majorversion" = 2 ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+<html>
+<head>
+<title>export-html-utf8</title>
+<meta charset="utf-8" />
+</head>
+<body>
+<table>
+<tr><th>⌨</th><th>code</th><th>meaning</th></tr>
+<tr><td>&#9970;</td><td>1F347</td><td>FOUNTAIN</td></tr>
+<tr><td>&#9971;</td><td>1F349</td><td>FLAG IN HOLE</td></tr>
+<tr><td>&#9973;</td><td>1F352</td><td>SAILBOAT</td></tr>
+</table>
+</body>
+</html>
+DATA
+else
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+<html>
+<head>
+<title>export-html-utf8</title>
+<meta charset="utf-8" />
+</head>
+<body>
+<table>
+<tr><th>⌨</th><th>code</th><th>meaning</th></tr>
+<tr><td>⛲</td><td>1F347</td><td>FOUNTAIN</td></tr>
+<tr><td>⛳</td><td>1F349</td><td>FLAG IN HOLE</td></tr>
+<tr><td>⛵</td><td>1F352</td><td>SAILBOAT</td></tr>
+</table>
+</body>
+</html>
+DATA
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.html"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.html"

tests/export-html.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+<html>
+<head>
+<title>export-html</title>
+<meta charset="utf-8" />
+</head>
+<body>
+<table>
+<tr><th>a</th><th>b</th><th>c</th></tr>
+<tr><td>1</td><td>2</td><td>3</td></tr>
+<tr><td>0</td><td>0</td><td>0</td></tr>
+<tr><td>$</td><td>\</td><td>&apos;</td></tr>
+</table>
+</body>
+</html>
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.html"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.html"

tests/export-ods-utf8.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,"FLAG IN HOLE"
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.ods"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.ods" "${t}.csv" &&
+  mv "${t}.csv.1" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-ods.sh

@@ -0,0 +1,47 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ -z "$(command -v ssconvert 2> /dev/null)" ]] ; then
+  echo 1>&2 "ERROR: This test requires ssconvert (gnumeric)"
+  exit 127
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.ods"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.ods" "${t}.csv" &&
+  mv "${t}.csv.1" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-tsv-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.tsv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.tsv"

tests/export-tsv.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.tsv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.tsv"

tests/export-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-xls-utf8.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.xls"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.xls" "${t}.csv" &&
+  mv "${t}.csv" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-xls.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.xls"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.xls" "${t}.csv" &&
+  mv "${t}.csv" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-xlsx-utf8.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.xlsx"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.xlsx" "${t}.csv" &&
+  mv "${t}.csv" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
+
+

tests/export-xlsx.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.xlsx"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.xlsx" "${t}.csv" &&
+  mv "${t}.csv" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/format-create-separator.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+a;b;c
+1;2;3
+0;0;0
+$;\;'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --format "csv"  --separator ";"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/format-create.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --format "line-based"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/format-export-output.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --format "csv" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/format-export.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --format "csv" > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/help.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Script to provide a command line interface to an OpenRefine server.
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --help | sed '3q;d' > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/info-utf8.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+          column 001: 🔣
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --info "${t}" | grep 'column 001' > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/info.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+          column 002: b
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --info "${t}" | grep 'column 002' > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/list-utf8.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+ ${t} biểu tượng cảm xúc ⛲
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --projectName "${t} biểu tượng cảm xúc ⛲"
+${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/list.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+ ${t}
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/template-facets.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+email,name,state,gender,purchase
+danny.baron@example1.com,Danny Baron,CA,M,TV
+melanie.white@example2.edu,Melanie White,NC,F,iPhone
+danny.baron@example1.com,D. Baron,CA,M,Winter jacket
+ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
+arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
+danny.baron@example1.com,Daniel Baron,CA,M,Bike
+jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
+melanie.white@example2.edu,Melanie White,NC,F,iPad
+ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
+arthur.duff@example4.com,Arthur Duff,OR,M,Night table
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "events" : [
+    { "name" : "Melanie White", "purchase" : "iPhone" },
+    { "name" : "Jean Griffith", "purchase" : "Power drill" },
+    { "name" : "Melanie White", "purchase" : "iPad" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "events" : [
+' \
+--template '    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--facets '{"type":"list","name":"gender","columnName":"gender","expression":"value","omitBlank":false,"omitError":false,"selection":[{"v":{"v":"F","l":"F"}}],"selectBlank":false,"selectError":false,"invert":false}' \
+--output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/template-filterQuery-utf8.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "emojis" : [
+    { "symbol" : "🍇", "meaning" : "GRAPES" },
+    { "symbol" : "🍉", "meaning" : "WATERMELON" },
+    { "symbol" : "🍍", "meaning" : "PINEAPPLE" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "emojis" : [
+' \
+--template '    { "symbol" : {{jsonize(with(row.columnNames[0],cn,cells[cn].value))}}, "meaning" : {{jsonize(cells["meaning"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--filterQuery '^1F34' \
+--filterColumn 'code' \
+--output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/template-filterQuery.sh

@@ -0,0 +1,59 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+email,name,state,gender,purchase
+danny.baron@example1.com,Danny Baron,CA,M,TV
+melanie.white@example2.edu,Melanie White,NC,F,iPhone
+danny.baron@example1.com,D. Baron,CA,M,Winter jacket
+ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
+arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
+danny.baron@example1.com,Daniel Baron,CA,M,Bike
+jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
+melanie.white@example2.edu,Melanie White,NC,F,iPad
+ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
+arthur.duff@example4.com,Arthur Duff,OR,M,Night table
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "events" : [
+    { "name" : "Melanie White", "purchase" : "iPhone" },
+    { "name" : "Jean Griffith", "purchase" : "Power drill" },
+    { "name" : "Melanie White", "purchase" : "iPad" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "events" : [
+' \
+--template '    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--filterQuery '^F$' \
+--filterColumn 'gender' \
+--output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"