Merge pull request #21 from mohammed78620/master

moved csrf token check into urlopen_json

.gitignore

@@ -4,4 +4,6 @@ dist
 .*
 openrefine_client.egg-info
 refine.spec
-README.html
+openrefine-*
+openrefine-client_*
+tests-cli.log

MANIFEST.in

@@ -2,4 +2,3 @@ include README.md
 include COPYING.txt
 recursive-include tests/data *.csv
 recursive-include tests      *.py
-prune docker

README.md

@@ -1,71 +1,758 @@
-# OpenRefine Python Client with extended command line interface
+# OpenRefine Python Client with extended command line interface (⌨️ for 💎)
 
-The [OpenRefine Python Client Library from PaulMakepeace](https://github.com/PaulMakepeace/refine-client-py) provides an interface to communicating with an [OpenRefine](http://openrefine.org) server. This fork extends the command line interface (CLI) and supports communication between docker containers.
+[![Codacy Badge](https://app.codacy.com/project/badge/Grade/43ad9bfd707b4627bd45e5c5f912a8e0)](https://www.codacy.com/gh/opencultureconsulting/openrefine-client/dashboard) [![Docker](https://img.shields.io/microbadger/image-size/felixlohmeier/openrefine-client?label=docker)](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [![PyPI](https://img.shields.io/pypi/v/openrefine-client)](https://pypi.org/project/openrefine-client/) [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master)
+
+The [OpenRefine Python Client from PaulMakepeace](https://github.com/PaulMakepeace/refine-client-py) provides a library for communicating with an [OpenRefine](http://openrefine.org) server.
+This fork extends the command line interface (CLI) and is distributed as a convenient one-file-executable (Windows, Linux, macOS).
+It is also available via Docker Hub, PyPI and Binder.
+
+works with OpenRefine 2.7, 2.8, 3.0, 3.1, 3.2, 3.3, 3.4, 3.4.1, 3.5.0
 
 ## Download
 
 One-file-executables:
 
-* Linux: [openrefine-client_0-3-4_linux-64bit](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.4/openrefine-client_0-3-4_linux-64bit) (4,7 MB)
-* Windows: [openrefine-client_0-3-4_windows.exe](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.4/openrefine-client_0-3-4_windows.exe) (4,9 MB)
-* Mac: [openrefine-client_0-3-4_mac](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.4/openrefine-client_0-3-4_mac) (4,4 MB)
+- Windows: [openrefine-client_0-3-10_windows.exe](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_windows.exe) (~5 MB)
+- macOS: [openrefine-client_0-3-10_macos](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_macos) (~5 MB)
+- Linux: [openrefine-client_0-3-10_linux](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_linux) (~5 MB)
 
-For native Python installation on Windows, Mac or Linux see [Installation](#installation) below.
+For [Docker](#docker) containers, native [Python](#python) installation and free [Binder](#binder) on-demand server see the corresponding chapters below.
 
 ## Peek
 
-A short video loop that demonstrates the basic features (list, create, apply, export)
+A short video loop that demonstrates the basic features (list, create, apply, export):
 
 ![video loop that demonstrates basic features](openrefine-client-peek.gif)
 
 ## Usage
 
-Command line interface:
+Ensure you have [OpenRefine](http://openrefine.org) running (i.e. available at http://localhost:3333 or [another URL](#change-url)).
+
+To use the client:
+
+1. Open a terminal pointing to the folder where you have [downloaded](#download) the one-file-executable (e.g. Downloads in your home directory).
+
+   - Windows: Open PowerShell and enter following command
+
+     ```sh
+     cd ~\Downloads
+     ```
+
+   - macOS: Open Terminal (Finder > Applications > Utilities > Terminal) and enter following command
+
+     ```sh
+     cd ~/Downloads
+     ```
+
+   - Linux: Open terminal app (Terminal, Konsole, xterm, ...) and enter following command
+
+     ```sh
+     cd ~/Downloads
+     ```
+
+2. Make the file executable.
+
+   - Windows: not necessary
+
+   - macOS:
+
+     ```sh
+     chmod +x openrefine-client_0-3-10_macos
+     ```
+
+   - Linux:
+
+     ```sh
+     chmod +x openrefine-client_0-3-10_linux
+     ```
+
+3. Execute the file.
+
+   - Windows:
+
+     ```sh
+     .\openrefine-client_0-3-10_windows.exe
+     ```
+
+   - macOS:
+
+     ```sh
+     ./openrefine-client_0-3-10_macos
+     ```
+
+   - Linux:
+
+     ```sh
+     ./openrefine-client_0-3-10_linux
+     ```
+
+Using tab completion and command history is highly recommended:
+
+- autocomplete filenames: enter a few characters and press `↹`
+- recall previous command: press `↑`
+
+### Basic commands
+
+Execute the client by entering its filename followed by the desired command.
+
+The following example will download two small files ([duplicates.csv](https://raw.githubusercontent.com/opencultureconsulting/openrefine-client/master/tests/data/duplicates.csv) and [duplicates-deletion.json](https://raw.githubusercontent.com/opencultureconsulting/openrefine-client/master/tests/data/duplicates-deletion.json)) into the current directory and will create a new OpenRefine project from file duplicates.csv.
+
+Download example data (`--download`) and create project from file (`--create`):
+
+- Windows:
+
+  ```sh
+  .\openrefine-client_0-3-10_windows.exe --download "https://git.io/fj5hF" --output=duplicates.csv
+  .\openrefine-client_0-3-10_windows.exe --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+  .\openrefine-client_0-3-10_windows.exe --create duplicates.csv
+  ```
+
+- macOS:
+
+  ```sh
+  ./openrefine-client_0-3-10_macos --download "https://git.io/fj5hF" --output=duplicates.csv
+  ./openrefine-client_0-3-10_macos --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+  ./openrefine-client_0-3-10_macos --create duplicates.csv
+  ```
+
+- Linux:
+
+  ```sh
+  ./openrefine-client_0-3-10_linux --download "https://git.io/fj5hF" --output=duplicates.csv
+  ./openrefine-client_0-3-10_linux --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+  ./openrefine-client_0-3-10_linux --create duplicates.csv
+  ```
+
+Other commands:
 
 - list all projects: `--list`
-- create project from file: `--create [FILE]`
-- apply [rules from json file](http://kb.refinepro.com/2012/06/google-refine-json-and-my-notepad-or.html): `--apply [FILE.json] [PROJECTID/PROJECTNAME]`
-- export project to file: `--export [PROJECTID/PROJECTNAME] --output=FILE.tsv`
-- templating export: `--export "My Address Book" --template='{ "friend" : {{jsonize(cells["friend"].value)}}, "address" : {{jsonize(cells["address"].value)}} }' --prefix='{ "address" : [' --rowSeparator ',' --suffix '] }' --filterQuery="^mary$"`
-- show project metadata: `--info [PROJECTID/PROJECTNAME]`
-- delete project: `--delete [PROJECTID/PROJECTNAME]`
-- check `--help` for further options...
+- show project metadata: `--info "duplicates"`
+- export project to terminal: `--export "duplicates"`
+- apply [rules from json file](http://kb.refinepro.com/2012/06/google-refine-json-and-my-notepad-or.html): `--apply duplicates-deletion.json "duplicates"`
+- export project to file: `--export --output=deduped.xls "duplicates"`
+- delete project: `--delete "duplicates"`
 
-If you are familiar with python you may try all functions interactively (`python -i refine.py`) or use this library in your own python scripts. Some Examples:
+### Getting help
 
-* show version of OpenRefine server: `refine.RefineServer().get_version()`
-* show total rows of project 2151545447855: `refine.RefineProject(refine.RefineServer(),'2151545447855').do_json('get-rows')['total']`
-* compute clusters of project 2151545447855 and column key: `refine.RefineProject(refine.RefineServer(),'2151545447855').compute_clusters('key')`
+Check `--help` for further options.
 
-## Configuration
+Please file an [issue](https://github.com/opencultureconsulting/openrefine-client/issues) if you miss some features in the command line interface or if you have tracked a bug.
+And you are welcome to ask any questions!
 
-By default the OpenRefine server URL is [http://127.0.0.1:3333](http://127.0.0.1:3333)
+### Change URL
 
-The environment variables `OPENREFINE_HOST` and `OPENREFINE_PORT` enable overriding the host & port as well as the command line options `-H` and `-P`.
+By default the client connects to the usual URL of OpenRefine [http://localhost:3333](http://localhost:3333).
+If your OpenRefine server is running somewhere else then you may set hostname and port with additional command line options (e.g. http://example.com):
 
-## Installation
+- set host: `-H example.com`
+- set port: `-P 80`
 
-```
-pip install openrefine-client
+### Templating
+
+The OpenRefine [Templating](https://github.com/OpenRefine/OpenRefine/wiki/Export-As-YAML) supports exporting data in any text format (i.e. to construct JSON or XML).
+The graphical user interface offers four input fields:
+
+1. prefix
+2. row template
+   - supports [GREL](https://github.com/OpenRefine/OpenRefine/wiki/General-Refine-Expression-Language) inside two curly brackets, e.g. `{{jsonize(cells["name"].value)}}`
+3. row separator
+4. suffix
+
+This templating functionality is available via the openrefine-client command line interface.
+It even provides an additional feature for splitting results into multiple files.
+
+To try out the functionality create another project from the example file above.
+
+```sh
+--create duplicates.csv --projectName=advanced
 ```
 
-(requires Python 2.x, depends on urllib2_file>=0.2.1)
+The following example code will export...
 
-## Tests
+- the columns "name" and "purchase" in JSON format
+- from the project "advanced"
+- for rows matching the regex text filter `^F$` in column "gender"
 
-Ensure you have a Refine server running somewhere and, if necessary, set the environment vars as above.
+macOS/Linux Terminal (multi-line input with `\` ):
 
-Run tests, build, and install:
-
-```
-python setup.py test # to do a subset, e.g., --test-suite tests.test_facet
-
-python setup.py build
-
-python setup.py install
+```sh
+"advanced" \
+--prefix='{ "events" : [
+' \
+--template='    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
+--rowSeparator=',
+' \
+--suffix='
+] }' \
+--filterQuery='^F$' \
+--filterColumn='gender'
 ```
 
-There is a Makefile that will do this too, and more.
+Windows PowerShell (multi-line input with `` ` ``; quotes needs to be doubled):
+
+```sh
+"advanced" `
+--prefix='{ ""events"" : [
+' `
+--template='    { ""name"" : {{jsonize(cells[""name""].value)}}, ""purchase"" : {{jsonize(cells[""purchase""].value)}} }' `
+--rowSeparator=',
+' `
+--suffix='
+] }' `
+--filterQuery='^F$' `
+--filterColumn='gender'
+```
+
+Add the following options to the last command (recall with `↑`) to store the results in multiple files.
+Each file will contain the prefix, an processed row, and the suffix.
+
+```sh
+--output=advanced.json --splitToFiles=true
+```
+
+Filenames are suffixed with the row number by default (e.g. `advanced_1.json`, `advanced_2.json` etc.).
+There is another option to use the value in the first column instead:
+
+```sh
+--output=advanced.json --splitToFiles=true --suffixById=true
+```
+
+Because our project "advanced" contains duplicates in the first column "email" this command will overwrite files (e.g. `advanced_melanie.white@example2.edu.json`).
+When using this option, the first column should contain unique identifiers.
+
+### Append data to an existing project
+
+OpenRefine does not support appending rows to an existing project.
+As long as the [feature request](https://github.com/OpenRefine/OpenRefine/issues/715) is not yet implemented, you can use the openrefine-client to script a workaround:
+
+1. export existing project as csv
+2. put old and new data into a zip archive
+3. create new project by importing the zip archive
+
+Here is an example that replaces the existing project:
+
+```
+openrefine-client --export myproject --output old.csv
+openrefine-client --delete myproject
+zip combined.zip old.csv new.csv
+openrefine-client --create combined.zip --format csv --projectName myproject
+```
+
+Note that the project id will change.
+If you want to distinguish between old and new data, you can use the additional flag includeFileSources:
+
+```
+openrefine-client --create combined.zip --format csv --projectName myproject --includeFileSources true
+```
+
+### See also
+
+- Linux Bash script to run OpenRefine in batch mode (import, transform, export): [openrefine-batch](https://github.com/opencultureconsulting/openrefine-batch)
+- [Jupyter notebook demonstrating usage in Linux Bash](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/notebooks/openrefine-client-bash.ipynb)
+- Use case [HOS-MetadataTransformations](https://github.com/subhh/HOS-MetadataTransformations): Automated workflow for harvesting, transforming and indexing of metadata using metha, OpenRefine and Solr. Part of the Hamburg Open Science "Schaufenster" software stack.
+- Use case [Data processing of ILS data to facilitate a new discovery layer for the German Literature Archive (DLA)](https://doi.org/10.5281/zenodo.2678113): Custom data processing pipeline based on Pandas (a Python library) and OpenRefine.
+
+## Docker
+
+[felixlohmeier/openrefine-client](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [![Docker](https://img.shields.io/microbadger/image-size/felixlohmeier/openrefine-client?label=docker)](https://hub.docker.com/r/felixlohmeier/openrefine-client/)
+
+```sh
+docker pull felixlohmeier/openrefine-client:v0.3.10
+```
+
+### Option 1: Dockerized client
+
+Run client and mount current directory as workspace:
+
+```sh
+docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10
+```
+
+The docker option `--network=host` allows you to connect to a local or remote OpenRefine via the host network:
+
+- list projects on default URL (http://localhost:3333)
+
+  ```sh
+  docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --list
+  ```
+
+- list projects on a remote server (http://example.com)
+
+  ```sh
+  docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H example.com -P 80 --list
+  ```
+
+Usage: same commands as explained above (see [Basic Commands](#basic-commands) and [Advanced Templating](#advanced-templating))
+
+### Option 2: Dockerized client and dockerized OpenRefine
+
+Run openrefine-client linked to a dockerized OpenRefine ([felixlohmeier/openrefine](https://hub.docker.com/r/felixlohmeier/openrefine/) [![Docker](https://img.shields.io/microbadger/image-size/felixlohmeier/openrefine?label=docker)](https://hub.docker.com/r/felixlohmeier/openrefine)):
+
+1. Create docker network
+
+   ```sh
+   docker network create openrefine
+   ```
+
+2. Run server (will be available at http://localhost:3333)
+
+   ```sh
+   docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.5.0
+   ```
+
+3. Run client with some [basic commands](#basic-commands): 1. download example files, 2. create project from file, 3. list projects, 4. show metadata, 5. export to terminal, 6. apply transformation rules (deduplication), 7. export again to terminal, 8. export to xls file and 9. delete project
+
+   ```sh
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --download "https://git.io/fj5hF" --output=duplicates.csv
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --create duplicates.csv
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --list
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --info "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --apply duplicates-deletion.json "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export --output=deduped.xls "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --delete "duplicates"
+   ```
+
+4. Stop and delete server:
+
+   ```sh
+   docker stop openrefine-server
+   docker rm openrefine-server
+   ```
+
+5. Delete docker network:
+
+   ```sh
+   docker network rm openrefine
+   ```
+
+Customize OpenRefine server:
+
+- If you want to add an OpenRefine startup option you need to repeat the default commands (cf. [Dockerfile](https://hub.docker.com/r/felixlohmeier/openrefine/dockerfile))
+  - `-i 0.0.0.0` sets OpenRefine to be accessible from outside the container, i.e. from host
+  - `-d /data` sets OpenRefine workspace
+
+- Example for [allocating more memory](https://github.com/OpenRefine/OpenRefine/wiki/FAQ#out-of-memory-errors---feels-slow---could-not-reserve-enough-space-for-object-heap) to OpenRefine with additional option `-m 4G`
+
+  ```sh
+  docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.5.0 -i 0.0.0.0 -d /data -m 4G
+  ```
+
+- The OpenRefine version is defined by the docker tag.
+  Check the [DockerHub repository](https://hub.docker.com/r/felixlohmeier/openrefine) for available tags.
+  Example for OpenRefine `2.8` with same options as above:
+
+  ```sh
+  docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:2.8 -i 0.0.0.0 -d /data -m 4G
+  ```
+
+- If you want OpenRefine to read and write persistent data in host directory (i.e. store projects) you can mount the container path `/data`. Example for host directory `/home/felix/refine`:
+
+  ```sh
+  docker run -d -p 3333:3333 -v /home/felix/refine:/data:z --network=openrefine name=openrefine-server felixlohmeier/openrefine:2.8 -i 0.0.0.0 -d /data -m 4G
+  ```
+
+See also:
+
+- [GitHub Repository](https://github.com/opencultureconsulting/openrefine-docker) for docker container `felixlohmeier/openrefine`
+- Linux Bash script to run OpenRefine in batch mode (import, transform, export) with docker containers: [openrefine-batch-docker.sh](https://github.com/opencultureconsulting/openrefine-batch/#docker)
+
+## Python
+
+[openrefine-client](https://pypi.org/project/openrefine-client/) [![PyPI](https://img.shields.io/pypi/v/openrefine-client)](https://pypi.org/project/openrefine-client/) (requires Python 2.x)
+
+```sh
+python2 -m pip install openrefine-client --user
+```
+
+This will install the package `openrefine-client` containing modules in `google.refine`.
+
+A command line script `openrefine-client` will also be installed.
+
+### Option 1: command line script
+
+```sh
+openrefine-client --help
+```
+
+Usage: same commands as explained above (see [Basic Commands](#basic-commands) and [Advanced Templating](#advanced-templating))
+
+### Option 2: using cli functions in Python 2.x environment
+
+Import module cli:
+
+```python
+from google.refine import cli
+```
+
+Change URL (if necessary):
+
+```python
+cli.refine.REFINE_HOST = 'localhost'
+cli.refine.REFINE_PORT = '3333'
+```
+
+Help screen:
+
+```python
+help(cli)
+```
+
+Commands:
+
+* download (e.g. example data):
+
+  ```python
+  cli.download('https://git.io/fj5hF','duplicates.csv')
+  cli.download('https://git.io/fj5ju','duplicates-deletion.json')
+  ```
+
+* list projects:
+
+  ```python
+  cli.ls()
+  ```
+
+* create project:
+
+  ```python
+  p1 = cli.create('duplicates.csv')
+  ```
+
+* show metadata:
+
+  ```python
+  cli.info(p1.project_id)
+  ```
+
+* apply rules from file to project:
+
+  ```python
+  cli.apply(p1.project_id, 'duplicates-deletion.json')
+  ```
+
+* export project to terminal:
+
+  ```python
+  cli.export(p1.project_id)
+  ```
+
+* export project to file in xls format:
+
+  ```python
+  cli.export(p1.project_id, 'deduped.xls')
+  ```
+
+* export templating (see [Advanced Templating](#advanced-templating) above):
+
+  ```python
+  cli.templating(
+      p1.project_id,
+      prefix='''{ "events" : [
+  ''',template='''    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }''',
+      rowSeparator=''',
+  ''',suffix='''
+  ] }''')
+  ```
+
+* delete project:
+
+  ```python
+  cli.delete(p1.project_id)
+  ```
+
+### Option 3: the upstream way
+
+This fork can be used in the same way as the upstream [Python client library](https://github.com/PaulMakepeace/refine-client-py/).
+
+Some functions in the python client library are not yet compatible with OpenRefine >=3.0 (cf. [issue #19 in refine-client-py](https://github.com/paulmakepeace/refine-client-py/issues/19)).
+
+Import module refine:
+
+```python
+from google.refine import refine
+```
+
+Server Commands:
+
+* set up connection:
+
+  ```python
+  server1 = refine.Refine('http://localhost:3333')
+  ```
+
+- show version:
+
+  ```python
+  server1.server.get_version()
+  server1.server.version
+  ```
+
+- list projects:
+
+  ```python
+  server1.list_projects()
+  ```
+
+  - pretty print the returned dict with json.dumps:
+
+    ```python
+    import json
+    print(json.dumps(server1.list_projects(), indent=1))
+    ```
+
+- create project:
+
+  ```python
+  server1.new_project(project_file='duplicates.csv')
+  ```
+
+  * create and open the returned project in one step:
+
+    ```python
+    project1 = server1.new_project(project_file='duplicates.csv')
+    ```
+
+Project commands:
+
+* open project:
+
+  ```python
+  project1 = server1.open_project('1234567890123')
+  ```
+
+* print full URL to project:
+
+  ```python
+  project1.project_url()
+  ```
+
+* list columns:
+
+  ```python
+  project1.columns
+  ```
+
+* compute text facet on first column (**fails with OpenRefine >=3.2**):
+
+  ```python
+  project1.compute_facets(facet.TextFacet(project1.columns[0]))
+  ```
+
+  * print returned object
+
+    ```python
+    facets = project1.compute_facets(facet.TextFacet(project1.columns[0])).facets[0]
+    for k in sorted(facets.choices, key=lambda k: facets.choices[k].count, reverse=True):
+        print(facets.choices[k].count, k)
+    ```
+
+* compute clusters on first column:
+
+  ```python
+  project1.compute_clusters(project1.columns[0])
+  ```
+
+* apply rules from file to project:
+
+  ```python
+  project1.apply_operations('duplicates-deletion.json')
+  ```
+
+* export project:
+
+  ```python
+  project1.export(export_format='tsv')
+  ```
+
+  * print the returned fileobject:
+
+    ```python
+    print(project1.export(export_format='tsv').read())
+    ```
+
+  * save the returned fileobject to file:
+
+    ```python
+    with open('export.tsv', 'wb') as f:
+        f.write(project1.export(export_format='tsv').read())
+    ```
+
+* templating export (**function was added in this fork**, see [Advanced Templating](#advanced-templating) above):
+
+  ```python
+  data = project1.export_templating(
+      prefix='''{ "events" : [
+  ''',template='''    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }''',
+      rowSeparator=''',
+  ''',suffix='''
+  ] }''')
+  print(data.read())
+  ```
+
+* print help screen with available commands (many more!):
+
+  ```python
+  help(project1)
+  ```
+
+* example for custom commands:
+
+  ```python
+  project1.do_json('get-rows')['total']
+  ```
+
+* delete project:
+
+  ```python
+  project1.delete()
+  ```
+
+See also:
+
+- Jupyter notebook by Trevor Muñoz (2013-08-18): [Programmatic Use of Open Refine to Facet and Cluster Names of 'Dishes' from NYPL's What's on the menu?](https://nbviewer.jupyter.org/gist/trevormunoz/6265360)
+- Jupyter notebook by Tony Hirst (2019-01-09) [Notebook demonstrating how to control OpenRefine via a Python client.](https://nbviewer.jupyter.org/github/ouseful-PR/openrefineder/blob/4cef25a4ca6077536c5f49cafb531499fbcad96e/notebooks/OpenRefine%20Demos.ipynb)
+- Unittests [test_refine.py](tests/test_refine.py) and [test_tutorial.py](tests/test_tutorial.py) (both importing [refinetest.py](tests/refinetest.py))
+- [OpenRefine API](https://github.com/OpenRefine/OpenRefine/wiki/OpenRefine-API) in official OpenRefine wiki
+
+## Binder
+
+[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master)
+
+- free to use on-demand server with Jupyter notebook, OpenRefine and Bash
+- no registration needed, will start within a few minutes
+- [restricted](https://mybinder.readthedocs.io/en/latest/faq.html#how-much-memory-am-i-given-when-using-binder) to 2 GB RAM and server will be deleted after 10 minutes of inactivity
+- [bash_kernel demo notebook](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/notebooks/openrefine-client-bash.ipynb) for using the openrefine-client in a Linux Bash environment [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master?urlpath=/tree/notebooks/openrefine-client-bash.ipynb)
+- [python2 demo notebook](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/notebooks/openrefine-client-python.ipynb) for using the openrefine-client in a Python 2 environment [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master?urlpath=/tree/notebooks/openrefine-client-python.ipynb)
+
+## Development
+
+If you would like to contribute to the Python client library please consider a pull request to the upstream repository [refine-client-py](https://github.com/PaulMakepeace/refine-client-py/).
+
+### Tests
+
+Ensure you have OpenRefine running (i.e. available at http://localhost:3333). If necessary set the environment variables `OPENREFINE_HOST` and `OPENREFINE_PORT` to change the URL.
+
+The Python client library includes several unit tests.
+
+- run all tests
+
+  ```sh
+  python setup.py test
+  ```
+
+- run subset test_facet
+
+  ```sh
+  python setup.py --test-suite tests.test_facet
+  ```
+
+There is also a script that uses docker images to run the unit tests with different versions of OpenRefine.
+
+- run tests on all OpenRefine versions (from 2.0 up to 3.5.0)
+
+  ```sh
+  ./tests.sh -a
+  ```
+
+- run tests on tag 3.5.0
+
+  ```sh
+  ./tests.sh -t 3.5.0
+  ```
+
+- run tests on tag 3.5.0 interactively (pause before and after tests)
+
+  ```sh
+  ./tests.sh -t 3.5.0 -i
+  ```
+
+- run tests on tags 3.5.0 and 2.7
+
+  ```sh
+  ./tests.sh -t 3.5.0 -t 2.7
+  ```
+
+For Linux there are also functional tests for all command line options.
+
+- run all functional tests on OpenRefine 3.5.0
+
+  ```sh
+  ./tests-cli.sh 3.5.0
+  ```
+
+- run all functional tests on OpenRefine 3.5.0 with one-file-executable
+
+  ```sh
+  ./tests-cli.sh 3.5.0 openrefine-client_0-3-7_linux
+  ```
+
+### Distributing
+
+Note to myself: When releasing a new version...
+
+1. Run functional tests
+
+   ```sh
+   for v in 2.7 2.8 3.0 3.1 3.2 3.3 3.4 3.4.1 3.5.0; do
+      ./tests-cli.sh $v
+   done
+   ```
+
+2. Make final changes in Git
+
+   - update versions (e.g. 0.3.7 und 0-3-7) in [README.md](https://github.com/opencultureconsulting/openrefine-client/blob/master/README.md#download)
+   - update version in [setup.py](https://github.com/opencultureconsulting/openrefine-client/blob/master/setup.py)
+   - check if [Dockerfile](https://github.com/opencultureconsulting/openrefine-client/blob/master/docker/Dockerfile) needs to be changed
+
+3. Build executables with PyInstaller
+
+   - Run PyInstaller in Python 2 environments on native Windows, macOS and Linux. Should be "the oldest version of the OS you need to support"! Current release is built with: 
+
+     - Ubuntu 16.04 LTS (64-bit)
+     - macOS Sierra 10.12 (64-bit)
+     - Windows 7 (32-bit)
+
+   - One-file-executables will be available in `dist/`.
+
+     ```sh
+     git clone https://github.com/opencultureconsulting/openrefine-client.git
+     cd openrefine-client
+     python2 -m pip install pyinstaller --user
+     python2 -m pip install urllib2_file --user
+     python2 -m PyInstaller --onefile refine.py --hidden-import google.refine.__main__
+     ```
+
+4. Run functional tests with Linux executable
+
+   ```sh
+   for v in 2.7 2.8 3.0 3.1 3.2 3.3 3.4 3.4.1 3.5.0; do
+      ./tests-cli.sh $v openrefine-client_0-3-7_linux
+   done
+   ```
+
+5. Create release in GitHub
+
+   - draft [release notes](https://github.com/opencultureconsulting/openrefine-client/releases) and attach one-file-executables
+
+6. Build package and upload to PyPI
+
+   ```sh
+   python3 setup.py sdist bdist_wheel
+   python3 -m twine upload dist/*
+   ```
+
+7. Update Docker container
+
+   - add new autobuild for release version
+   - trigger latest build
+
+8. Bump openrefine-client version in related projects
+
+   - openrefine-batch: [openrefine-batch.sh](https://github.com/opencultureconsulting/openrefine-batch/blob/master/openrefine-batch.sh#L7) and [openrefine-batch-docker.sh](https://github.com/opencultureconsulting/openrefine-batch/blob/master/openrefine-batch-docker.sh)
+   - openrefineder: [postBuild](https://github.com/felixlohmeier/openrefineder/blob/master/binder/postBuild)
 
 ## Credits
 
@@ -77,7 +764,7 @@ David Huynh, [initial cut](<http://markmail.org/message/jsxzlcu3gn6drtb7)
 
 [Felix Lohmeier](https://felixlohmeier.de), extended the CLI features
 
-Some data used in the test suite has been used from publicly available sources,
+Some data used in the test suite has been used from publicly available sources:
 
 - louisiana-elected-officials.csv: from http://www.sos.louisiana.gov/tabid/136/Default.aspx
 

docker/Dockerfile

@@ -1,4 +1,4 @@
-FROM alpine:latest
+FROM alpine:3.11
 LABEL maintainer="felixlohmeier@opencultureconsulting.com"
 # The OpenRefine Python Client Library from PaulMakepeace provides an interface to communicating with an OpenRefine server. This fork extends the command line interface (CLI) and supports communication between docker containers.
 # Source: https://github.com/opencultureconsulting/openrefine-client

docker/README.md

@@ -1,89 +0,0 @@
-## batch processing with python-client
-
-There are some client libraries for OpenRefine that communicate with the [OpenRefine API](https://github.com/OpenRefine/OpenRefine/wiki/OpenRefine-API). I have prepared a docker container on top of the [Python Library from PaulMakepeace](https://github.com/PaulMakepeace/refine-client-py/) and extended the CLI with some options to create new OpenRefine projects from files.
-
-If you are looking for a ready to use command line interface to OpenRefine for batch processing then you might be interested in the following bash shell script: [felixlohmeier/openrefine-batch](https://github.com/felixlohmeier/openrefine-batch)
-
-### basic usage
-
-**1) start server:**
-> docker run -d --name=openrefine-server felixlohmeier/openrefine
-
-**2) run client with one of the following commands:**
-
-list projects:
-> docker run --rm --link openrefine-server felixlohmeier/openrefine-client --list
-
-create project from file:
-> docker run --rm --link openrefine-server felixlohmeier/openrefine-client --create [FILE]
-
-apply [rules from json file](http://kb.refinepro.com/2012/06/google-refine-json-and-my-notepad-or.html):
-> docker run --rm --link openrefine-server felixlohmeier/openrefine-client --apply [FILE.json] [PROJECTID]
-
-export project to file:
-> docker run --rm --link openrefine-server felixlohmeier/openrefine-client --export [PROJECTID] --output=FILE.tsv
-
-check help screen for more options:
-> docker run --rm --link openrefine-server felixlohmeier/openrefine-client --help
-
-**3) cleanup:**
-> docker stop openrefine-server && docker rm openrefine-server
-
-### example for customized run commands in interactive mode (e.g. for usage in terminals)
-
-**1) start server in terminal A:**
-
-```docker run --rm --name=openrefine-server -p 80:3333 -v /home/felix/refine:/data:z felixlohmeier/openrefine -i 0.0.0.0 -m 4G -d /data```
-
-* automatically remove docker container when it exits
-* set name "openrefine" for docker container
-* publish internal port 3333 to host port 80
-* mount host directory /home/felix/refine as working directory
-* make openrefine available in the network
-* increase java heap size to 4 GB
-* set refine workspace to /data
-* OpenRefine should be available at http://localhost
-
-**2) start client in terminal B (prints help screen):**
-
-```docker run --rm --link openrefine-server -v /home/felix/refine:/data:z felixlohmeier/openrefine-client```
-
-* automatically remove docker container when it exits
-* build up network connection with docker container "openrefine"
-* mount host directory /home/felix/refine as working directory
-* apply history in file /home/felix/refine/history.json to project with id 1234567890123
-
-### example for customized run commands in detached mode (e.g. for usage in shell scripts)
-
-**1) define variables (bring your own example data)**
-> workingdir=/home/felix/refine
-> inputfile=example.csv
-> jsonfile=test.json
-
-**2) start server**
-
- ```docker run -d --name=openrefine-server -v ${workingdir}:/data:z felixlohmeier/openrefine -i 0.0.0.0 -m 4G -d /data```
-
-**3) wait until server is ready**
-
-```until docker run --rm --link openrefine-server --entrypoint /usr/bin/curl felixlohmeier/openrefine-client --silent -N http://openrefine-server:3333 | cat | grep -q -o "OpenRefine" ; do sleep 1; done```
-
-**4) create project (import file)**
-
-```docker run --rm --link openrefine-server -v ${workingdir}:/data:z felixlohmeier/openrefine-client --create $inputfile```
-
-**5) get project id**
-
-```project=($(docker run --rm --link openrefine-server -v ${workingdir}:/data felixlohmeier/openrefine-client --list | cut -c 2-14))```
-
-**6) apply transformations from json file**
-
-```docker run --rm --link openrefine-server -v ${workingdir}:/data felixlohmeier/openrefine-client --apply ${jsonfile} ${project}```
-
-**7) export project to file**
-
-```docker run --rm --link openrefine-server -v ${workingdir}:/data felixlohmeier/openrefine-client --export --output=${project}.tsv ${project}```
-
-**8) cleanup**
-
-```docker stop -t=500 openrefine-server && docker rm openrefine-server```

google/refine/__main__.py

@@ -1,6 +1,6 @@
 #! /usr/bin/env python
 """
-Script to provide a command line interface to a OpenRefine server.
+Script to provide a command line interface to a Refine server.
 """
 
 # Copyright (c) 2011 Paul Makepeace, Real Programmers. All rights reserved.
@@ -20,198 +20,259 @@ Script to provide a command line interface to a OpenRefine server.
 
 
 import optparse
-import os
-import sys
 
 from google.refine import refine
-from google.refine import client
+from google.refine import cli
 
-reload(sys)
-sys.setdefaultencoding('utf-8')
 
 class myParser(optparse.OptionParser):
 
     def format_epilog(self, formatter):
         return self.epilog
 
+
 PARSER = \
-    myParser(description='Script to provide a command line interface to an OpenRefine server.',
+    myParser(description=('Script to provide a command line interface to an '
+                          'OpenRefine server.'),
              usage='usage: %prog [--help | OPTIONS]',
              epilog="""
-Examples:
-  --list # show list of projects (id: name)
+Example data:
+  --download "https://git.io/fj5hF" --output=duplicates.csv
+  --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+
+Basic commands:
+  --list # list all projects
   --list -H 127.0.0.1 -P 80 # specify hostname and port
-  --info 2161595260364 # show metadata of project 
-  --info "christmas gifts"
-  --create example.csv # create new project from file example.csv
+  --create duplicates.csv # create new project from file
+  --info "duplicates" # show project metadata
+  --apply duplicates-deletion.json "duplicates" # apply rules in file to project
+  --export "duplicates" # export project to terminal in tsv format
+  --export --output=deduped.xls "duplicates" # export project to file in xls format
+  --delete "duplicates" # delete project
+
+Some more examples:
+  --info 1234567890123 # specify project by id
   --create example.tsv --encoding=UTF-8
   --create example.xml --recordPath=collection --recordPath=record
   --create example.json --recordPath=_ --recordPath=_
   --create example.xlsx --sheets=0
   --create example.ods --sheets=0
-  --apply trim.json 2161595260364 # apply rules in trim.json to project 1234...
-  --apply trim.json "christmas gifts"
-  --export 2161595260364 > project.tsv # export project 2161595260364 in tsv format
-  --export "christmas gifts" > project.tsv
-  --export --output=project.xlsx 2161595260364 # export project in xlsx format
-  --export --output=project.xlsx "christmas gifts"
-  --export "My Address Book" --template='{ "friend" : {{jsonize(cells["friend"].value)}}, "address" : {{jsonize(cells["address"].value)}} }' --prefix='{ "rows" : [' --rowSeparator ',' --suffix '] }' --filterQuery="^mary$"
-  --delete 2161595260364 # delete project 
-  --delete "christmas gifts"
+
+Example for Templating Export:
+  Cf. https://github.com/opencultureconsulting/openrefine-client#advanced-templating
 """)
 
 group1 = optparse.OptionGroup(PARSER, 'Connection options')
-group1.add_option('-H', '--host', dest='host', metavar='127.0.0.1',
+group1.add_option('-H', '--host', dest='host',
+                  metavar='127.0.0.1',
                   help='OpenRefine hostname (default: 127.0.0.1)')
-group1.add_option('-P', '--port', dest='port', metavar='3333',
+group1.add_option('-P', '--port', dest='port',
+                  metavar='3333',
                   help='OpenRefine port (default: 3333)')
 PARSER.add_option_group(group1)
 
 group2 = optparse.OptionGroup(PARSER, 'Commands')
-group2.add_option('-c', '--create', dest='create', metavar='[FILE]',
+group2.add_option('-c', '--create', dest='create',
+                  metavar='[FILE]',
                   help='Create project from file. The filename ending (e.g. .csv) defines the input format (csv,tsv,xml,json,txt,xls,xlsx,ods)')
-group2.add_option('-l', '--list', dest='list', action='store_true',
+group2.add_option('-l', '--list', dest='list',
+                  action='store_true',
                   help='List projects')
+group2.add_option('--download', dest='download',
+                  metavar='[URL]',
+                  help='Download file from URL (e.g. example data). Combine with --output to specify a filename.')
 PARSER.add_option_group(group2)
 
-group3 = optparse.OptionGroup(PARSER, 'Commands with argument [PROJECTID/PROJECTNAME]')
-group3.add_option('-d', '--delete', dest='delete', action='store_true',
+group3 = optparse.OptionGroup(
+    PARSER, 'Commands with argument [PROJECTID/PROJECTNAME]')
+group3.add_option('-d', '--delete', dest='delete',
+                  action='store_true',
                   help='Delete project')
-group3.add_option('-f', '--apply', dest='apply', metavar='[FILE]',
+group3.add_option('-f', '--apply', dest='apply',
+                  metavar='[FILE]',
                   help='Apply JSON rules to OpenRefine project')
-group3.add_option('-E', '--export', dest='export', action='store_true',
+group3.add_option('-E', '--export', dest='export',
+                  action='store_true',
                   help='Export project in tsv format to stdout.')
-group3.add_option('-o', '--output', dest='output', metavar='[FILE]',
+group3.add_option('-o', '--output', dest='output',
+                  metavar='[FILE]',
                   help='Export project to file. The filename ending (e.g. .tsv) defines the output format (csv,tsv,xls,xlsx,html)')
-group3.add_option('--info', dest='info', action='store_true',
+group3.add_option('--template', dest='template',
+                  metavar='[STRING]',
+                  help='Export project with templating. Provide (big) text string that you enter in the *row template* textfield in the export/templating menu in the browser app)')
+group3.add_option('--info', dest='info',
+                  action='store_true',
                   help='show project metadata')
 PARSER.add_option_group(group3)
 
-group4 = optparse.OptionGroup(PARSER, 'Create options')
-group4.add_option('--columnWidths', dest='columnWidths',
-                  help='(txt/fixed-width) please provide widths separated by comma (e.g. 7,5)')
-group4.add_option('--encoding', dest='encoding',
-                  help='(csv,tsv,txt), please provide short encoding name (e.g. UTF-8)')
-group4.add_option('--guessCellValueTypes', dest='guessCellValueTypes', metavar='true/false', choices=('true', 'false'),
-                  help='(xml,csv,tsv,txt,json, default: false)')
-group4.add_option('--headerLines', dest='headerLines', type="int",
-                  help='(csv,tsv,txt/fixed-width,xls,xlsx,ods), default: 1, default txt/fixed-width: 0')
-group4.add_option('--ignoreLines', dest='ignoreLines', type="int",
-                  help='(csv,tsv,txt,xls,xlsx,ods), default: -1')
-group4.add_option('--includeFileSources', dest='includeFileSources', metavar='true/false', choices=('true', 'false'),
-                  help='(all formats), default: false')
-group4.add_option('--limit', dest='limit', type="int",
-                  help='(all formats), default: -1')
-group4.add_option('--linesPerRow', dest='linesPerRow', type="int",
-                  help='(txt/line-based), default: 1')
-group4.add_option('--processQuotes', dest='processQuotes', metavar='true/false', choices=('true', 'false'),
-                  help='(csv,tsv), default: true')
-group4.add_option('--projectName', dest='project_name',
-                  help='(all formats), default: filename')
-group4.add_option('--recordPath', dest='recordPath', action='append',
-                  help='(xml,json), please provide path in multiple arguments without slashes, e.g. /collection/record/ should be entered like this: --recordPath=collection --recordPath=record, default xml: record, default json: _ _')
-group4.add_option('--separator', dest='separator',
-                  help='(csv,tsv), default csv: , default tsv: \\t')
-group4.add_option('--sheets', dest='sheets', action='append', type="int",
-                  help='(xls,xlsx,ods), please provide sheets in multiple arguments, e.g. --sheets=0 --sheets=1, default: 0 (first sheet)')
-group4.add_option('--skipDataLines', dest='skipDataLines', type="int",
-                  help='(csv,tsv,txt,xls,xlsx,ods), default: 0, default line-based: -1')
-group4.add_option('--storeBlankRows', dest='storeBlankRows', metavar='true/false', choices=('true', 'false'),
-                  help='(csv,tsv,txt,xls,xlsx,ods), default: true')
-group4.add_option('--storeBlankCellsAsNulls', dest='storeBlankCellsAsNulls', metavar='true/false', choices=('true', 'false'),
-                  help='(csv,tsv,txt,xls,xlsx,ods), default: true')
-group4.add_option('--storeEmptyStrings', dest='storeEmptyStrings', metavar='true/false', choices=('true', 'false'),
-                  help='(xml,json), default: true')
-group4.add_option('--trimStrings', dest='trimStrings', metavar='true/false', choices=('true', 'false'),
-                  help='(xml,json), default: false')
+group4 = optparse.OptionGroup(PARSER, 'General options')
+group4.add_option('--format', dest='file_format',
+                  help='Override file detection (import: csv,tsv,xml,json,line-based,fixed-width,xls,xlsx,ods; export: csv,tsv,html,xls,xlsx,ods)')
 PARSER.add_option_group(group4)
 
-group5 = optparse.OptionGroup(PARSER, 'Legacy options')
-group5.add_option('--format', dest='input_format',
-help='Specify input format (csv,tsv,xml,json,line-based,fixed-width,xls,xlsx,ods)')
+group5 = optparse.OptionGroup(PARSER, 'Create options')
+group5.add_option('--columnWidths', dest='columnWidths',
+                  action='append',
+                  type='int',
+                  help='(txt/fixed-width), please provide widths in multiple arguments, e.g. --columnWidths=7 --columnWidths=5')
+group5.add_option('--encoding', dest='encoding',
+                  help='(csv,tsv,txt), please provide short encoding name (e.g. UTF-8)')
+group5.add_option('--guessCellValueTypes', dest='guessCellValueTypes',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(xml,csv,tsv,txt,json, default: false)')
+group5.add_option('--headerLines', dest='headerLines',
+                  type="int",
+                  help='(csv,tsv,txt/fixed-width,xls,xlsx,ods), default: 1, default txt/fixed-width: 0')
+group5.add_option('--ignoreLines', dest='ignoreLines',
+                  type="int",
+                  help='(csv,tsv,txt,xls,xlsx,ods), default: -1')
+group5.add_option('--includeFileSources', dest='includeFileSources',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(all formats), default: false')
+group5.add_option('--limit', dest='limit',
+                  type="int",
+                  help='(all formats), default: -1')
+group5.add_option('--linesPerRow', dest='linesPerRow',
+                  type="int",
+                  help='(txt/line-based), default: 1')
+group5.add_option('--processQuotes', dest='processQuotes',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(csv,tsv), default: true')
+group5.add_option('--projectName', dest='projectName',
+                  help='(all formats), default: filename')
+group5.add_option('--projectTags', dest='projectTags',
+                  action='append',
+                  help='(all formats), please provide tags in multiple arguments, e.g. --projectTags=beta --projectTags=client1')
+group5.add_option('--recordPath', dest='recordPath',
+                  action='append',
+                  help='(xml,json), please provide path in multiple arguments, e.g. /collection/record/ should be entered: --recordPath=collection --recordPath=record, default xml: root element, default json: _ _')
+group5.add_option('--separator', dest='separator',
+                  help='(csv,tsv), default csv: , default tsv: \\t')
+group5.add_option('--sheets', dest='sheets',
+                  action='append',
+                  type="int",
+                  help='(xls,xlsx,ods), please provide sheets in multiple arguments, e.g. --sheets=0 --sheets=1, default: 0 (first sheet)')
+group5.add_option('--skipDataLines', dest='skipDataLines',
+                  type="int",
+                  help='(csv,tsv,txt,xls,xlsx,ods), default: 0, default line-based: -1')
+group5.add_option('--storeBlankCellsAsNulls', dest='storeBlankCellsAsNulls',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(csv,tsv,txt,xls,xlsx,ods), default: true')
+group5.add_option('--storeBlankRows', dest='storeBlankRows',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(csv,tsv,txt,xls,xlsx,ods), default: true')
+group5.add_option('--storeEmptyStrings', dest='storeEmptyStrings',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(xml,json), default: true')
+group5.add_option('--trimStrings', dest='trimStrings',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='(xml,json), default: false')
 PARSER.add_option_group(group5)
 
-group6= optparse.OptionGroup(PARSER, 'Templating export options')
-group6.add_option('--template', dest='template',
-help='mandatory; (big) text string that you enter in the *row template* textfield in the export/templating menu in the browser app)')
-group6.add_option('--mode', dest='mode', metavar='row-based/record-based', choices=('row-based', 'record-based'),
-help='engine mode (default: row-based)')
+group6 = optparse.OptionGroup(PARSER, 'Templating options')
+group6.add_option('--mode', dest='mode',
+                  metavar='row-based/record-based',
+                  choices=('row-based', 'record-based'),
+                  help='engine mode (default: row-based)')
 group6.add_option('--prefix', dest='prefix',
-help='text string that you enter in the *prefix* textfield in the browser app')
+                  help='text string that you enter in the *prefix* textfield in the browser app')
 group6.add_option('--rowSeparator', dest='rowSeparator',
-help='text string that you enter in the *row separator* textfield in the browser app')
+                  help='text string that you enter in the *row separator* textfield in the browser app')
 group6.add_option('--suffix', dest='suffix',
-help='text string that you enter in the *suffix* textfield in the browser app')
-group6.add_option('--filterQuery', dest='filterQuery', metavar='REGEX',
-help='Simple RegEx text filter on filterColumn, e.g. ^12015$'),
-group6.add_option('--filterColumn', dest='filterColumn', metavar='COLUMNNAME',
-help='column name for filterQuery (default: name of first column)')
+                  help='text string that you enter in the *suffix* textfield in the browser app')
+group6.add_option('--filterQuery', dest='filterQuery',
+                  metavar='REGEX',
+                  help='Simple RegEx text filter on filterColumn, e.g. ^12015$'),
+group6.add_option('--filterColumn', dest='filterColumn',
+                  metavar='COLUMNNAME',
+                  help='column name for filterQuery (default: name of first column)')
 group6.add_option('--facets', dest='facets',
-help='facets config in json format (may be extracted with browser dev tools in browser app)')
-group6.add_option('--splitToFiles', dest='splitToFiles', metavar='true/false', choices=('true', 'false'),
-help='will split each row/record into a single file; it specifies a presumably unique character series for splitting; --prefix and --suffix will be applied to all files; filename-prefix can be specified with --output (default: %Y%m%d)')
-group6.add_option('--suffixById', dest='suffixById', metavar='true/false', choices=('true', 'false'),
-help='enhancement option for --splitToFiles; will generate filename-suffix from values in key column')
+                  help='facets config in json format (may be extracted with browser dev tools in browser app)')
+group6.add_option('--splitToFiles', dest='splitToFiles',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='will split each row/record into a single file; it specifies a presumably unique character series for splitting; --prefix and --suffix will be applied to all files; filename-prefix can be specified with --output (default: %Y%m%d)')
+group6.add_option('--suffixById', dest='suffixById',
+                  metavar='true/false', choices=('true', 'false'),
+                  help='enhancement option for --splitToFiles; will generate filename-suffix from values in key column')
 PARSER.add_option_group(group6)
 
-#noinspection PyPep8Naming
+
 def main():
     """Command line interface."""
 
-    # get environment variables in docker network
-    docker_host = os.environ.get('OPENREFINE_SERVER_PORT_3333_TCP_ADDR')
-    if docker_host:
-        os.environ["OPENREFINE_HOST"] = docker_host
-        refine.REFINE_HOST = docker_host
-    docker_port = os.environ.get('OPENREFINE_SERVER_PORT_3333_TCP_PORT')
-    if docker_port:
-        os.environ["OPENREFINE_PORT"] = docker_port
-        refine.REFINE_PORT = docker_port
-
     options, args = PARSER.parse_args()
-    commands_dict = { group2_arg.dest : getattr(options, group2_arg.dest) for group2_arg in group2.option_list }
-    commands_dict.update({ group3_arg.dest : getattr(options, group3_arg.dest) for group3_arg in group3.option_list })
-    commands_dict = { k: v for k, v in commands_dict.items() if v != None }
-    if not commands_dict:
-        PARSER.print_usage()
-        return
+
+    # set environment
     if options.host:
         refine.REFINE_HOST = options.host
     if options.port:
         refine.REFINE_PORT = options.port
+
+    # get project_id
     if args and not str.isdigit(args[0]):
         projects = refine.Refine(refine.RefineServer()).list_projects().items()
         idlist = []
         for project_id, project_info in projects:
-            if args[0] == project_info['name']:
+            if args[0].decode('UTF-8') == project_info['name']:
                 idlist.append(str(project_id))
         if len(idlist) > 1:
-            raise Exception('Found at least two projects. Please specify project by id.')
+            print('Error: Found %s projects with name %s.\n'
+                  'Please specify project by id.' % (len(idlist), args[0]))
+            for i in idlist:
+                print('')
+                cli.info(i)
+            return
         else:
-            args[0] = idlist[0]
+            try:
+                project_id = idlist[0]
+            except IndexError:
+                print('Error: No project found with name %s.\n'
+                      'Try command --list' % args[0])
+                return
+    elif args:
+        project_id = args[0]
 
+    # commands without args
     if options.list:
-        client.list_projects()
-    if options.create:
-        client.create_project(options)
-    if options.delete:
-        project = refine.RefineProject(args[0])
-        project.delete()
-    if options.apply:
-        project = refine.RefineProject(args[0])
-        response = project.apply_operations(options.apply)
-        if response != 'ok':
-            print >> sys.stderr, 'Failed to apply %s: %s' \
-                % (options.apply, response)
-        return project
-    if options.export or options.output:
-        project = refine.RefineProject(args[0])
-        client.export_project(project, options)
-        return project
-    if options.info:
-        client.info(args[0])
-        project = refine.RefineProject(args[0])
-        return project
+        cli.ls()
+    elif options.download:
+        cli.download(options.download, output_file=options.output)
+    elif options.create:
+        group5_dict = {group5_arg.dest: getattr(options, group5_arg.dest)
+                       for group5_arg in group5.option_list}
+        kwargs = {k: v for k, v in group5_dict.items()
+                  if v is not None and v not in ['true', 'false']}
+        kwargs.update({k: True for k, v in group5_dict.items()
+                       if v == 'true'})
+        kwargs.update({k: False for k, v in group5_dict.items()
+                       if v == 'false'})
+        if options.file_format:
+            kwargs.update({'project_format': options.file_format})
+        cli.create(options.create, **kwargs)
+    # commands with args
+    elif args and options.info:
+        cli.info(project_id)
+    elif args and options.delete:
+        cli.delete(project_id)
+    elif args and options.apply:
+        cli.apply(project_id, options.apply)
+    elif args and options.template:
+        group6_dict = {group6_arg.dest: getattr(options, group6_arg.dest)
+                       for group6_arg in group6.option_list}
+        kwargs = {k: v for k, v in group6_dict.items()
+                  if v is not None and v not in ['true', 'false']}
+        kwargs.update({k: True for k, v in group6_dict.items()
+                       if v == 'true'})
+        kwargs.update({k: False for k, v in group6_dict.items()
+                       if v == 'false'})
+        cli.templating(project_id, options.template,
+                       output_file=options.output, **kwargs)
+    elif args and (options.export or options.output):
+        cli.export(project_id, output_file=options.output,
+                   export_format=options.file_format)
+    else:
+        PARSER.print_usage()
+
 
 if __name__ == "__main__":
     # execute only if run as a script

google/refine/cli.py

@@ -0,0 +1,335 @@
+#! /usr/bin/env python
+"""
+Functions used by the command line interface (CLI)
+"""
+
+# Copyright (c) 2011 Paul Makepeace, Real Programmers. All rights reserved.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+
+import json
+import os
+import ssl
+import sys
+import time
+import urllib
+from xml.etree import ElementTree
+
+from google.refine import refine
+
+
+def apply(project_id, history_file):
+    """Apply OpenRefine history from json file to project."""
+    project = refine.RefineProject(project_id)
+    response = project.apply_operations(history_file)
+    if response != 'ok':
+        raise Exception('Failed to apply %s to %s: %s' %
+                        (history_file, project_id, response))
+    else:
+        print('File %s has been successfully applied to project %s' %
+              (history_file, project_id))
+
+def create(project_file,
+           project_format=None,
+           columnWidths=None,
+           encoding=None,
+           guessCellValueTypes=False,
+           headerLines=None,
+           ignoreLines=None,
+           includeFileSources=False,
+           limit=None,
+           linesPerRow=None,
+           processQuotes=True,
+           projectName=None,
+           projectTags=None,
+           recordPath=None,
+           separator=None,
+           sheets=None,
+           skipDataLines=None,
+           storeBlankCellsAsNulls=True,
+           storeBlankRows=True,
+           storeEmptyStrings=True,
+           trimStrings=False
+           ):
+    """Create a new project from file."""
+    # guess format from file extension
+    if not project_format:
+        project_format = os.path.splitext(project_file)[1][1:].lower()
+        if project_format == 'txt':
+            try:
+                columnWidths[0]
+                project_format = 'fixed-width'
+            except TypeError:
+                project_format = 'line-based'
+    # defaults for each file type
+    if project_format == 'xml':
+        project_format = 'text/xml'
+        if not recordPath:
+            recordPath = [ElementTree.parse(project_file).getroot().tag]
+    elif project_format == 'csv':
+        project_format = 'text/line-based/*sv'
+    elif project_format == 'tsv':
+        project_format = 'text/line-based/*sv'
+        if not separator:
+            separator = '\t'
+    elif project_format == 'line-based':
+        project_format = 'text/line-based'
+        if not skipDataLines:
+            skipDataLines = -1
+    elif project_format == 'fixed-width':
+        project_format = 'text/line-based/fixed-width'
+        if not headerLines:
+            headerLines = 0
+    elif project_format == 'json':
+        project_format = 'text/json'
+        if not recordPath:
+            recordPath = ['_', '_']
+    elif project_format == 'xls':
+        project_format = 'binary/text/xml/xls/xlsx'
+        if not sheets:
+            sheets = [0]
+            # TODO: new format for sheets option introduced in OpenRefine 2.8
+    elif project_format == 'xlsx':
+        project_format = 'binary/text/xml/xls/xlsx'
+        if not sheets:
+            sheets = [0]
+            # TODO: new format for sheets option introduced in OpenRefine 2.8
+    elif project_format == 'ods':
+        project_format = 'text/xml/ods'
+        if not sheets:
+            sheets = [0]
+            # TODO: new format for sheets option introduced in OpenRefine 2.8
+    # execute
+    kwargs = {k: v for k, v in vars().items() if v is not None}
+    project = refine.Refine(refine.RefineServer()).new_project(
+        guess_cell_value_types=guessCellValueTypes,
+        ignore_lines=ignoreLines,
+        header_lines=headerLines,
+        skip_data_lines=skipDataLines,
+        store_blank_rows=storeBlankRows,
+        process_quotes=processQuotes,
+        project_name=projectName,
+        store_blank_cells_as_nulls=storeBlankCellsAsNulls,
+        include_file_sources=includeFileSources,
+        **kwargs)
+    rows = project.do_json('get-rows')['total']
+    if rows > 0:
+        print('{0}: {1}'.format('id', project.project_id))
+        print('{0}: {1}'.format('rows', rows))
+        return project
+    else:
+        raise Exception(
+            'Project contains 0 rows. Please check --help for mandatory '
+            'arguments for xml, json, xlsx and ods')
+
+
+def delete(project_id):
+    """Delete project."""
+    project = refine.RefineProject(project_id)
+    response = project.delete()
+    if response != True:
+        raise Exception('Failed to delete %s: %s' %
+                        (project_id, response))
+    else:
+        print('Project %s has been successfully deleted' % project_id)
+
+
+def download(url, output_file=None):
+    """Integrated download function for your convenience."""
+    if not output_file:
+        output_file = os.path.basename(url)
+    if os.path.exists(output_file):
+        print('Error: File %s already exists.\n'
+              'Delete existing file or try command --output '
+              'to specify a different filename.' % output_file)
+        return
+    # Workaround for SSL verification problems in one-file-executables
+    context = ssl._create_unverified_context()
+    urllib.urlretrieve(url, output_file, context=context)
+    print('Download to file %s complete' % output_file)
+
+
+def export(project_id, encoding=None, output_file=None, export_format=None):
+    """Dump a project to stdout or file."""
+    project = refine.RefineProject(project_id)
+    if not output_file:
+        if not export_format:
+            export_format = 'tsv'
+        if export_format in ['csv', 'tsv', 'txt']:
+                encoding = 'UTF-8'
+        sys.stdout.write(project.export(
+            export_format=export_format, encoding=encoding).read())
+    else:
+        ext = os.path.splitext(output_file)[1][1:]
+        if ext and not export_format:
+            export_format = ext.lower()
+        if not export_format:
+            export_format = 'tsv'
+        if export_format in ['csv', 'tsv', 'txt']:
+            encoding = 'UTF-8'
+        with open(output_file, 'wb') as f:
+            f.write(project.export(
+                export_format=export_format, encoding=encoding).read())
+        print('Export to file %s complete' % output_file)
+
+
+def info(project_id):
+    """Show project metadata"""
+    projects = refine.Refine(refine.RefineServer()).list_projects()
+    if project_id in projects.keys():
+        print('{0:>20}: {1}'.format('id', project_id))
+        print('{0:>20}: {1}'.format('url', 'http://' +
+                                    refine.REFINE_HOST + ':' +
+                                    refine.REFINE_PORT +
+                                    '/project?project=' + project_id))
+        for k, v in projects[project_id].items():
+            if v:
+                    print(u'{0:>20}: {1}'.format(k, v))
+        project_model = refine.RefineProject(project_id).get_models()
+        columns = [c['name'] for c in project_model['columnModel']['columns']]
+        for (i, v) in enumerate(columns, start=1):
+            print(u'{0:>20}: {1}'.format(u'column ' + str(i).zfill(3), v).encode('utf-8'))
+    else:
+        print('Error: No project found with id %s.\n'
+              'Check existing projects with command --list' % (project_id))
+
+
+def ls():
+    """Query the server and list projects sorted by mtime."""
+    projects = refine.Refine(refine.RefineServer()).list_projects().items()
+
+    def date_to_epoch(json_dt):
+        """Convert a JSON date time into seconds-since-epoch."""
+        return time.mktime(time.strptime(json_dt, '%Y-%m-%dT%H:%M:%SZ'))
+    projects.sort(key=lambda v: date_to_epoch(v[1]['modified']), reverse=True)
+    if projects:
+        for project_id, project_info in projects:
+            print(u'{0:>14}: {1}'.format(project_id, project_info['name']).encode('utf-8'))
+    else:
+        print('Error: No projects found')
+
+
+def templating(project_id,
+               template,
+               encoding='UTF-8',
+               output_file=None,
+               mode=None,
+               prefix='',
+               rowSeparator='\n',
+               suffix='',
+               filterQuery=None,
+               filterColumn=None,
+               facets=None,
+               splitToFiles=False,
+               suffixById=None
+               ):
+    """Dump a project to stdout or file with templating."""
+    project = refine.RefineProject(project_id)
+
+    # basic config
+    templateconfig = {'prefix': prefix,
+                      'suffix': suffix,
+                      'template': template,
+                      'rowSeparator': rowSeparator,
+                      'encoding': encoding}
+
+    # construct the engine config
+    if mode == 'record-based':
+        engine = {'facets': [], 'mode': 'record-based'}
+    else:
+        engine = {'facets': [], 'mode': 'row-based'}
+    if facets:
+        engine['facets'].append(json.loads(facets))
+    if filterQuery:
+        if not filterColumn:
+            filterColumn = project.get_models()['columnModel']['keyColumnName']
+        textFilter = {'type': 'text',
+                      'name': filterColumn,
+                      'columnName': filterColumn,
+                      'mode': 'regex',
+                      'caseSensitive': False,
+                      'query': filterQuery}
+        engine['facets'].append(textFilter)
+    templateconfig.update({'engine': json.dumps(engine)})
+
+    if not splitToFiles:
+        # normal output
+        if not output_file:
+            sys.stdout.write(project.export_templating(
+                             **templateconfig).read())
+        else:
+            with open(output_file, 'wb') as f:
+                f.write(project.export_templating(**templateconfig).read())
+            print('Export to file %s complete' % output_file)
+    else:
+        # splitToFiles functionality
+        prefix = templateconfig['prefix']
+        suffix = templateconfig['suffix']
+        split = '===|||THISISTHEBEGINNINGOFANEWRECORD|||==='
+        if not output_file:
+            output_file = time.strftime('%Y%m%d')
+        else:
+            base = os.path.splitext(output_file)[0]
+            ext = os.path.splitext(output_file)[1][1:]
+        if not ext:
+            ext = 'txt'
+        # generate config for subfeature suffixById
+        if suffixById:
+            ids_template = ('{{forNonBlank(' +
+                            'with(row.columnNames[0],cn,cells[cn].value),' +
+                            'v,v,"")}}')
+            ids_templateconfig = {'engine': json.dumps(engine),
+                                  'template': ids_template,
+                                  'rowSeparator': '\n',
+                                  'encoding': encoding}
+            ids = [line.rstrip('\n') for line in project.export_templating(
+                   **ids_templateconfig) if line.rstrip('\n')]
+        # generate common config
+        if mode == 'record-based':
+            # record-based: split-character into template
+            #               if key column is not blank (=record)
+            template = ('{{forNonBlank(' +
+                         'with(row.columnNames[0],cn,cells[cn].value),' +
+                         'v,"' + split + '", "")}}' +
+                        templateconfig['template'])
+            templateconfig.update({'prefix': '',
+                                   'suffix': '',
+                                   'template': template,
+                                   'rowSeparator': ''})
+        else:
+            # row-based: split-character into template
+            template = split + templateconfig['template']
+            templateconfig.update({'prefix': '',
+                                   'suffix': '',
+                                   'template': template,
+                                   'rowSeparator': ''})
+        # execute
+        records = project.export_templating(
+            **templateconfig).read().split(split)
+        del records[0]  # skip first blank entry
+        if suffixById:
+            for index, record in enumerate(records):
+                output_file = base + '_' + ids[index] + '.' + ext
+                with open(output_file, 'wb') as f:
+                    f.writelines([prefix, record, suffix])
+            print('Export to files complete. Last file: %s' % output_file)
+        else:
+            zeros = len(str(len(records)))
+            for index, record in enumerate(records):
+                output_file = base + '_' + \
+                    str(index + 1).zfill(zeros) + '.' + ext
+                with open(output_file, 'wb') as f:
+                    f.writelines([prefix, record, suffix])
+            print('Export to files complete. Last file: %s' % output_file)

google/refine/client.py

@@ -1,154 +0,0 @@
-#! /usr/bin/env python
-"""
-Commands provided for the CLI and interactive usage
-"""
-
-# Copyright (c) 2011 Paul Makepeace, Real Programmers. All rights reserved.
-
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>
-
-
-import os
-import sys
-import time
-import json
-
-from google.refine import refine
-
-def list_projects():
-    """Query the OpenRefine server and list projects by ID: name."""
-    projects = refine.Refine(refine.RefineServer()).list_projects().items()
-
-    def date_to_epoch(json_dt):
-        """Convert a JSON date time into seconds-since-epoch."""
-        return time.mktime(time.strptime(json_dt, '%Y-%m-%dT%H:%M:%SZ'))
-    projects.sort(key=lambda v: date_to_epoch(v[1]['modified']), reverse=True)
-    for project_id, project_info in projects:
-        print('{0:>14}: {1}'.format(project_id, project_info['name']))
-
-def info(project_id):
-    """Show project metadata"""
-    projects = refine.Refine(refine.RefineServer()).list_projects().items()
-    for projects_id, projects_info in projects:
-        if project_id == projects_id:
-            print('{0}: {1}'.format('id', projects_id))
-            print('{0}: {1}'.format('name', projects_info['name']))
-            print('{0}: {1}'.format('created', projects_info['created']))
-            print('{0}: {1}'.format('modified', projects_info['modified']))
-
-def create_project(options):
-    """Create a new project from options.create file."""
-    # general defaults are defined in google/refine/refine.py new_project
-    # additional defaults for each file type
-    defaults = {}
-    defaults['xml'] = { 'project_format' : 'text/xml', 'recordPath' : 'record' }
-    defaults['csv'] = { 'project_format' : 'text/line-based/*sv', 'separator' : ',' }
-    defaults['tsv'] = { 'project_format' : 'text/line-based/*sv', 'separator' : '\t' }
-    defaults['line-based'] = { 'project_format' : 'text/line-based', 'skipDataLines' : -1 }
-    defaults['fixed-width'] = { 'project_format' : 'text/line-based/fixed-width', 'headerLines' : 0 }
-    defaults['json'] = { 'project_format' : 'text/json', 'recordPath' : ('_', '_') }
-    defaults['xls'] = { 'project_format' : 'binary/text/xml/xls/xlsx', 'sheets' : 0 }
-    defaults['xlsx'] = { 'project_format' : 'binary/text/xml/xls/xlsx', 'sheets' : 0 }
-    defaults['ods'] = { 'project_format' : 'text/xml/ods', 'sheets' : 0 }
-    # guess format from file extension (or legacy option --format)
-    input_format = os.path.splitext(options.create)[1][1:].lower()
-    if input_format == 'txt' and options.columnWidths:
-        input_format = 'fixed-width'
-    if input_format == 'txt' and not options.columnWidths:
-        input_format = 'line-based'
-    if options.input_format:
-        input_format = options.input_format
-    # defaults for selected format
-    input_dict = defaults[input_format]
-    # user input
-    input_user = { group4_arg.dest : getattr(options, group4_arg.dest) for group4_arg in group4.option_list }
-    input_user['strings'] = { k: v for k, v in input_user.items() if v != None and v not in ['true', 'false'] }
-    input_user['trues'] = { k: True for k, v in input_user.items() if v == 'true' }
-    input_user['falses'] = { k: False for k, v in input_user.items() if v == 'false' }
-    input_user_eval = input_user['strings']
-    input_user_eval.update(input_user['trues'])
-    input_user_eval.update(input_user['falses'])
-    # merge defaults with user input
-    input_dict.update(input_user_eval)
-    input_dict['project_file'] = options.create
-    refine.Refine(refine.RefineServer()).new_project(**input_dict)
-
-def export_project(project, options):
-    """Dump a project to stdout or options.output file."""
-    export_format = 'tsv'
-    if options.output and not options.splitToFiles == 'true':
-        ext = os.path.splitext(options.output)[1][1:]
-        if ext:
-            export_format = ext.lower()
-        output = open(options.output, 'wb')
-    else:
-        output = sys.stdout
-    if options.template:
-        templateconfig = { group6_arg.dest : getattr(options, group6_arg.dest) for group6_arg in group6.option_list if group6_arg.dest in ['prefix', 'template', 'rowSeparator', 'suffix'] }
-        if options.mode == 'record-based':
-            engine = { 'facets':[], 'mode':'record-based' }
-        else:
-            engine = { 'facets':[], 'mode':'row-based' }
-        if options.facets:
-            engine['facets'].append(json.loads(options.facets))
-        if options.filterQuery:
-            if not options.filterColumn:
-                filterColumn = project.get_models()['columnModel']['keyColumnName']
-            else:
-                filterColumn = options.filterColumn
-            textFilter = { 'type':'text', 'name':filterColumn, 'columnName':filterColumn, 'mode':'regex', 'caseSensitive':False, 'query':options.filterQuery }
-            engine['facets'].append(textFilter)
-        templateconfig.update({ 'engine': json.dumps(engine) })
-        if options.splitToFiles == 'true':
-            # common config for row-based and record-based
-            prefix = templateconfig['prefix']
-            suffix = templateconfig['suffix']
-            split = '===|||THISISTHEBEGINNINGOFANEWRECORD|||==='
-            keyColumn = project.get_models()['columnModel']['keyColumnName']
-            if not options.output:
-                filename = time.strftime('%Y%m%d')
-            else:
-                filename = os.path.splitext(options.output)[0]
-                ext = os.path.splitext(options.output)[1][1:]
-            if not ext:
-                ext = 'txt'
-            if options.suffixById:
-                ids_template = '{{forNonBlank(cells["' + keyColumn + '"].value, v, v, "")}}'
-                ids_templateconfig = { 'engine': json.dumps(engine), 'template': ids_template, 'rowSeparator':'\n' }
-                ids = [line.rstrip('\n') for line in project.export_templating(**ids_templateconfig) if line.rstrip('\n')]
-            if options.mode == 'record-based':
-                # record-based: split-character into template if key column is not blank (=record)
-                template = '{{forNonBlank(cells["' + keyColumn + '"].value, v, "' + split + '", "")}}' + templateconfig['template']
-                templateconfig.update({ 'prefix': '', 'suffix': '', 'template': template, 'rowSeparator':'' })
-            else:
-                # row-based: split-character into template
-                template = split + templateconfig['template']
-                templateconfig.update({ 'prefix': '', 'suffix': '', 'template': template, 'rowSeparator':'' })
-            records = project.export_templating(**templateconfig).read().split(split)
-            del records[0] # skip first blank entry
-            if options.suffixById:
-                for index, record in enumerate(records):
-                    output = open(filename + '_' + ids[index] + '.' + ext, 'wb')
-                    output.writelines([prefix, record, suffix])
-            else:
-                zeros = len(str(len(records)))
-                for index, record in enumerate(records):
-                    output = open(filename + '_' + str(index+1).zfill(zeros) + '.' + ext, 'wb')
-                    output.writelines([prefix, record, suffix])
-        else:
-            output.writelines(project.export_templating(**templateconfig))
-            output.close()
-    else:
-        output.writelines(project.export(export_format=export_format))
-        output.close()

google/refine/refine.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 """
-Client library to communicate with a OpenRefine server.
+Client library to communicate with a Refine server.
 """
 
 # Copyright (c) 2011 Paul Makepeace, Real Programmers. All rights reserved.
@@ -38,11 +38,11 @@ REFINE_PORT = os.environ.get('OPENREFINE_PORT', os.environ.get('GOOGLE_REFINE_PO
 
 
 class RefineServer(object):
-    """Communicate with a OpenRefine server."""
+    """Communicate with a Refine server."""
 
     @staticmethod
     def url():
-        """Return the URL to the OpenRefine server."""
+        """Return the URL to the Refine server."""
         server = 'http://' + REFINE_HOST
         if REFINE_PORT != '80':
             server += ':' + REFINE_PORT
@@ -53,9 +53,21 @@ class RefineServer(object):
             server = self.url()
         self.server = server[:-1] if server.endswith('/') else server
         self.__version = None     # see version @property below
+        self.token = None         # CSRF token introduced in OpenRefine 3.3
+        self.get_csrf_token()
+
+    def get_csrf_token(self):
+        """Return csrf token."""
+        try:
+            url = self.server + '/command/core/get-csrf-token'
+            response = json.loads(urllib2.urlopen(url).read())
+            self.token = response['token']
+            return self.token
+        except:
+            pass # fail silently to not disturb usage of OpenRefine <3.3
 
     def urlopen(self, command, data=None, params=None, project_id=None):
-        """Open a OpenRefine URL and with optional query params and POST data.
+        """Open a Refine URL and with optional query params and POST data.
 
         data: POST data dict
         param: query params dict
@@ -73,6 +85,9 @@ class RefineServer(object):
                 data['project'] = project_id
             else:
                 params['project'] = project_id
+        # be lazy and send the token for each API call (even when not needed)
+        if self.token:
+            params['csrf_token'] = self.token
         if params:
             url += '?' + urllib.urlencode(params)
         req = urllib2.Request(url)
@@ -82,10 +97,11 @@ class RefineServer(object):
         try:
             response = urllib2.urlopen(req)
         except urllib2.HTTPError as e:
-            raise Exception('HTTP %d "%s" for %s\n\t%s' % (e.code, e.msg, e.geturl(), data))
+            raise Exception('HTTP %d "%s" for %s\n\t%s' %
+                            (e.code, e.msg, e.geturl(), data))
         except urllib2.URLError as e:
             raise urllib2.URLError(
-                '%s for %s. No OpenRefine server reachable/running; ENV set?' %
+                '%s for %s. No Refine server reachable/running; ENV set?' %
                 (e.reason, self.server))
         if response.info().get('Content-Encoding', None) == 'gzip':
             # Need a seekable filestream for gzip
@@ -95,9 +111,13 @@ class RefineServer(object):
         return response
 
     def urlopen_json(self, *args, **kwargs):
-        """Open a OpenRefine URL, optionally POST data, and return parsed JSON."""
+        """Open a Refine URL, optionally POST data, and return parsed JSON."""
         response = json.loads(self.urlopen(*args, **kwargs).read())
         if 'code' in response and response['code'] not in ('ok', 'pending'):
+            if 'Missing or invalid csrf_token parameter' == response['message']:
+                self.get_csrf_token()
+                response = json.loads(self.urlopen(*args, **kwargs).read())
+                return response
             error_message = ('server ' + response['code'] + ': ' +
                              response.get('message', response.get('stack', response)))
             raise Exception(error_message)
@@ -118,7 +138,7 @@ class RefineServer(object):
 
 
 class Refine:
-    """Class representing a connection to a OpenRefine server."""
+    """Class representing a connection to a Refine server."""
     def __init__(self, server):
         if isinstance(server, RefineServer):
             self.server = server
@@ -144,17 +164,121 @@ class Refine:
         return projects[project_id]['name']
 
     def open_project(self, project_id):
-        """Open a OpenRefine project."""
+        """Open a Refine project."""
         return RefineProject(self.server, project_id)
 
-    def new_project(self, project_file=None, project_name=None,
-                    project_format='', **kwargs):
-        """Create a OpenRefine project."""
-        defaults = { 'guessCellValueTypes' : False, 'headerLines' : 1, 'ignoreLines' : -1, 'includeFileSources' : False, 'limit' : -1, 'linesPerRow' : 1, 'processQuotes' : True, 'separator' : ',', 'skipDataLines' : 0, 'storeBlankCellsAsNulls' : True, 'storeBlankRows' : True, 'storeEmptyStrings' : True, 'trimStrings' : False }
+    # These aren't used yet but are included for reference
+    new_project_defaults = {
+        'text/line-based/*sv': {
+            'encoding': '',
+            'separator': ',',
+            'ignore_lines': -1,
+            'header_lines': 1,
+            'skip_data_lines': 0,
+            'limit': -1,
+            'store_blank_rows': True,
+            'guess_cell_value_types': True,
+            'process_quotes': True,
+            'store_blank_cells_as_nulls': True,
+            'include_file_sources': False},
+        'text/line-based': {
+            'encoding': '',
+            'lines_per_row': 1,
+            'ignore_lines': -1,
+            'limit': -1,
+            'skip_data_lines': -1,
+            'store_blank_rows': True,
+            'store_blank_cells_as_nulls': True,
+            'include_file_sources': False},
+        'text/line-based/fixed-width': {
+            'encoding': '',
+            'column_widths': [20],
+            'ignore_lines': -1,
+            'header_lines': 0,
+            'skip_data_lines': 0,
+            'limit': -1,
+            'guess_cell_value_types': False,
+            'store_blank_rows': True,
+            'store_blank_cells_as_nulls': True,
+            'include_file_sources': False},
+        'text/line-based/pc-axis': {
+            'encoding': '',
+            'limit': -1,
+            'skip_data_lines': -1,
+            'include_file_sources': False},
+        'text/rdf+n3': {'encoding': ''},
+        'text/xml/ods': {
+            'sheets': [],
+            'ignore_lines': -1,
+            'header_lines': 1,
+            'skip_data_lines': 0,
+            'limit': -1,
+            'store_blank_rows': True,
+            'store_blank_cells_as_nulls': True,
+            'include_file_sources': False},
+        'binary/xls': {
+            'xml_based': False,
+            'sheets': [],
+            'ignore_lines': -1,
+            'header_lines': 1,
+            'skip_data_lines': 0,
+            'limit': -1,
+            'store_blank_rows': True,
+            'store_blank_cells_as_nulls': True,
+            'include_file_sources': False}
+    }
 
-        # options
-        options = { 'format': project_format }
-        if project_file is not None:
+    def new_project(self, project_file=None, project_url=None, project_name=None, project_format='text/line-based/*sv',
+                    encoding='',
+                    separator=',',
+                    ignore_lines=-1,
+                    header_lines=1,
+                    skip_data_lines=0,
+                    limit=-1,
+                    store_blank_rows=True,
+                    guess_cell_value_types=False,
+                    process_quotes=True,
+                    store_blank_cells_as_nulls=True,
+                    include_file_sources=False,
+                    **opts):
+
+        if (project_file and project_url) or (not project_file and not project_url):
+            raise ValueError('One (only) of project_file and project_url must be set')
+
+        def s(opt):
+            if isinstance(opt, bool):
+                return 'true' if opt else 'false'
+            if opt is None:
+                return ''
+            return str(opt)
+
+        # the new APIs requires a json in the 'option' POST or GET argument
+        # POST is broken at the moment, so we send it in the URL
+        new_style_options = dict(opts, **{
+            'encoding': s(encoding),
+            'separator': s(separator)
+        })
+        params = {
+            'options': json.dumps(new_style_options),
+        }
+
+        # old style options
+        options = {
+            'format': project_format,
+            'ignore-lines': s(ignore_lines),
+            'header-lines': s(header_lines),
+            'skip-data-lines': s(skip_data_lines),
+            'limit': s(limit),
+            'guess-value-type': s(guess_cell_value_types),
+            'process-quotes': s(process_quotes),
+            'store-blank-rows': s(store_blank_rows),
+            'store-blank-cells-as-nulls': s(store_blank_cells_as_nulls),
+            'include-file-sources': s(include_file_sources)
+        }
+
+        if project_url is not None:
+            options['url'] = project_url
+        elif project_file is not None:
             options['project-file'] = {
                 'fd': open(project_file),
                 'filename': project_file,
@@ -164,13 +288,6 @@ class Refine:
             project_name = (project_file or 'New project').rsplit('.', 1)[0]
             project_name = os.path.basename(project_name)
         options['project-name'] = project_name
-
-        # params (the API requires a json in the 'option' POST argument)
-        params = defaults
-        params.update(kwargs)
-        params = { 'options': json.dumps(params) }
-
-        # submit
         response = self.server.urlopen(
             'create-project-from-upload', options, params
         )
@@ -179,17 +296,11 @@ class Refine:
             urlparse.urlparse(response.geturl()).query)
         if 'project' in url_params:
             project_id = url_params['project'][0]
-            # check number of rows
-            rows = RefineProject(RefineServer(),project_id).do_json('get-rows')['total']
-            if rows > 0:
-                print('{0}: {1}'.format('id', project_id))
-                print('{0}: {1}'.format('rows', rows))
-                return RefineProject(self.server, project_id)
-            else:
-                raise Exception('Project contains 0 rows. Please check --help for mandatory arguments for xml, json, xlsx and ods')
+            return RefineProject(self.server, project_id)
         else:
             raise Exception('Project not created')
 
+
 def RowsResponseFactory(column_index):
     """Factory for the parsing the output from get_rows().
 
@@ -251,7 +362,7 @@ class RefineProject:
                 server = RefineServer(server)
         self.server = server
         if not project_id:
-            raise Exception('Missing OpenRefine project ID')
+            raise Exception('Missing Refine project ID')
         self.project_id = project_id
         self.engine = facet.Engine()
         self.sorting = facet.Sorting()
@@ -307,7 +418,10 @@ class RefineProject:
         for i, column in enumerate(column_model['columns']):
             name = column['name']
             self.column_order[name] = i
-            column_index[name] = column['cellIndex']
+            try:
+                column_index[name] = column['cellIndex']
+            except KeyError:
+                column_index[name] = i
         self.key_column = column_model['keyColumnName']
         self.has_records = response['recordModel'].get('hasRecords', False)
         self.rows_response_factory = RowsResponseFactory(column_index)
@@ -336,17 +450,31 @@ class RefineProject:
             return 'ok'
         return response_json['code']  # can be 'ok' or 'pending'
 
-    def export(self, export_format='tsv'):
+    def export(self, encoding=None, export_format='tsv'):
         """Return a fileobject of a project's data."""
-        url = ('export-rows/' + urllib.quote(self.project_name()) + '.' +
-               export_format)
-        return self.do_raw(url, data={'format': export_format})
+        url = ('export-rows/' +
+               urllib.quote(self.project_name().encode('utf8')) +
+               '.' + export_format)
+        data = {'format': export_format}
+        if encoding:
+            data['encoding'] = encoding
+        return self.do_raw(url, data)
 
-    def export_templating(self, export_format='txt', engine='', prefix='', template='', rowSeparator='', suffix=''):
-        """Return a fileobject of a project's data."""
-        url = ('export-rows/' + urllib.quote(self.project_name()) + '.' +
-               export_format)
-        return self.do_raw(url, data={'format': 'template', 'template': template, 'engine': engine, 'prefix': prefix, 'suffix': suffix, 'separator': rowSeparator } )
+    def export_templating(self, encoding=None, engine='', prefix='',
+                          template='', rowSeparator='\n', suffix=''):
+        """Return a fileobject of a project's data in templating mode."""
+        url = ('export-rows/' +
+               urllib.quote(self.project_name().encode('utf8')) +
+               '.' + 'txt')
+        data = {'format': 'template',
+                'template': template,
+                'engine': engine,
+                'prefix': prefix,
+                'suffix': suffix,
+                'separator': rowSeparator}
+        if encoding:
+            data['encoding'] = encoding
+        return self.do_raw(url, data)
 
     def export_rows(self, **kwargs):
         """Return an iterable of parsed rows of a project's data."""

refine.py

@@ -1,6 +1,6 @@
-#! /usr/bin/env python
+#!/usr/bin/env python
 """
-Script to provide a command line interface to a OpenRefine server.
+Script to provide a command line interface to a Refine server.
 """
 
 # Copyright (c) 2011 Paul Makepeace, Real Programmers. All rights reserved.
@@ -18,9 +18,18 @@ Script to provide a command line interface to a OpenRefine server.
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>
 
+import sys
 
-from google.refine import __main__
+from google.refine import __main__, cli, refine
+
+# workaround for pyinstaller
+if getattr(sys, 'frozen', False) and hasattr(sys, '_MEIPASS'):
+    reload(sys)
+    sys.setdefaultencoding('utf-8')
+    if sys.platform == "win32":
+        import codecs
+        codecs.register(lambda name: codecs.lookup(
+            'utf-8') if name == 'cp65001' else None)
 
 if __name__ == '__main__':
-    # return project so that it's available interactively, python -i refine.py
-    refine_project = __main__.main()
+    __main__.main()

setup.py

@@ -25,7 +25,7 @@ def read(filename):
     return open(os.path.join(os.path.dirname(__file__), filename)).read()
 
 setup(name='openrefine-client',
-      version='0.3.6',
+      version='0.3.10',
       description=('The OpenRefine Python Client Library provides an '
                    'interface to communicating with an OpenRefine server. '
                    'This fork extends the command line interface (CLI).'),
@@ -36,7 +36,7 @@ setup(name='openrefine-client',
       url='https://github.com/opencultureconsulting/openrefine-client',
       packages=find_packages(exclude=['tests']),
       install_requires=['urllib2_file'],
-      python_requires='>2.6, !=3.*',
+      python_requires='>=2.7, !=3.*',
       entry_points={
           'console_scripts': [ 'openrefine-client = google.refine.__main__:main' ]
       },

tests-cli.sh

@@ -0,0 +1,123 @@
+#!/bin/bash
+# Script for running functional tests against the CLI
+
+# Copyright (c) 2011 Paul Makepeace, Real Programmers. All rights reserved.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+# ================================== CONFIG ================================== #
+
+cd "${BASH_SOURCE%/*}/" || exit 1
+
+port=3334
+
+if [[ ${1} ]]; then
+  version="${1}"
+else
+  version="3.2"
+fi
+refine="openrefine-${version}/refine"
+
+if [[ ${2} ]]; then
+  client="$(readlink -e "${2}")"
+else
+  client="python2 $(readlink -e refine.py)"
+fi
+cmd="${client} -H localhost -P ${port}"
+
+if [[ ${3} ]]; then
+  filename="${3%%.*}"
+else
+  filename=""
+fi
+cmd="${client} -H localhost -P ${port}"
+
+# =============================== REQUIREMENTS =============================== #
+
+# check existence of java and cURL
+if [[ -z "$(command -v java 2> /dev/null)" ]] ; then
+  echo 1>&2 "ERROR: OpenRefine requires JAVA runtime environment (jre)" \
+    "https://openjdk.java.net/install/"
+  exit 1
+fi
+if [[ -z "$(command -v curl 2> /dev/null)" ]] ; then
+  echo 1>&2 "ERROR: This shell script requires cURL" \
+    "https://curl.haxx.se/download.html"
+  exit 1
+fi
+# download OpenRefine
+if [[ -z "$(readlink -e "${refine}")" ]]; then
+  echo "Download OpenRefine ${version}..."
+  mkdir -p "$(dirname "${refine}")"
+  curl -L --output openrefine.tar.gz \
+    "https://github.com/OpenRefine/OpenRefine/releases/download/${version}/openrefine-linux-${version}.tar.gz"
+  echo "Install OpenRefine ${version} in subdirectory $(dirname "${refine}")..."
+  tar -xzf openrefine.tar.gz -C "$(dirname "${refine}")" --strip 1 --totals
+  rm -f openrefine.tar.gz
+  # do not try to open OpenRefine in browser
+  sed -i '$ a JAVA_OPTIONS=-Drefine.headless=true' \
+    "$(dirname "${refine}")"/refine.ini
+  # set autosave period from 5 minutes to 25 hours
+  sed -i 's/#REFINE_AUTOSAVE_PERIOD=60/REFINE_AUTOSAVE_PERIOD=1500/' \
+    "$(dirname "${refine}")"/refine.ini
+  echo
+fi
+
+# ================================== SETUP =================================== #
+
+dir="$(readlink -f "tests/tmp")"
+mkdir -p "${dir}"
+rm -f tests-cli.log
+
+echo "start OpenRefine ${version}..."
+${refine} -v warn -p ${port} -d "${dir}" &>> tests-cli.log &
+pid_server=${!}
+timeout 30s bash -c "until curl -s 'http://localhost:3334' \
+  | cat | grep -q -o 'OpenRefine' ; do sleep 1; done" \
+  || error "starting OpenRefine server failed!"
+echo
+
+# ================================== TESTS =================================== #
+
+echo "running tests, please wait..."
+tests=()
+results=()
+for t in tests/*${filename}*.sh; do
+  tests+=("${t}")
+  echo "======================= ${t} =======================" &>> tests-cli.log
+  bash "${t}" "${cmd}" "${version}" &>> tests-cli.log
+  results+=(${?})
+done
+echo
+
+# ================================= TEARDOWN ================================= #
+
+echo "cleanup..."
+{ kill -9 "${pid_server}" && wait "${pid_server}"; } 2>/dev/null
+rm -rf "${dir}"
+echo
+
+# ================================= SUMMARY ================================== #
+
+printf "%s\t%s\n" "code" "test"
+printf "%s\t%s\n" "----" "----------------"
+for i in "${!tests[@]}"; do
+  printf "%s\t%s\n" "${results[$i]}" "${tests[$i]}"
+done
+echo
+if [[ " ${results[*]} " =~ [1-9] ]]; then
+  echo "failed tests! check tests-cli.log for debugging"; echo
+else
+  echo "all tests passed!"; echo
+fi

tests.sh

@@ -0,0 +1,130 @@
+#!/bin/bash
+# Script for running tests with different OpenRefine and Java versions based on Docker images.
+
+# Copyright (c) 2011 Paul Makepeace, Real Programmers. All rights reserved.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+# defaults:
+all=(3.5.0 3.4.1 3.4 3.3 3.2-java12 3.2-java11 3.2-java10 3.2-java9 3.2 3.1-java9 3.1 3.0-java9 3.0 2.8-java9 2.8 2.8-java7 2.7 2.7-java7 2.5-java7 2.5-java6 2.1-java6 2.0-java6)
+main=(3.5.0 3.4.1 3.4 3.3 3.2 3.1 3.0 2.8 2.7 2.5-java6 2.1-java6 2.0-java6)
+interactively=false
+port="3333"
+
+# help screen
+function usage () {
+    cat <<EOF
+Usage: ./tests.sh [-t TAG] [-i] [-p] [-a] [-h]
+
+Script for running tests with different OpenRefine and Java versions.
+It uses docker images from https://hub.docker.com/r/felixlohmeier/openrefine.
+
+Examples:
+./tests.sh -a              # run tests on all OpenRefine versions (from 2.0 up to 3.5.0)
+./tests.sh -t 3.5.0        # run tests on tag 3.5.0
+./tests.sh -t 3.5.0 -i     # run tests on tag 3.5.0 interactively (pause before and after tests)
+./tests.sh -t 3.5.0 -t 2.7 # run tests on tags 3.5.0 and 2.7
+
+Advanced:
+./tests.sh -j                # run tests on all OpenRefine versions and each with all supported Java versions (requires a lot of docker images to be downloaded!)
+./tests.sh -t 3.1 -i -p 3334 # run tests on tag 3.1 interactively on port 3334
+
+Running tests interactively (-i) allows you to examine OpenRefine GUI at http://localhost:3333.
+Execute the script concurrently in another terminal on another port (-p 3334) to compare changes in the OpenRefine GUI at http://localhost:3333 and http://localhost:3334.
+
+Available tags (java 8 if java not mentioned in tag):
+EOF
+    for t in ${all[*]} ; do
+        echo "$t"
+    done
+    exit 1
+}
+
+# check input
+NUMARGS=$#
+if [ "$NUMARGS" -eq 0 ]; then
+  usage
+fi
+
+# check system requirements
+DOCKER="$(command -v docker 2> /dev/null)"
+if [ -z "$DOCKER" ] ; then
+    echo 1>&2 "This action requires you to have 'docker' installed and present in your PATH. You can download it for free at http://www.docker.com/"
+    exit 1
+fi
+DOCKERINFO="$(docker info 2>/dev/null | grep 'Server Version')"
+if [ -z "$DOCKERINFO" ]
+then
+    echo "command 'docker info' failed, trying again with sudo..."
+    DOCKERINFO="$(sudo docker info 2>/dev/null | grep 'Server Version')"
+    echo "OK"
+    docker=(sudo docker)
+    if [ -z "$DOCKERINFO" ] ; then
+        echo 1>&2 "This action requires you to start the docker daemon. Try 'sudo systemctl start docker' or 'sudo start docker'. If the docker daemon is already running then maybe some security privileges are missing to run docker commands.'"
+        exit 1
+    fi
+else
+    docker=(docker)
+fi
+CURLINFO="$(command -v curl 2>/dev/null)"
+if [ -z "$CURLINFO" ] ; then
+    echo 1>&2 "This action requires you to have 'curl' installed and present in your PATH."
+    exit 1
+fi
+
+# get user input
+options="t:p:iajh"
+while getopts $options opt; do
+   case $opt in
+   t )  tags+=("${OPTARG}");;
+   p )  port="${OPTARG}";export OPENREFINE_PORT="$port";;
+   i )  interactively=true;;
+   a )  tags=("${main[*]}");;
+   j )  tags=("${all[*]}");;
+   h )  usage ;;
+   \? ) echo 1>&2 "Unknown option: -$OPTARG"; usage; exit 1;;
+   :  ) echo 1>&2 "Missing option argument for -$OPTARG"; usage; exit 1;;
+   *  ) echo 1>&2 "Unimplemented option: -$OPTARG"; usage; exit 1;;
+   esac
+done
+shift $((OPTIND - 1))
+
+# print config
+echo "Tags: ${tags[*]}"
+echo "Port: $port"
+echo ""
+
+# safe cleanup handler
+cleanup()
+{
+    echo "cleanup..."
+    ${docker[*]} stop "$t"
+}
+trap "cleanup;exit" SIGHUP SIGINT SIGQUIT SIGTERM
+
+# run setup.py tests for each docker tag
+for t in ${tags[*]} ; do
+    echo "=== Tests for $t ==="
+    echo ""
+    echo "Begin: $(date)"
+    ${docker[*]} run -d -p "$port":3333 --rm --name "$t" felixlohmeier/openrefine:"$t"
+    until curl --silent -N http://localhost:"$port" | cat | grep -q -o "Refine" ; do sleep 1; done
+    echo "Refine running at http://localhost:${port}"
+    if [ $interactively = true ]; then read -r -p "Press [Enter] key to start tests..."; fi
+    python2 setup.py test
+    if [ $interactively = true ]; then read -r -p "Press [Enter] key to stop OpenRefine..."; fi
+    ${docker[*]} stop "$t"
+    echo "End: $(date)"
+    echo ""
+done

tests/apply-utf8.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-addition",
+    "engineConfig": {
+      "mode": "row-based"
+    },
+    "newColumnName": "apply",
+    "columnInsertIndex": 2,
+    "baseColumnName": "b",
+    "expression": "grel:value.replace('2','⛲')",
+    "onError": "set-to-blank"
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	apply	c
+1	2	⛲	3
+0	0	0	0
+$	\	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/apply.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-addition",
+    "engineConfig": {
+      "mode": "row-based"
+    },
+    "newColumnName": "apply",
+    "columnInsertIndex": 2,
+    "baseColumnName": "b",
+    "expression": "grel:value.replace('2','TEST')",
+    "onError": "set-to-blank"
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	apply	c
+1	2	TEST	3
+0	0	0	0
+$	\	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-encoding.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}-utf8.csv"
+a,b,c
+1,2,3
+ä,é,ß
+$,\,'
+DATA
+iconv -f UTF-8 -t ISO-8859-1 "tmp/${t}/${t}-utf8.csv" > "tmp/${t}/${t}.csv"
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+ä	é	ß
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --encoding "ISO-8859-1"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-guessCellValueTypes.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+01,02,03
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+1	2	3
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --guessCellValueTypes "true"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-headerLines.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1	Column 2	Column 3
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --headerLines "0"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-ignoreLines.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --ignoreLines "1"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-limit.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --limit "2"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-processQuotes.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,"2,0",3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c	Column 4
+1	2	0	3
+0	0	0	
+$	\	'	
+DATA
+
+# ================================== ACTION ================================== #
+
+# OpenRefine 4.x fails without manually set headerLines
+${cmd} --create "tmp/${t}/${t}.csv" --processQuotes "false" --headerLines 1
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-projectTags.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version:0:1}" = "2" ]]; then
+  echo "projectTags were introduced in OpenRefine 3.0"
+  exit 200
+else
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+                tags: [u'beta', u'client1']
+DATA
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --projectTags "beta" --projectTags "client1"
+${cmd} --info "${t}" | grep ' tags: ' > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-separator.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a;b;c
+1;2;3
+0;0;0
+$;\;'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --separator ";"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-skipDataLines.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --skipDataLines "1"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-storeBlankCellsAsNulls.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,,0
+$,\,'
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "b",
+    "expression": "grel:isNull(value)",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	false	3
+0	false	0
+$	false	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --storeBlankCellsAsNulls "false"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-storeBlankRows.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+,,
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --storeBlankRows "false"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-utf8.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨	code	meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --projectName "${t} biểu tượng cảm xúc ⛲"
+${cmd} --export "${t} biểu tượng cảm xúc ⛲" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json-recordPath.sh

@@ -0,0 +1,55 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+{
+   "rows":[
+      {
+         "a":1,
+         "b":2,
+         "c":3
+      },
+      {
+         "a":0,
+         "b":0,
+         "c":0
+      },
+      {
+         "a":"$",
+         "b":"\\",
+         "c":"\""
+      }
+   ]
+}
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - a	_ - b	_ - c
+1	2	3
+0	0	0
+$	\	""""
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json" --recordPath "_" --recordPath "rows" --recordPath "_"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json-storeEmptyStrings.sh

@@ -0,0 +1,52 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+[
+  {
+    "a": 1,
+    "b": 2,
+    "c": 3
+  },
+  {
+    "a": "",
+    "b": "",
+    "c": ""
+  },
+  {
+    "a": "$",
+    "b": "\\",
+    "c": "\""
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - a	_ - b	_ - c
+1	2	3
+$	\	""""
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json" --storeEmptyStrings "false"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json-trimStrings.sh

@@ -0,0 +1,62 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+[
+  {
+    "a": 1,
+    "b": 2,
+    "c": 3
+  },
+  {
+    "a": "0",
+    "b": " 0",
+    "c": "0 "
+  },
+  {
+    "a": "$",
+    "b": "\\",
+    "c": "\""
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version:0:1}" = "2" || "${version}" = "3.0" || "${version}" = "3.1" || "${version}" = "3.2" || "${version}" = "3.3" ]]; then
+  echo "trimStrings option does not work in OpenRefine <=3.3"
+  echo "https://github.com/OpenRefine/OpenRefine/issues/2409"
+  exit 200
+else
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - a	_ - b	_ - c
+1	2	3
+0	0	0
+$	\	""""
+DATA
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json" --trimStrings "true"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json-utf8.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+[
+  {
+    "⌨": "⛲",
+    "code": "1F347",
+    "meaning": "FOUNTAIN"
+  },
+  {
+    "⌨": "⛳",
+    "code": "1F349",
+    "meaning": "FLAG IN HOLE"
+  },
+  {
+    "⌨": "⛵",
+    "code": "1F352",
+    "meaning": "SAILBOAT"
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - ⌨	_ - code	_ - meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+[
+  {
+    "a": 1,
+    "b": 2,
+    "c": 3
+  },
+  {
+    "a": 0,
+    "b": 0,
+    "c": 0
+  },
+  {
+    "a": "$",
+    "b": "\\",
+    "c": "\""
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - a	_ - b	_ - c
+1	2	3
+0	0	0
+$	\	""""
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-ods-sheets-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.ods" "tmp/${t}/${t}.ods"
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨	code	meaning	Column	Column 5	Column 6	Column 7	Column 8
+⛲	1F347	FOUNTAIN					
+⛳	1F349	FLAG IN HOLE					
+⛵	1F352	SAILBOAT					
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.ods" --sheets 1
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-ods.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.ods" "tmp/${t}/${t}.ods"
+#a	b	c
+#1	2	3
+#0	0	0
+#$	\	'
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c	Column	Column 5	Column 6	Column 7	Column 8
+1.0	2.0	3.0					
+0.0	0.0	0.0					
+$	\	'					
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.ods"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-tsv-utf8.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+⌨	code	meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.csv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.csv"

tests/create-tsv.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.csv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.csv"

tests/create-txt-fixed-width-headerLines.sh

@@ -0,0 +1,80 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+1     2     3
+mon   tue   wed
+$2    $300  $1
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "1",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "2",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "3",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+1	2	3
+mon	tue	wed
+$2	$300	$1
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --columnWidths "6" --columnWidths "6" --columnWidths "6" --headerLines "1"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-txt-fixed-width-utf8.sh

@@ -0,0 +1,81 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+⛲     1F347 FOUNTAIN
+⛳     1F349 FLAG IN HOLE
+⛵     1F352 SAILBOAT
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 1",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 2",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 3",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1	Column 2	Column 3
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --columnWidths "6" --columnWidths "6" --columnWidths "60"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-txt-fixed-width.sh

@@ -0,0 +1,81 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+1     2     3
+mon   tue   wed
+$2    $300  $1
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 1",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 2",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 3",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1	Column 2	Column 3
+1	2	3
+mon	tue	wed
+$2	$300	$1
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --columnWidths "6" --columnWidths "6" --columnWidths "6"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-txt-linesPerRow.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+mon tue wed
+$2 $300 $1
+thu fri sat
+$70 $20 $50
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1	Column 2
+mon tue wed	$2 $300 $1
+thu fri sat	$70 $20 $50
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --linesPerRow "2"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-txt.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+1 2 3
+mon tue wed
+$2 $300 $1
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1
+1 2 3
+mon tue wed
+$2 $300 $1
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xls-sheets-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.xls" "tmp/${t}/${t}.xls"
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨	code	meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xls" --sheets 1
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xls.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.xls" "tmp/${t}/${t}.xls"
+#a	b	c
+#1	2	3
+#0	0	0
+#$	\	'
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1.0	2.0	3.0
+0.0	0.0	0.0
+$	\	'
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xls"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xlsx-sheets-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.xlsx" "tmp/${t}/${t}.xlsx"
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨	code	meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xlsx" --sheets 1
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xlsx.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.xlsx" "tmp/${t}/${t}.xlsx"
+#a	b	c
+#1	2	3
+#0	0	0
+#$	\	'
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1.0	2.0	3.0
+0.0	0.0	0.0
+$	\	'
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xlsx"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xml-recordPath.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.xml"
+<?xml version="1.0" encoding="UTF-8"?>
+<root>
+  <record>
+    <a>1</a>
+    <b>2</b>
+    <c>3</c>
+  </record>
+  <record>
+    <a>0</a>
+    <b>0</b>
+    <c>0</c>
+  </record>
+  <record>
+    <a>$</a>
+    <b>\</b>
+    <c>'</c>
+  </record>
+</root>
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-reorder",
+    "columnNames": [
+      "record - a",
+      "record - b",
+      "record - c"
+    ],
+    "description": "Reorder columns"
+  },
+  {
+    "op": "core/row-removal",
+    "engineConfig": {
+      "facets": [
+        {
+          "type": "list",
+          "name": "Blank Rows",
+          "expression": "(filter(row.columnNames,cn,isNonBlank(cells[cn].value)).length()==0).toString()",
+          "columnName": "",
+          "invert": false,
+          "omitBlank": false,
+          "omitError": false,
+          "selection": [
+            {
+              "v": {
+                "v": "true",
+                "l": "true"
+              }
+            }
+          ],
+          "selectBlank": false,
+          "selectError": false
+        }
+      ],
+      "mode": "record-based"
+    }
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+record - a	record - b	record - c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xml" --recordPath "root" --recordPath "record"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xml-utf8.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.xml"
+<?xml version="1.0" encoding="UTF-8"?>
+<root>
+  <record>
+    <icon>⛲</icon>
+    <code>1F347</code>
+    <meaning>FOUNTAIN</meaning>
+  </record>
+  <record>
+    <icon>⛳</icon>
+    <code>1F349</code>
+    <meaning>FLAG IN HOLE</meaning>
+  </record>
+  <record>
+    <icon>⛵</icon>
+    <code>1F352</code>
+    <meaning>SAILBOAT</meaning>
+  </record>
+</root>
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-reorder",
+    "columnNames": [
+      "root - record - icon",
+      "root - record - code",
+      "root - record - meaning"
+    ],
+    "description": "Reorder columns"
+  },
+  {
+    "op": "core/row-removal",
+    "engineConfig": {
+      "facets": [
+        {
+          "type": "list",
+          "name": "Blank Rows",
+          "expression": "(filter(row.columnNames,cn,isNonBlank(cells[cn].value)).length()==0).toString()",
+          "columnName": "",
+          "invert": false,
+          "omitBlank": false,
+          "omitError": false,
+          "selection": [
+            {
+              "v": {
+                "v": "true",
+                "l": "true"
+              }
+            }
+          ],
+          "selectBlank": false,
+          "selectError": false
+        }
+      ],
+      "mode": "record-based"
+    }
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+root - record - icon	root - record - code	root - record - meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xml"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xml.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.xml"
+<?xml version="1.0" encoding="UTF-8"?>
+<root>
+  <record>
+    <a>1</a>
+    <b>2</b>
+    <c>3</c>
+  </record>
+  <record>
+    <a>0</a>
+    <b>0</b>
+    <c>0</c>
+  </record>
+  <record>
+    <a>$</a>
+    <b>\</b>
+    <c>'</c>
+  </record>
+</root>
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-reorder",
+    "columnNames": [
+      "root - record - a",
+      "root - record - b",
+      "root - record - c"
+    ],
+    "description": "Reorder columns"
+  },
+  {
+    "op": "core/row-removal",
+    "engineConfig": {
+      "facets": [
+        {
+          "type": "list",
+          "name": "Blank Rows",
+          "expression": "(filter(row.columnNames,cn,isNonBlank(cells[cn].value)).length()==0).toString()",
+          "columnName": "",
+          "invert": false,
+          "omitBlank": false,
+          "omitError": false,
+          "selection": [
+            {
+              "v": {
+                "v": "true",
+                "l": "true"
+              }
+            }
+          ],
+          "selectBlank": false,
+          "selectError": false
+        }
+      ],
+      "mode": "record-based"
+    }
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+root - record - a	root - record - b	root - record - c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xml"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-zip-includeFileSources.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}-1.csv"
+a,b,c
+1,2,3
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}-2.csv"
+a,b,c
+4,5,6
+DATA
+
+zip "tmp/${t}/${t}.zip" "tmp/${t}/${t}-1.csv" "tmp/${t}/${t}-2.csv"
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+File	a	b	c
+tmp/${t}/${t}-1.csv	1	2	3
+tmp/${t}/${t}-2.csv	4	5	6
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.zip" --includeFileSources "true"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-zip.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}-1.csv"
+a,b,c
+1,2,3
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}-2.csv"
+a,b,c
+4,5,6
+DATA
+
+zip "tmp/${t}/${t}.zip" "tmp/${t}/${t}-1.csv" "tmp/${t}/${t}-2.csv"
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+4	5	6
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.zip"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/data/cli/duplicates.csv

@@ -0,0 +1,11 @@
+email,name,state,gender,purchase,count,date
+danny.baron@example1.com,Danny Baron,CA,M,TV (UTF-8: 📺),1,"Wed, 4 Jul 2001"
+melanie.white@example2.edu,Melanie White,NC,F,<iPhone>,1,2001-07-04T12:08:56
+danny.baron@example1.com,   D.	("Tab") Baron,CA,M,Winter jacket,1,2001-07-04
+ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight,1,2001/07/04
+arthur.duff@example4.com,Arthur Duff,OR,M,Dining table,1,2001-07
+danny.baron@example1.com,Daniel Baron,,,Bike,1,2001
+jean.griffith@example5.org,Jean Griffith,WA,F,Power drill,1,2000
+melanie.white@example2.edu,Melanie White,NC,F,'iPad',1,1999
+ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier,1,1998
+arthur.duff@example4.com,Arthur Duff,OR,M,Night table,1,1997

tests/data/cli/duplicates.json

@@ -0,0 +1,92 @@
+[
+  {
+    "email": "danny.baron@example1.com",
+    "name": "Danny Baron",
+    "state": "CA",
+    "gender": "M",
+    "purchase": "TV (UTF-8: 📺)",
+    "count": 1,
+    "date": "Wed, 4 Jul 2001"
+  },
+  {
+    "email": "melanie.white@example2.edu",
+    "name": "Melanie White",
+    "state": "NC",
+    "gender": "F",
+    "purchase": "<iPhone>",
+    "count": 1,
+    "date": "2001-07-04T12:08:56"
+  },
+  {
+    "email": "danny.baron@example1.com",
+    "name": "   D.\t(\"Tab\") Baron",
+    "state": "CA",
+    "gender": "M",
+    "purchase": "Winter jacket",
+    "count": 1,
+    "date": "2001-07-04"
+  },
+  {
+    "email": "ben.tyler@example3.org",
+    "name": "Ben Tyler",
+    "state": "NV",
+    "gender": "M",
+    "purchase": "Flashlight",
+    "count": 1,
+    "date": "2001/07/04"
+  },
+  {
+    "email": "arthur.duff@example4.com",
+    "name": "Arthur Duff",
+    "state": "OR",
+    "gender": "M",
+    "purchase": "Dining table",
+    "count": 1,
+    "date": "2001-07"
+  },
+  {
+    "email": "danny.baron@example1.com",
+    "name": "Daniel Baron",
+    "state": "",
+    "gender": "",
+    "purchase": "Bike",
+    "count": 1,
+    "date": 2001
+  },
+  {
+    "email": "jean.griffith@example5.org",
+    "name": "Jean Griffith",
+    "state": "WA",
+    "gender": "F",
+    "purchase": "Power drill",
+    "count": 1,
+    "date": 2000
+  },
+  {
+    "email": "melanie.white@example2.edu",
+    "name": "Melanie White",
+    "state": "NC",
+    "gender": "F",
+    "purchase": "'iPad'",
+    "count": 1,
+    "date": 1999
+  },
+  {
+    "email": "ben.morisson@example6.org",
+    "name": "Ben Morisson",
+    "state": "FL",
+    "gender": "M",
+    "purchase": "Amplifier",
+    "count": 1,
+    "date": 1998
+  },
+  {
+    "email": "arthur.duff@example4.com",
+    "name": "Arthur Duff",
+    "state": "OR",
+    "gender": "M",
+    "purchase": "Night table",
+    "count": 1,
+    "date": 1997
+  }
+]

tests/data/cli/duplicates.tsv

@@ -0,0 +1,11 @@
+email	name	state	gender	purchase	count	date
+danny.baron@example1.com	Danny Baron	CA	M	TV (UTF-8: 📺)	1	Wed, 4 Jul 2001
+melanie.white@example2.edu	Melanie White	NC	F	<iPhone>	1	2001-07-04T12:08:56
+danny.baron@example1.com	"D.	(""Tab"") Baron"	CA	M	Winter jacket	1	2001-07-04
+ben.tyler@example3.org	Ben Tyler	NV	M	Flashlight	1	2001/07/04
+arthur.duff@example4.com	Arthur Duff	OR	M	Dining table	1	2001-07
+danny.baron@example1.com	Daniel Baron			Bike	1	2001
+jean.griffith@example5.org	Jean Griffith	WA	F	Power drill	1	2000
+melanie.white@example2.edu	Melanie White	NC	F	'iPad'	1	1999
+ben.morisson@example6.org	Ben Morisson	FL	M	Amplifier	1	1998
+arthur.duff@example4.com	Arthur Duff	OR	M	Night table	1	1997

tests/data/cli/duplicates.txt

@@ -0,0 +1,11 @@
+email                      name                 state gender purchase       count date              
+danny.baron@example1.com   Danny Baron          CA    M      TV (UTF-8: 📺) 1     Wed, 4 Jul 2001   
+melanie.white@example2.edu Melanie White        NC    F      <iPhone>       1     2001-07-04T12:08:5
+danny.baron@example1.com      D.	("Tab") Baron  CA    M      Winter jacket  1     2001-07-04        
+ben.tyler@example3.org     Ben Tyler            NV    M      Flashlight     1     2001/07/04        
+arthur.duff@example4.com   Arthur Duff          OR    M      Dining table   1     2001-07           
+danny.baron@example1.com   Daniel Baron                      Bike           1     2001              
+jean.griffith@example5.org Jean Griffith        WA    F      Power drill    1     2000              
+melanie.white@example2.edu Melanie White        NC    F      'iPad'         1     1999              
+ben.morisson@example6.org  Ben Morisson         FL    M      Amplifier      1     1998              
+arthur.duff@example4.com   Arthur Duff          OR    M      Night table    1     1997              

tests/data/cli/duplicates.xml

@@ -0,0 +1,93 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<root>
+  <record>
+    <email>danny.baron@example1.com</email>
+    <name>Danny Baron</name>
+    <state>CA</state>
+    <gender>M</gender>
+    <purchase>TV (UTF-8: 📺)</purchase>
+    <count>1</count>
+    <date>Wed, 4 Jul 2001</date>
+  </record>
+  <record>
+    <email>melanie.white@example2.edu</email>
+    <name>Melanie White</name>
+    <state>NC</state>
+    <gender>F</gender>
+    <purchase>&lt;iPhone&gt;</purchase>
+    <count>1</count>
+    <date>2001-07-04T12:08:56</date>
+  </record>
+  <record>
+    <email>danny.baron@example1.com</email>
+    <name>   D.	(&quot;Tab&quot;) Baron</name>
+    <state>CA</state>
+    <gender>M</gender>
+    <purchase>Winter jacket</purchase>
+    <count>1</count>
+    <date>2001-07-04</date>
+  </record>
+  <record>
+    <email>ben.tyler@example3.org</email>
+    <name>Ben Tyler</name>
+    <state>NV</state>
+    <gender>M</gender>
+    <purchase>Flashlight</purchase>
+    <count>1</count>
+    <date>2001/07/04</date>
+  </record>
+  <record>
+    <email>arthur.duff@example4.com</email>
+    <name>Arthur Duff</name>
+    <state>OR</state>
+    <gender>M</gender>
+    <purchase>Dining table</purchase>
+    <count>1</count>
+    <date>2001-07</date>
+  </record>
+  <record>
+    <email>danny.baron@example1.com</email>
+    <name>Daniel Baron</name>
+    <state></state>
+    <gender></gender>
+    <purchase>Bike</purchase>
+    <count>1</count>
+    <date>2001</date>
+  </record>
+  <record>
+    <email>jean.griffith@example5.org</email>
+    <name>Jean Griffith</name>
+    <state>WA</state>
+    <gender>F</gender>
+    <purchase>Power drill</purchase>
+    <count>1</count>
+    <date>2000</date>
+  </record>
+  <record>
+    <email>melanie.white@example2.edu</email>
+    <name>Melanie White</name>
+    <state>NC</state>
+    <gender>F</gender>
+    <purchase>&apos;iPad&apos;</purchase>
+    <count>1</count>
+    <date>1999</date>
+  </record>
+  <record>
+    <email>ben.morisson@example6.org</email>
+    <name>Ben Morisson</name>
+    <state>FL</state>
+    <gender>M</gender>
+    <purchase>Amplifier</purchase>
+    <count>1</count>
+    <date>1998</date>
+  </record>
+  <record>
+    <email>arthur.duff@example4.com</email>
+    <name>Arthur Duff</name>
+    <state>OR</state>
+    <gender>M</gender>
+    <purchase>Night table</purchase>
+    <count>1</count>
+    <date>1997</date>
+  </record>
+</root>

tests/data/cli/duplicates1.xml

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<record>
+  <email>danny.baron@example1.com</email>
+  <name>Danny Baron</name>
+  <state>CA</state>
+  <gender>M</gender>
+  <purchase>TV (UTF-8: 📺)</purchase>
+  <count>1</count>
+  <date>Wed, 4 Jul 2001</date>
+</record>

tests/data/cli/evil-fruits.tsv

@@ -0,0 +1,6 @@
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE

tests/data/duplicates-deletion.json

@@ -0,0 +1,69 @@
+[
+  {
+    "op": "core/row-reorder",
+    "description": "Reorder rows",
+    "mode": "record-based",
+    "sorting": {
+      "criteria": [
+        {
+          "errorPosition": 1,
+          "caseSensitive": false,
+          "valueType": "string",
+          "column": "email",
+          "blankPosition": 2,
+          "reverse": false
+        }
+      ]
+    }
+  },
+  {
+    "op": "core/column-addition",
+    "description": "Create column count at index 1 based on column email using expression grel:facetCount(value, \"value\", \"email\")",
+    "engineConfig": {
+      "mode": "row-based",
+      "facets": []
+    },
+    "newColumnName": "count",
+    "columnInsertIndex": 1,
+    "baseColumnName": "email",
+    "expression": "grel:facetCount(value, \"value\", \"email\")",
+    "onError": "set-to-blank"
+  },
+  {
+    "op": "core/blank-down",
+    "description": "Blank down cells in column email",
+    "engineConfig": {
+      "mode": "row-based",
+      "facets": []
+    },
+    "columnName": "email"
+  },
+  {
+    "op": "core/row-removal",
+    "description": "Remove rows",
+    "engineConfig": {
+      "mode": "row-based",
+      "facets": [
+        {
+          "omitError": false,
+          "expression": "isBlank(value)",
+          "selectBlank": false,
+          "selection": [
+            {
+              "v": {
+                "v": true,
+                "l": "true"
+              }
+            }
+          ],
+          "selectError": false,
+          "invert": false,
+          "name": "email",
+          "omitBlank": false,
+          "type": "list",
+          "columnName": "email"
+        }
+      ]
+    }
+  }
+]

tests/delete-utf8.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --projectName "${t} biểu tượng cảm xúc ⛲"
+${cmd} --list | grep "${t}" || exit 1
+${cmd} --delete "${t} biểu tượng cảm xúc ⛲"
+${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/delete.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --list | grep "${t}" || exit 1
+${cmd} --delete "${t}"
+${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/download.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# ================================== ACTION ================================== #
+
+${cmd} --download "https://git.io/fj5ju" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "data/duplicates-deletion.json" "tmp/${t}/${t}.output"

tests/export-csv-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --export "${t}" --output "tmp/${t}/${t} biểu tượng cảm xúc 🍉.csv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t} biểu tượng cảm xúc 🍉.csv"

tests/export-csv.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.csv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.csv"

tests/export-html-utf8.sh

@@ -0,0 +1,72 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  majorversion="${2%%.*}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+if [[ "$majorversion" = 2 ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+<html>
+<head>
+<title>export-html-utf8</title>
+<meta charset="utf-8" />
+</head>
+<body>
+<table>
+<tr><th>⌨</th><th>code</th><th>meaning</th></tr>
+<tr><td>&#9970;</td><td>1F347</td><td>FOUNTAIN</td></tr>
+<tr><td>&#9971;</td><td>1F349</td><td>FLAG IN HOLE</td></tr>
+<tr><td>&#9973;</td><td>1F352</td><td>SAILBOAT</td></tr>
+</table>
+</body>
+</html>
+DATA
+else
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+<html>
+<head>
+<title>export-html-utf8</title>
+<meta charset="utf-8" />
+</head>
+<body>
+<table>
+<tr><th>⌨</th><th>code</th><th>meaning</th></tr>
+<tr><td>⛲</td><td>1F347</td><td>FOUNTAIN</td></tr>
+<tr><td>⛳</td><td>1F349</td><td>FLAG IN HOLE</td></tr>
+<tr><td>⛵</td><td>1F352</td><td>SAILBOAT</td></tr>
+</table>
+</body>
+</html>
+DATA
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.html"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.html"

tests/export-html.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+<html>
+<head>
+<title>export-html</title>
+<meta charset="utf-8" />
+</head>
+<body>
+<table>
+<tr><th>a</th><th>b</th><th>c</th></tr>
+<tr><td>1</td><td>2</td><td>3</td></tr>
+<tr><td>0</td><td>0</td><td>0</td></tr>
+<tr><td>$</td><td>\</td><td>&apos;</td></tr>
+</table>
+</body>
+</html>
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.html"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.html"

tests/export-ods-utf8.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,"FLAG IN HOLE"
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.ods"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.ods" "${t}.csv" &&
+  mv "${t}.csv.1" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-ods.sh

@@ -0,0 +1,47 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ -z "$(command -v ssconvert 2> /dev/null)" ]] ; then
+  echo 1>&2 "ERROR: This test requires ssconvert (gnumeric)"
+  exit 127
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.ods"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.ods" "${t}.csv" &&
+  mv "${t}.csv.1" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-tsv-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.tsv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.tsv"

tests/export-tsv.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.tsv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.tsv"

tests/export-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-xls-utf8.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.xls"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.xls" "${t}.csv" &&
+  mv "${t}.csv" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-xls.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.xls"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.xls" "${t}.csv" &&
+  mv "${t}.csv" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-xlsx-utf8.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.xlsx"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.xlsx" "${t}.csv" &&
+  mv "${t}.csv" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
+
+

tests/export-xlsx.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.xlsx"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.xlsx" "${t}.csv" &&
+  mv "${t}.csv" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/format-create-separator.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+a;b;c
+1;2;3
+0;0;0
+$;\;'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --format "csv"  --separator ";"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/format-create.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --format "line-based"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/format-export-output.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --format "csv" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/format-export.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --format "csv" > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/help.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Script to provide a command line interface to an OpenRefine server.
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --help | sed '3q;d' > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/info-utf8.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+          column 001: 🔣
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --info "${t}" | grep 'column 001' > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/info.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+          column 002: b
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --info "${t}" | grep 'column 002' > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/list-utf8.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+ ${t} biểu tượng cảm xúc ⛲
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --projectName "${t} biểu tượng cảm xúc ⛲"
+${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/list.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+ ${t}
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/template-facets.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+email,name,state,gender,purchase
+danny.baron@example1.com,Danny Baron,CA,M,TV
+melanie.white@example2.edu,Melanie White,NC,F,iPhone
+danny.baron@example1.com,D. Baron,CA,M,Winter jacket
+ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
+arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
+danny.baron@example1.com,Daniel Baron,CA,M,Bike
+jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
+melanie.white@example2.edu,Melanie White,NC,F,iPad
+ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
+arthur.duff@example4.com,Arthur Duff,OR,M,Night table
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "events" : [
+    { "name" : "Melanie White", "purchase" : "iPhone" },
+    { "name" : "Jean Griffith", "purchase" : "Power drill" },
+    { "name" : "Melanie White", "purchase" : "iPad" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "events" : [
+' \
+--template '    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--facets '{"type":"list","name":"gender","columnName":"gender","expression":"value","omitBlank":false,"omitError":false,"selection":[{"v":{"v":"F","l":"F"}}],"selectBlank":false,"selectError":false,"invert":false}' \
+--output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/template-filterQuery-utf8.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "emojis" : [
+    { "symbol" : "🍇", "meaning" : "GRAPES" },
+    { "symbol" : "🍉", "meaning" : "WATERMELON" },
+    { "symbol" : "🍍", "meaning" : "PINEAPPLE" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "emojis" : [
+' \
+--template '    { "symbol" : {{jsonize(with(row.columnNames[0],cn,cells[cn].value))}}, "meaning" : {{jsonize(cells["meaning"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--filterQuery '^1F34' \
+--filterColumn 'code' \
+--output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/template-filterQuery.sh

@@ -0,0 +1,59 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+email,name,state,gender,purchase
+danny.baron@example1.com,Danny Baron,CA,M,TV
+melanie.white@example2.edu,Melanie White,NC,F,iPhone
+danny.baron@example1.com,D. Baron,CA,M,Winter jacket
+ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
+arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
+danny.baron@example1.com,Daniel Baron,CA,M,Bike
+jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
+melanie.white@example2.edu,Melanie White,NC,F,iPad
+ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
+arthur.duff@example4.com,Arthur Duff,OR,M,Night table
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "events" : [
+    { "name" : "Melanie White", "purchase" : "iPhone" },
+    { "name" : "Jean Griffith", "purchase" : "Power drill" },
+    { "name" : "Melanie White", "purchase" : "iPad" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "events" : [
+' \
+--template '    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--filterQuery '^F$' \
+--filterColumn 'gender' \
+--output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/template-splitToFiles-mode.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+email,name,state,gender,purchase
+arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
+,Arthur Duff,OR,M,Night table
+ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
+ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
+danny.baron@example1.com,Daniel Baron,CA,M,Bike
+,Danny Baron,CA,M,TV
+,D. Baron,CA,M,Winter jacket
+jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
+melanie.white@example2.edu,Melanie White,NC,F,iPad
+,Melanie White,NC,F,iPhone
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "events" : [
+    { "name" : "Melanie White", "purchase" : "iPad" }    { "name" : "Melanie White", "purchase" : "iPhone" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "events" : [
+' \
+--template '    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--mode "record-based" \
+--splitToFiles true \
+--output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+ls "tmp/${t}"
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}_6.output"