Merge pull request #21 from mohammed78620/master

moved csrf token check into urlopen_json

.gitignore

@@ -4,4 +4,6 @@ dist
 .*
 openrefine_client.egg-info
 refine.spec
-README.html
+openrefine-*
+openrefine-client_*
+tests-cli.log

README.md

@@ -1,18 +1,20 @@
-# OpenRefine Python Client with extended command line interface
+# OpenRefine Python Client with extended command line interface (⌨️ for 💎)
 
-[![Codacy Badge](https://api.codacy.com/project/badge/Grade/33129bd15cdc4ece88c8012caab8d347)](https://www.codacy.com/app/felixlohmeier/openrefine-client?utm_source=github.com&utm_medium=referral&utm_content=opencultureconsulting/openrefine-client&utm_campaign=Badge_Grade) [![Docker](https://img.shields.io/microbadger/image-size/felixlohmeier/openrefine-client?label=docker)](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [![PyPI](https://img.shields.io/pypi/v/openrefine-client)](https://pypi.org/project/openrefine-client/) [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/betatim/openrefineder/22fbb07?filepath=openrefine-client.ipynb)
+[![Codacy Badge](https://app.codacy.com/project/badge/Grade/43ad9bfd707b4627bd45e5c5f912a8e0)](https://www.codacy.com/gh/opencultureconsulting/openrefine-client/dashboard) [![Docker](https://img.shields.io/microbadger/image-size/felixlohmeier/openrefine-client?label=docker)](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [![PyPI](https://img.shields.io/pypi/v/openrefine-client)](https://pypi.org/project/openrefine-client/) [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master)
 
 The [OpenRefine Python Client from PaulMakepeace](https://github.com/PaulMakepeace/refine-client-py) provides a library for communicating with an [OpenRefine](http://openrefine.org) server.
 This fork extends the command line interface (CLI) and is distributed as a convenient one-file-executable (Windows, Linux, macOS).
 It is also available via Docker Hub, PyPI and Binder.
 
+works with OpenRefine 2.7, 2.8, 3.0, 3.1, 3.2, 3.3, 3.4, 3.4.1, 3.5.0
+
 ## Download
 
 One-file-executables:
 
-- Windows: [openrefine-client_0-3-7_windows.exe](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.7/openrefine-client_0-3-7_windows.exe) (~5 MB)
-- macOS: [openrefine-client_0-3-7_macos](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.7/openrefine-client_0-3-7_macos) (~5 MB)
-- Linux: [openrefine-client_0-3-7_linux](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.7/openrefine-client_0-3-7_linux) (~5 MB)
+- Windows: [openrefine-client_0-3-10_windows.exe](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_windows.exe) (~5 MB)
+- macOS: [openrefine-client_0-3-10_macos](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_macos) (~5 MB)
+- Linux: [openrefine-client_0-3-10_linux](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_linux) (~5 MB)
 
 For [Docker](#docker) containers, native [Python](#python) installation and free [Binder](#binder) on-demand server see the corresponding chapters below.
 
@@ -32,19 +34,19 @@ To use the client:
 
    - Windows: Open PowerShell and enter following command
 
-     ```
+     ```sh
      cd ~\Downloads
      ```
 
    - macOS: Open Terminal (Finder > Applications > Utilities > Terminal) and enter following command
 
-     ```
+     ```sh
      cd ~/Downloads
      ```
 
    - Linux: Open terminal app (Terminal, Konsole, xterm, ...) and enter following command
 
-     ```
+     ```sh
      cd ~/Downloads
      ```
 
@@ -54,34 +56,34 @@ To use the client:
 
    - macOS:
 
-     ```
-     chmod +x openrefine-client_0-3-7_macos
+     ```sh
+     chmod +x openrefine-client_0-3-10_macos
      ```
 
    - Linux:
 
-     ```
-     chmod +x openrefine-client_0-3-7_linux
+     ```sh
+     chmod +x openrefine-client_0-3-10_linux
      ```
 
 3. Execute the file.
 
    - Windows:
 
-     ```
-     .\openrefine-client_0-3-7_windows.exe
+     ```sh
+     .\openrefine-client_0-3-10_windows.exe
      ```
 
    - macOS:
 
-     ```
-     ./openrefine-client_0-3-7_macos
+     ```sh
+     ./openrefine-client_0-3-10_macos
      ```
 
    - Linux:
 
-     ```
-     ./openrefine-client_0-3-7_linux
+     ```sh
+     ./openrefine-client_0-3-10_linux
      ```
 
 Using tab completion and command history is highly recommended:
@@ -99,26 +101,26 @@ Download example data (`--download`) and create project from file (`--create`):
 
 - Windows:
 
-  ```
-  .\openrefine-client_0-3-7_windows.exe --download "https://git.io/fj5hF" --output=duplicates.csv
-  .\openrefine-client_0-3-7_windows.exe --download "https://git.io/fj5ju" --output=duplicates-deletion.json
-  .\openrefine-client_0-3-7_windows.exe --create duplicates.csv
+  ```sh
+  .\openrefine-client_0-3-10_windows.exe --download "https://git.io/fj5hF" --output=duplicates.csv
+  .\openrefine-client_0-3-10_windows.exe --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+  .\openrefine-client_0-3-10_windows.exe --create duplicates.csv
   ```
 
 - macOS:
 
-  ```
-  ./openrefine-client_0-3-7_macos --download "https://git.io/fj5hF" --output=duplicates.csv
-  ./openrefine-client_0-3-7_macos --download "https://git.io/fj5ju" --output=duplicates-deletion.json
-  ./openrefine-client_0-3-7_macos --create duplicates.csv
+  ```sh
+  ./openrefine-client_0-3-10_macos --download "https://git.io/fj5hF" --output=duplicates.csv
+  ./openrefine-client_0-3-10_macos --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+  ./openrefine-client_0-3-10_macos --create duplicates.csv
   ```
 
 - Linux:
 
-  ```
-  ./openrefine-client_0-3-7_linux --download "https://git.io/fj5hF" --output=duplicates.csv
-  ./openrefine-client_0-3-7_linux --download "https://git.io/fj5ju" --output=duplicates-deletion.json
-  ./openrefine-client_0-3-7_linux --create duplicates.csv
+  ```sh
+  ./openrefine-client_0-3-10_linux --download "https://git.io/fj5hF" --output=duplicates.csv
+  ./openrefine-client_0-3-10_linux --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+  ./openrefine-client_0-3-10_linux --create duplicates.csv
   ```
 
 Other commands:
@@ -145,7 +147,7 @@ If your OpenRefine server is running somewhere else then you may set hostname an
 - set host: `-H example.com`
 - set port: `-P 80`
 
-### Advanced Templating
+### Templating
 
 The OpenRefine [Templating](https://github.com/OpenRefine/OpenRefine/wiki/Export-As-YAML) supports exporting data in any text format (i.e. to construct JSON or XML).
 The graphical user interface offers four input fields:
@@ -161,20 +163,20 @@ It even provides an additional feature for splitting results into multiple files
 
 To try out the functionality create another project from the example file above.
 
-```
+```sh
 --create duplicates.csv --projectName=advanced
 ```
 
 The following example code will export...
 
 - the columns "name" and "purchase" in JSON format
-- from the project "duplicates"
-- for rows matching the regex text filter `^CA$` in column "state"
+- from the project "advanced"
+- for rows matching the regex text filter `^F$` in column "gender"
 
 macOS/Linux Terminal (multi-line input with `\` ):
 
-```
---export "advanced" \
+```sh
+"advanced" \
 --prefix='{ "events" : [
 ' \
 --template='    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
@@ -182,14 +184,14 @@ macOS/Linux Terminal (multi-line input with `\` ):
 ' \
 --suffix='
 ] }' \
---filterQuery='^CA$' \
---filterColumn='state'
+--filterQuery='^F$' \
+--filterColumn='gender'
 ```
 
 Windows PowerShell (multi-line input with `` ` ``; quotes needs to be doubled):
 
-```
---export "advanced" `
+```sh
+"advanced" `
 --prefix='{ ""events"" : [
 ' `
 --template='    { ""name"" : {{jsonize(cells[""name""].value)}}, ""purchase"" : {{jsonize(cells[""purchase""].value)}} }' `
@@ -197,31 +199,56 @@ Windows PowerShell (multi-line input with `` ` ``; quotes needs to be doubled):
 ' `
 --suffix='
 ] }' `
---filterQuery='^CA$' `
---filterColumn='state'
+--filterQuery='^F$' `
+--filterColumn='gender'
 ```
 
 Add the following options to the last command (recall with `↑`) to store the results in multiple files.
 Each file will contain the prefix, an processed row, and the suffix.
 
-```
+```sh
 --output=advanced.json --splitToFiles=true
 ```
 
 Filenames are suffixed with the row number by default (e.g. `advanced_1.json`, `advanced_2.json` etc.).
 There is another option to use the value in the first column instead:
 
-```
+```sh
 --output=advanced.json --splitToFiles=true --suffixById=true
 ```
 
-Because our project "advanced" contains duplicates in the first column "email" this command will store only one file `advanced_danny.baron@example1.com.json`.
+Because our project "advanced" contains duplicates in the first column "email" this command will overwrite files (e.g. `advanced_melanie.white@example2.edu.json`).
 When using this option, the first column should contain unique identifiers.
 
+### Append data to an existing project
+
+OpenRefine does not support appending rows to an existing project.
+As long as the [feature request](https://github.com/OpenRefine/OpenRefine/issues/715) is not yet implemented, you can use the openrefine-client to script a workaround:
+
+1. export existing project as csv
+2. put old and new data into a zip archive
+3. create new project by importing the zip archive
+
+Here is an example that replaces the existing project:
+
+```
+openrefine-client --export myproject --output old.csv
+openrefine-client --delete myproject
+zip combined.zip old.csv new.csv
+openrefine-client --create combined.zip --format csv --projectName myproject
+```
+
+Note that the project id will change.
+If you want to distinguish between old and new data, you can use the additional flag includeFileSources:
+
+```
+openrefine-client --create combined.zip --format csv --projectName myproject --includeFileSources true
+```
+
 ### See also
 
 - Linux Bash script to run OpenRefine in batch mode (import, transform, export): [openrefine-batch](https://github.com/opencultureconsulting/openrefine-batch)
-- [Jupyter notebook demonstrating usage in Linux Bash](https://nbviewer.jupyter.org/github/betatim/openrefineder/blob/master/openrefine-client.ipynb)
+- [Jupyter notebook demonstrating usage in Linux Bash](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/notebooks/openrefine-client-bash.ipynb)
 - Use case [HOS-MetadataTransformations](https://github.com/subhh/HOS-MetadataTransformations): Automated workflow for harvesting, transforming and indexing of metadata using metha, OpenRefine and Solr. Part of the Hamburg Open Science "Schaufenster" software stack.
 - Use case [Data processing of ILS data to facilitate a new discovery layer for the German Literature Archive (DLA)](https://doi.org/10.5281/zenodo.2678113): Custom data processing pipeline based on Pandas (a Python library) and OpenRefine.
 
@@ -229,30 +256,30 @@ When using this option, the first column should contain unique identifiers.
 
 [felixlohmeier/openrefine-client](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [![Docker](https://img.shields.io/microbadger/image-size/felixlohmeier/openrefine-client?label=docker)](https://hub.docker.com/r/felixlohmeier/openrefine-client/)
 
-```
-docker pull felixlohmeier/openrefine-client:v0.3.7
+```sh
+docker pull felixlohmeier/openrefine-client:v0.3.10
 ```
 
 ### Option 1: Dockerized client
 
 Run client and mount current directory as workspace:
 
-```
-docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.7
+```sh
+docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10
 ```
 
 The docker option `--network=host` allows you to connect to a local or remote OpenRefine via the host network:
 
 - list projects on default URL (http://localhost:3333)
 
-  ```
-  docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.7 --list
+  ```sh
+  docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --list
   ```
 
 - list projects on a remote server (http://example.com)
 
-  ```
-  docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.7 -H example.com -P 80 --list
+  ```sh
+  docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H example.com -P 80 --list
   ```
 
 Usage: same commands as explained above (see [Basic Commands](#basic-commands) and [Advanced Templating](#advanced-templating))
@@ -263,41 +290,41 @@ Run openrefine-client linked to a dockerized OpenRefine ([felixlohmeier/openrefi
 
 1. Create docker network
 
-   ```
+   ```sh
    docker network create openrefine
    ```
 
 2. Run server (will be available at http://localhost:3333)
 
-   ```
-   docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.2
+   ```sh
+   docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.5.0
    ```
 
 3. Run client with some [basic commands](#basic-commands): 1. download example files, 2. create project from file, 3. list projects, 4. show metadata, 5. export to terminal, 6. apply transformation rules (deduplication), 7. export again to terminal, 8. export to xls file and 9. delete project
 
-   ```
-   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.7 --download "https://git.io/fj5hF" --output=duplicates.csv
-   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.7 --download "https://git.io/fj5ju" --output=duplicates-deletion.json
-   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.7 -H openrefine-server --create duplicates.csv
-   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.7 -H openrefine-server --list
-   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.7 -H openrefine-server --info "duplicates"
-   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.7 -H openrefine-server --export "duplicates"
-   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.7 -H openrefine-server --apply duplicates-deletion.json "duplicates"
-   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.7 -H openrefine-server --export "duplicates"
-   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.7 -H openrefine-server --export --output=deduped.xls "duplicates"
-   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.7 -H openrefine-server --delete "duplicates"
+   ```sh
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --download "https://git.io/fj5hF" --output=duplicates.csv
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --download "https://git.io/fj5ju" --output=duplicates-deletion.json
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --create duplicates.csv
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --list
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --info "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --apply duplicates-deletion.json "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export --output=deduped.xls "duplicates"
+   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --delete "duplicates"
    ```
 
 4. Stop and delete server:
 
-   ```
+   ```sh
    docker stop openrefine-server
    docker rm openrefine-server
    ```
 
 5. Delete docker network:
 
-   ```
+   ```sh
    docker network rm openrefine
    ```
 
@@ -309,21 +336,21 @@ Customize OpenRefine server:
 
 - Example for [allocating more memory](https://github.com/OpenRefine/OpenRefine/wiki/FAQ#out-of-memory-errors---feels-slow---could-not-reserve-enough-space-for-object-heap) to OpenRefine with additional option `-m 4G`
 
-  ```
-  docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.2 -i 0.0.0.0 -d /data -m 4G
+  ```sh
+  docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.5.0 -i 0.0.0.0 -d /data -m 4G
   ```
 
 - The OpenRefine version is defined by the docker tag.
   Check the [DockerHub repository](https://hub.docker.com/r/felixlohmeier/openrefine) for available tags.
   Example for OpenRefine `2.8` with same options as above:
 
-  ```
+  ```sh
   docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:2.8 -i 0.0.0.0 -d /data -m 4G
   ```
 
 - If you want OpenRefine to read and write persistent data in host directory (i.e. store projects) you can mount the container path `/data`. Example for host directory `/home/felix/refine`:
 
-  ```
+  ```sh
   docker run -d -p 3333:3333 -v /home/felix/refine:/data:z --network=openrefine name=openrefine-server felixlohmeier/openrefine:2.8 -i 0.0.0.0 -d /data -m 4G
   ```
 
@@ -336,8 +363,8 @@ See also:
 
 [openrefine-client](https://pypi.org/project/openrefine-client/) [![PyPI](https://img.shields.io/pypi/v/openrefine-client)](https://pypi.org/project/openrefine-client/) (requires Python 2.x)
 
-```
-pip install openrefine-client
+```sh
+python2 -m pip install openrefine-client --user
 ```
 
 This will install the package `openrefine-client` containing modules in `google.refine`.
@@ -346,7 +373,7 @@ A command line script `openrefine-client` will also be installed.
 
 ### Option 1: command line script
 
-```
+```sh
 openrefine-client --help
 ```
 
@@ -356,20 +383,20 @@ Usage: same commands as explained above (see [Basic Commands](#basic-commands) a
 
 Import module cli:
 
-```
+```python
 from google.refine import cli
 ```
 
 Change URL (if necessary):
 
-```
-refine.REFINE_HOST = 'localhost'
-refine.REFINE_PORT = 3333
+```python
+cli.refine.REFINE_HOST = 'localhost'
+cli.refine.REFINE_PORT = '3333'
 ```
 
 Help screen:
 
-```
+```python
 help(cli)
 ```
 
@@ -377,59 +404,62 @@ Commands:
 
 * download (e.g. example data):
 
-  ```
+  ```python
   cli.download('https://git.io/fj5hF','duplicates.csv')
   cli.download('https://git.io/fj5ju','duplicates-deletion.json')
   ```
 
 * list projects:
 
-  ```
+  ```python
   cli.ls()
   ```
 
 * create project:
 
-  ```
+  ```python
   p1 = cli.create('duplicates.csv')
   ```
 
 * show metadata:
 
-  ```
+  ```python
   cli.info(p1.project_id)
   ```
 
 * apply rules from file to project:
 
-  ```
+  ```python
   cli.apply(p1.project_id, 'duplicates-deletion.json')
   ```
 
 * export project to terminal:
 
-  ```
+  ```python
   cli.export(p1.project_id)
   ```
 
 * export project to file in xls format:
 
-  ```
+  ```python
   cli.export(p1.project_id, 'deduped.xls')
   ```
 
 * export templating (see [Advanced Templating](#advanced-templating) above):
 
-  ```
-  cli.templating(p1.project_id, prefix='''{ "events" : [
-  ''', template='''    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }''', rowSeparator=''',
-  ''', suffix='''
+  ```python
+  cli.templating(
+      p1.project_id,
+      prefix='''{ "events" : [
+  ''',template='''    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }''',
+      rowSeparator=''',
+  ''',suffix='''
   ] }''')
   ```
 
 * delete project:
 
-  ```
+  ```python
   cli.delete(p1.project_id)
   ```
 
@@ -441,7 +471,7 @@ Some functions in the python client library are not yet compatible with OpenRefi
 
 Import module refine:
 
-```
+```python
 from google.refine import refine
 ```
 
@@ -449,39 +479,39 @@ Server Commands:
 
 * set up connection:
 
-  ```
+  ```python
   server1 = refine.Refine('http://localhost:3333')
   ```
 
 - show version:
 
-  ```
+  ```python
   server1.server.get_version()
   server1.server.version
   ```
 
 - list projects:
 
-  ```
+  ```python
   server1.list_projects()
   ```
 
   - pretty print the returned dict with json.dumps:
 
-    ```
+    ```python
     import json
     print(json.dumps(server1.list_projects(), indent=1))
     ```
 
-- create project (**function was edited in this fork**):
+- create project:
 
-  ```
+  ```python
   server1.new_project(project_file='duplicates.csv')
   ```
 
   * create and open the returned project in one step:
 
-    ```
+    ```python
     project1 = server1.new_project(project_file='duplicates.csv')
     ```
 
@@ -489,31 +519,31 @@ Project commands:
 
 * open project:
 
-  ```
+  ```python
   project1 = server1.open_project('1234567890123')
   ```
 
 * print full URL to project:
 
-  ```
+  ```python
   project1.project_url()
   ```
 
 * list columns:
 
-  ```
+  ```python
   project1.columns
   ```
 
 * compute text facet on first column (**fails with OpenRefine >=3.2**):
 
-  ```
+  ```python
   project1.compute_facets(facet.TextFacet(project1.columns[0]))
   ```
 
   * print returned object
 
-    ```
+    ```python
     facets = project1.compute_facets(facet.TextFacet(project1.columns[0])).facets[0]
     for k in sorted(facets.choices, key=lambda k: facets.choices[k].count, reverse=True):
         print(facets.choices[k].count, k)
@@ -521,60 +551,62 @@ Project commands:
 
 * compute clusters on first column:
 
-  ```
+  ```python
   project1.compute_clusters(project1.columns[0])
   ```
 
 * apply rules from file to project:
 
-  ```
+  ```python
   project1.apply_operations('duplicates-deletion.json')
   ```
 
 * export project:
 
-  ```
+  ```python
   project1.export(export_format='tsv')
   ```
 
   * print the returned fileobject:
 
-    ```
+    ```python
     print(project1.export(export_format='tsv').read())
     ```
 
   * save the returned fileobject to file:
 
-    ```
+    ```python
     with open('export.tsv', 'wb') as f:
         f.write(project1.export(export_format='tsv').read())
     ```
 
 * templating export (**function was added in this fork**, see [Advanced Templating](#advanced-templating) above):
 
-  ```
-  data = project1.export_templating(prefix='''{ "events" : [
-  ''', template='''    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }''', rowSeparator=''',
-  ''', suffix='''
+  ```python
+  data = project1.export_templating(
+      prefix='''{ "events" : [
+  ''',template='''    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }''',
+      rowSeparator=''',
+  ''',suffix='''
   ] }''')
   print(data.read())
   ```
 
 * print help screen with available commands (many more!):
 
-  ```
+  ```python
   help(project1)
   ```
 
 * example for custom commands:
 
-  ```
+  ```python
   project1.do_json('get-rows')['total']
   ```
 
 * delete project:
 
-  ```
+  ```python
   project1.delete()
   ```
 
@@ -587,12 +619,13 @@ See also:
 
 ## Binder
 
-[openrefineder](https://github.com/betatim/openrefineder) [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/betatim/openrefineder/22fbb07?filepath=openrefine-client.ipynb)
+[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master)
 
 - free to use on-demand server with Jupyter notebook, OpenRefine and Bash
 - no registration needed, will start within a few minutes
 - [restricted](https://mybinder.readthedocs.io/en/latest/faq.html#how-much-memory-am-i-given-when-using-binder) to 2 GB RAM and server will be deleted after 10 minutes of inactivity
-- includes [demo notebook](https://nbviewer.jupyter.org/github/betatim/openrefineder/blob/master/openrefine-client.ipynb) for using openrefine-client with Linux Bash
+- [bash_kernel demo notebook](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/notebooks/openrefine-client-bash.ipynb) for using the openrefine-client in a Linux Bash environment [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master?urlpath=/tree/notebooks/openrefine-client-bash.ipynb)
+- [python2 demo notebook](https://nbviewer.jupyter.org/github/felixlohmeier/openrefineder/blob/master/notebooks/openrefine-client-python.ipynb) for using the openrefine-client in a Python 2 environment [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/felixlohmeier/openrefineder/master?urlpath=/tree/notebooks/openrefine-client-python.ipynb)
 
 ## Development
 
@@ -606,55 +639,72 @@ The Python client library includes several unit tests.
 
 - run all tests
 
-  ```
+  ```sh
   python setup.py test
   ```
 
 - run subset test_facet
 
-  ```
+  ```sh
   python setup.py --test-suite tests.test_facet
   ```
 
 There is also a script that uses docker images to run the unit tests with different versions of OpenRefine.
 
-- run tests on all OpenRefine versions (from 2.0 up to 3.2)
+- run tests on all OpenRefine versions (from 2.0 up to 3.5.0)
 
-  ```
+  ```sh
   ./tests.sh -a
   ```
 
-- run tests on tag 3.2
+- run tests on tag 3.5.0
 
-  ```
-  ./tests.sh -t 3.2
+  ```sh
+  ./tests.sh -t 3.5.0
   ```
 
-- run tests on tag 3.2 interactively (pause before and after tests)
+- run tests on tag 3.5.0 interactively (pause before and after tests)
 
-  ```
-  ./tests.sh -t 3.2 -i
+  ```sh
+  ./tests.sh -t 3.5.0 -i
   ```
 
-- run tests on tags 3.2 and 2.7
+- run tests on tags 3.5.0 and 2.7
 
+  ```sh
+  ./tests.sh -t 3.5.0 -t 2.7
   ```
-  ./tests.sh -t 3.2 -t 2.7
+
+For Linux there are also functional tests for all command line options.
+
+- run all functional tests on OpenRefine 3.5.0
+
+  ```sh
+  ./tests-cli.sh 3.5.0
+  ```
+
+- run all functional tests on OpenRefine 3.5.0 with one-file-executable
+
+  ```sh
+  ./tests-cli.sh 3.5.0 openrefine-client_0-3-7_linux
   ```
 
 ### Distributing
 
 Note to myself: When releasing a new version...
 
-1. Run tests
+1. Run functional tests
 
-   ```
-   ./tests.sh -a
+   ```sh
+   for v in 2.7 2.8 3.0 3.1 3.2 3.3 3.4 3.4.1 3.5.0; do
+      ./tests-cli.sh $v
+   done
    ```
 
-2. Make final changes in GitHub
+2. Make final changes in Git
 
-   - update versions and download links (guess in advance) in [README.md](https://github.com/opencultureconsulting/openrefine-client/blob/master/README.md#download)
+   - update versions (e.g. 0.3.7 und 0-3-7) in [README.md](https://github.com/opencultureconsulting/openrefine-client/blob/master/README.md#download)
+   - update version in [setup.py](https://github.com/opencultureconsulting/openrefine-client/blob/master/setup.py)
    - check if [Dockerfile](https://github.com/opencultureconsulting/openrefine-client/blob/master/docker/Dockerfile) needs to be changed
 
 3. Build executables with PyInstaller
@@ -662,40 +712,47 @@ Note to myself: When releasing a new version...
    - Run PyInstaller in Python 2 environments on native Windows, macOS and Linux. Should be "the oldest version of the OS you need to support"! Current release is built with: 
 
      - Ubuntu 16.04 LTS (64-bit)
-     - macOS Sierra 10.12
-     - Windows 10
+     - macOS Sierra 10.12 (64-bit)
+     - Windows 7 (32-bit)
 
    - One-file-executables will be available in `dist/`.
 
-     ```
+     ```sh
      git clone https://github.com/opencultureconsulting/openrefine-client.git
      cd openrefine-client
-     python -m pip install . --user
-     python -m pip install pyinstaller --user
-     pyinstaller --onefile refine.py --hidden-import google.refine.__main__
+     python2 -m pip install pyinstaller --user
+     python2 -m pip install urllib2_file --user
+     python2 -m PyInstaller --onefile refine.py --hidden-import google.refine.__main__
      ```
 
-4. Create release in GitHub
+4. Run functional tests with Linux executable
+
+   ```sh
+   for v in 2.7 2.8 3.0 3.1 3.2 3.3 3.4 3.4.1 3.5.0; do
+      ./tests-cli.sh $v openrefine-client_0-3-7_linux
+   done
+   ```
+
+5. Create release in GitHub
 
    - draft [release notes](https://github.com/opencultureconsulting/openrefine-client/releases) and attach one-file-executables
 
-5. Build package and upload to PyPI
+6. Build package and upload to PyPI
 
-   ```
+   ```sh
    python3 setup.py sdist bdist_wheel
    python3 -m twine upload dist/*
    ```
 
-6. Update Docker container
+7. Update Docker container
 
    - add new autobuild for release version
    - trigger latest build
 
-7. Bump openrefine-client version in related projects
+8. Bump openrefine-client version in related projects
 
    - openrefine-batch: [openrefine-batch.sh](https://github.com/opencultureconsulting/openrefine-batch/blob/master/openrefine-batch.sh#L7) and [openrefine-batch-docker.sh](https://github.com/opencultureconsulting/openrefine-batch/blob/master/openrefine-batch-docker.sh)
-
-   - openrefineder: [postBuild](https://github.com/betatim/openrefineder/blob/master/postBuild)
+   - openrefineder: [postBuild](https://github.com/felixlohmeier/openrefineder/blob/master/binder/postBuild)
 
 ## Credits
 

docker/Dockerfile

@@ -1,4 +1,4 @@
-FROM alpine:latest
+FROM alpine:3.11
 LABEL maintainer="felixlohmeier@opencultureconsulting.com"
 # The OpenRefine Python Client Library from PaulMakepeace provides an interface to communicating with an OpenRefine server. This fork extends the command line interface (CLI) and supports communication between docker containers.
 # Source: https://github.com/opencultureconsulting/openrefine-client

google/refine/__main__.py

@@ -138,14 +138,14 @@ group5.add_option('--linesPerRow', dest='linesPerRow',
 group5.add_option('--processQuotes', dest='processQuotes',
                   metavar='true/false', choices=('true', 'false'),
                   help='(csv,tsv), default: true')
-group5.add_option('--projectName', dest='project_name',
+group5.add_option('--projectName', dest='projectName',
                   help='(all formats), default: filename')
 group5.add_option('--projectTags', dest='projectTags',
                   action='append',
                   help='(all formats), please provide tags in multiple arguments, e.g. --projectTags=beta --projectTags=client1')
 group5.add_option('--recordPath', dest='recordPath',
                   action='append',
-                  help='(xml,json), please provide path in multiple arguments without slashes, e.g. /collection/record/ should be entered like this: --recordPath=collection --recordPath=record, default xml: record, default json: _ _')
+                  help='(xml,json), please provide path in multiple arguments, e.g. /collection/record/ should be entered: --recordPath=collection --recordPath=record, default xml: root element, default json: _ _')
 group5.add_option('--separator', dest='separator',
                   help='(csv,tsv), default csv: , default tsv: \\t')
 group5.add_option('--sheets', dest='sheets',
@@ -213,7 +213,7 @@ def main():
         projects = refine.Refine(refine.RefineServer()).list_projects().items()
         idlist = []
         for project_id, project_info in projects:
-            if args[0] == project_info['name']:
+            if args[0].decode('UTF-8') == project_info['name']:
                 idlist.append(str(project_id))
         if len(idlist) > 1:
             print('Error: Found %s projects with name %s.\n'

google/refine/cli.py

@@ -25,6 +25,7 @@ import ssl
 import sys
 import time
 import urllib
+from xml.etree import ElementTree
 
 from google.refine import refine
 
@@ -40,10 +41,8 @@ def apply(project_id, history_file):
         print('File %s has been successfully applied to project %s' %
               (history_file, project_id))
 
-
 def create(project_file,
            project_format=None,
-           project_name=None,
            columnWidths=None,
            encoding=None,
            guessCellValueTypes=False,
@@ -54,6 +53,7 @@ def create(project_file,
            linesPerRow=None,
            processQuotes=True,
            projectName=None,
+           projectTags=None,
            recordPath=None,
            separator=None,
            sheets=None,
@@ -69,15 +69,15 @@ def create(project_file,
         project_format = os.path.splitext(project_file)[1][1:].lower()
         if project_format == 'txt':
             try:
-                columnWidths
+                columnWidths[0]
                 project_format = 'fixed-width'
-            except NameError:
+            except TypeError:
                 project_format = 'line-based'
     # defaults for each file type
     if project_format == 'xml':
         project_format = 'text/xml'
         if not recordPath:
-            recordPath = 'record'
+            recordPath = [ElementTree.parse(project_file).getroot().tag]
     elif project_format == 'csv':
         project_format = 'text/line-based/*sv'
     elif project_format == 'tsv':
@@ -95,22 +95,35 @@ def create(project_file,
     elif project_format == 'json':
         project_format = 'text/json'
         if not recordPath:
-            recordPath = ('_', '_')
+            recordPath = ['_', '_']
     elif project_format == 'xls':
         project_format = 'binary/text/xml/xls/xlsx'
         if not sheets:
-            sheets = 0
+            sheets = [0]
+            # TODO: new format for sheets option introduced in OpenRefine 2.8
     elif project_format == 'xlsx':
         project_format = 'binary/text/xml/xls/xlsx'
         if not sheets:
-            sheets = 0
+            sheets = [0]
+            # TODO: new format for sheets option introduced in OpenRefine 2.8
     elif project_format == 'ods':
         project_format = 'text/xml/ods'
         if not sheets:
-            sheets = 0
+            sheets = [0]
+            # TODO: new format for sheets option introduced in OpenRefine 2.8
     # execute
     kwargs = {k: v for k, v in vars().items() if v is not None}
-    project = refine.Refine(refine.RefineServer()).new_project(**kwargs)
+    project = refine.Refine(refine.RefineServer()).new_project(
+        guess_cell_value_types=guessCellValueTypes,
+        ignore_lines=ignoreLines,
+        header_lines=headerLines,
+        skip_data_lines=skipDataLines,
+        store_blank_rows=storeBlankRows,
+        process_quotes=processQuotes,
+        project_name=projectName,
+        store_blank_cells_as_nulls=storeBlankCellsAsNulls,
+        include_file_sources=includeFileSources,
+        **kwargs)
     rows = project.do_json('get-rows')['total']
     if rows > 0:
         print('{0}: {1}'.format('id', project.project_id))
@@ -148,43 +161,46 @@ def download(url, output_file=None):
     print('Download to file %s complete' % output_file)
 
 
-def export(project_id, output_file=None, export_format=None):
+def export(project_id, encoding=None, output_file=None, export_format=None):
     """Dump a project to stdout or file."""
     project = refine.RefineProject(project_id)
-    if not export_format:
-        export_format = 'tsv'
     if not output_file:
-        sys.stdout.write(project.export(export_format=export_format).read())
+        if not export_format:
+            export_format = 'tsv'
+        if export_format in ['csv', 'tsv', 'txt']:
+                encoding = 'UTF-8'
+        sys.stdout.write(project.export(
+            export_format=export_format, encoding=encoding).read())
     else:
         ext = os.path.splitext(output_file)[1][1:]
-        if ext:
+        if ext and not export_format:
             export_format = ext.lower()
+        if not export_format:
+            export_format = 'tsv'
+        if export_format in ['csv', 'tsv', 'txt']:
+            encoding = 'UTF-8'
         with open(output_file, 'wb') as f:
-            f.write(project.export(export_format).read())
+            f.write(project.export(
+                export_format=export_format, encoding=encoding).read())
         print('Export to file %s complete' % output_file)
 
 
 def info(project_id):
     """Show project metadata"""
-    projects = refine.Refine(refine.RefineServer()).list_projects().items()
-    if project_id in [item[0] for item in projects]:
-        project = refine.RefineProject(project_id)
-        projects = refine.Refine(refine.RefineServer()).list_projects().items()
-        for projects_id, projects_info in projects:
-            if project_id == projects_id:
-                print('{0:>20}: {1}'.format('id', project_id))
-                print('{0:>20}: {1}'.format('url', 'http://' +
-                                            refine.REFINE_HOST + ':' +
-                                            refine.REFINE_PORT +
-                                            '/project?project=' + project_id))
-                for k, v in projects_info.items():
-                    if v:
-                        print('{0:>20}: {1}'.format(k, v))
-        project_model = project.get_models()
-        column_model = project_model['columnModel']
-        columns = [column['name'] for column in column_model['columns']]
+    projects = refine.Refine(refine.RefineServer()).list_projects()
+    if project_id in projects.keys():
+        print('{0:>20}: {1}'.format('id', project_id))
+        print('{0:>20}: {1}'.format('url', 'http://' +
+                                    refine.REFINE_HOST + ':' +
+                                    refine.REFINE_PORT +
+                                    '/project?project=' + project_id))
+        for k, v in projects[project_id].items():
+            if v:
+                    print(u'{0:>20}: {1}'.format(k, v))
+        project_model = refine.RefineProject(project_id).get_models()
+        columns = [c['name'] for c in project_model['columnModel']['columns']]
         for (i, v) in enumerate(columns, start=1):
-            print('{0:>20}: {1}'.format('column ' + str(i).zfill(3), v))
+            print(u'{0:>20}: {1}'.format(u'column ' + str(i).zfill(3), v).encode('utf-8'))
     else:
         print('Error: No project found with id %s.\n'
               'Check existing projects with command --list' % (project_id))
@@ -200,13 +216,14 @@ def ls():
     projects.sort(key=lambda v: date_to_epoch(v[1]['modified']), reverse=True)
     if projects:
         for project_id, project_info in projects:
-            print('{0:>14}: {1}'.format(project_id, project_info['name']))
+            print(u'{0:>14}: {1}'.format(project_id, project_info['name']).encode('utf-8'))
     else:
         print('Error: No projects found')
 
 
 def templating(project_id,
                template,
+               encoding='UTF-8',
                output_file=None,
                mode=None,
                prefix='',
@@ -225,7 +242,8 @@ def templating(project_id,
     templateconfig = {'prefix': prefix,
                       'suffix': suffix,
                       'template': template,
-                      'rowSeparator': rowSeparator}
+                      'rowSeparator': rowSeparator,
+                      'encoding': encoding}
 
     # construct the engine config
     if mode == 'record-based':
@@ -246,8 +264,8 @@ def templating(project_id,
         engine['facets'].append(textFilter)
     templateconfig.update({'engine': json.dumps(engine)})
 
-    # normal output or some refinable magic for splitToFiles functionality
     if not splitToFiles:
+        # normal output
         if not output_file:
             sys.stdout.write(project.export_templating(
                              **templateconfig).read())
@@ -256,11 +274,10 @@ def templating(project_id,
                 f.write(project.export_templating(**templateconfig).read())
             print('Export to file %s complete' % output_file)
     else:
-        # common config for row-based and record-based
+        # splitToFiles functionality
         prefix = templateconfig['prefix']
         suffix = templateconfig['suffix']
         split = '===|||THISISTHEBEGINNINGOFANEWRECORD|||==='
-        keyColumn = project.get_models()['columnModel']['keyColumnName']
         if not output_file:
             output_file = time.strftime('%Y%m%d')
         else:
@@ -268,23 +285,24 @@ def templating(project_id,
             ext = os.path.splitext(output_file)[1][1:]
         if not ext:
             ext = 'txt'
+        # generate config for subfeature suffixById
         if suffixById:
-            ids_template = ('{{forNonBlank(cells["' +
-                            keyColumn +
-                            '"].value, v, v, "")}}')
+            ids_template = ('{{forNonBlank(' +
+                            'with(row.columnNames[0],cn,cells[cn].value),' +
+                            'v,v,"")}}')
             ids_templateconfig = {'engine': json.dumps(engine),
                                   'template': ids_template,
-                                  'rowSeparator': '\n'}
+                                  'rowSeparator': '\n',
+                                  'encoding': encoding}
             ids = [line.rstrip('\n') for line in project.export_templating(
                    **ids_templateconfig) if line.rstrip('\n')]
+        # generate common config
         if mode == 'record-based':
             # record-based: split-character into template
             #               if key column is not blank (=record)
-            template = ('{{forNonBlank(cells["' +
-                        keyColumn +
-                        '"].value, v, "' +
-                        split +
-                        '", "")}}' +
+            template = ('{{forNonBlank(' +
+                         'with(row.columnNames[0],cn,cells[cn].value),' +
+                         'v,"' + split + '", "")}}' +
                         templateconfig['template'])
             templateconfig.update({'prefix': '',
                                    'suffix': '',
@@ -297,6 +315,7 @@ def templating(project_id,
                                    'suffix': '',
                                    'template': template,
                                    'rowSeparator': ''})
+        # execute
         records = project.export_templating(
             **templateconfig).read().split(split)
         del records[0]  # skip first blank entry

google/refine/refine.py

@@ -53,6 +53,18 @@ class RefineServer(object):
             server = self.url()
         self.server = server[:-1] if server.endswith('/') else server
         self.__version = None     # see version @property below
+        self.token = None         # CSRF token introduced in OpenRefine 3.3
+        self.get_csrf_token()
+
+    def get_csrf_token(self):
+        """Return csrf token."""
+        try:
+            url = self.server + '/command/core/get-csrf-token'
+            response = json.loads(urllib2.urlopen(url).read())
+            self.token = response['token']
+            return self.token
+        except:
+            pass # fail silently to not disturb usage of OpenRefine <3.3
 
     def urlopen(self, command, data=None, params=None, project_id=None):
         """Open a Refine URL and with optional query params and POST data.
@@ -73,6 +85,9 @@ class RefineServer(object):
                 data['project'] = project_id
             else:
                 params['project'] = project_id
+        # be lazy and send the token for each API call (even when not needed)
+        if self.token:
+            params['csrf_token'] = self.token
         if params:
             url += '?' + urllib.urlencode(params)
         req = urllib2.Request(url)
@@ -82,7 +97,8 @@ class RefineServer(object):
         try:
             response = urllib2.urlopen(req)
         except urllib2.HTTPError as e:
-            raise Exception('HTTP %d "%s" for %s\n\t%s' % (e.code, e.msg, e.geturl(), data))
+            raise Exception('HTTP %d "%s" for %s\n\t%s' %
+                            (e.code, e.msg, e.geturl(), data))
         except urllib2.URLError as e:
             raise urllib2.URLError(
                 '%s for %s. No Refine server reachable/running; ENV set?' %
@@ -98,6 +114,10 @@ class RefineServer(object):
         """Open a Refine URL, optionally POST data, and return parsed JSON."""
         response = json.loads(self.urlopen(*args, **kwargs).read())
         if 'code' in response and response['code'] not in ('ok', 'pending'):
+            if 'Missing or invalid csrf_token parameter' == response['message']:
+                self.get_csrf_token()
+                response = json.loads(self.urlopen(*args, **kwargs).read())
+                return response
             error_message = ('server ' + response['code'] + ': ' +
                              response.get('message', response.get('stack', response)))
             raise Exception(error_message)
@@ -147,41 +167,127 @@ class Refine:
         """Open a Refine project."""
         return RefineProject(self.server, project_id)
 
-    def new_project(self, project_file=None, project_name=None,
-                    project_format='text/line-based/*sv', **kwargs):
-        """Create a Refine project."""
+    # These aren't used yet but are included for reference
+    new_project_defaults = {
+        'text/line-based/*sv': {
+            'encoding': '',
+            'separator': ',',
+            'ignore_lines': -1,
+            'header_lines': 1,
+            'skip_data_lines': 0,
+            'limit': -1,
+            'store_blank_rows': True,
+            'guess_cell_value_types': True,
+            'process_quotes': True,
+            'store_blank_cells_as_nulls': True,
+            'include_file_sources': False},
+        'text/line-based': {
+            'encoding': '',
+            'lines_per_row': 1,
+            'ignore_lines': -1,
+            'limit': -1,
+            'skip_data_lines': -1,
+            'store_blank_rows': True,
+            'store_blank_cells_as_nulls': True,
+            'include_file_sources': False},
+        'text/line-based/fixed-width': {
+            'encoding': '',
+            'column_widths': [20],
+            'ignore_lines': -1,
+            'header_lines': 0,
+            'skip_data_lines': 0,
+            'limit': -1,
+            'guess_cell_value_types': False,
+            'store_blank_rows': True,
+            'store_blank_cells_as_nulls': True,
+            'include_file_sources': False},
+        'text/line-based/pc-axis': {
+            'encoding': '',
+            'limit': -1,
+            'skip_data_lines': -1,
+            'include_file_sources': False},
+        'text/rdf+n3': {'encoding': ''},
+        'text/xml/ods': {
+            'sheets': [],
+            'ignore_lines': -1,
+            'header_lines': 1,
+            'skip_data_lines': 0,
+            'limit': -1,
+            'store_blank_rows': True,
+            'store_blank_cells_as_nulls': True,
+            'include_file_sources': False},
+        'binary/xls': {
+            'xml_based': False,
+            'sheets': [],
+            'ignore_lines': -1,
+            'header_lines': 1,
+            'skip_data_lines': 0,
+            'limit': -1,
+            'store_blank_rows': True,
+            'store_blank_cells_as_nulls': True,
+            'include_file_sources': False}
+    }
 
-        defaults = {'guessCellValueTypes': False,
-                    'headerLines': 1,
-                    'ignoreLines': -1,
-                    'includeFileSources': False,
-                    'limit': -1,
-                    'linesPerRow': 1,
-                    'processQuotes': True,
-                    'separator': ',',
-                    'skipDataLines': 0,
-                    'storeBlankCellsAsNulls': True,
-                    'storeBlankRows': True,
-                    'storeEmptyStrings': True,
-                    'trimStrings': False}
+    def new_project(self, project_file=None, project_url=None, project_name=None, project_format='text/line-based/*sv',
+                    encoding='',
+                    separator=',',
+                    ignore_lines=-1,
+                    header_lines=1,
+                    skip_data_lines=0,
+                    limit=-1,
+                    store_blank_rows=True,
+                    guess_cell_value_types=False,
+                    process_quotes=True,
+                    store_blank_cells_as_nulls=True,
+                    include_file_sources=False,
+                    **opts):
 
-        # options
-        options = {'format': project_format}
-        if project_file is not None:
-            options['project-file'] = {'fd': open(project_file),
-                                       'filename': project_file}
+        if (project_file and project_url) or (not project_file and not project_url):
+            raise ValueError('One (only) of project_file and project_url must be set')
+
+        def s(opt):
+            if isinstance(opt, bool):
+                return 'true' if opt else 'false'
+            if opt is None:
+                return ''
+            return str(opt)
+
+        # the new APIs requires a json in the 'option' POST or GET argument
+        # POST is broken at the moment, so we send it in the URL
+        new_style_options = dict(opts, **{
+            'encoding': s(encoding),
+            'separator': s(separator)
+        })
+        params = {
+            'options': json.dumps(new_style_options),
+        }
+
+        # old style options
+        options = {
+            'format': project_format,
+            'ignore-lines': s(ignore_lines),
+            'header-lines': s(header_lines),
+            'skip-data-lines': s(skip_data_lines),
+            'limit': s(limit),
+            'guess-value-type': s(guess_cell_value_types),
+            'process-quotes': s(process_quotes),
+            'store-blank-rows': s(store_blank_rows),
+            'store-blank-cells-as-nulls': s(store_blank_cells_as_nulls),
+            'include-file-sources': s(include_file_sources)
+        }
+
+        if project_url is not None:
+            options['url'] = project_url
+        elif project_file is not None:
+            options['project-file'] = {
+                'fd': open(project_file),
+                'filename': project_file,
+            }
         if project_name is None:
             # make a name for itself by stripping extension and directories
             project_name = (project_file or 'New project').rsplit('.', 1)[0]
             project_name = os.path.basename(project_name)
         options['project-name'] = project_name
-
-        # params
-        params_dict = dict(defaults)
-        params_dict.update(kwargs)
-        params = {'options': json.dumps(params_dict)}
-
-        # submit
         response = self.server.urlopen(
             'create-project-from-upload', options, params
         )
@@ -312,7 +418,10 @@ class RefineProject:
         for i, column in enumerate(column_model['columns']):
             name = column['name']
             self.column_order[name] = i
-            column_index[name] = column['cellIndex']
+            try:
+                column_index[name] = column['cellIndex']
+            except KeyError:
+                column_index[name] = i
         self.key_column = column_model['keyColumnName']
         self.has_records = response['recordModel'].get('hasRecords', False)
         self.rows_response_factory = RowsResponseFactory(column_index)
@@ -341,23 +450,31 @@ class RefineProject:
             return 'ok'
         return response_json['code']  # can be 'ok' or 'pending'
 
-    def export(self, export_format='tsv'):
+    def export(self, encoding=None, export_format='tsv'):
         """Return a fileobject of a project's data."""
-        url = ('export-rows/' + urllib.quote(self.project_name()) + '.' +
-               export_format)
-        return self.do_raw(url, data={'format': export_format})
+        url = ('export-rows/' +
+               urllib.quote(self.project_name().encode('utf8')) +
+               '.' + export_format)
+        data = {'format': export_format}
+        if encoding:
+            data['encoding'] = encoding
+        return self.do_raw(url, data)
 
-    def export_templating(self, engine='', prefix='',
+    def export_templating(self, encoding=None, engine='', prefix='',
                           template='', rowSeparator='\n', suffix=''):
         """Return a fileobject of a project's data in templating mode."""
-        url = ('export-rows/' + urllib.quote(self.project_name()) + '.' +
-               'txt')
-        return self.do_raw(url, data={'format': 'template',
-                                      'template': template,
-                                      'engine': engine,
-                                      'prefix': prefix,
-                                      'suffix': suffix,
-                                      'separator': rowSeparator})
+        url = ('export-rows/' +
+               urllib.quote(self.project_name().encode('utf8')) +
+               '.' + 'txt')
+        data = {'format': 'template',
+                'template': template,
+                'engine': engine,
+                'prefix': prefix,
+                'suffix': suffix,
+                'separator': rowSeparator}
+        if encoding:
+            data['encoding'] = encoding
+        return self.do_raw(url, data)
 
     def export_rows(self, **kwargs):
         """Return an iterable of parsed rows of a project's data."""

refine.py

@@ -18,8 +18,18 @@ Script to provide a command line interface to a Refine server.
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>
 
+import sys
 
 from google.refine import __main__, cli, refine
 
+# workaround for pyinstaller
+if getattr(sys, 'frozen', False) and hasattr(sys, '_MEIPASS'):
+    reload(sys)
+    sys.setdefaultencoding('utf-8')
+    if sys.platform == "win32":
+        import codecs
+        codecs.register(lambda name: codecs.lookup(
+            'utf-8') if name == 'cp65001' else None)
+
 if __name__ == '__main__':
     __main__.main()

setup.py

@@ -25,7 +25,7 @@ def read(filename):
     return open(os.path.join(os.path.dirname(__file__), filename)).read()
 
 setup(name='openrefine-client',
-      version='0.3.7',
+      version='0.3.10',
       description=('The OpenRefine Python Client Library provides an '
                    'interface to communicating with an OpenRefine server. '
                    'This fork extends the command line interface (CLI).'),

tests-cli.sh

@@ -0,0 +1,123 @@
+#!/bin/bash
+# Script for running functional tests against the CLI
+
+# Copyright (c) 2011 Paul Makepeace, Real Programmers. All rights reserved.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+# ================================== CONFIG ================================== #
+
+cd "${BASH_SOURCE%/*}/" || exit 1
+
+port=3334
+
+if [[ ${1} ]]; then
+  version="${1}"
+else
+  version="3.2"
+fi
+refine="openrefine-${version}/refine"
+
+if [[ ${2} ]]; then
+  client="$(readlink -e "${2}")"
+else
+  client="python2 $(readlink -e refine.py)"
+fi
+cmd="${client} -H localhost -P ${port}"
+
+if [[ ${3} ]]; then
+  filename="${3%%.*}"
+else
+  filename=""
+fi
+cmd="${client} -H localhost -P ${port}"
+
+# =============================== REQUIREMENTS =============================== #
+
+# check existence of java and cURL
+if [[ -z "$(command -v java 2> /dev/null)" ]] ; then
+  echo 1>&2 "ERROR: OpenRefine requires JAVA runtime environment (jre)" \
+    "https://openjdk.java.net/install/"
+  exit 1
+fi
+if [[ -z "$(command -v curl 2> /dev/null)" ]] ; then
+  echo 1>&2 "ERROR: This shell script requires cURL" \
+    "https://curl.haxx.se/download.html"
+  exit 1
+fi
+# download OpenRefine
+if [[ -z "$(readlink -e "${refine}")" ]]; then
+  echo "Download OpenRefine ${version}..."
+  mkdir -p "$(dirname "${refine}")"
+  curl -L --output openrefine.tar.gz \
+    "https://github.com/OpenRefine/OpenRefine/releases/download/${version}/openrefine-linux-${version}.tar.gz"
+  echo "Install OpenRefine ${version} in subdirectory $(dirname "${refine}")..."
+  tar -xzf openrefine.tar.gz -C "$(dirname "${refine}")" --strip 1 --totals
+  rm -f openrefine.tar.gz
+  # do not try to open OpenRefine in browser
+  sed -i '$ a JAVA_OPTIONS=-Drefine.headless=true' \
+    "$(dirname "${refine}")"/refine.ini
+  # set autosave period from 5 minutes to 25 hours
+  sed -i 's/#REFINE_AUTOSAVE_PERIOD=60/REFINE_AUTOSAVE_PERIOD=1500/' \
+    "$(dirname "${refine}")"/refine.ini
+  echo
+fi
+
+# ================================== SETUP =================================== #
+
+dir="$(readlink -f "tests/tmp")"
+mkdir -p "${dir}"
+rm -f tests-cli.log
+
+echo "start OpenRefine ${version}..."
+${refine} -v warn -p ${port} -d "${dir}" &>> tests-cli.log &
+pid_server=${!}
+timeout 30s bash -c "until curl -s 'http://localhost:3334' \
+  | cat | grep -q -o 'OpenRefine' ; do sleep 1; done" \
+  || error "starting OpenRefine server failed!"
+echo
+
+# ================================== TESTS =================================== #
+
+echo "running tests, please wait..."
+tests=()
+results=()
+for t in tests/*${filename}*.sh; do
+  tests+=("${t}")
+  echo "======================= ${t} =======================" &>> tests-cli.log
+  bash "${t}" "${cmd}" "${version}" &>> tests-cli.log
+  results+=(${?})
+done
+echo
+
+# ================================= TEARDOWN ================================= #
+
+echo "cleanup..."
+{ kill -9 "${pid_server}" && wait "${pid_server}"; } 2>/dev/null
+rm -rf "${dir}"
+echo
+
+# ================================= SUMMARY ================================== #
+
+printf "%s\t%s\n" "code" "test"
+printf "%s\t%s\n" "----" "----------------"
+for i in "${!tests[@]}"; do
+  printf "%s\t%s\n" "${results[$i]}" "${tests[$i]}"
+done
+echo
+if [[ " ${results[*]} " =~ [1-9] ]]; then
+  echo "failed tests! check tests-cli.log for debugging"; echo
+else
+  echo "all tests passed!"; echo
+fi

tests.sh

@@ -17,8 +17,8 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>
 
 # defaults:
-all=(3.2-java12 3.2-java11 3.2-java10 3.2-java9 3.2 3.1-java9 3.1 3.0-java9 3.0 2.8-java9 2.8 2.8-java7 2.7 2.7-java7 2.5-java7 2.5-java6 2.1-java6 2.0-java6)
-main=(3.2 3.1 3.0 2.8 2.7 2.5-java6 2.1-java6 2.0-java6)
+all=(3.5.0 3.4.1 3.4 3.3 3.2-java12 3.2-java11 3.2-java10 3.2-java9 3.2 3.1-java9 3.1 3.0-java9 3.0 2.8-java9 2.8 2.8-java7 2.7 2.7-java7 2.5-java7 2.5-java6 2.1-java6 2.0-java6)
+main=(3.5.0 3.4.1 3.4 3.3 3.2 3.1 3.0 2.8 2.7 2.5-java6 2.1-java6 2.0-java6)
 interactively=false
 port="3333"
 
@@ -31,10 +31,10 @@ Script for running tests with different OpenRefine and Java versions.
 It uses docker images from https://hub.docker.com/r/felixlohmeier/openrefine.
 
 Examples:
-./tests.sh -a            # run tests on all OpenRefine versions (from 2.0 up to 3.2)
-./tests.sh -t 3.2        # run tests on tag 3.2
-./tests.sh -t 3.2 -i     # run tests on tag 3.2 interactively (pause before and after tests)
-./tests.sh -t 3.2 -t 2.7 # run tests on tags 3.2 and 2.7
+./tests.sh -a              # run tests on all OpenRefine versions (from 2.0 up to 3.5.0)
+./tests.sh -t 3.5.0        # run tests on tag 3.5.0
+./tests.sh -t 3.5.0 -i     # run tests on tag 3.5.0 interactively (pause before and after tests)
+./tests.sh -t 3.5.0 -t 2.7 # run tests on tags 3.5.0 and 2.7
 
 Advanced:
 ./tests.sh -j                # run tests on all OpenRefine versions and each with all supported Java versions (requires a lot of docker images to be downloaded!)
@@ -122,7 +122,7 @@ for t in ${tags[*]} ; do
     until curl --silent -N http://localhost:"$port" | cat | grep -q -o "Refine" ; do sleep 1; done
     echo "Refine running at http://localhost:${port}"
     if [ $interactively = true ]; then read -r -p "Press [Enter] key to start tests..."; fi
-    python setup.py test
+    python2 setup.py test
     if [ $interactively = true ]; then read -r -p "Press [Enter] key to stop OpenRefine..."; fi
     ${docker[*]} stop "$t"
     echo "End: $(date)"

tests/apply-utf8.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-addition",
+    "engineConfig": {
+      "mode": "row-based"
+    },
+    "newColumnName": "apply",
+    "columnInsertIndex": 2,
+    "baseColumnName": "b",
+    "expression": "grel:value.replace('2','⛲')",
+    "onError": "set-to-blank"
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	apply	c
+1	2	⛲	3
+0	0	0	0
+$	\	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/apply.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-addition",
+    "engineConfig": {
+      "mode": "row-based"
+    },
+    "newColumnName": "apply",
+    "columnInsertIndex": 2,
+    "baseColumnName": "b",
+    "expression": "grel:value.replace('2','TEST')",
+    "onError": "set-to-blank"
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	apply	c
+1	2	TEST	3
+0	0	0	0
+$	\	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-encoding.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}-utf8.csv"
+a,b,c
+1,2,3
+ä,é,ß
+$,\,'
+DATA
+iconv -f UTF-8 -t ISO-8859-1 "tmp/${t}/${t}-utf8.csv" > "tmp/${t}/${t}.csv"
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+ä	é	ß
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --encoding "ISO-8859-1"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-guessCellValueTypes.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+01,02,03
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+1	2	3
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --guessCellValueTypes "true"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-headerLines.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1	Column 2	Column 3
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --headerLines "0"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-ignoreLines.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --ignoreLines "1"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-limit.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --limit "2"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-processQuotes.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,"2,0",3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c	Column 4
+1	2	0	3
+0	0	0	
+$	\	'	
+DATA
+
+# ================================== ACTION ================================== #
+
+# OpenRefine 4.x fails without manually set headerLines
+${cmd} --create "tmp/${t}/${t}.csv" --processQuotes "false" --headerLines 1
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-projectTags.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version:0:1}" = "2" ]]; then
+  echo "projectTags were introduced in OpenRefine 3.0"
+  exit 200
+else
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+                tags: [u'beta', u'client1']
+DATA
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --projectTags "beta" --projectTags "client1"
+${cmd} --info "${t}" | grep ' tags: ' > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-separator.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a;b;c
+1;2;3
+0;0;0
+$;\;'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --separator ";"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-skipDataLines.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --skipDataLines "1"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-storeBlankCellsAsNulls.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,,0
+$,\,'
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "b",
+    "expression": "grel:isNull(value)",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	false	3
+0	false	0
+$	false	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --storeBlankCellsAsNulls "false"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-storeBlankRows.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+,,
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --storeBlankRows "false"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv-utf8.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨	code	meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --projectName "${t} biểu tượng cảm xúc ⛲"
+${cmd} --export "${t} biểu tượng cảm xúc ⛲" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-csv.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json-recordPath.sh

@@ -0,0 +1,55 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+{
+   "rows":[
+      {
+         "a":1,
+         "b":2,
+         "c":3
+      },
+      {
+         "a":0,
+         "b":0,
+         "c":0
+      },
+      {
+         "a":"$",
+         "b":"\\",
+         "c":"\""
+      }
+   ]
+}
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - a	_ - b	_ - c
+1	2	3
+0	0	0
+$	\	""""
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json" --recordPath "_" --recordPath "rows" --recordPath "_"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json-storeEmptyStrings.sh

@@ -0,0 +1,52 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+[
+  {
+    "a": 1,
+    "b": 2,
+    "c": 3
+  },
+  {
+    "a": "",
+    "b": "",
+    "c": ""
+  },
+  {
+    "a": "$",
+    "b": "\\",
+    "c": "\""
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - a	_ - b	_ - c
+1	2	3
+$	\	""""
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json" --storeEmptyStrings "false"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json-trimStrings.sh

@@ -0,0 +1,62 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+[
+  {
+    "a": 1,
+    "b": 2,
+    "c": 3
+  },
+  {
+    "a": "0",
+    "b": " 0",
+    "c": "0 "
+  },
+  {
+    "a": "$",
+    "b": "\\",
+    "c": "\""
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version:0:1}" = "2" || "${version}" = "3.0" || "${version}" = "3.1" || "${version}" = "3.2" || "${version}" = "3.3" ]]; then
+  echo "trimStrings option does not work in OpenRefine <=3.3"
+  echo "https://github.com/OpenRefine/OpenRefine/issues/2409"
+  exit 200
+else
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - a	_ - b	_ - c
+1	2	3
+0	0	0
+$	\	""""
+DATA
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json" --trimStrings "true"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json-utf8.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+[
+  {
+    "⌨": "⛲",
+    "code": "1F347",
+    "meaning": "FOUNTAIN"
+  },
+  {
+    "⌨": "⛳",
+    "code": "1F349",
+    "meaning": "FLAG IN HOLE"
+  },
+  {
+    "⌨": "⛵",
+    "code": "1F352",
+    "meaning": "SAILBOAT"
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - ⌨	_ - code	_ - meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-json.sh

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.json"
+[
+  {
+    "a": 1,
+    "b": 2,
+    "c": 3
+  },
+  {
+    "a": 0,
+    "b": 0,
+    "c": 0
+  },
+  {
+    "a": "$",
+    "b": "\\",
+    "c": "\""
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+_ - a	_ - b	_ - c
+1	2	3
+0	0	0
+$	\	""""
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.json"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-ods-sheets-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.ods" "tmp/${t}/${t}.ods"
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨	code	meaning	Column	Column 5	Column 6	Column 7	Column 8
+⛲	1F347	FOUNTAIN					
+⛳	1F349	FLAG IN HOLE					
+⛵	1F352	SAILBOAT					
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.ods" --sheets 1
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-ods.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.ods" "tmp/${t}/${t}.ods"
+#a	b	c
+#1	2	3
+#0	0	0
+#$	\	'
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c	Column	Column 5	Column 6	Column 7	Column 8
+1.0	2.0	3.0					
+0.0	0.0	0.0					
+$	\	'					
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.ods"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-tsv-utf8.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+⌨	code	meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.csv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.csv"

tests/create-tsv.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.csv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.csv"

tests/create-txt-fixed-width-headerLines.sh

@@ -0,0 +1,80 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+1     2     3
+mon   tue   wed
+$2    $300  $1
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "1",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "2",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "3",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+1	2	3
+mon	tue	wed
+$2	$300	$1
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --columnWidths "6" --columnWidths "6" --columnWidths "6" --headerLines "1"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-txt-fixed-width-utf8.sh

@@ -0,0 +1,81 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+⛲     1F347 FOUNTAIN
+⛳     1F349 FLAG IN HOLE
+⛵     1F352 SAILBOAT
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 1",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 2",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 3",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1	Column 2	Column 3
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --columnWidths "6" --columnWidths "6" --columnWidths "60"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-txt-fixed-width.sh

@@ -0,0 +1,81 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+1     2     3
+mon   tue   wed
+$2    $300  $1
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 1",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 2",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  },
+  {
+    "op": "core/text-transform",
+    "engineConfig": {
+      "facets": [],
+      "mode": "row-based"
+    },
+    "columnName": "Column 3",
+    "expression": "grel:value.trim()",
+    "onError": "keep-original",
+    "repeat": false,
+    "repeatCount": 10
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1	Column 2	Column 3
+1	2	3
+mon	tue	wed
+$2	$300	$1
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --columnWidths "6" --columnWidths "6" --columnWidths "6"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-txt-linesPerRow.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+mon tue wed
+$2 $300 $1
+thu fri sat
+$70 $20 $50
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1	Column 2
+mon tue wed	$2 $300 $1
+thu fri sat	$70 $20 $50
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --linesPerRow "2"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-txt.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+1 2 3
+mon tue wed
+$2 $300 $1
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1
+1 2 3
+mon tue wed
+$2 $300 $1
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xls-sheets-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.xls" "tmp/${t}/${t}.xls"
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨	code	meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xls" --sheets 1
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xls.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.xls" "tmp/${t}/${t}.xls"
+#a	b	c
+#1	2	3
+#0	0	0
+#$	\	'
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1.0	2.0	3.0
+0.0	0.0	0.0
+$	\	'
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xls"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xlsx-sheets-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.xlsx" "tmp/${t}/${t}.xlsx"
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨	code	meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xlsx" --sheets 1
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xlsx.sh

@@ -0,0 +1,48 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  version="${2}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cp "data/example.xlsx" "tmp/${t}/${t}.xlsx"
+#a	b	c
+#1	2	3
+#0	0	0
+#$	\	'
+
+# ================================= ASSERTION ================================ #
+
+if [[ "${version}" = "2.7" ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1.0	2.0	3.0
+0.0	0.0	0.0
+$	\	'
+DATA
+else
+  #TODO
+  echo "https://github.com/opencultureconsulting/openrefine-client/issues/4"
+  exit 200
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xlsx"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xml-recordPath.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.xml"
+<?xml version="1.0" encoding="UTF-8"?>
+<root>
+  <record>
+    <a>1</a>
+    <b>2</b>
+    <c>3</c>
+  </record>
+  <record>
+    <a>0</a>
+    <b>0</b>
+    <c>0</c>
+  </record>
+  <record>
+    <a>$</a>
+    <b>\</b>
+    <c>'</c>
+  </record>
+</root>
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-reorder",
+    "columnNames": [
+      "record - a",
+      "record - b",
+      "record - c"
+    ],
+    "description": "Reorder columns"
+  },
+  {
+    "op": "core/row-removal",
+    "engineConfig": {
+      "facets": [
+        {
+          "type": "list",
+          "name": "Blank Rows",
+          "expression": "(filter(row.columnNames,cn,isNonBlank(cells[cn].value)).length()==0).toString()",
+          "columnName": "",
+          "invert": false,
+          "omitBlank": false,
+          "omitError": false,
+          "selection": [
+            {
+              "v": {
+                "v": "true",
+                "l": "true"
+              }
+            }
+          ],
+          "selectBlank": false,
+          "selectError": false
+        }
+      ],
+      "mode": "record-based"
+    }
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+record - a	record - b	record - c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xml" --recordPath "root" --recordPath "record"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xml-utf8.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.xml"
+<?xml version="1.0" encoding="UTF-8"?>
+<root>
+  <record>
+    <icon>⛲</icon>
+    <code>1F347</code>
+    <meaning>FOUNTAIN</meaning>
+  </record>
+  <record>
+    <icon>⛳</icon>
+    <code>1F349</code>
+    <meaning>FLAG IN HOLE</meaning>
+  </record>
+  <record>
+    <icon>⛵</icon>
+    <code>1F352</code>
+    <meaning>SAILBOAT</meaning>
+  </record>
+</root>
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-reorder",
+    "columnNames": [
+      "root - record - icon",
+      "root - record - code",
+      "root - record - meaning"
+    ],
+    "description": "Reorder columns"
+  },
+  {
+    "op": "core/row-removal",
+    "engineConfig": {
+      "facets": [
+        {
+          "type": "list",
+          "name": "Blank Rows",
+          "expression": "(filter(row.columnNames,cn,isNonBlank(cells[cn].value)).length()==0).toString()",
+          "columnName": "",
+          "invert": false,
+          "omitBlank": false,
+          "omitError": false,
+          "selection": [
+            {
+              "v": {
+                "v": "true",
+                "l": "true"
+              }
+            }
+          ],
+          "selectBlank": false,
+          "selectError": false
+        }
+      ],
+      "mode": "record-based"
+    }
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+root - record - icon	root - record - code	root - record - meaning
+⛲	1F347	FOUNTAIN
+⛳	1F349	FLAG IN HOLE
+⛵	1F352	SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xml"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-xml.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.xml"
+<?xml version="1.0" encoding="UTF-8"?>
+<root>
+  <record>
+    <a>1</a>
+    <b>2</b>
+    <c>3</c>
+  </record>
+  <record>
+    <a>0</a>
+    <b>0</b>
+    <c>0</c>
+  </record>
+  <record>
+    <a>$</a>
+    <b>\</b>
+    <c>'</c>
+  </record>
+</root>
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}.transform"
+[
+  {
+    "op": "core/column-reorder",
+    "columnNames": [
+      "root - record - a",
+      "root - record - b",
+      "root - record - c"
+    ],
+    "description": "Reorder columns"
+  },
+  {
+    "op": "core/row-removal",
+    "engineConfig": {
+      "facets": [
+        {
+          "type": "list",
+          "name": "Blank Rows",
+          "expression": "(filter(row.columnNames,cn,isNonBlank(cells[cn].value)).length()==0).toString()",
+          "columnName": "",
+          "invert": false,
+          "omitBlank": false,
+          "omitError": false,
+          "selection": [
+            {
+              "v": {
+                "v": "true",
+                "l": "true"
+              }
+            }
+          ],
+          "selectBlank": false,
+          "selectError": false
+        }
+      ],
+      "mode": "record-based"
+    }
+  }
+]
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+root - record - a	root - record - b	root - record - c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.xml"
+${cmd} --apply "tmp/${t}/${t}.transform" "${t}"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-zip-includeFileSources.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}-1.csv"
+a,b,c
+1,2,3
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}-2.csv"
+a,b,c
+4,5,6
+DATA
+
+zip "tmp/${t}/${t}.zip" "tmp/${t}/${t}-1.csv" "tmp/${t}/${t}-2.csv"
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+File	a	b	c
+tmp/${t}/${t}-1.csv	1	2	3
+tmp/${t}/${t}-2.csv	4	5	6
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.zip" --includeFileSources "true"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/create-zip.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}-1.csv"
+a,b,c
+1,2,3
+DATA
+
+cat << "DATA" > "tmp/${t}/${t}-2.csv"
+a,b,c
+4,5,6
+DATA
+
+zip "tmp/${t}/${t}.zip" "tmp/${t}/${t}-1.csv" "tmp/${t}/${t}-2.csv"
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+4	5	6
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.zip"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/data/cli/duplicates.csv

@@ -0,0 +1,11 @@
+email,name,state,gender,purchase,count,date
+danny.baron@example1.com,Danny Baron,CA,M,TV (UTF-8: 📺),1,"Wed, 4 Jul 2001"
+melanie.white@example2.edu,Melanie White,NC,F,<iPhone>,1,2001-07-04T12:08:56
+danny.baron@example1.com,   D.	("Tab") Baron,CA,M,Winter jacket,1,2001-07-04
+ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight,1,2001/07/04
+arthur.duff@example4.com,Arthur Duff,OR,M,Dining table,1,2001-07
+danny.baron@example1.com,Daniel Baron,,,Bike,1,2001
+jean.griffith@example5.org,Jean Griffith,WA,F,Power drill,1,2000
+melanie.white@example2.edu,Melanie White,NC,F,'iPad',1,1999
+ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier,1,1998
+arthur.duff@example4.com,Arthur Duff,OR,M,Night table,1,1997

tests/data/cli/duplicates.json

@@ -0,0 +1,92 @@
+[
+  {
+    "email": "danny.baron@example1.com",
+    "name": "Danny Baron",
+    "state": "CA",
+    "gender": "M",
+    "purchase": "TV (UTF-8: 📺)",
+    "count": 1,
+    "date": "Wed, 4 Jul 2001"
+  },
+  {
+    "email": "melanie.white@example2.edu",
+    "name": "Melanie White",
+    "state": "NC",
+    "gender": "F",
+    "purchase": "<iPhone>",
+    "count": 1,
+    "date": "2001-07-04T12:08:56"
+  },
+  {
+    "email": "danny.baron@example1.com",
+    "name": "   D.\t(\"Tab\") Baron",
+    "state": "CA",
+    "gender": "M",
+    "purchase": "Winter jacket",
+    "count": 1,
+    "date": "2001-07-04"
+  },
+  {
+    "email": "ben.tyler@example3.org",
+    "name": "Ben Tyler",
+    "state": "NV",
+    "gender": "M",
+    "purchase": "Flashlight",
+    "count": 1,
+    "date": "2001/07/04"
+  },
+  {
+    "email": "arthur.duff@example4.com",
+    "name": "Arthur Duff",
+    "state": "OR",
+    "gender": "M",
+    "purchase": "Dining table",
+    "count": 1,
+    "date": "2001-07"
+  },
+  {
+    "email": "danny.baron@example1.com",
+    "name": "Daniel Baron",
+    "state": "",
+    "gender": "",
+    "purchase": "Bike",
+    "count": 1,
+    "date": 2001
+  },
+  {
+    "email": "jean.griffith@example5.org",
+    "name": "Jean Griffith",
+    "state": "WA",
+    "gender": "F",
+    "purchase": "Power drill",
+    "count": 1,
+    "date": 2000
+  },
+  {
+    "email": "melanie.white@example2.edu",
+    "name": "Melanie White",
+    "state": "NC",
+    "gender": "F",
+    "purchase": "'iPad'",
+    "count": 1,
+    "date": 1999
+  },
+  {
+    "email": "ben.morisson@example6.org",
+    "name": "Ben Morisson",
+    "state": "FL",
+    "gender": "M",
+    "purchase": "Amplifier",
+    "count": 1,
+    "date": 1998
+  },
+  {
+    "email": "arthur.duff@example4.com",
+    "name": "Arthur Duff",
+    "state": "OR",
+    "gender": "M",
+    "purchase": "Night table",
+    "count": 1,
+    "date": 1997
+  }
+]

tests/data/cli/duplicates.tsv

@@ -0,0 +1,11 @@
+email	name	state	gender	purchase	count	date
+danny.baron@example1.com	Danny Baron	CA	M	TV (UTF-8: 📺)	1	Wed, 4 Jul 2001
+melanie.white@example2.edu	Melanie White	NC	F	<iPhone>	1	2001-07-04T12:08:56
+danny.baron@example1.com	"D.	(""Tab"") Baron"	CA	M	Winter jacket	1	2001-07-04
+ben.tyler@example3.org	Ben Tyler	NV	M	Flashlight	1	2001/07/04
+arthur.duff@example4.com	Arthur Duff	OR	M	Dining table	1	2001-07
+danny.baron@example1.com	Daniel Baron			Bike	1	2001
+jean.griffith@example5.org	Jean Griffith	WA	F	Power drill	1	2000
+melanie.white@example2.edu	Melanie White	NC	F	'iPad'	1	1999
+ben.morisson@example6.org	Ben Morisson	FL	M	Amplifier	1	1998
+arthur.duff@example4.com	Arthur Duff	OR	M	Night table	1	1997

tests/data/cli/duplicates.txt

@@ -0,0 +1,11 @@
+email                      name                 state gender purchase       count date              
+danny.baron@example1.com   Danny Baron          CA    M      TV (UTF-8: 📺) 1     Wed, 4 Jul 2001   
+melanie.white@example2.edu Melanie White        NC    F      <iPhone>       1     2001-07-04T12:08:5
+danny.baron@example1.com      D.	("Tab") Baron  CA    M      Winter jacket  1     2001-07-04        
+ben.tyler@example3.org     Ben Tyler            NV    M      Flashlight     1     2001/07/04        
+arthur.duff@example4.com   Arthur Duff          OR    M      Dining table   1     2001-07           
+danny.baron@example1.com   Daniel Baron                      Bike           1     2001              
+jean.griffith@example5.org Jean Griffith        WA    F      Power drill    1     2000              
+melanie.white@example2.edu Melanie White        NC    F      'iPad'         1     1999              
+ben.morisson@example6.org  Ben Morisson         FL    M      Amplifier      1     1998              
+arthur.duff@example4.com   Arthur Duff          OR    M      Night table    1     1997              

tests/data/cli/duplicates.xml

@@ -0,0 +1,93 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<root>
+  <record>
+    <email>danny.baron@example1.com</email>
+    <name>Danny Baron</name>
+    <state>CA</state>
+    <gender>M</gender>
+    <purchase>TV (UTF-8: 📺)</purchase>
+    <count>1</count>
+    <date>Wed, 4 Jul 2001</date>
+  </record>
+  <record>
+    <email>melanie.white@example2.edu</email>
+    <name>Melanie White</name>
+    <state>NC</state>
+    <gender>F</gender>
+    <purchase>&lt;iPhone&gt;</purchase>
+    <count>1</count>
+    <date>2001-07-04T12:08:56</date>
+  </record>
+  <record>
+    <email>danny.baron@example1.com</email>
+    <name>   D.	(&quot;Tab&quot;) Baron</name>
+    <state>CA</state>
+    <gender>M</gender>
+    <purchase>Winter jacket</purchase>
+    <count>1</count>
+    <date>2001-07-04</date>
+  </record>
+  <record>
+    <email>ben.tyler@example3.org</email>
+    <name>Ben Tyler</name>
+    <state>NV</state>
+    <gender>M</gender>
+    <purchase>Flashlight</purchase>
+    <count>1</count>
+    <date>2001/07/04</date>
+  </record>
+  <record>
+    <email>arthur.duff@example4.com</email>
+    <name>Arthur Duff</name>
+    <state>OR</state>
+    <gender>M</gender>
+    <purchase>Dining table</purchase>
+    <count>1</count>
+    <date>2001-07</date>
+  </record>
+  <record>
+    <email>danny.baron@example1.com</email>
+    <name>Daniel Baron</name>
+    <state></state>
+    <gender></gender>
+    <purchase>Bike</purchase>
+    <count>1</count>
+    <date>2001</date>
+  </record>
+  <record>
+    <email>jean.griffith@example5.org</email>
+    <name>Jean Griffith</name>
+    <state>WA</state>
+    <gender>F</gender>
+    <purchase>Power drill</purchase>
+    <count>1</count>
+    <date>2000</date>
+  </record>
+  <record>
+    <email>melanie.white@example2.edu</email>
+    <name>Melanie White</name>
+    <state>NC</state>
+    <gender>F</gender>
+    <purchase>&apos;iPad&apos;</purchase>
+    <count>1</count>
+    <date>1999</date>
+  </record>
+  <record>
+    <email>ben.morisson@example6.org</email>
+    <name>Ben Morisson</name>
+    <state>FL</state>
+    <gender>M</gender>
+    <purchase>Amplifier</purchase>
+    <count>1</count>
+    <date>1998</date>
+  </record>
+  <record>
+    <email>arthur.duff@example4.com</email>
+    <name>Arthur Duff</name>
+    <state>OR</state>
+    <gender>M</gender>
+    <purchase>Night table</purchase>
+    <count>1</count>
+    <date>1997</date>
+  </record>
+</root>

tests/data/cli/duplicates1.xml

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<record>
+  <email>danny.baron@example1.com</email>
+  <name>Danny Baron</name>
+  <state>CA</state>
+  <gender>M</gender>
+  <purchase>TV (UTF-8: 📺)</purchase>
+  <count>1</count>
+  <date>Wed, 4 Jul 2001</date>
+</record>

tests/data/cli/evil-fruits.tsv

@@ -0,0 +1,6 @@
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE

tests/delete-utf8.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --projectName "${t} biểu tượng cảm xúc ⛲"
+${cmd} --list | grep "${t}" || exit 1
+${cmd} --delete "${t} biểu tượng cảm xúc ⛲"
+${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/delete.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --list | grep "${t}" || exit 1
+${cmd} --delete "${t}"
+${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/download.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# ================================== ACTION ================================== #
+
+${cmd} --download "https://git.io/fj5ju" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "data/duplicates-deletion.json" "tmp/${t}/${t}.output"

tests/export-csv-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --export "${t}" --output "tmp/${t}/${t} biểu tượng cảm xúc 🍉.csv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t} biểu tượng cảm xúc 🍉.csv"

tests/export-csv.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.csv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.csv"

tests/export-html-utf8.sh

@@ -0,0 +1,72 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ ${2} ]]; then
+  majorversion="${2%%.*}"
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+if [[ "$majorversion" = 2 ]]; then
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+<html>
+<head>
+<title>export-html-utf8</title>
+<meta charset="utf-8" />
+</head>
+<body>
+<table>
+<tr><th>⌨</th><th>code</th><th>meaning</th></tr>
+<tr><td>&#9970;</td><td>1F347</td><td>FOUNTAIN</td></tr>
+<tr><td>&#9971;</td><td>1F349</td><td>FLAG IN HOLE</td></tr>
+<tr><td>&#9973;</td><td>1F352</td><td>SAILBOAT</td></tr>
+</table>
+</body>
+</html>
+DATA
+else
+  cat << "DATA" > "tmp/${t}/${t}.assert"
+<html>
+<head>
+<title>export-html-utf8</title>
+<meta charset="utf-8" />
+</head>
+<body>
+<table>
+<tr><th>⌨</th><th>code</th><th>meaning</th></tr>
+<tr><td>⛲</td><td>1F347</td><td>FOUNTAIN</td></tr>
+<tr><td>⛳</td><td>1F349</td><td>FLAG IN HOLE</td></tr>
+<tr><td>⛵</td><td>1F352</td><td>SAILBOAT</td></tr>
+</table>
+</body>
+</html>
+DATA
+fi
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.html"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.html"

tests/export-html.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+<html>
+<head>
+<title>export-html</title>
+<meta charset="utf-8" />
+</head>
+<body>
+<table>
+<tr><th>a</th><th>b</th><th>c</th></tr>
+<tr><td>1</td><td>2</td><td>3</td></tr>
+<tr><td>0</td><td>0</td><td>0</td></tr>
+<tr><td>$</td><td>\</td><td>&apos;</td></tr>
+</table>
+</body>
+</html>
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.html"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.html"

tests/export-ods-utf8.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,"FLAG IN HOLE"
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.ods"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.ods" "${t}.csv" &&
+  mv "${t}.csv.1" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-ods.sh

@@ -0,0 +1,47 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+if [[ -z "$(command -v ssconvert 2> /dev/null)" ]] ; then
+  echo 1>&2 "ERROR: This test requires ssconvert (gnumeric)"
+  exit 127
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.ods"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.ods" "${t}.csv" &&
+  mv "${t}.csv.1" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-tsv-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.tsv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.tsv"

tests/export-tsv.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.tsv"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.tsv"

tests/export-utf8.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-xls-utf8.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.xls"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.xls" "${t}.csv" &&
+  mv "${t}.csv" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-xls.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.xls"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.xls" "${t}.csv" &&
+  mv "${t}.csv" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export-xlsx-utf8.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+⌨,code,meaning
+⛲,1F347,FOUNTAIN
+⛳,1F349,FLAG IN HOLE
+⛵,1F352,SAILBOAT
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.xlsx"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.xlsx" "${t}.csv" &&
+  mv "${t}.csv" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"
+
+

tests/export-xlsx.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.xlsx"
+(cd tmp/"${t}" &&
+  ssconvert -S "${t}.xlsx" "${t}.csv" &&
+  mv "${t}.csv" "${t}.output")
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/export.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/format-create-separator.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.txt"
+a;b;c
+1;2;3
+0;0;0
+$;\;'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a	b	c
+1	2	3
+0	0	0
+$	\	'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.txt" --format "csv"  --separator ";"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/format-create.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Column 1
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --format "line-based"
+${cmd} --export "${t}" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/format-export-output.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --format "csv" --output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/format-export.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+a,b,c
+1,2,3
+0,0,0
+$,\,'
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" --format "csv" > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/help.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+Script to provide a command line interface to an OpenRefine server.
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --help | sed '3q;d' > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/info-utf8.sh

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.tsv"
+🔣	code	meaning
+🍇	1F347	GRAPES
+🍉	1F349	WATERMELON
+🍒	1F352	CHERRIES
+🍓	1F353	STRAWBERRY
+🍍	1F34D	PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+          column 001: 🔣
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.tsv"
+${cmd} --info "${t}" | grep 'column 001' > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/info.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+          column 002: b
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --info "${t}" | grep 'column 002' > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/list-utf8.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+ ${t} biểu tượng cảm xúc ⛲
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv" --projectName "${t} biểu tượng cảm xúc ⛲"
+${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/list.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+a,b,c
+1,2,3
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << DATA > "tmp/${t}/${t}.assert"
+ ${t}
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --list | grep "${t}" | cut -d ':' -f 2 > "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/template-facets.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+email,name,state,gender,purchase
+danny.baron@example1.com,Danny Baron,CA,M,TV
+melanie.white@example2.edu,Melanie White,NC,F,iPhone
+danny.baron@example1.com,D. Baron,CA,M,Winter jacket
+ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
+arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
+danny.baron@example1.com,Daniel Baron,CA,M,Bike
+jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
+melanie.white@example2.edu,Melanie White,NC,F,iPad
+ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
+arthur.duff@example4.com,Arthur Duff,OR,M,Night table
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "events" : [
+    { "name" : "Melanie White", "purchase" : "iPhone" },
+    { "name" : "Jean Griffith", "purchase" : "Power drill" },
+    { "name" : "Melanie White", "purchase" : "iPad" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "events" : [
+' \
+--template '    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--facets '{"type":"list","name":"gender","columnName":"gender","expression":"value","omitBlank":false,"omitError":false,"selection":[{"v":{"v":"F","l":"F"}}],"selectBlank":false,"selectError":false,"invert":false}' \
+--output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/template-filterQuery-utf8.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "emojis" : [
+    { "symbol" : "🍇", "meaning" : "GRAPES" },
+    { "symbol" : "🍉", "meaning" : "WATERMELON" },
+    { "symbol" : "🍍", "meaning" : "PINEAPPLE" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "emojis" : [
+' \
+--template '    { "symbol" : {{jsonize(with(row.columnNames[0],cn,cells[cn].value))}}, "meaning" : {{jsonize(cells["meaning"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--filterQuery '^1F34' \
+--filterColumn 'code' \
+--output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/template-filterQuery.sh

@@ -0,0 +1,59 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+email,name,state,gender,purchase
+danny.baron@example1.com,Danny Baron,CA,M,TV
+melanie.white@example2.edu,Melanie White,NC,F,iPhone
+danny.baron@example1.com,D. Baron,CA,M,Winter jacket
+ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
+arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
+danny.baron@example1.com,Daniel Baron,CA,M,Bike
+jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
+melanie.white@example2.edu,Melanie White,NC,F,iPad
+ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
+arthur.duff@example4.com,Arthur Duff,OR,M,Night table
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "events" : [
+    { "name" : "Melanie White", "purchase" : "iPhone" },
+    { "name" : "Jean Griffith", "purchase" : "Power drill" },
+    { "name" : "Melanie White", "purchase" : "iPad" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "events" : [
+' \
+--template '    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--filterQuery '^F$' \
+--filterColumn 'gender' \
+--output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}.output"

tests/template-splitToFiles-mode.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+email,name,state,gender,purchase
+arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
+,Arthur Duff,OR,M,Night table
+ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
+ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
+danny.baron@example1.com,Daniel Baron,CA,M,Bike
+,Danny Baron,CA,M,TV
+,D. Baron,CA,M,Winter jacket
+jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
+melanie.white@example2.edu,Melanie White,NC,F,iPad
+,Melanie White,NC,F,iPhone
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "events" : [
+    { "name" : "Melanie White", "purchase" : "iPad" }    { "name" : "Melanie White", "purchase" : "iPhone" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "events" : [
+' \
+--template '    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--mode "record-based" \
+--splitToFiles true \
+--output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+ls "tmp/${t}"
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}_6.output"

tests/template-splitToFiles-suffixById-utf8.sh

@@ -0,0 +1,52 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "emojis" : [
+    { "symbol" : "🍍", "meaning" : "PINEAPPLE" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "emojis" : [
+' \
+--template '    { "symbol" : {{jsonize(with(row.columnNames[0],cn,cells[cn].value))}}, "meaning" : {{jsonize(cells["meaning"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--splitToFiles true \
+--suffixById true \
+--output "tmp/${t}/trái cây.json"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/trái cây_🍍.json"

tests/template-splitToFiles-suffixById.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+email,name,state,gender,purchase
+danny.baron@example1.com,Danny Baron,CA,M,TV
+melanie.white@example2.edu,Melanie White,NC,F,iPhone
+danny.baron@example1.com,D. Baron,CA,M,Winter jacket
+ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
+arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
+danny.baron@example1.com,Daniel Baron,CA,M,Bike
+jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
+melanie.white@example2.edu,Melanie White,NC,F,iPad
+ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
+arthur.duff@example4.com,Arthur Duff,OR,M,Night table
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "events" : [
+    { "name" : "Arthur Duff", "purchase" : "Night table" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "events" : [
+' \
+--template '    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--splitToFiles true \
+--suffixById true \
+--output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+ls "tmp/${t}"
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}_arthur.duff@example4.com.output"

tests/template-splitToFiles-utf8.sh

@@ -0,0 +1,51 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+🔣,code,meaning
+🍇,1F347,GRAPES
+🍉,1F349,WATERMELON
+🍒,1F352,CHERRIES
+🍓,1F353,STRAWBERRY
+🍍,1F34D,PINEAPPLE
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "emojis" : [
+    { "symbol" : "🍍", "meaning" : "PINEAPPLE" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "emojis" : [
+' \
+--template '    { "symbol" : {{jsonize(with(row.columnNames[0],cn,cells[cn].value))}}, "meaning" : {{jsonize(cells["meaning"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--splitToFiles true \
+--output "tmp/${t}/trái cây.json"
+
+# =================================== TEST =================================== #
+
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/trái cây_5.json"

tests/template-splitToFiles.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# =============================== ENVIRONMENT ================================ #
+
+if [[ ${1} ]]; then
+  cmd="${1}"
+else
+  echo 1>&2 "execute tests-cli.sh to run all tests"; exit 1
+fi
+
+t="$(basename "${BASH_SOURCE[0]}" .sh)"
+cd "${BASH_SOURCE%/*}/" || exit 1
+mkdir -p "tmp/${t}"
+
+# =================================== DATA =================================== #
+
+cat << "DATA" > "tmp/${t}/${t}.csv"
+email,name,state,gender,purchase
+danny.baron@example1.com,Danny Baron,CA,M,TV
+melanie.white@example2.edu,Melanie White,NC,F,iPhone
+danny.baron@example1.com,D. Baron,CA,M,Winter jacket
+ben.tyler@example3.org,Ben Tyler,NV,M,Flashlight
+arthur.duff@example4.com,Arthur Duff,OR,M,Dining table
+danny.baron@example1.com,Daniel Baron,CA,M,Bike
+jean.griffith@example5.org,Jean Griffith,WA,F,Power drill
+melanie.white@example2.edu,Melanie White,NC,F,iPad
+ben.morisson@example6.org,Ben Morisson,FL,M,Amplifier
+arthur.duff@example4.com,Arthur Duff,OR,M,Night table
+DATA
+
+# ================================= ASSERTION ================================ #
+
+cat << "DATA" > "tmp/${t}/${t}.assert"
+{ "events" : [
+    { "name" : "Arthur Duff", "purchase" : "Night table" }
+] }
+DATA
+
+# ================================== ACTION ================================== #
+
+${cmd} --create "tmp/${t}/${t}.csv"
+${cmd} --export "${t}" \
+--prefix '{ "events" : [
+' \
+--template '    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
+--rowSeparator ',
+' \
+--suffix '
+] }
+' \
+--splitToFiles true \
+--output "tmp/${t}/${t}.output"
+
+# =================================== TEST =================================== #
+
+ls "tmp/${t}"
+diff -u "tmp/${t}/${t}.assert" "tmp/${t}/${t}_10.output"