OpenRefine Python Client with extended command line interface

The OpenRefine Python Client from PaulMakepeace provides a library for communicating with an OpenRefine server. This fork extends the command line interface (CLI) and is distributed as a convenient one-file-executable (Windows, Linux, macOS). It is also available via Docker Hub, PyPI and Binder.

works with OpenRefine 2.7, 2.8, 3.0, 3.1, 3.2

Download

One-file-executables:

• Windows: openrefine-client_0-3-9_windows.exe (~5 MB)
• macOS: openrefine-client_0-3-9_macos (~5 MB)
• Linux: openrefine-client_0-3-9_linux (~5 MB)

For Docker containers, native Python installation and free Binder on-demand server see the corresponding chapters below.

Peek

A short video loop that demonstrates the basic features (list, create, apply, export):

Usage

Ensure you have OpenRefine running (i.e. available at http://localhost:3333 or another URL).

To use the client:

1. Open a terminal pointing to the folder where you have downloaded the one-file-executable (e.g. Downloads in your home directory).

   • Windows: Open PowerShell and enter following command

     cd ~\Downloads
     

   • macOS: Open Terminal (Finder > Applications > Utilities > Terminal) and enter following command

     cd ~/Downloads
     

   • Linux: Open terminal app (Terminal, Konsole, xterm, ...) and enter following command

     cd ~/Downloads
     

2. Make the file executable.

   • Windows: not necessary

   • macOS:

     chmod +x openrefine-client_0-3-9_macos
     

   • Linux:

     chmod +x openrefine-client_0-3-9_linux
     

3. Execute the file.

   • Windows:

     .\openrefine-client_0-3-9_windows.exe
     

   • macOS:

     ./openrefine-client_0-3-9_macos
     

   • Linux:

     ./openrefine-client_0-3-9_linux
     

Using tab completion and command history is highly recommended:

• autocomplete filenames: enter a few characters and press ↹
• recall previous command: press ↑

Basic commands

Execute the client by entering its filename followed by the desired command.

The following example will download two small files (duplicates.csv and duplicates-deletion.json) into the current directory and will create a new OpenRefine project from file duplicates.csv.

Download example data (--download) and create project from file (--create):

• Windows:

  .\openrefine-client_0-3-9_windows.exe --download "https://git.io/fj5hF" --output=duplicates.csv
  .\openrefine-client_0-3-9_windows.exe --download "https://git.io/fj5ju" --output=duplicates-deletion.json
  .\openrefine-client_0-3-9_windows.exe --create duplicates.csv
  

• macOS:

  ./openrefine-client_0-3-9_macos --download "https://git.io/fj5hF" --output=duplicates.csv
  ./openrefine-client_0-3-9_macos --download "https://git.io/fj5ju" --output=duplicates-deletion.json
  ./openrefine-client_0-3-9_macos --create duplicates.csv
  

• Linux:

  ./openrefine-client_0-3-9_linux --download "https://git.io/fj5hF" --output=duplicates.csv
  ./openrefine-client_0-3-9_linux --download "https://git.io/fj5ju" --output=duplicates-deletion.json
  ./openrefine-client_0-3-9_linux --create duplicates.csv
  

Other commands:

• list all projects: --list
• show project metadata: --info "duplicates"
• export project to terminal: --export "duplicates"
• apply rules from json file: --apply duplicates-deletion.json "duplicates"
• export project to file: --export --output=deduped.xls "duplicates"
• delete project: --delete "duplicates"

Getting help

Check --help for further options.

Please file an issue if you miss some features in the command line interface or if you have tracked a bug. And you are welcome to ask any questions!

Change URL

By default the client connects to the usual URL of OpenRefine http://localhost:3333. If your OpenRefine server is running somewhere else then you may set hostname and port with additional command line options (e.g. http://example.com):

• set host: -H example.com
• set port: -P 80

Advanced Templating

The OpenRefine Templating supports exporting data in any text format (i.e. to construct JSON or XML). The graphical user interface offers four input fields:

1. prefix
2. row template
   • supports GREL inside two curly brackets, e.g. {{jsonize(cells["name"].value)}}
3. row separator
4. suffix

This templating functionality is available via the openrefine-client command line interface. It even provides an additional feature for splitting results into multiple files.

To try out the functionality create another project from the example file above.

--create duplicates.csv --projectName=advanced

The following example code will export...

• the columns "name" and "purchase" in JSON format
• from the project "advanced"
• for rows matching the regex text filter ^F$ in column "gender"

macOS/Linux Terminal (multi-line input with \ ):

"advanced" \
--prefix='{ "events" : [
' \
--template='    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }' \
--rowSeparator=',
' \
--suffix='
] }' \
--filterQuery='^F$' \
--filterColumn='gender'

Windows PowerShell (multi-line input with `; quotes needs to be doubled):

"advanced" `
--prefix='{ ""events"" : [
' `
--template='    { ""name"" : {{jsonize(cells[""name""].value)}}, ""purchase"" : {{jsonize(cells[""purchase""].value)}} }' `
--rowSeparator=',
' `
--suffix='
] }' `
--filterQuery='^F$' `
--filterColumn='gender'

Add the following options to the last command (recall with ↑) to store the results in multiple files. Each file will contain the prefix, an processed row, and the suffix.

--output=advanced.json --splitToFiles=true

Filenames are suffixed with the row number by default (e.g. advanced_1.json, advanced_2.json etc.). There is another option to use the value in the first column instead:

--output=advanced.json --splitToFiles=true --suffixById=true

Because our project "advanced" contains duplicates in the first column "email" this command will overwrite files (e.g. advanced_melanie.white@example2.edu.json). When using this option, the first column should contain unique identifiers.

See also

• Linux Bash script to run OpenRefine in batch mode (import, transform, export): openrefine-batch
• Jupyter notebook demonstrating usage in Linux Bash
• Use case HOS-MetadataTransformations: Automated workflow for harvesting, transforming and indexing of metadata using metha, OpenRefine and Solr. Part of the Hamburg Open Science "Schaufenster" software stack.
• Use case Data processing of ILS data to facilitate a new discovery layer for the German Literature Archive (DLA): Custom data processing pipeline based on Pandas (a Python library) and OpenRefine.

Docker

felixlohmeier/openrefine-client

docker pull felixlohmeier/openrefine-client:v0.3.9

Option 1: Dockerized client

Run client and mount current directory as workspace:

docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9

The docker option --network=host allows you to connect to a local or remote OpenRefine via the host network:

• list projects on default URL (http://localhost:3333)

  docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 --list
  

• list projects on a remote server (http://example.com)

  docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H example.com -P 80 --list
  

Usage: same commands as explained above (see Basic Commands and Advanced Templating)

Option 2: Dockerized client and dockerized OpenRefine

Run openrefine-client linked to a dockerized OpenRefine (felixlohmeier/openrefine ):

1. Create docker network

   docker network create openrefine
   

2. Run server (will be available at http://localhost:3333)

   docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.2
   

3. Run client with some basic commands: 1. download example files, 2. create project from file, 3. list projects, 4. show metadata, 5. export to terminal, 6. apply transformation rules (deduplication), 7. export again to terminal, 8. export to xls file and 9. delete project

   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 --download "https://git.io/fj5hF" --output=duplicates.csv
   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 --download "https://git.io/fj5ju" --output=duplicates-deletion.json
   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --create duplicates.csv
   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --list
   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --info "duplicates"
   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --export "duplicates"
   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --apply duplicates-deletion.json "duplicates"
   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --export "duplicates"
   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --export --output=deduped.xls "duplicates"
   docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --delete "duplicates"
   

4. Stop and delete server:

   docker stop openrefine-server
   docker rm openrefine-server
   

5. Delete docker network:

   docker network rm openrefine
   

Customize OpenRefine server:

• If you want to add an OpenRefine startup option you need to repeat the default commands (cf. Dockerfile)

  • -i 0.0.0.0 sets OpenRefine to be accessible from outside the container, i.e. from host
  • -d /data sets OpenRefine workspace

• Example for allocating more memory to OpenRefine with additional option -m 4G

  docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.2 -i 0.0.0.0 -d /data -m 4G
  

• The OpenRefine version is defined by the docker tag. Check the DockerHub repository for available tags. Example for OpenRefine 2.8 with same options as above:

  docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:2.8 -i 0.0.0.0 -d /data -m 4G
  

• If you want OpenRefine to read and write persistent data in host directory (i.e. store projects) you can mount the container path /data. Example for host directory /home/felix/refine:

  docker run -d -p 3333:3333 -v /home/felix/refine:/data:z --network=openrefine name=openrefine-server felixlohmeier/openrefine:2.8 -i 0.0.0.0 -d /data -m 4G
  

See also:

• GitHub Repository for docker container felixlohmeier/openrefine
• Linux Bash script to run OpenRefine in batch mode (import, transform, export) with docker containers: openrefine-batch-docker.sh

Python

openrefine-client (requires Python 3.x)

python3 -m pip install openrefine-client --user

This will install the package openrefine-client containing modules in google.refine.

A command line script openrefine-client will also be installed.

Option 1: command line script

openrefine-client --help

Usage: same commands as explained above (see Basic Commands and Advanced Templating)

Option 2: using cli functions in Python 3.x environment

Import module cli:

from google.refine import cli

Change URL (if necessary):

cli.refine.REFINE_HOST = 'localhost'
cli.refine.REFINE_PORT = '3333'

Help screen:

help(cli)

Commands:

• download (e.g. example data):

  cli.download('https://git.io/fj5hF','duplicates.csv')
  cli.download('https://git.io/fj5ju','duplicates-deletion.json')
  

• list projects:

  cli.ls()
  

• create project:

  p1 = cli.create('duplicates.csv')
  

• show metadata:

  cli.info(p1.project_id)
  

• apply rules from file to project:

  cli.apply(p1.project_id, 'duplicates-deletion.json')
  

• export project to terminal:

  cli.export(p1.project_id)
  

• export project to file in xls format:

  cli.export(p1.project_id, 'deduped.xls')
  

• export templating (see Advanced Templating above):

  cli.templating(
      p1.project_id,
      prefix='''{ "events" : [
  ''',template='''    { "name" : {{jsonize(cells["name"].value)}}, "purchase" : {{jsonize(cells["purchase"].value)}} }''',
      rowSeparator=''',
  ''',suffix='''
  ] }''')
  

• delete project:

  cli.delete(p1.project_id)
  

See also:

• Jupyter notebook by Tony Hirst (2019-01-09) Notebook demonstrating how to control OpenRefine via a Python client.
• OpenRefine API in official OpenRefine wiki

Binder

• free to use on-demand server with Jupyter notebook, OpenRefine and Bash
• no registration needed, will start within a few minutes
• restricted to 2 GB RAM and server will be deleted after 10 minutes of inactivity
• bash_kernel demo notebook for using the openrefine-client in a Linux Bash environment

Development

Tests

TODO

There is also a script that uses docker images to run the unit tests with different versions of OpenRefine.

• run tests on all OpenRefine versions (from 2.0 up to 3.2)

  ./tests.sh -a
  

• run tests on tag 3.2

  ./tests.sh -t 3.2
  

• run tests on tag 3.2 interactively (pause before and after tests)

  ./tests.sh -t 3.2 -i
  

• run tests on tags 3.2 and 2.7

  ./tests.sh -t 3.2 -t 2.7
  

Distributing

Note to myself: When releasing a new version...

1. Run tests

   ./tests.sh -a
   

2. Make final changes in Git

   • update versions (e.g. 0.3.7 und 0-3-7) in README.md
   • update version in setup.py
   • check if Dockerfile needs to be changed

3. Build executables with PyInstaller

   • Run PyInstaller in Python 3 environments on native Windows, macOS and Linux. Should be "the oldest version of the OS you need to support"! Current release is built with:

     • Ubuntu 16.04 LTS (64-bit)
     • macOS Sierra 10.12 (64-bit)
     • Windows 7 (32-bit)

   • One-file-executables will be available in dist/.

     git clone https://github.com/opencultureconsulting/openrefine-client.git
     cd openrefine-client
     python3 -m pip install . --user
     python3 -m pip install pyinstaller --user
     python3 -m pip PyInstaller --onefile refine.py --hidden-import google.refine.__main__
     

4. Run test with Linux executable

   ./tests.sh -a
   

5. Create release in GitHub

   • draft release notes and attach one-file-executables

6. Build package and upload to PyPI

   TODO
   python3 setup.py sdist bdist_wheel
   python3 -m twine upload dist/*
   

7. Update Docker container

   • add new autobuild for release version
   • trigger latest build

8. Bump openrefine-client version in related projects

   • openrefine-batch: openrefine-batch.sh and openrefine-batch-docker.sh
   • openrefineder: postBuild and openrefine-client-bash.ipynb
   • openrefineder: postBuild

Credits

Paul Makepeace, author

David Huynh, [initial cut](<http://markmail.org/message/jsxzlcu3gn6drtb7)

Felix Lohmeier, CLI features

Wolf Vollprecht, port to python 3