openrefine-client/docker

batch processing with python-client

There are some client libraries for OpenRefine that communicate with the OpenRefine API. I have prepared a docker container on top of the Python Library from PaulMakepeace and extended the CLI with some options to create new OpenRefine projects from files.

basic usage

1. start server: docker run -d --name=openrefine-server felixlohmeier/openrefine

2. run client with one of the following commands:

• list projects: docker run --rm --link openrefine-server felixlohmeier/openrefine-client --list
• create project from file: docker run --rm --link openrefine-server felixlohmeier/openrefine-client --create [FILE] [PROJECTID]
• apply rules from json file: docker run --rm --link openrefine-server felixlohmeier/openrefine-client --apply [FILE.json] [PROJECTID]
• export project to file: docker run --rm --link openrefine-server felixlohmeier/openrefine-client --export [PROJECTID] --output=FILE.tsv
• check help screen for more options: docker run --rm --link openrefine-server felixlohmeier/openrefine-client --help

3. cleanup: docker stop openrefine-server && docker rm openrefine-server

example for customized run commands in interactive mode (e.g. for usage in terminals)

1. start server in terminal A:

docker run --rm --name=openrefine-server -p 80:3333 -v /home/felix/refine:/data:z felixlohmeier/openrefine -i 0.0.0.0 -m 4G -d /data

• automatically remove docker container when it exits
• set name "openrefine" for docker container
• publish internal port 3333 to host port 80
• mount host directory /home/felix/refine as working directory
• make openrefine available in the network
• increase java heap size to 4 GB
• set refine workspace to /data
• OpenRefine should be available at http://localhost

2. start client in terminal B (prints help screen):

docker run --rm --link openrefine-server -v /home/felix/refine:/data:z felixlohmeier/openrefine-client

• automatically remove docker container when it exits
• build up network connection with docker container "openrefine"
• mount host directory /home/felix/refine as working directory
• apply history in file /home/felix/refine/history.json to project with id 1234567890123

example for customized run commands in detached mode (e.g. for usage in shell scripts)

1. define variables (bring your own example data)

workingdir=/home/felix/refine inputfile=example.csv jsonfile=test.json

2. start server

docker run -d --name=openrefine-server -v ${workingdir}:/data:z felixlohmeier/openrefine -i 0.0.0.0 -m 4G -d /data

3. create project (import file)

docker run --rm --link openrefine-server -v ${workingdir}:/data:z felixlohmeier/openrefine-client --create $inputfile

4. get project id

project=($(docker run --rm --link openrefine-server -v ${workingdir}:/data felixlohmeier/openrefine-client --list | cut -c 2-14))

5. apply transformations from json file

docker run --rm --link openrefine-server -v ${workingdir}:/data felixlohmeier/openrefine-client --apply ${jsonfile} ${project}

6. export project to file

docker run --rm --link openrefine-server -v ${workingdir}:/data felixlohmeier/openrefine-client --export --output=${project}.tsv ${project}

7. cleanup

docker stop -t=500 openrefine-server && docker rm openrefine-server