opencultureconsulting
/
openrefine-client
mirror of https://github.com/opencultureconsulting/openrefine-client.git
2
0
Fork 0
Code Issues Releases Activity

release 0.3.10

This commit is contained in:
Felix Lohmeier 2021-01-04 14:01:54 +01:00
parent 9eeebce47c
commit a0274f6166
3 changed files with 62 additions and 48 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

96
README.md
View File

@ -12,9 +12,9 @@ works with OpenRefine 2.7, 2.8, 3.0, 3.1, 3.2, 3.3, 3.4, 3.4.1
One-file-executables: One-file-executables:
- Windows: [openrefine-client_0-3-9_windows.exe](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.9/openrefine-client_0-3-9_windows.exe) (~5 MB) - Windows: [openrefine-client_0-3-10_windows.exe](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_windows.exe) (~5 MB)
- macOS: [openrefine-client_0-3-9_macos](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.9/openrefine-client_0-3-9_macos) (~5 MB) - macOS: [openrefine-client_0-3-10_macos](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_macos) (~5 MB)
- Linux: [openrefine-client_0-3-9_linux](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.9/openrefine-client_0-3-9_linux) (~5 MB) - Linux: [openrefine-client_0-3-10_linux](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.10/openrefine-client_0-3-10_linux) (~5 MB)
For [Docker](#docker) containers, native [Python](#python) installation and free [Binder](#binder) on-demand server see the corresponding chapters below. For [Docker](#docker) containers, native [Python](#python) installation and free [Binder](#binder) on-demand server see the corresponding chapters below.
@ -57,13 +57,13 @@ To use the client:
- macOS: - macOS:
```sh ```sh
chmod +x openrefine-client_0-3-9_macos chmod +x openrefine-client_0-3-10_macos
``` ```
- Linux: - Linux:
```sh ```sh
chmod +x openrefine-client_0-3-9_linux chmod +x openrefine-client_0-3-10_linux
``` ```
3. Execute the file. 3. Execute the file.
@ -71,19 +71,19 @@ To use the client:
- Windows: - Windows:
```sh ```sh
.\openrefine-client_0-3-9_windows.exe .\openrefine-client_0-3-10_windows.exe
``` ```
- macOS: - macOS:
```sh ```sh
./openrefine-client_0-3-9_macos ./openrefine-client_0-3-10_macos
``` ```
- Linux: - Linux:
```sh ```sh
./openrefine-client_0-3-9_linux ./openrefine-client_0-3-10_linux
``` ```
Using tab completion and command history is highly recommended: Using tab completion and command history is highly recommended:
@ -102,25 +102,25 @@ Download example data (`--download`) and create project from file (`--create`):
- Windows: - Windows:
```sh ```sh
.\openrefine-client_0-3-9_windows.exe --download "https://git.io/fj5hF" --output=duplicates.csv .\openrefine-client_0-3-10_windows.exe --download "https://git.io/fj5hF" --output=duplicates.csv
.\openrefine-client_0-3-9_windows.exe --download "https://git.io/fj5ju" --output=duplicates-deletion.json .\openrefine-client_0-3-10_windows.exe --download "https://git.io/fj5ju" --output=duplicates-deletion.json
.\openrefine-client_0-3-9_windows.exe --create duplicates.csv .\openrefine-client_0-3-10_windows.exe --create duplicates.csv
``` ```
- macOS: - macOS:
```sh ```sh
./openrefine-client_0-3-9_macos --download "https://git.io/fj5hF" --output=duplicates.csv ./openrefine-client_0-3-10_macos --download "https://git.io/fj5hF" --output=duplicates.csv
./openrefine-client_0-3-9_macos --download "https://git.io/fj5ju" --output=duplicates-deletion.json ./openrefine-client_0-3-10_macos --download "https://git.io/fj5ju" --output=duplicates-deletion.json
./openrefine-client_0-3-9_macos --create duplicates.csv ./openrefine-client_0-3-10_macos --create duplicates.csv
``` ```
- Linux: - Linux:
```sh ```sh
./openrefine-client_0-3-9_linux --download "https://git.io/fj5hF" --output=duplicates.csv ./openrefine-client_0-3-10_linux --download "https://git.io/fj5hF" --output=duplicates.csv
./openrefine-client_0-3-9_linux --download "https://git.io/fj5ju" --output=duplicates-deletion.json ./openrefine-client_0-3-10_linux --download "https://git.io/fj5ju" --output=duplicates-deletion.json
./openrefine-client_0-3-9_linux --create duplicates.csv ./openrefine-client_0-3-10_linux --create duplicates.csv
``` ```
Other commands: Other commands:
@ -257,7 +257,7 @@ openrefine-client --create combined.zip --format csv --projectName myproject --i
[felixlohmeier/openrefine-client](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [![Docker](https://img.shields.io/microbadger/image-size/felixlohmeier/openrefine-client?label=docker)](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [felixlohmeier/openrefine-client](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [![Docker](https://img.shields.io/microbadger/image-size/felixlohmeier/openrefine-client?label=docker)](https://hub.docker.com/r/felixlohmeier/openrefine-client/)
```sh ```sh
docker pull felixlohmeier/openrefine-client:v0.3.9 docker pull felixlohmeier/openrefine-client:v0.3.10
``` ```
### Option 1: Dockerized client ### Option 1: Dockerized client
@ -265,7 +265,7 @@ docker pull felixlohmeier/openrefine-client:v0.3.9
Run client and mount current directory as workspace: Run client and mount current directory as workspace:
```sh ```sh
docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10
``` ```
The docker option `--network=host` allows you to connect to a local or remote OpenRefine via the host network: The docker option `--network=host` allows you to connect to a local or remote OpenRefine via the host network:
@ -273,13 +273,13 @@ The docker option `--network=host` allows you to connect to a local or remote Op
- list projects on default URL (http://localhost:3333) - list projects on default URL (http://localhost:3333)
```sh ```sh
docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 --list docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --list
``` ```
- list projects on a remote server (http://example.com) - list projects on a remote server (http://example.com)
```sh ```sh
docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H example.com -P 80 --list docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H example.com -P 80 --list
``` ```
Usage: same commands as explained above (see [Basic Commands](#basic-commands) and [Advanced Templating](#advanced-templating)) Usage: same commands as explained above (see [Basic Commands](#basic-commands) and [Advanced Templating](#advanced-templating))
@ -297,22 +297,22 @@ Run openrefine-client linked to a dockerized OpenRefine ([felixlohmeier/openrefi
2. Run server (will be available at http://localhost:3333) 2. Run server (will be available at http://localhost:3333)
```sh ```sh
docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.2 docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.4.1
``` ```
3. Run client with some [basic commands](#basic-commands): 1. download example files, 2. create project from file, 3. list projects, 4. show metadata, 5. export to terminal, 6. apply transformation rules (deduplication), 7. export again to terminal, 8. export to xls file and 9. delete project 3. Run client with some [basic commands](#basic-commands): 1. download example files, 2. create project from file, 3. list projects, 4. show metadata, 5. export to terminal, 6. apply transformation rules (deduplication), 7. export again to terminal, 8. export to xls file and 9. delete project
```sh ```sh
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 --download "https://git.io/fj5hF" --output=duplicates.csv docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --download "https://git.io/fj5hF" --output=duplicates.csv
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 --download "https://git.io/fj5ju" --output=duplicates-deletion.json docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 --download "https://git.io/fj5ju" --output=duplicates-deletion.json
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --create duplicates.csv docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --create duplicates.csv
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --list docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --list
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --info "duplicates" docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --info "duplicates"
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --export "duplicates" docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export "duplicates"
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --apply duplicates-deletion.json "duplicates" docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --apply duplicates-deletion.json "duplicates"
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --export "duplicates" docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export "duplicates"
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --export --output=deduped.xls "duplicates" docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --export --output=deduped.xls "duplicates"
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --delete "duplicates" docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.10 -H openrefine-server --delete "duplicates"
``` ```
4. Stop and delete server: 4. Stop and delete server:
@ -337,7 +337,7 @@ Customize OpenRefine server:
- Example for [allocating more memory](https://github.com/OpenRefine/OpenRefine/wiki/FAQ#out-of-memory-errors---feels-slow---could-not-reserve-enough-space-for-object-heap) to OpenRefine with additional option `-m 4G` - Example for [allocating more memory](https://github.com/OpenRefine/OpenRefine/wiki/FAQ#out-of-memory-errors---feels-slow---could-not-reserve-enough-space-for-object-heap) to OpenRefine with additional option `-m 4G`
```sh ```sh
docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.2 -i 0.0.0.0 -d /data -m 4G docker run -d -p 3333:3333 --network=openrefine --name=openrefine-server felixlohmeier/openrefine:3.4.1 -i 0.0.0.0 -d /data -m 4G
``` ```
- The OpenRefine version is defined by the docker tag. - The OpenRefine version is defined by the docker tag.
@ -651,28 +651,42 @@ The Python client library includes several unit tests.
There is also a script that uses docker images to run the unit tests with different versions of OpenRefine. There is also a script that uses docker images to run the unit tests with different versions of OpenRefine.
- run tests on all OpenRefine versions (from 2.0 up to 3.2) - run tests on all OpenRefine versions (from 2.0 up to 3.4.1)
```sh ```sh
./tests.sh -a ./tests.sh -a
``` ```
- run tests on tag 3.2 - run tests on tag 3.4.1
```sh ```sh
./tests.sh -t 3.2 ./tests.sh -t 3.4.1
``` ```
- run tests on tag 3.2 interactively (pause before and after tests) - run tests on tag 3.4.1 interactively (pause before and after tests)
```sh ```sh
./tests.sh -t 3.2 -i ./tests.sh -t 3.4.1 -i
``` ```
- run tests on tags 3.2 and 2.7 - run tests on tags 3.4.1 and 2.7
```sh ```sh
./tests.sh -t 3.2 -t 2.7 ./tests.sh -t 3.4.1 -t 2.7
```
For Linux there are also functional tests for all command line options.
- run all functional tests on OpenRefine 3.4
```sh
./tests-cli.sh 3.4.1
```
- run all functional tests on OpenRefine 3.4 with one-file-executable
```sh
./tests-cli.sh 3.4.1 openrefine-client_0-3-7_linux
``` ```
### Distributing ### Distributing
@ -682,7 +696,7 @@ Note to myself: When releasing a new version...
1. Run functional tests 1. Run functional tests
```sh ```sh
for v in 2.7 2.8 3.0 3.1 3.2; do for v in 2.7 2.8 3.0 3.1 3.2 3.3 3.4 3.4.1; do
./tests-cli.sh $v ./tests-cli.sh $v
done done
``` ```

2
setup.py
View File

@ -25,7 +25,7 @@ def read(filename):
return open(os.path.join(os.path.dirname(__file__), filename)).read() return open(os.path.join(os.path.dirname(__file__), filename)).read()
setup(name='openrefine-client', setup(name='openrefine-client',
version='0.3.9', version='0.3.10',
description=('The OpenRefine Python Client Library provides an ' description=('The OpenRefine Python Client Library provides an '
'interface to communicating with an OpenRefine server. ' 'interface to communicating with an OpenRefine server. '
'This fork extends the command line interface (CLI).'), 'This fork extends the command line interface (CLI).'),

12
tests.sh
View File

@ -17,8 +17,8 @@
# along with this program. If not, see <http://www.gnu.org/licenses/> # along with this program. If not, see <http://www.gnu.org/licenses/>
# defaults: # defaults:
all=(3.2-java12 3.2-java11 3.2-java10 3.2-java9 3.2 3.1-java9 3.1 3.0-java9 3.0 2.8-java9 2.8 2.8-java7 2.7 2.7-java7 2.5-java7 2.5-java6 2.1-java6 2.0-java6) all=(3.4.1 3.4 3.3 3.2-java12 3.2-java11 3.2-java10 3.2-java9 3.2 3.1-java9 3.1 3.0-java9 3.0 2.8-java9 2.8 2.8-java7 2.7 2.7-java7 2.5-java7 2.5-java6 2.1-java6 2.0-java6)
main=(3.2 3.1 3.0 2.8 2.7 2.5-java6 2.1-java6 2.0-java6) main=(3.4.1 3.4 3.3 3.2 3.1 3.0 2.8 2.7 2.5-java6 2.1-java6 2.0-java6)
interactively=false interactively=false
port="3333" port="3333"
@ -31,10 +31,10 @@ Script for running tests with different OpenRefine and Java versions.
It uses docker images from https://hub.docker.com/r/felixlohmeier/openrefine. It uses docker images from https://hub.docker.com/r/felixlohmeier/openrefine.
Examples: Examples:
./tests.sh -a # run tests on all OpenRefine versions (from 2.0 up to 3.2) ./tests.sh -a # run tests on all OpenRefine versions (from 2.0 up to 3.4.1)
./tests.sh -t 3.2 # run tests on tag 3.2 ./tests.sh -t 3.4.1 # run tests on tag 3.4.1
./tests.sh -t 3.2 -i # run tests on tag 3.2 interactively (pause before and after tests) ./tests.sh -t 3.4.1 -i # run tests on tag 3.4.1 interactively (pause before and after tests)
./tests.sh -t 3.2 -t 2.7 # run tests on tags 3.2 and 2.7 ./tests.sh -t 3.4.1 -t 2.7 # run tests on tags 3.4.1 and 2.7
Advanced: Advanced:
./tests.sh -j # run tests on all OpenRefine versions and each with all supported Java versions (requires a lot of docker images to be downloaded!) ./tests.sh -j # run tests on all OpenRefine versions and each with all supported Java versions (requires a lot of docker images to be downloaded!)