opencultureconsulting
/
openrefine-client
mirror of https://github.com/opencultureconsulting/openrefine-client.git
2
0
Fork 0
Code Issues Releases Activity

Merge branch 'master' into py3_port

This commit is contained in:
Felix Lohmeier 2020-08-08 11:53:35 +02:00 committed by GitHub
parent b519a971e5 1c812d1253
commit f06fc54e12
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 40 additions and 38 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

72
README.md
View File

@ -6,13 +6,15 @@ The [OpenRefine Python Client from PaulMakepeace](https://github.com/PaulMakepea
This fork extends the command line interface (CLI) and is distributed as a convenient one-file-executable (Windows, Linux, macOS). This fork extends the command line interface (CLI) and is distributed as a convenient one-file-executable (Windows, Linux, macOS).
It is also available via Docker Hub, PyPI and Binder. It is also available via Docker Hub, PyPI and Binder.
works with OpenRefine 2.7, 2.8, 3.0, 3.1, 3.2
## Download ## Download
One-file-executables: One-file-executables:
- Windows: [openrefine-client_0-3-8_windows.exe](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.8/openrefine-client_0-3-8_windows.exe) (~5 MB) - Windows: [openrefine-client_0-3-9_windows.exe](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.9/openrefine-client_0-3-9_windows.exe) (~5 MB)
- macOS: [openrefine-client_0-3-8_macos](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.8/openrefine-client_0-3-8_macos) (~5 MB) - macOS: [openrefine-client_0-3-9_macos](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.9/openrefine-client_0-3-9_macos) (~5 MB)
- Linux: [openrefine-client_0-3-8_linux](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.8/openrefine-client_0-3-8_linux) (~5 MB) - Linux: [openrefine-client_0-3-9_linux](https://github.com/opencultureconsulting/openrefine-client/releases/download/v0.3.9/openrefine-client_0-3-9_linux) (~5 MB)
For [Docker](#docker) containers, native [Python](#python) installation and free [Binder](#binder) on-demand server see the corresponding chapters below. For [Docker](#docker) containers, native [Python](#python) installation and free [Binder](#binder) on-demand server see the corresponding chapters below.
@ -55,13 +57,13 @@ To use the client:
- macOS: - macOS:
```sh ```sh
chmod +x openrefine-client_0-3-8_macos chmod +x openrefine-client_0-3-9_macos
``` ```
- Linux: - Linux:
```sh ```sh
chmod +x openrefine-client_0-3-8_linux chmod +x openrefine-client_0-3-9_linux
``` ```
3. Execute the file. 3. Execute the file.
@ -69,19 +71,19 @@ To use the client:
- Windows: - Windows:
```sh ```sh
.\openrefine-client_0-3-8_windows.exe .\openrefine-client_0-3-9_windows.exe
``` ```
- macOS: - macOS:
```sh ```sh
./openrefine-client_0-3-8_macos ./openrefine-client_0-3-9_macos
``` ```
- Linux: - Linux:
```sh ```sh
./openrefine-client_0-3-8_linux ./openrefine-client_0-3-9_linux
``` ```
Using tab completion and command history is highly recommended: Using tab completion and command history is highly recommended:
@ -100,25 +102,25 @@ Download example data (`--download`) and create project from file (`--create`):
- Windows: - Windows:
```sh ```sh
.\openrefine-client_0-3-8_windows.exe --download "https://git.io/fj5hF" --output=duplicates.csv .\openrefine-client_0-3-9_windows.exe --download "https://git.io/fj5hF" --output=duplicates.csv
.\openrefine-client_0-3-8_windows.exe --download "https://git.io/fj5ju" --output=duplicates-deletion.json .\openrefine-client_0-3-9_windows.exe --download "https://git.io/fj5ju" --output=duplicates-deletion.json
.\openrefine-client_0-3-8_windows.exe --create duplicates.csv .\openrefine-client_0-3-9_windows.exe --create duplicates.csv
``` ```
- macOS: - macOS:
```sh ```sh
./openrefine-client_0-3-8_macos --download "https://git.io/fj5hF" --output=duplicates.csv ./openrefine-client_0-3-9_macos --download "https://git.io/fj5hF" --output=duplicates.csv
./openrefine-client_0-3-8_macos --download "https://git.io/fj5ju" --output=duplicates-deletion.json ./openrefine-client_0-3-9_macos --download "https://git.io/fj5ju" --output=duplicates-deletion.json
./openrefine-client_0-3-8_macos --create duplicates.csv ./openrefine-client_0-3-9_macos --create duplicates.csv
``` ```
- Linux: - Linux:
```sh ```sh
./openrefine-client_0-3-8_linux --download "https://git.io/fj5hF" --output=duplicates.csv ./openrefine-client_0-3-9_linux --download "https://git.io/fj5hF" --output=duplicates.csv
./openrefine-client_0-3-8_linux --download "https://git.io/fj5ju" --output=duplicates-deletion.json ./openrefine-client_0-3-9_linux --download "https://git.io/fj5ju" --output=duplicates-deletion.json
./openrefine-client_0-3-8_linux --create duplicates.csv ./openrefine-client_0-3-9_linux --create duplicates.csv
``` ```
Other commands: Other commands:
@ -230,7 +232,7 @@ When using this option, the first column should contain unique identifiers.
[felixlohmeier/openrefine-client](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [![Docker](https://img.shields.io/microbadger/image-size/felixlohmeier/openrefine-client?label=docker)](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [felixlohmeier/openrefine-client](https://hub.docker.com/r/felixlohmeier/openrefine-client/) [![Docker](https://img.shields.io/microbadger/image-size/felixlohmeier/openrefine-client?label=docker)](https://hub.docker.com/r/felixlohmeier/openrefine-client/)
```sh ```sh
docker pull felixlohmeier/openrefine-client:v0.3.8 docker pull felixlohmeier/openrefine-client:v0.3.9
``` ```
### Option 1: Dockerized client ### Option 1: Dockerized client
@ -238,7 +240,7 @@ docker pull felixlohmeier/openrefine-client:v0.3.8
Run client and mount current directory as workspace: Run client and mount current directory as workspace:
```sh ```sh
docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9
``` ```
The docker option `--network=host` allows you to connect to a local or remote OpenRefine via the host network: The docker option `--network=host` allows you to connect to a local or remote OpenRefine via the host network:
@ -246,13 +248,13 @@ The docker option `--network=host` allows you to connect to a local or remote Op
- list projects on default URL (http://localhost:3333) - list projects on default URL (http://localhost:3333)
```sh ```sh
docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 --list docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 --list
``` ```
- list projects on a remote server (http://example.com) - list projects on a remote server (http://example.com)
```sh ```sh
docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H example.com -P 80 --list docker run --rm --network=host -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H example.com -P 80 --list
``` ```
Usage: same commands as explained above (see [Basic Commands](#basic-commands) and [Advanced Templating](#advanced-templating)) Usage: same commands as explained above (see [Basic Commands](#basic-commands) and [Advanced Templating](#advanced-templating))
@ -276,16 +278,16 @@ Run openrefine-client linked to a dockerized OpenRefine ([felixlohmeier/openrefi
3. Run client with some [basic commands](#basic-commands): 1. download example files, 2. create project from file, 3. list projects, 4. show metadata, 5. export to terminal, 6. apply transformation rules (deduplication), 7. export again to terminal, 8. export to xls file and 9. delete project 3. Run client with some [basic commands](#basic-commands): 1. download example files, 2. create project from file, 3. list projects, 4. show metadata, 5. export to terminal, 6. apply transformation rules (deduplication), 7. export again to terminal, 8. export to xls file and 9. delete project
```sh ```sh
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 --download "https://git.io/fj5hF" --output=duplicates.csv docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 --download "https://git.io/fj5hF" --output=duplicates.csv
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 --download "https://git.io/fj5ju" --output=duplicates-deletion.json docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 --download "https://git.io/fj5ju" --output=duplicates-deletion.json
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --create duplicates.csv docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --create duplicates.csv
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --list docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --list
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --info "duplicates" docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --info "duplicates"
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --export "duplicates" docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --export "duplicates"
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --apply duplicates-deletion.json "duplicates" docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --apply duplicates-deletion.json "duplicates"
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --export "duplicates" docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --export "duplicates"
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --export --output=deduped.xls "duplicates" docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --export --output=deduped.xls "duplicates"
docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.8 -H openrefine-server --delete "duplicates" docker run --rm --network=openrefine -v ${PWD}:/data:z felixlohmeier/openrefine-client:v0.3.9 -H openrefine-server --delete "duplicates"
``` ```
4. Stop and delete server: 4. Stop and delete server:
@ -503,8 +505,8 @@ Note to myself: When releasing a new version...
- Run PyInstaller in Python 3 environments on native Windows, macOS and Linux. Should be "the oldest version of the OS you need to support"! Current release is built with: - Run PyInstaller in Python 3 environments on native Windows, macOS and Linux. Should be "the oldest version of the OS you need to support"! Current release is built with:
- Ubuntu 16.04 LTS (64-bit) - Ubuntu 16.04 LTS (64-bit)
- macOS Sierra 10.12 - macOS Sierra 10.12 (64-bit)
- Windows 10 - Windows 7 (32-bit)
- One-file-executables will be available in `dist/`. - One-file-executables will be available in `dist/`.
@ -513,7 +515,7 @@ Note to myself: When releasing a new version...
cd openrefine-client cd openrefine-client
python3 -m pip install . --user python3 -m pip install . --user
python3 -m pip install pyinstaller --user python3 -m pip install pyinstaller --user
pyinstaller --onefile refine.py --hidden-import google.refine.__main__ python3 -m pip PyInstaller --onefile refine.py --hidden-import google.refine.__main__
``` ```
4. Run test with Linux executable 4. Run test with Linux executable
@ -542,8 +544,8 @@ Note to myself: When releasing a new version...
8. Bump openrefine-client version in related projects 8. Bump openrefine-client version in related projects
- openrefine-batch: [openrefine-batch.sh](https://github.com/opencultureconsulting/openrefine-batch/blob/master/openrefine-batch.sh#L7) and [openrefine-batch-docker.sh](https://github.com/opencultureconsulting/openrefine-batch/blob/master/openrefine-batch-docker.sh) - openrefine-batch: [openrefine-batch.sh](https://github.com/opencultureconsulting/openrefine-batch/blob/master/openrefine-batch.sh#L7) and [openrefine-batch-docker.sh](https://github.com/opencultureconsulting/openrefine-batch/blob/master/openrefine-batch-docker.sh)
- openrefineder: [postBuild](https://github.com/felixlohmeier/openrefineder/blob/master/postBuild) and [openrefine-client-bash.ipynb](https://github.com/felixlohmeier/openrefineder/blob/master/openrefine-client-python.ipynb) - openrefineder: [postBuild](https://github.com/felixlohmeier/openrefineder/blob/master/postBuild) and [openrefine-client-bash.ipynb](https://github.com/felixlohmeier/openrefineder/blob/master/openrefine-client-python.ipynb)
- openrefineder: [postBuild](https://github.com/felixlohmeier/openrefineder/blob/master/postBuild)
## Credits ## Credits

4
google/refine/refine.py
View File

@ -248,6 +248,7 @@ class Refine:
# POST is broken at the moment, so we send it in the URL # POST is broken at the moment, so we send it in the URL
new_style_options = dict(opts, **{ new_style_options = dict(opts, **{
'encoding': s(encoding), 'encoding': s(encoding),
'separator': s(separator)
}) })
params = { params = {
'options': json.dumps(new_style_options), 'options': json.dumps(new_style_options),
@ -256,7 +257,6 @@ class Refine:
# old style options # old style options
options = { options = {
'format': project_format, 'format': project_format,
'separator': s(separator),
'ignore-lines': s(ignore_lines), 'ignore-lines': s(ignore_lines),
'header-lines': s(header_lines), 'header-lines': s(header_lines),
'skip-data-lines': s(skip_data_lines), 'skip-data-lines': s(skip_data_lines),
@ -265,7 +265,7 @@ class Refine:
'process-quotes': s(process_quotes), 'process-quotes': s(process_quotes),
'store-blank-rows': s(store_blank_rows), 'store-blank-rows': s(store_blank_rows),
'store-blank-cells-as-nulls': s(store_blank_cells_as_nulls), 'store-blank-cells-as-nulls': s(store_blank_cells_as_nulls),
'include-file-sources': s(include_file_sources), 'include-file-sources': s(include_file_sources)
} }
files = None files = None

2
setup.py
View File

@ -25,7 +25,7 @@ def read(filename):
return open(os.path.join(os.path.dirname(__file__), filename)).read() return open(os.path.join(os.path.dirname(__file__), filename)).read()
setup(name='openrefine-client', setup(name='openrefine-client',
version='0.3.8', version='0.3.9',
description=('The OpenRefine Python Client Library provides an ' description=('The OpenRefine Python Client Library provides an '
'interface to communicating with an OpenRefine server. ' 'interface to communicating with an OpenRefine server. '
'This fork extends the command line interface (CLI).'), 'This fork extends the command line interface (CLI).'),