Data Curator is a simple desktop CSV editor to help describe, validate and share usable open data.
Open data creators are increasingly focusing on improving open data publishing so that data can be used to create insight and drive positive change.
Open data is more likely to be used if data consumers can:
- understand why and how the data was collected
- understand the structure of the data
- look up the meaning of codes used in the data
- understand the quality of the data
- access the data in an open machine-readable format
- know how the data is licensed and how it can be reused
Using Data Curator open data producers can:
- create and edit tabular data from scratch or from a template
- open data from a CSV or Microsoft Excel file
- open multiple related data tables from a Data Package
- automatically correct common problems found in CSV and Excel files
Using data from any of these sources, you can:
- automatically create a schema that describes the data fields
- refine the schema to include extra data validation rules
- describe the provenance of your data
- save data as a valid CSV file in various CSV dialects
The schema enables you to:
- validate the whole table at once
- validate a column at a time (planned)
Once the data is described and validated, you can share the data and its description by exporting a Data Package to:
- publish on your open data portal
- use as a template for others to make similar data
Open data consumers can use published Data Packages to:
- view the data structure and provenance information to help determine if the data is fit for their purpose
- download the data together with its metadata in a single zip file
- use a suite of tools to work with the data
Interested in this project? Subscribe to Data Curator News to get occasional updates on our progress and hear about each release.
We welcome all types of contributions to Data Curator:
We acknowledge the great work of others. We are:
- inspired by the ODI Comma Chameleon experiment.
- using the Open Knowledge International Frictionless Data specification and code libraries
- adopting W3C Data on the Web Best Practices
Data Curator proudly uses open source software, including:
| software | organisation | licence | support |
|---|---|---|---|
| Comma Chameleon | The Open Data Institute | MIT | join |
| datapackage.js | Open Knowledge | MIT | donate |
| tableschema-js | Open Knowledge | MIT | donate |
| Electron | GitHub | MIT | contribute |
| Node.js | Node.js | licence | contribute |
| Chromium | The Chromium Authors | licence | contribute |
| Vue.js | Yuxi (Evan) You | MIT | donate |
| electron-vue | SimulatedGREG (Greg Holguin) | MIT | donate |
| Handsontable | Handsontable | MIT | Buy Pro |
- Choose a platform from the Releases page.
- Drag the application to your applications folder.
If you encounter a warning message informing you the application cannot be opened due to being from an unknown developer, try:
- Right click the app
- then option + click on Open.
This warning occurs due to macOS quarantining applications when it cannot determine the certificate used to sign the application. We're planning to sign the application so this goes away.
We develop against the 'develop' branch. The 'master' branch contains tagged releases. We are currently using this branching model by Vincent Driessen.
node
yarn
electron
You can use yarn to install all relevant packages and development dependencies. (Install yarn)
We're keeping our dependencies up to date with Dependabot.
To open the app on your local machine and run Data Curator in development mode:
- change to your local Data Curator directory
yarn(pulls down all dependencies)yarn run dev
Data Curator will launch with an extra Developer menu.
Data Curator is built using Electron, a framework that allows developers to build desktop applications using web technology.
There are two parts of the application, the main process and the renderer process. The main process deals with things like carrying out file operations, validating CSVs, and rendering views. The renderer acts very much like client side javascript in a web browser, dealing with things like presentation, and user interactions.
Electron passes and listens for messages between main and renderer using the IPC module, one for the main process and one for the renderer process.
We have adopted Standard JS as our JavaScript coding standard.
Tools to automate testing are being established.
We have:
- defined Acceptance tests using the Gherkin language
- shared Acceptance tests in a pretty format using Relish
To push the acceptance tests (.feature files) to Relish:
relish push odi-australia/data-curator path /your-local-path/data-curator/test/features
We may link acceptance tests to lower level tests (cucumber-js will probably be helpful).
We're considering:
- End-to-end tests: Spectron, a purpose built Electron testing framework
- Unit tests: Karma, a browser test runner, designed for low-level/unit testing
- Comma Chameleon currently uses Electron-Mocha, Chai and Sinon for unit tests
- Spectron and Karma can be used with any testing library, including Mocha, Chai and Jasmine
- Travis and Appveyor for continuous integration
To run tests and launch Data Curator:
yarn run test
To build Data Curator locally:
yarn run build
To automate deployment, we are using:
To trigger the automated build and draft release, create and push a git tag, following the versioning pattern used in current releases. Ensure that any changes in this tag are also released back to develop and master branches.
Changes are recorded in the Change Log.