pkcakeout / dtoolcore

Core API to manage (scientific) data

Manage scientific data sets

• Documentation: http://dtoolcore.readthedocs.io
• GitHub: https://github.com/JIC-CSB/dtoolcore
• PyPI: https://pypi.python.org/pypi/dtoolcore
• Free software: MIT License

Features

• Core API for adding different types of metadata to files on disk
• Automatic generation of structural metadata
• Programmatic discovery and access of items in a dataset
• Structural metadata includes hash, size and modification time for subsequent integrity checks
• Ability to annotate individual files with arbitrary metadata
• Metadata stored on disk as plain text files, i.e. disk datasets generated using this API can be accessed without special tools
• Ability to create plugins for custom storage solutions
• Plugins for iRODS and Microsoft Azure storage backends available
• Cross-platform: Linux, Mac and Windows are all supported
• Works with Python 2.7, 3.5 and 3.6
• No external dependencies

Overview

The dtoolcore project provides a Python API for managing (scientific) data. It aims to help in three areas:

1. Adding structure and meta data to your project and files
2. Providing programmatic discovery of your data
3. Verifying the integrity of your data