keitaroinc / ckanext-big-resources

ckanext-big-resources

CKAN extension that overwrites the upload and download of the resources. The extension breaks the resources into chunks and streams the response. This extension is usefull if We want to configure the chunk size of upload and download in order to use the server resources (RAM and CPU) in more efficient way.

Requirements

Compatibility with core CKAN versions:

CKAN version	Compatible?
2.6 and earlier	not tested
2.7	not tested
2.8	not tested
2.9	Yes

Installation

To install ckanext-big-resources:

1. Activate your CKAN virtual environment, for example:

   . /usr/lib/ckan/default/bin/activate

2. Clone the source and install it on the virtualenv

   git clone https://github.com/keitaroinc/ckanext-big-resources.git    
   cd ckanext-big-resources    
   python setup.py develop    
   pip install -r dev-requirements.txt
   

3. Add big_resources to the ckan.plugins setting in your CKAN config file (by default the config file is located at /etc/ckan/default/ckan.ini).

4. Restart CKAN. For example if you've deployed CKAN with Apache on Ubuntu:

   sudo service apache2 reload

Config settings

The following settings (which set the upload and the download chunk in bytes) should be present in the ckan.ini file.

ckanext.big_resources.chunk_upload = number_in_bytes
ckanext.big_resources.chunk_download = number_in_bytes

If thease settings are not found in the ckan.ini than the default value will be set by the extension. The default value for both upload and download of the resource is 2048

Developer installation

To install ckanext-big-resources for development, activate your CKAN virtualenv and do:

git clone https://github.com/keitaroinc/ckanext-big-resources.git
cd ckanext-big-resources
python setup.py develop
pip install -r dev-requirements.txt

Tests

To run the tests, do:

pytest --ckan-ini=test.ini

Notice

This extension can not be combined together with a cloud storage CKAN extension (like ckanext-cloudstorage or ckanext-s3filestore). If the CKAN is extended with a cloud storage extensions than the upload and download of the resources is taken over by the cloud storage extension and the ckanext-big-resources becomes obsolete.

Releasing a new version of ckanext-big-resources

If ckanext-big-resources should be available on PyPI you can follow these steps to publish a new version:

1. Update the version number in the setup.py file. See PEP 440 for how to choose version numbers.

2. Make sure you have the latest version of necessary packages:

   pip install --upgrade setuptools wheel twine

3. Create a source and binary distributions of the new version:

   python setup.py sdist bdist_wheel && twine check dist/*
   

   Fix any errors you get.

4. Upload the source distribution to PyPI:

   twine upload dist/*
   

5. Commit any outstanding changes:

   git commit -a
   git push
   

6. Tag the new release of the project on GitHub with the version number from the setup.py file. For example if the version number in setup.py is 0.0.1 then do:

   git tag 0.0.1
   git push --tags
   

License

AGPL