[2021 Theme Proposal] Increase max block size / defualt to blake2b-256
lizelive opened this issue · comments
Note, this is part of the 2021 IPFS project planning process - feel free to add other potential 2021 themes for the IPFS project by opening a new issue or discuss this proposed theme in the comments, especially other example workstreams that could fit under this theme for 2021. Please also review others’ proposed themes and leave feedback here!
Theme description
One thing that limits IPFS usage for large datasets is its very small block size. 256kb is too small, and the largest size (at least for files) is 1mb. For better disk performance a minimum of 4mb would be needed. This will also drastically improve performance when using cloud backings for data storage.
Hypothesis
Large datasets for repeatable research are a pain to move around, ipfs can fix this.
Vision statement
allow bitswap to handel 4mb files.
Why focus this year
The useage is growing and moving to a 4mb default will result in a 16x reduction in overhead. Also makes it performant to use hdd as backing.
Example workstreams
Please list relevant workstreams, development milestones, and a high-level timeline for these efforts.
Other content
Please include links to other relevant content, notes, etc.
Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:
- "Priority" labels will show how urgent this is for the team.
- "Status" labels will show if this is ready to be worked on, blocked, or in progress.
- "Need" labels will indicate if additional input or analysis is required.
Finally, remember to use https://discuss.ipfs.io if you just need general support.