With this fork of chiapos I try to add some threads to improve plot creation time.
Important: the main changes are done for enabled bitmap plotting.
For now sorting is done with threads. Here it is an example of how it improved with thread when no IO limits:
size: 0.11GiB, threads: 2, times: read:1.112s, total: 1.998s
size: 0.141GiB, threads: 3, times: read:0.494s, total: 1.033s
size: 0.141GiB, threads: 4, times: read:0.346s, total: 0.758s
size: 0.141GiB, threads: 5, times: read:0.330s, total: 0.690s
size: 0.141GiB, threads: 6, times: read:0.261s, total: 0.575s
size: 0.141GiB, threads: 7, times: read:0.239s, total: 0.536s
size: 0.141GiB, threads: 8, times: read:0.269s, total: 0.512s
size: 0.141GiB, threads: 9, times: read:0.240s, total: 0.504s
size: 0.141GiB, threads: 10, times: read:0.203s, total: 0.463s
In case of enough RAM next bucket presorted in parallel.
Threads added to phase2 and computational stage 2 of phase 3.
Bitmap of table 7 is calculated on write of the table and rewriting of table 7 is done on read of it in phase 3, than no need of table 7 stage in phase 2.
In addition added IO compaction that can lead to faster creation with HDD plotting or prolonge SSD life with less writtings. But compaction do some more work than on slow CPUs or in systems where IO is not a botleneck creation will be slower. To use all implemented compactions buckets number should be at least 256. The IO compaction allows less ram to plot on ramdisk. By my measures to create k32 it needed around 170GiB space in temp directory when using compaction with 256 (or more) buckets.
Free buffer can be used as write caching that allows to write less. For my measures for k32 each 1GiB of ram up to 22Gib saves around 16GiB of writtings and 25GiB buffers saves ~45% of writtings, 46GiB buffer saves ~63% of writtings and 110GiB buffer saves ~83% of writtings ( all this for 256 bukets and compaction enabled ).
Known issue with using buffer as cache is a memory fragmentation.
It is almost fixed when creating one plot at a time, but if python
calls the plotter many times the memory usage can be increased
drastically (2-3 times) for second and more plots.
This is the reason it is disabled by default.
It is possible to enable cache also by prefixing temp dir with :CACHE:
i.e. providing "-t :CACHE:/my/temp/dir" parameter
I tested ram plotting on amazon R6i.8xlarge instance that is 256GiB RAM and 30 vcores, the time of creation k32 plot is 33 minutes.
How to use? Get it from github
git clone https://github.com/vladimirttt123/chiaposht.git
Compile as original one. Then you have ProofOfSpace that not very usefull, but you can replace original chiapos with this one in installed chia. In order to do so you need replace file chiapos.cpython-310-x86_64-linux-gnu.so (the name depends on your python version) in chia installation sub dir venv/lib/python3.10/site-packages in addtion add there 2 more files: libfse.a and libuint128.a. Now when you run plots as regular your will run by this plotter.
Chia's proof of space is written in C++. Includes a plotter, prover, and verifier. It exclusively runs on 64 bit architectures. Read the Proof of Space document to learn about what proof of space is and how it works.
# Requires cmake 3.14+
mkdir -p build && cd build
cmake ../
cmake --build . -- -j 6mkdir -p build && cd build
cmake -DBUILD_PROOF_OF_SPACE_STATICALLY=ON ../
cmake --build . -- -j 6./RunTests./ProofOfSpace -k 25 -f "plot.dat" -m "0x1234" create
./ProofOfSpace -k 25 -f "final-plot.dat" -m "0x4567" -t TMPDIR -2 SECOND_TMPDIR create
./ProofOfSpace -f "plot.dat" prove <32 byte hex challenge>
./ProofOfSpace -k 25 verify <hex proof> <32 byte hex challenge>
./ProofOfSpace -f "plot.dat" check <iterations>time ./ProofOfSpace -k 25 createThere is an experimental implementation which implements some of the Hellman Attacks that can provide significant space savings for the final file.
./HellmanAttacks -k 18 -f "plot.dat" -m "0x1234" create
./HellmanAttacks -f "plot.dat" check <iterations>Finally, python bindings are provided in the python-bindings directory.
python3 -m venv .venv
. .venv/bin/activate
pip3 install .Testings uses pytest. Linting uses flake8 and mypy.
py.test ./tests -s -vThe primary build process for this repository is to use GitHub Actions to
build binary wheels for MacOS, Linux (x64 and aarch64), and Windows and publish
them with a source wheel on PyPi. See .github/workflows/build.yml. CMake uses
FetchContent
to download pybind11. Building is then
managed by cibuildwheel. Further
installation is then available via pip install chiapos e.g.
Contributions are welcome and more details are available in chia-blockchain's CONTRIBUTING.md.
The main branch is usually the currently released latest version on PyPI. Note that at times chiapos will be ahead of the release version that chia-blockchain requires in it's main/release version in preparation for a new chia-blockchain release. Please branch or fork main and then create a pull request to the main branch. Linear merging is enforced on main and merging requires a completed review. PRs will kick off a GitHub actions ci build and analysis of chiapos at lgtm.com. Please make sure your build is passing and that it does not increase alerts at lgtm.