Native Released to System Repositories/Stores
cmhulbert opened this issue · comments
Native releases should be available in platform specific ways:L
- Windows Store?
- Apple Store?
- Debian PPA
Needs dependency management:
- libblosc1
I understand that @hubbardp has an active Apple developer key
We may qualify for a fee waiver for an Apple membership fee waiver since HHMI is a non-profit. I have not been able to clarify if applying for the waiver would violate a HHMi policy or not.
https://developer.apple.com/support/membership-fee-waiver/
Blosc1 availability summary
The latest release of c-blosc is 1.21.4. Importantly, this upgraded Zstd from 1.5.2 to 1.5.5 due to a known rare compression bug in Zstandard pre-1.5.5.
Ubuntu LTS and Debian stable contain out dated versions of libblosc and zstandard. The latest release of libblosc1 in Debian unstable is 1.21.3+ds-1.
Homebrew and MacPorts have c-blosc 1.21.4. The dependency graph is not clear to me at the moment. Homebrew and MacPorts also ship an up-to-date zstandard 1.5.5
I have not found a "native" Windows based distribution of a modular built blosc on either Chocholatey, win-get, or the Microsoft Store. conda-forge contains an update to date recipe of blosc 1.21.4.
conda-forge contains blosc 1.21.4 for all platforms.
https://github.com/conda-forge/blosc-feedstock/blob/main/recipe/meta.yaml
https://github.com/conda-forge/blosc-feedstock/blob/main/recipe/bld.bat
https://github.com/conda-forge/blosc-feedstock/blob/main/recipe/build.sh
Ubuntu libblosc1 version 1.21.1 in Ubuntu 22.04 LTS
https://packages.ubuntu.com/jammy/libblosc1
liblz4-1 (>= 0.0~r130)
libsnappy1v5 (>= 1.1.7) [not riscv64]
libzstd1 (>= 1.3.2)
zlib1g (>= 1:1.1.4)
Debian libblosc1 version 1.21.3 in Debian bookworm stable
https://packages.debian.org/bookworm/libblosc1
liblz4-1 (>= 0.0~r130)
libsnappy1v5 (>= 1.1.9)
libzstd1 (>= 1.5.2)
dep: zlib1g (>= 1:1.1.4)
Homebrew c-blosc 1.21.4
https://formulae.brew.sh/formula/c-blosc
MacPorts 1.21.4
https://ports.macports.org/port/blosc/
Chocholatey
https://community.chocolatey.org/packages?q=blosc
(none exists)
win-get
https://winget.run/search?query=blosc
(none exists)
conda-forge
https://anaconda.org/conda-forge/blosc
osx-arm64 1.21.4
osx-arm64 blosc 1.21.4
linux-64 1.21.4
linux-aarch64 1.21.4
osx-64 1.21.4 (Intel)
win-64 1.21.4
For Windows, I recommend obtaining the DLLs in this fashion via a conda enabled PowerShell:
conda create -n blosc_env -c conda-forge blosc
conda activate blosc_env
cd $ENV:CONDA_PREFIX\Library\bin
dir
Copy and paste all the DLLs into the Paintera bin directory. This is the Windows way.
Directory: C:\conda\3\envs\blosc_env\Library\bin
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 5/6/2022 11:21 21984 api-ms-win-core-console-l1-1-0.dll
-a---- 5/6/2022 11:29 21976 api-ms-win-core-console-l1-2-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-core-datetime-l1-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-core-debug-l1-1-0.dll
-a---- 5/6/2022 11:21 21976 api-ms-win-core-errorhandling-l1-1-0.dll
-a---- 5/6/2022 11:21 21992 api-ms-win-core-fibers-l1-1-0.dll
-a---- 5/6/2022 11:21 26088 api-ms-win-core-file-l1-1-0.dll
-a---- 5/6/2022 11:21 21976 api-ms-win-core-file-l1-2-0.dll
-a---- 5/6/2022 11:22 21976 api-ms-win-core-file-l2-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-core-handle-l1-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-core-heap-l1-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-core-interlocked-l1-1-0.dll
-a---- 5/6/2022 11:22 22008 api-ms-win-core-libraryloader-l1-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-core-localization-l1-2-0.dll
-a---- 5/6/2022 11:21 21976 api-ms-win-core-memory-l1-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-core-namedpipe-l1-1-0.dll
-a---- 5/6/2022 11:21 22008 api-ms-win-core-processenvironment-l1-1-0.dll
-a---- 5/6/2022 11:21 21976 api-ms-win-core-processthreads-l1-1-0.dll
-a---- 5/6/2022 11:21 21976 api-ms-win-core-processthreads-l1-1-1.dll
-a---- 5/6/2022 11:21 21992 api-ms-win-core-profile-l1-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-core-rtlsupport-l1-1-0.dll
-a---- 5/6/2022 11:21 21976 api-ms-win-core-string-l1-1-0.dll
-a---- 5/6/2022 11:21 21976 api-ms-win-core-synch-l1-1-0.dll
-a---- 5/6/2022 11:22 21976 api-ms-win-core-synch-l1-2-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-core-sysinfo-l1-1-0.dll
-a---- 5/6/2022 11:21 21976 api-ms-win-core-timezone-l1-1-0.dll
-a---- 5/6/2022 11:21 21992 api-ms-win-core-util-l1-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-crt-conio-l1-1-0.dll
-a---- 5/6/2022 11:21 26080 api-ms-win-crt-convert-l1-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-crt-environment-l1-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-crt-filesystem-l1-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-crt-heap-l1-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-crt-locale-l1-1-0.dll
-a---- 5/6/2022 11:21 30184 api-ms-win-crt-math-l1-1-0.dll
-a---- 5/6/2022 11:21 30176 api-ms-win-crt-multibyte-l1-1-0.dll
-a---- 5/6/2022 11:21 75232 api-ms-win-crt-private-l1-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-crt-process-l1-1-0.dll
-a---- 5/6/2022 11:21 26104 api-ms-win-crt-runtime-l1-1-0.dll
-a---- 5/6/2022 11:21 26080 api-ms-win-crt-stdio-l1-1-0.dll
-a---- 5/6/2022 11:22 26072 api-ms-win-crt-string-l1-1-0.dll
-a---- 5/6/2022 11:21 21984 api-ms-win-crt-time-l1-1-0.dll
-a---- 5/6/2022 11:21 22000 api-ms-win-crt-utility-l1-1-0.dll
-a---- 5/16/2023 07:09 69632 blosc.dll
-a---- 5/10/2023 03:01 327576 concrt140.dll
-a---- 1/26/2023 05:00 118784 liblz4.dll
-a---- 7/7/2023 05:26 525312 libzstd.dll
-a---- 1/26/2023 05:01 161792 lz4.exe
-a---- 5/10/2023 03:01 578384 msvcp140.dll
-a---- 5/10/2023 03:01 35704 msvcp140_1.dll
-a---- 5/10/2023 03:01 267160 msvcp140_2.dll
-a---- 5/10/2023 03:01 50072 msvcp140_atomic_wait.dll
-a---- 5/10/2023 03:01 31640 msvcp140_codecvt_ids.dll
-a---- 3/11/2023 06:39 75264 snappy.dll
-a---- 5/6/2022 11:22 1123808 ucrtbase.dll
-a---- 5/10/2023 03:01 414104 vcamp140.dll
-a---- 5/10/2023 03:01 346008 vccorlib140.dll
-a---- 5/10/2023 03:01 191864 vcomp140.dll
-a---- 5/10/2023 03:01 109440 vcruntime140.dll
-a---- 5/10/2023 03:01 49560 vcruntime140_1.dll
-a---- 6/12/2023 09:09 89088 zlib.dll
-a---- 7/7/2023 05:26 525312 zstd.dll
-a---- 7/7/2023 05:26 129024 zstd.exe
Wider Linux Distribution Scan for blosc packages:
https://pkgs.org/search/?q=blosc
https://pkgs.org/search/?q=libblosc1
https://pkgs.org/download/blosc
https://pkgs.org/download/c-blosc
I was also finally note that we really need to move to Blosc2 for better upstream support from the authors as well as the second generation frame format. Blosc1 is becoming increasingly difficult to support.
Thanks @mkitti for the info. I think what we ideally will use is the platform specific standard auto-updating sites, to make using Paintera as straight-forward from the end user. For example, if we package Paintera on windows, in a way that requires we get the dlls from conda first, It would still require that the user has conda installed (unless we can have it as a dependency?). Perhaps we can do this at package time on the windows runner, and just provide the dlls to the installer?
As for blosc2, I understand it's backwards compatible; Is it easier to package and move to blosc2 say, than maintain blosc1?
No, the user would not need conda. We would just use conda to obtain the binaries and include them in the Windows package. The expectation on Windows is for us to vendor such dependencies into the package.
Thanks @mkitti! We had a productive chat to discuss this more, a brief summary as follows:
- Since we have an ubuntu PPA, and ubuntu provides blosc dependency at a platform level, we should prioritize releasing on Ubuntu first
- Should also consider Debian, since it likely will not be difficult after this
- Apple Store is probably the best target for Macs
- Needs signing keys at least. Not clear if this is best to sign by individual, lab, or as HHMI. HHMI may already have signing keys, and this may be broadly more useful to try and get, even if there currently are not.
- Blosc is available through homebrew and macports, but if we are publishing to Apple Store, it may be best to package the libraries with Paintera
- Windows target is not clear at the moment
- It's likely that windows store is ideal, but I'm not familiar enough at the moment
- Also we will likely need to package the blosc library with Paintera
So next steps as far as I have is to generate the deb package with the blosc dependency, and release to our ppa. That should be fairly straight forward. After, can discuss more specifics around Apple/Windows stores. In the meantime, I can also work on publishing the SNAPSHOT releases on github like we currently are, but with the bundled bloscl library. Likely the best way to do this is as @mkitti mentioned above, by pulling them from the conda recipes in the github action runner, and package them.
Some other notes, we will want to setup Paintera to publish SNAPSHOT releases on an M1 mac runner as welll.
Just an update for now, but the deb package built and released here ubuntu-latest should now depend on libblosc1