Required CPU flags for sharp 0.28.2/libvips-8.10.6

[Sebastian]

Hi

first of all: thank you very much for your work and this awesome lib. I am happy that libvips-8.10.6 supports a broader range of prebuilds, e.g. libvips-8.10.6-linux-arm64v8 for the Raspberry Pi ;-) very cool!

I am trying sharp 0.28.1 on Debian Buster and receive an segmentation fault and can reproduce it via docker. Same for sharp 0.28.0.

Unfortunately sharp 0.27.2 does not support prebuilds on Raspberry Pi with arm64 OS yet :'(

I used Debian/Buster 10.9, docker 18.09.1 and nodejs 14

$ docker run -ti --rm node:14-buster /bin/sh
# npm install sharp@0.28.1

> sharp@0.28.1 install /node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)

sharp: Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.10.6/libvips-8.10.6-linux-x64.tar.br
npm WARN saveError ENOENT: no such file or directory, open '/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open '/package.json'
npm WARN !invalid#1 No description
npm WARN !invalid#1 No repository field.
npm WARN !invalid#1 No README data
npm WARN !invalid#1 No license field.

+ sharp@0.28.1
added 69 packages from 185 contributors and audited 69 packages in 9.389s

6 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

# node
Welcome to Node.js v14.11.0.
Type ".help" for more information.
> require('sharp')
Illegal instruction (core dumped)
#

Any help and hints are appreciated.

[Sebastian]

Maybe it is a CPU issue? Here are mine CPU flags of AMD Phenom(tm) II X4 955 Processor:

# cat /proc/cpuinfo | grep flags
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt hw_pstate vmmcall npt lbrv svm_lock nrip_save

[Lovell Fuller]

Yes, since libvips v8.10.6 (sharp v0.28.0) the x64 prebuilt binaries are compiled to expect more modern CPUs that provide SSE4.2 - see #88

(I think AMD implemented different SSE4 intrinsics to Intel on their K10 CPUs. I used to have a machine with the same processor as you as these were relatively easy to overclock, although this was ~12 years ago.)

[Sebastian]

Thank you for the confirmation. I will check for the sse4_2 CPU flag from /proc/cpuinfo on Linux in postinstall script to decide if sharp binary is CPU compatible or a build from source installation is required.

[Sebastian]

Is there a way to skip prebuild binaries of libvips while installing sharp from sources? I guess sharp still uses the machine incompatible precompiled libvips?

I would like to use the system libvips package...

[Lovell Fuller]

https://sharp.pixelplumbing.com/install#custom-libvips

To use a custom, globally-installed version of libvips instead of the provided binaries, make sure it is at least the version listed under config.libvips in the package.json file and that it can be located using pkg-config --modversion vips-cpp.

[Sebastian]

Thank you very much for the hint.

Instead of building libvips on my host I gave the docker version a try. Unfortunately nothing was compiled into the current directory and I stuck again.

I did:

git clone https://github.com/lovell/sharp-libvips.git
cd sharp-libvips
git checkout -b v8.10.6
./build.sh $(cat LIBVIPS_VERSION) linux-x64
...
Successfully built 9eb094fda9bb
Successfully tagged vips-dev-linux-x64:latest
glib version 2.68.0 has been superseded by 2.68.1
gdkpixbuf version 2.42.4 has been superseded by 2.42.6
pango version 1.48.3 has been superseded by 1.48.4

If this succeeds I can customize the libvips and build it for my older CPU. Any help appreciated.

[Lovell Fuller]

Ah, checking out previous tags will still trigger the version_latest logic. We should probably add some kind of check/flag to bypass this and allow previous tags to build, perhaps a VERSION_LATEST_REQUIRED environment variable (defaulting to true).

For now try the following patch:

--- a/build/lin.sh
+++ b/build/lin.sh
@@ -134,6 +134,7 @@ without_patch() {
 # Check for newer versions
 ALL_AT_VERSION_LATEST=true
 version_latest() {
+  return
   VERSION_LATEST=$($CURL "https://release-monitoring.org/api/v2/versions/?project_id=$3" | jq -j ".stable_versions[0]")
   if [ "$VERSION_LATEST" != "$2" ]; then
     ALL_AT_VERSION_LATEST=false

[Sebastian]

Yeah! That worked. I think I am now able to build a customized workaround for my old CPU. For now, monkey patching build/lin.sh and linux-x64/Dockerfile are fine for me.

Thank you very much for this project, your time and support.

[Sebastian]

I have no more questions and I am closing this issue.

Thank you

[Lovell Fuller]

Commit c415d26 adds an optional VERSION_LATEST_REQUIRED=false to skip the latest version check.

[Sebastian]

Awesome. Thank you