freetype not linked correctly in latest macOS Python 3.11 wheels (1.3.17 and 1.3.18)
devbanu opened this issue · comments
Starting with 1.3.17 the wheels for Python 3.11 on macOS are not linked correctly:
import aggdraw
ImportError: dlopen(/Users/alex/SOMEPATH/venv/lib/python3.11/site-packages/aggdraw.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_FT_Attach_File'
When using otool on the .so file:
(venv) alex@Trinity site-packages % otool -L aggdraw.cpython-311-darwin.so
aggdraw.cpython-311-darwin.so:
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
one can see that libfreetype is not listed, hence the missing symbols.
The latest working wheels for this configuration was for version 1.3.16.
I suspect it is related to these changes here: 0c5124c.
Thanks for letting me know.
This is a difficult one to fix and decide what the right solution is. I don't think I ever intentionally installed freetype. You can see in the CI config I install freetype for linux, but have never done anything for osx or Windows. It looks like cibuildwheel uses pipx
to build in different Python environments but I'm not exactly sure why it can't find a version of freetype that is installed on the system.
If I go to the work of installing freetype on the osx builds then I want to be sure it is as compatible as it used to be.
In your tests did you only change the version of aggdraw? Or did the rest of your environment change too?
Do you have suggestions for where I should install freetype from?
I see in the latest build log that it finds:
Repairing wheel...
+ delocate-wheel --require-archs x86_64 -w /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-x74ze_f6/cp39-macosx_x86_64/repaired_wheel -v /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-x74ze_f6/cp39-macosx_x86_64/built_wheel/aggdraw-1.3.18-cp39-cp39-macosx_10_9_x86_64.whl
INFO:delocate.delocating:Copying library /usr/local/Cellar/freetype/2.13.2/lib/libfreetype.6.dylib to aggdraw.dylibs/libfreetype.6.dylib
INFO:delocate.delocating:Copying library /usr/local/Cellar/libpng/1.6.40/lib/libpng16.16.dylib to aggdraw.dylibs/libpng16.16.dylib
INFO:delocate.delocating:Modifying install name in aggdraw.dylibs/libfreetype.6.dylib from /usr/local/opt/libpng/lib/libpng16.16.dylib to @loader_path/libpng16.16.dylib
INFO:delocate.delocating:Modifying install name in aggdraw.cpython-39-darwin.so from /usr/local/opt/freetype/lib/libfreetype.6.dylib to @loader_path/aggdraw.dylibs/libfreetype.6.dylib
Fixing: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-x74ze_f6/cp39-macosx_x86_64/built_wheel/aggdraw-1.3.18-cp39-cp39-macosx_10_9_x86_64.whl
Copied to package .dylibs directory:
/usr/local/Cellar/freetype/2.13.2/lib/libfreetype.6.dylib
/usr/local/Cellar/libpng/1.6.40/lib/libpng16.16.dylib
This suggests that the wheels should have freetype in them.
Hm it looks like the x86 osx wheels have the libraries, but not the ARM ones. 😕
Oh! We never used to provide ARM64 OSX wheels!
https://pypi.org/project/aggdraw/1.3.16/#files
You've been building from source before 1.3.17. Add --no-binary
to your pip install aggdraw
command (pip install --no-binary aggdraw
) and it should be a short term workaround.
From what I can tell I could install a separate instance of homebrew and install packages there. I'm not sure I want to go down that road, but hopefully someone has a better idea and can comment here.
David,
You are right, until 1.3.16 the installation was from source. I did not realize that. So the temporary solution from #93 seem the right approach for now.
On my Mac pip seems to pick up libfreetype from /opt/homebrew
.
I will try to better understand the CI system and freetype on macOS. Note sure yet what is the first-class solution to building/providing it in such context.
I'm not familiar enough with how reliable wheels are when they bundle versions of shared libraries internally. I, for some reason, thought it would just require the user have the library installed but it seems they are bundled inside the wheel. So I'm not sure if that makes things better or worse or easier or riskier.
After some thinking:
-
Including libfreetype and libpng dynamic libraries in the wheel seems to be the correct thing to do. Many others do the same, e.g. numpy and scipy include libopenblas and libquadmath.
-
The
delocate-wheel
tool takes care of copying the dynamic libraries and relinking the binaries to refer to new locations. It works as advertised. For thex86_64
wheel it copies over the appropriate libraries from/usr/local/Cellar
on the macOS runner. The issue is that it cannot do the same for thearm64
wheel because the libs in/usr/local/Cellar
are for forx86_64
only. -
Do we know whether the macOS runner executing the GitHub action has an Intel (
x86_64
) or Apple Silicon (arm64
) CPU? The easy way would be to have anarm64
macOS runner to build thearm64
wheels, similarly how it is done forx86_64
. -
Alternatively, as you mentioned, one can try installing
arm64
homebrew onx86_64
system, like discussed here.
Once the macOS runner has the arm64
freetype and libpng installed, the arm64
wheels should build correctly.
Yes, this is the conclusion I was coming to also. Github does not currently provide arm64 runners. All non-native wheel builds are emulated (arm64 running on x86_64). So yeah the options are either no wheels for arm64 or go through the complicate process of installing an arm version of homebrew to get access to the arm64 freetype.
I am not familiar with the Github infrastructure.
Does Github provide free or very-low cost runners?
Isn't this announcement about an arm64 native macOS runner? This one is definitely not free though.
That looks arm-based, yes. Definitely not free. The other option is to use a different CI provider that does provide arm runners, but I really don't want to maintain two CI services for a project like this.
Let me know if you need any help with figuring this out, I have access to both arm64 and x86_64 Macs.
So I could either make a new release that doesn't include the ARM64 wheels or I could remove the ARM64 wheels for the existing release. Opinions?
I do not plan on doing the fancy homebrew stuff to get freetype for ARM64 and/or other architectures. If someone wants to do that and try that out, go for it. I'd be curious what it looks like in github actions.
So I could either make a new release that doesn't include the ARM64 wheels or I could remove the ARM64 wheels for the existing release. Opinions?
For users with newish Macs that come with M1, M2, or M3 CPUs, which is an ever growing number, there is no difference between these two options. Either way they have to have freetype installed and xcode tools installed in order to install aggdraw.
I do not plan on doing the fancy homebrew stuff to get freetype for ARM64 and/or other architectures. If someone wants to do that and try that out, go for it. I'd be curious what it looks like in github actions.
I will look into this.
Several important goals that I see:
- keeping all wheels well encapsulated (contain what they need)
- keeping all wheels similar (all contains the same libs)
- not requiring users to install a compiler chain to install aggdraw
Agreed. And also: keeping maintenance burden to a minimum 😉
Oops, got formatting wrong on my previous comment, updated it now.
Please see PR #94 with fix for this issue.
Yep, tested, and it works!