Linking multiple SPM dynamic libraries towards the same target ends up in failure.

Question

Linking multiple SPM dynamic libraries towards the same target ends up in failure.

nikita-leonov opened this issue 2 years ago · comments

Short description of the issue:
When the package depends on multiple dynamic targets from RxSwift package it fails to compile with ld error.

Expected outcome:
I would expect the package to compile.

What actually happens:
Building of swift-dynamic-test provided in the example below will fail with Command Ld failed with a nonzero exit code, and the actual reason underneath will be that ld uses a path /Build/Products/Debug-iphoneos/PackageFrameworks/RxSwift-Dynamic.framework/RxSwift-Dynamic while there is no RxSwift-Dynamic in RxSwift-Dynamic.framework. Meanwhile, there is RxSwift in the folder RxSwift.framework folder next to it. Moreover manually moving and renaming the file to the place where ld tries to find it results in a successful build. Note: There is no use of RxSwift product library in the package file and it is unclear how the whole RxSwift.framework is being built as well as why RxSwift-Dynamic.framework exists on a disk but RxSwift-Dynamic is not there, while other things are.

Errors details:

Ld /Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Products/Debug-iphoneos/swift-dynamic-testTests.xctest/swift-dynamic-testTests normal (in target 'swift-dynamic-testTests' from project 'swift-dynamic-test')
    cd /Users/nleonov/projects/Temp/swift-dynamic-test
    /Applications/Xcode13.3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -target arm64-apple-ios9.0 -bundle -isysroot /Applications/Xcode13.3.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS15.4.sdk -L/Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Products/Debug-iphoneos -L/Applications/Xcode13.3.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/usr/lib -F/Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Products/Debug-iphoneos/PackageFrameworks -F/Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Products/Debug-iphoneos/PackageFrameworks -F/Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Products/Debug-iphoneos/PackageFrameworks -F/Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Products/Debug-iphoneos/PackageFrameworks -F/Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Products/Debug-iphoneos/PackageFrameworks -F/Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Products/Debug-iphoneos -F/Applications/Xcode13.3.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/Frameworks -iframework /Applications/Xcode13.3.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/Frameworks -iframework /Applications/Xcode13.3.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS15.4.sdk/Developer/Library/Frameworks -filelist /Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Intermediates.noindex/swift-dynamic-test.build/Debug-iphoneos/swift-dynamic-testTests.build/Objects-normal/arm64/swift-dynamic-testTests.LinkFileList -Xlinker -rpath -Xlinker /usr/lib/swift -Xlinker -rpath -Xlinker @loader_path/Frameworks -Xlinker -rpath -Xlinker @loader_path/../Frameworks -Xlinker -rpath -Xlinker @loader_path/Frameworks -dead_strip -Xlinker -object_path_lto -Xlinker /Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Intermediates.noindex/swift-dynamic-test.build/Debug-iphoneos/swift-dynamic-testTests.build/Objects-normal/arm64/swift-dynamic-testTests_lto.o -Xlinker -export_dynamic -Xlinker -no_deduplicate -fobjc-link-runtime -L/Applications/Xcode13.3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/iphoneos -L/usr/lib/swift -Xlinker -add_ast_path -Xlinker /Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Intermediates.noindex/swift-dynamic-test.build/Debug-iphoneos/swift-dynamic-testTests.build/Objects-normal/arm64/swift_dynamic_testTests.swiftmodule -framework XCTest /Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Products/Debug-iphoneos/PackageFrameworks/RxSwift-Dynamic.framework/RxSwift-Dynamic /Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Products/Debug-iphoneos/PackageFrameworks/RxCocoa-Dynamic.framework/RxCocoa-Dynamic -Xlinker -dependency_info -Xlinker /Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Intermediates.noindex/swift-dynamic-test.build/Debug-iphoneos/swift-dynamic-testTests.build/Objects-normal/arm64/swift-dynamic-testTests_dependency_info.dat -o /Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Products/Debug-iphoneos/swift-dynamic-testTests.xctest/swift-dynamic-testTests -Xlinker -add_ast_path -Xlinker /Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Intermediates.noindex/swift-dynamic-test.build/Debug-iphoneos/swift-dynamic-test.build/Objects-normal/arm64/swift_dynamic_test.swiftmodule

clang: error: no such file or directory: '/Users/nleonov/Library/Developer/Xcode/DerivedData/swift-dynamic-test-hjhlavumahezasfjpkyaquybqqke/Build/Products/Debug-iphoneos/PackageFrameworks/RxSwift-Dynamic.framework/RxSwift-Dynamic'
Command Ld failed with a nonzero exit code

Self-contained code example that reproduces the issue:

A library within a package depends on RxSwift-Dynamic and RxCocoa-Dynamic which are defined as dynamic in the RxSwift package.

let package = Package(
    name: "swift-dynamic-test",
    products: [
        .library(
            name: "swift-dynamic-test",
            targets: ["swift-dynamic-test"]),
    ],
    dependencies: [
        .package(url: "https://github.com/ReactiveX/RxSwift.git", from: "6.5.0"),
    ],
    targets: [
        .target(
            name: "swift-dynamic-test",
            dependencies: [
                .product(name: "RxSwift-Dynamic", package: "RxSwift"),
                .product(name: "RxCocoa-Dynamic", package: "RxSwift")
            ]
        )
    ]
)

Platform/Environment

How easy is it to reproduce? (chances of successful reproduce after running the self-contained code)

easy, 100% repro
sometimes, 10%-100%
hard, 2% - 10%
extremely hard, %0 - 2%

Xcode version:

13.3.1

Installation method:

SPM
CocoaPods
Carthage
Git submodules

I have multiple versions of Xcode installed:
(so we can know if this is a potential cause of your issue)

yes (which ones)
no

Level of RxSwift knowledge:
(this is so we can understand your level of knowledge
and formulate the response in an appropriate manner)

just starting
I have a small codebase
I have a significant codebase

Nikita Leonov · Answer 1 · Sat May 21 2022 02:47:07 GMT+0800 (China Standard Time)

The issue reported on Swift Forums too, but without much of activities so far — https://forums.swift.org/t/linking-multiple-dynamic-libraries-that-depend-on-the-same-target-fails-with-ld-failing-due-to-missing-files/57382/2

Shai Mishali · Answer 2 · Tue May 24 2022 12:56:42 GMT+0800 (China Standard Time)

Thanks for reporting @nikita-leonov :)
That's pretty frustrating, but I think it's an SPM linking issue. I'm not entirely sure what we can do in the project to resolve it, but if you have any thoughts I'm happy to further discuss.

Nikita Leonov · Answer 3 · Wed May 25 2022 08:36:00 GMT+0800 (China Standard Time)

@freak4pc indeed seems like an SPM issue. Although I think there is an opportunity to improve the current products layout that may indirectly fix the issue. It seems like all dynamic products within the package depend on the same RxSwift target. This will likely end up in symbol duplication if you link multiple dynamic libraries such as RxCocoa-Dynamic and RxRelay-Dynamic into the single end product. To fix issues like this common dependency products should be not within the same package, but in separate packages. This means RxSwift likely needs to be defined as a separate package and in this way, it can be reused as dynamic across multiple Dynamic products within RxSwift infrastructure itself. This is definitely a suboptimal way of doing things, but the current state of SPM is likely the only experiment to try that may unblock this issue.
This fix is simple to validate, but I do not want to invest in it much unless we both agree it worth trying and adopting in case if fixes the problem.

Nikita Leonov · Answer 4 · Thu May 26 2022 11:50:36 GMT+0800 (China Standard Time)

I validated the setup where there is a separate package RxSwift with RxSwift-Dynamic target and where the top layer package where RxCocoa and others refer to the RxSwift. It does solve the problem above, as well as a general issue with RxSwift symbols linked into multiple dynamic libraries.

Shai Mishali · Answer 5 · Thu May 26 2022 12:04:16 GMT+0800 (China Standard Time)

I validated the setup where there is a separate package RxSwift with RxSwift-Dynamic target and where the top layer package where RxCocoa and others refer to the RxSwift. It does solve the problem above, as well as a general issue with RxSwift symbols linked into multiple dynamic libraries.

Thanks for looking into it.
Can you share how that looks ?

Shai Mishali · Answer 6 · Sat Jun 04 2022 03:33:55 GMT+0800 (China Standard Time)

@nikita-leonov ping

Nikita Leonov · Answer 7 · Fri Jun 10 2022 03:49:11 GMT+0800 (China Standard Time)

@freak4pc Hmm, so I have a mixed set of news. I did update the post here https://forums.swift.org/t/linking-multiple-dynamic-libraries-that-depend-on-the-same-target-fails-with-ld-failing-due-to-missing-files/57382/2 it seems the issue I experienced was not SPM specific, but probably some environment issue. After some change in the environment that I was unable to trace yet the issue is not reproducible anymore. I still believe that libs layout where multiple products could be static and dynamic, meanwhile all of them depend on RxSwift static target underneath may end up in the symbols duplication, but it is probably a discussion for a separate ticket. RxSwift is an important dependency in our product, and we are working through SPM migration now, so be sure if there anything poping up I will follow up here :) Although I do think we can close this particular ticket for now unless I will find what actually happened :O

Shai Mishali · Answer 8 · Fri Jun 10 2022 03:52:54 GMT+0800 (China Standard Time)

Happy to circle back to this. We also recently migrated everything over to SPM & Tuist and experienced no issues as well, so we're happy to hear anything you pop into.

Closing this for the time being, but feel free to comment or open a new issue if you find anything.
Thanks!