artsy / eigen

The Art World in Your Pocket or Your Trendy Tech Company's Tote, Artsy's mobile app.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

dyld crash at launch

alloy opened this issue · comments

  • A colleague ran into a situation where the app would crash on launch. Important to note is that she was also seeing it with other apps, such as LinkedIn.app.
  • A restart of her device ‘fixed’ the issue.
  • These crashes were not being reported to HockeyApp, meaning it was definitely happening very early in the process.
  • Extracting the crash reports directly from the device revealed a dyld error.
  • Checking iTunesConnect for crash reports revealed that we had ~80 similar crash reports.

TL;DR

This appears to happen when the device is out of memory, a restart of the device ‘fixes’ it. Presumably this neat trick does so as well.

Info about the errors

Dyld Error Message:
Dyld Message: Library not loaded: @rpath/CocoaLumberjack.framework/CocoaLumberjack
  Referenced from: /var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Artsy
  Reason: no suitable image found.  Did find:
    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack:
    mremap_encrypted() => -1, errno=12 for /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack

    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack:
    mremap_encrypted() => -1, errno=12 for /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/CocoaLumberjack.framework/CocoaLumberjack

    /private/var/mobile/Containers/Bundle/Application/D693664B-F6D0-43CB-A839-358E15CAF9E6/Artsy.app/Frameworks/C
  Dyld Version: 370.1
  • The error message and codes are always the same.
  • The error is not specific to a framework, it happens with various frameworks.
  • It appears to have started from iOS 9.1 and up (but that could very well be iOS 9).

Known affected versions

These are the various versions for which we have crash reports.

iOS versions

~/D/dyld crash » ack -h 'OS Version:' **/*.crash | sort | uniq
OS Version:          iOS 9.1 (13B143)
OS Version:          iOS 9.2 (13C75)
OS Version:          iOS 9.2.1 (13D15)
OS Version:          iOS 9.2.1 (13D20)
OS Version:          iOS 9.3 (13E5214d)
OS Version:          iOS 9.3 (13E5225a)

dyld versions

~/D/dyld crash » ack -h 'Dyld Version:' **/*.crash | sort | uniq
  Dyld Version: 370.1
  Dyld Version: 370.6
  Dyld Version: 390.7

Device versions

~/D/dyld crash » ack -h 'Hardware Model:' **/*.crash | sort | uniq
Hardware Model:      iPad3,4
Hardware Model:      iPad4,4
Hardware Model:      iPad4,7
Hardware Model:      iPad5,3
Hardware Model:      iPad5,4
Hardware Model:      iPad6,7
Hardware Model:      iPad6,8
Hardware Model:      iPhone6,1
Hardware Model:      iPhone7,1
Hardware Model:      iPhone7,2
Hardware Model:      iPhone8,1
Hardware Model:      iPhone8,2

Retracing

  1. The error is raised here in dyld.
  2. Only place where mremap_encrypted could return -1 is here.
  3. vn_getpath calls build_path, which in turn has a few possible locations where it could be returning -1:
  4. But trying to hunt that further down would take a lot more time, suffice to say that:

Caveats

Alas, the latest dyld sources to be released (at the time of writing) appear to be for version 360.18.

Taking a cursory look at the decompiled code for ImageLoaderMachOCompressed::registerEncryption from a firmware image for iOS 9.2 for iPad3,4, does not appear to have any real changes.

~/t/ios-9.2-iPad3,4 » unzip iPad3,4_9.2_13C75_Restore.ipsw 
Archive:  iPad3,4_9.2_13C75_Restore.ipsw
  inflating: 058-25903-078.dmg       
  inflating: 058-26242-078.dmg       
  inflating: 058-26268-078.dmg
[…]

~/t/ios-9.2-iPad3,4 » ~/Code/ObjectiveC/Debugging/xpwn/dmg/dmg extract 058-26268-078.dmg extracted.dmg -k 7ccd940dfda4b304c87050997e9fd70b458f21b5f3845262d985730b23f41e84aa25dae3
Writing out data..
[…]

~/t/ios-9.2-iPad3,4 » open extracted.dmg
~/t/ios-9.2-iPad3,4 » hopper -e /Volumes/Castlerock13C75.P101OS/usr/lib/dyld
int ImageLoaderMachOCompressed::registerEncryption(encryption_info_command const*, ImageLoader::LinkContext const&)(void * arg0, void * arg1) {
    r0 = arg0;
    r7 = (sp - 0x4 - 0x4 - 0x4 - 0x4 - 0x4) + 0xc;
    sp = sp - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4 - 0x4;
    r6 = arg1;
    r8 = r2;
    r4 = r0;
    if (r6 != 0x0) {
            r5 = 0x0;
            do {
                    r2 = *(*r4 + 0xe8);
                    if ((r2)(r4, r5, r2) != 0x0) goto (null);
                    r5 = r5 + 0x1;
            } while (true);
            r1 = r5;
            r2 = *(*r4 + 0x104);
            r0 = (r2)(r4, r1, r2);
            r10 = r6 + 0x8;
            asm{ ldm.w      sl, {r1, r5, sl} };
            r2 = *(r8 + 0xb2);
            r8 = *(r0 + 0x4);
            r6 = r0 + r1;
            r11 = *(r0 + 0x8);
            if (r2 != 0x0) {
                    dyld::log("                      0x%08lX->0x%08lX configured for FairPlay decryption\n");
            }
            r0 = _mremap_encrypted();
            r5 = r0;
            if (r5 != 0x0) {
                    r0 = ___error();
                    r3 = *(r4 + 0x4);
                    r2 = *r0;
                    r1 = r5;
                    r0 = dyld::throwf("mremap_encrypted() => %d, errno=%d for %s\n");
                    // …
            }
    }
    return r0;
}

Conclusion

The crash at launch definitely feels like a very bad experience and I feel like iOS should handle this more gracefully. Maybe show an alert? Will have a think about this and possibly file a radar.

I’m not sure what/if we can do about this. We could probably improve it somewhat by trimming some framework fat from the bundle, but I don’t think these are large anyways.

It would be interesting to know if such failures occurred prior to iOS 9 and if they were labelled differently.

Status

Dynamic library sizes:

$ cd DerivedData/Artsy/Build/Products/Debug-iphoneos/Artsy.app/Frameworks
$ find . -perm +u=x -type f ! -iname '*.strings' ! -iname '*.ttf' ! -iname '*.otf' -print0 | xargs -0 ls -lh | awk '{print $5, $9}' | column -t | gsort -h -r -k1
13M   ./libswiftCore.dylib
1.5M  ./libswiftFoundation.dylib
1.4M  ./HockeySDK_Source.framework/HockeySDK_Source
1.0M  ./ReactiveCocoa.framework/ReactiveCocoa
956K  ./FBSDKCoreKit.framework/FBSDKCoreKit
656K  ./AFNetworking.framework/AFNetworking
395K  ./libswiftCoreGraphics.dylib
392K  ./SDWebImage.framework/SDWebImage
368K  ./CocoaLumberjack.framework/CocoaLumberjack
363K  ./Adjust.framework/Adjust
337K  ./libswiftDarwin.dylib
294K  ./libswiftCoreAudio.dylib
294K  ./Analytics.framework/Analytics
288K  ./FBSDKLoginKit.framework/FBSDKLoginKit
273K  ./Bolts.framework/Bolts
255K  ./libswiftObjectiveC.dylib
248K  ./libswiftUIKit.dylib
248K  ./libswiftCoreMedia.dylib
241K  ./MMMarkdown.framework/MMMarkdown
240K  ./libswiftDispatch.dylib
240K  ./libswiftAVFoundation.dylib
224K  ./Mantle.framework/Mantle
210K  ./ARAnalytics.framework/ARAnalytics
208K  ./libswiftWebKit.dylib
208K  ./libswiftCoreLocation.dylib
208K  ./libswiftCoreData.dylib
208K  ./libswiftContacts.dylib
202K  ./libswiftCoreImage.dylib
163K  ./NAMapKit.framework/NAMapKit
151K  ./AFOAuth1Client.framework/AFOAuth1Client
146K  ./iRate.framework/iRate
145K  ./ALPValidator.framework/ALPValidator
138K  ./ARTiledImageView.framework/ARTiledImageView
136K  ./VCRURLConnection.framework/VCRURLConnection
135K  ./Interstellar.framework/Interstellar
135K  ./ARGenericTableViewController.framework/ARGenericTableViewController
131K  ./ARCollectionViewMasonryLayout.framework/ARCollectionViewMasonryLayout
128K  ./FLKAutoLayout.framework/FLKAutoLayout
118K  ./EDColor.framework/EDColor
111K  ./JSDecoupledAppDelegate.framework/JSDecoupledAppDelegate
109K  ./ObjectiveSugar.framework/ObjectiveSugar
109K  ./Keys.framework/Keys
108K  ./FXBlurView.framework/FXBlurView
106K  ./Artsy_UILabels.framework/Artsy_UILabels
106K  ./Artsy_UIButtons.framework/Artsy_UIButtons
105K  ./KSDeferred.framework/KSDeferred
104K  ./ORStackView.framework/ORStackView
104K  ./Aerodramus.framework/Aerodramus
103K  ./PBDCarouselCollectionViewLayout.framework/PBDCarouselCollectionViewLayout
103K  ./JLRoutes.framework/JLRoutes
101K  ./MARKRangeSlider.framework/MARKRangeSlider
101K  ./JSBadgeView.framework/JSBadgeView
100K  ./ISO8601DateFormatter.framework/ISO8601DateFormatter
99K   ./UICKeyChainStore.framework/UICKeyChainStore
96K   ./RSSwizzle.framework/RSSwizzle
81K   ./ORKeyboardReactingApplication.framework/ORKeyboardReactingApplication
81K   ./ARASCIISwizzle.framework/ARASCIISwizzle
80K   ./UIView_BooleanAnimations.framework/UIView_BooleanAnimations
80K   ./DRKonamiCode.framework/DRKonamiCode
80K   ./AFNetworkActivityLogger.framework/AFNetworkActivityLogger
79K   ./UIAlertView_Blocks.framework/UIAlertView_Blocks
79K   ./Artsy_UIFonts.framework/Artsy_UIFonts
78K   ./NPKeyboardLayoutGuide.framework/NPKeyboardLayoutGuide
76K   ./MultiDelegate.framework/MultiDelegate
76K   ./Artsy_UIColors.framework/Artsy_UIColors
75K   ./Then.framework/Then
74K   ./DHCShakeNotifier.framework/DHCShakeNotifier

There is very few mentions about this crash that I could find, maybe because people don’t realise it is happening?


Sadly enough, two of those cases appear to get their app rejected, 1 followed up to say they recompiled and resubmitted and it was accepted. I bet that these were just cases of the reviewer’s device running out of memory and that the second case also got accepted after resubmitting.

@JaviSoto had a discussion on twitter last week about the same issue but it looks like no solution surfaced. 🙁

@0xced Aye, he has let me know, it also sparked others chiming in. I have added a ‘Status’ section to the bottom of the OP.

commented

The jetsam/memory pressure process is described here for reference: http://newosxbook.com/articles/MemoryPressure.html

If the issue can be reliably reproduced it may be worth testing explicit weak linking of some frameworks as described in SDK based development

My radar has been closed as a duplicate of 24278648.

@robinsenior Did you get any updates on your DTS?

They initially told me it was fixed in 9.3, but an Apple dev on Twitter told me otherwise. I see crash logs for it in beta 7 but not the full release (AFAIK, I haven't checked in a few days).

@robinsenior I have crash reports for 9.3 builds 13E233 and 13E234, so definitely not fixed.

If you don’t have any atm and want to follow-up with DTS I can send you some.

@alloy thanks; I just checked and we have some with those build numbers so I'll email them to DTS.

As an aside, my phone is much faster under 9.3. The RAM clearing trick used to be essential just to get apps to launch quickly but now is unnecessary. Something related to reclaiming memory must have been fixed.

Ace 👍

As an aside, my phone is much faster under 9.3. The RAM clearing trick used to be essential just to get apps to launch quickly but now is unnecessary. Something related to reclaiming memory must have been fixed.

That’s great news! Might update my personal device sooner after all 😉

Apparently there have been some changes to dyld in 9.3 which resulted in significant performance improvements: https://github.com/stepanhruda/dyld-image-loading-performance
Didn't fix the crash from this issue though 😕

Yeah, that’s about #586.

Just got this update from DTS:

We’ve made some progress on this one. It seems to be related to the number of embedded frameworks an app has. I noticed that your app has 40 frameworks (12 of which are the Swift dylibs). You could reduce the chance of your app hitting this bug by combining some of those frameworks. Even cutting the number of frameworks in half would make a difference.

/facepalm

Oh jeez… So their progress is basically all of what this ticket already describes ¯_(ツ)_/¯

I do get that they want to give you something that’s actionable for you right now, but it would have been nice to know a bit more what their status is on the real issue. (rdar://24278648, which is the one mine got closed in favour of.)

Hi Alloy - Have you heard anything more about iOS dynamic library loading crasher? or are you guys making any changes to combine frameworks etc?
Thanks in advance for any insight.
-chris

@ctava Nope, if/when we would hear more, or decide a different course of action, it all goes public in this ticket. I don’t think combining frameworks would help, as it would probably still run out of memory and die in the same way. The only possibility to reduce (not fully get rid of) these issues would be to get rid of ‘fat’ (number of larger frameworks), which is not something we will be doing at this time.

To summarise, imo it’s clearly a bug in iOS, and the radar that mine was closed in favour of is still open, so they haven’t ruled it as “works as intended” yet. Until that time, I don’t think the number of crashes vs the number of users warrants a lot of work on our part. YMMV.

@robinsenior Thinking about that answer a bit more, could you ask them if they do in fact mean the number of libs vs the size of those libs? If they are positive that reducing the number of libs, regardless of size, is a way to workaround the issue, then that would definitely be helpful info.

@alloy done. I'll keep you posted.

Ace, thanks 👍

For another data point: we're seeing pretty commonly in our app as well since we switched to using dynamic frameworks with CocoaPods. We have 41 frameworks averaging about 1.5MB each. It's a pretty significant issue for us, so we're going back to a static library in our next release.

@maxmeyers Are you using Swift yet?

@alloy Fortunately not in production yet.

@maxmeyers Ok cool. Just wanted to check if the Swift dylibs were not causing this issue for you, but that explains it then. Thanks for the feedback 👍

@alloy DTS says it's the number of libs, not the size.

@robinsenior Well… that’s unexpected. Are you going to spend time on reducing the number of dynamic libs?

@alloy I'd rather not; our build system is already pretty convoluted due to shared frameworks across multiple projects, primarily built using Cocoapods. Do you know of a simple approach for mashing everything together?

@robinsenior I assume you’re using frameworks because you have some libs that use Swift? Do you have many of those vs objc ones?

@alloy of our 39 frameworks:

  • 12 are libswift
  • 17 are external dependencies
  • 10 are internal frameworks, 7 of which contain Swift

@alloy Do you think it would be possible to trigger this issue even after a device reboot? Had this issue come to light today in our latest app update with a colleague "fixing it" by restarting their device but another still appearing to run into problems on launch.

@robinsenior Something like https://github.com/johnno1962/Accelerator is a ‘simple’ approach. Alternatively you could maybe have a separate Podfile for your objc pods and statically build those, you might need to do integration into your Xcode project yourself then, though. But make sure that the libs you would be doing this with will be able to lookup their resources, if they have any (wrt to NSBundle lookup).

If/when I decided to do this for our app, I’ll post a follow up with my steps.

@DarrylBayliss Seeing as DTS is reporting that it has to do with the number of libs, it might well be. Have you retrieved the crash reports from the device after that crash to verify it’s the same?

Want to also note a useful tweet about this issue

@alloy We have less than 20 frameworks including the swift ones and we’re still seeing the issue.

@alloy Managed to grab some logs and we actually had two problems. The dyld issue mentioned here and something unrelated we didn't catch. So nothing new to add here.

Is this still a thing for 9.3.1? I don't see it listed above but I haven't seen anything to confirm its fixed in 9.3.1. Users are experiencing this in our app as well

@ryang1428 This happens on 9.3.1 as well.

thanks @alloy .. did this mean to be 9.3.1 in your original post? OS Version: iOS 9.3 (13E5225a) You have 9.3 listed twice I now noticed

Nope, 13E5214d & 13E5225a are both 9.3 betas iirc

@alloy I would like to thank you so much for doing such great writeup!! And also thanks Artsy for doing open source by default. It's great that you can share all details about an investigation of a weird crash! It helps others a lot!

We have noticed same issue with our app. I wonder if anybody is going to bring this topic to the labs at WWDC.

@yas375 You’re most welcome 🙇

That’s a good question. I’m personally not going, but maybe @orta is?

Not to beat a dead horse, but I've had this crash happen to me in several apps lately (including big ones like LinkedIn). Our app is getting pounded in app store reviews because of it. How is there not a bigger stink about it?

Just want to chime in on the conversation here. I ran into this issue this morning - the app crashed during startup. I reinstalled the app from app store and still to no avail. And then I found some other apps (not all) are having the same issue. Restarting my phone 'fixed' it.

Having looked at our crash reports in Xcode we do have a huge number of these crashes.

@robinsenior We're also getting destroyed, hundreds of 1 star reviews. 😢 I've been raising a fuss to everyone I know who could potentially help, and to no avail. It seems like many people don't realize they have these crashes. Since it's a crash on launch, their crash reporting tool (e.g. Crashlytics/Hockey) won't pick it up, and only iTunes Connect can relay it, they're probably living in ignorant bliss.

Hey folks, talked a bit to a DTS engineer that asked me to keep an eye out for this crash in iOS 9.3.2 beta 3. Might be nothing, but might as well add information here where I can.

Our app was also rejected from app review briefly for this specific problem which involved me coordinating between DTS engineering and the App Review team. Thankfully, we're approved and ready to go now, but things are getting hairy!

P.S. I'll buy whoever is at WWDC with this stack trace a beer as sympathy :)

@brianmichel @JaviSoto just mentioned that he’s still seeing it on 9.3.2: http://twitter.com/Javi/status/726882761106038784.

Thanks for putting it on the radar of the App Review team 👍

As of now, both @JaviSoto and I have only seen crashes up to 9.3.2 beta 1 (13F51a). This combined with @brianmichel’s DTS feedback to lookout for this issue in beta 3 seems very hopeful.

any update on this issue. for example, how hopeful is 9.3.2 beta in terms of fixing the iOS crasher?

We were just rejected for this issue this morning. App crashed on launch for reviewer, looked at the attached crash file, sure enough it was this issue. I didn't even bother with the appeal process, just submitted another build. Hopefully it gets approved.

Ironically, this is our first denial because of this issue, and I had just gotten done uninstalling 4 cocoapods we didn't absolutely need anymore.

This is really starting to worry me, and i'm wondering what percentage of our users are experiencing this and simply uninstalling the app and/or leaving bad reviews. We've already received a few 1 star reviews because of this.

We're seeing a number of 1 star reviews because of this too, so much so that in the next release notes we're going to have to specifically attribute this to being Apple/iOS's issue and not ours.

Really do hope that this is fixed in 9.3.2. If not it's definitely going to be something to mention at WWDC labs.

Here's what happened when we switched back from frameworks to static libraries (this is crashes as reported by iTunes Connect App Analytics):

screen shot 2016-05-11 at 11 01 45 am

@maxmeyers Did you not require frameworks for use with Swift or did you make use of https://github.com/johnno1962/Accelerator?

We didn't have any Swift dependencies. One option might be to install your Swift frameworks with Carthage and use CocoaPods for Objective-C dependencies (that's assuming you're currently using CocoaPods to install your frameworks).

@maxmeyers Do you use Swift in your app? If so, are you still getting this crash report (albeit in low numbers) due to libSwiftCore.dylib etc?

No Swift at all. This crash has completely disappeared.

One option might be to install your Swift frameworks with Carthage and use CocoaPods for Objective-C dependencies.

Or you could use a second Podfile for your Swift dependencies and use something like Rome, if you prefer CocoaPods ;)

@maxmeyers Ah, bugger, had hoped for an extra convincing case to Apple that shows that just using Swift leads to this issue. But great for you of course!

@alloy: That wouldn't solve the issue though would it? Only reduce the likelihood of the crash because of less frameworks. Either way, unfortunately our app is 50% Swift so most of our pods are used in both 😞

commented

Have any of you attempted to weak link frameworks and load them after launch using the NSBundle API?

With that technique, it sounds like you're starting to move into some of the territory covered in this issue: #586

commented

There are two, related issues at work here, both centering on the dynamic loader. Memory pressure and CPU. In the crash logs that I have seen memory pressure is the overriding concern. As the loader loads strongly linked frameworks at startup it has to verify their signatures. System processes involved in that increase memory pressure when it may have already been high on the device. The loader can't get enough memory to continue loading frameworks and their dependencies, the application can't be launched or is immediately terminated.

Weak-linking the frameworks and loading after launch using the NSBundle API gives the developer more control over this process.

Monitoring the performance of signature verification may also be helpful.

Apple has told us that weak linking does not help the problem.

We're having a similar problem and just wanted to reiterate the thanks for Artsy's open source ethos. This thread is very helpful.

In order to reproduce this crash consistently, we downloaded a number of apps from the App Store on an iOS 9.3.1 device (iPhone 6+). I opened 14 apps, and when trying to open the 15th app, I experienced a crash on launch. From there, any app I tried to open for the first time would crash on launch, while previously opened apps would continue to launch successfully.

I was able to reproduce this same sequence both after a reboot, and after manually killing all open apps via the app switcher.

After upgrading the phone iOS version to 9.3.2 Beta 4, I was unable to reproduce any crash. At the maximum, I opened 50+ apps successfully. Hopefully this means the issue is fixed or at least ameliorated with iOS 9.3.2 Beta 4.

@freshnitesh That’s great to hear, tanks for the update!

@alloy Happy to help :)

@freshnitesh thank you so much for the detailed instructions on how to reproduce the crash consistently. I'm also able to confirm that iOS 9.3.2 beta 4 fixes this issue. At some point, apps take longer to launch but no crash so far, which is way better.

9.3.2 is out. So far nothing is crashing, but will keep an eye on this in the coming days/weeks. 🙏 for success.

Yes, we believe the crashes on launch with mremap_encrypted -1 errno=12 are fixed in iOS 9.3.2 (as of beta 3, I think). Please file new bug reports if you see it again.

Alright, I'm calling this 👏

👍

Awesome! 👏

This is great! 🚀

That's an amazing thread.
Despite the fact that is fixed in 9.3.2 (and I hope iOS 10), there's still the issue we're supporting quite a lot of old version devices. What you do on those cases?

Aside from the manual work shown in:

A new CocoaPods plugin has been created by @Ruenzuo since then that will automate most of that work:

Other than that, there's no real choice to to write it off. At least 9.3 came with awesome emoji, and that got most of our users to update.

screen shot 2018-09-24 at 3 46 14 am

I am getting this error. App is working fine on simulator, but as I run it on device it crashes .I don't know why it's happening.