Crash on ScanState.restore

Question

Crash on ScanState.restore

03July opened this issue a year ago · comments

I have many crashes on :
Fatal Exception: java.lang.IllegalStateException
unread block data
org.altbeacon.beacon.service.ScanState.restore (ScanState.java:145)

This happen after the application update, it seems that the file "android-beacon-library-scan-state" is corrupted or incompatible with the update.

The exception should be caught and should be handled as if the serialized ScanState does not exist rather letting the app crash.

Beacon Library version: 2.19.5
This problem occurs on many manufacturer devices: Samsung, OPPO, Xiaomi, Huawei, Sony....
OS versions: 8,9,10,11,12 & 13

The full stack is
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode (ObjectInputStream.java:2602)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java:1472)
java.io.ObjectInputStream.defaultReadFields (ObjectInputStream.java:2142)
java.io.ObjectInputStream.readSerialData (ObjectInputStream.java:2066)
java.io.ObjectInputStream.readOrdinaryObject (ObjectInputStream.java:1927)
java.io.ObjectInputStream.readObject0 (ObjectInputStream.java:1440)
java.io.ObjectInputStream.readObject (ObjectInputStream.java:428)
org.altbeacon.beacon.service.ScanState.restore (ScanState.java:145)
org.altbeacon.beacon.service.ScanJobScheduler.applySettingsToScheduledJob (ScanJobScheduler.java:102)
org.altbeacon.beacon.BeaconManager.applyChangesToServices (BeaconManager.java:1368)
org.altbeacon.beacon.BeaconManager.startMonitoringBeaconsInRegion (BeaconManager.java:1209)
org.altbeacon.beacon.startup.RegionBootstrap$InternalBeaconConsumer.onBeaconServiceConnect (RegionBootstrap.java:218)
org.altbeacon.beacon.BeaconManager.bindInternal (BeaconManager.java:462)
org.altbeacon.beacon.BeaconManager.bind (BeaconManager.java:427)
org.altbeacon.beacon.startup.RegionBootstrap. (RegionBootstrap.java:140)
com.k.basemanager.BaseManager$1.onSuccess (BaseManager.java:421)
com.k.basemanager.BaseManager$1.onSuccess (BaseManager.java:2)
com.google.common.util.concurrent.Futures$CallbackListener.run (Futures.java:1076)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1137)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:637)
java.lang.Thread.run (Thread.java:1012)

03July · Answer 1 · Wed Feb 15 2023 19:18:01 GMT+0800 (China Standard Time)

Two other exceptions demonstrate that the file "android-beacon-library-scan-state" is corrupted or incompatible with the update.

-1-
Fatal Exception: java.lang.OutOfMemoryError
Failed to allocate a 4294967312 byte allocation with 1199208 free bytes and 252MB until OOM, max allowed footprint 4796832, growth limit 268435456

Of course there is not reason to try to allocate 4 Gb on the mobile device.

-2-
Fatal Exception: java.lang.NullPointerException
Attempt to invoke virtual method 'boolean java.lang.Class.isPrimitive()' on a null object reference
With the same stack trace.

File version / integrity should be checked first before trying to deserialize the state.

David G. Young · Answer 2 · Wed Feb 15 2023 23:13:29 GMT+0800 (China Standard Time)

I agree with these proposed changes.

I have also seen similar increases in crashes in an app I have in the Play Store -- it's unclear why this is happening now, but defensive programming seems like the only option to prevent it

Paolo Rotolo · Answer 3 · Sat Mar 04 2023 01:08:57 GMT+0800 (China Standard Time)

Hi! Would it be possible to release a quick bugfix update to address the IllegalStateException on restoring state and maybe fix the other issues later if they are more complex?

David G. Young · Answer 4 · Sat Mar 04 2023 01:20:02 GMT+0800 (China Standard Time)

See pull request here #1131 that shows my work so far. I have been busy with my day jobs, so this has been hold for the last couple of weeks. @paolorotolo if you can review the change and provide feedback that might speed things along. If you can do some end-to-end testing it would be even better -- that is the time consuming part.

Paolo Rotolo · Answer 5 · Sat Mar 04 2023 01:46:21 GMT+0800 (China Standard Time)

Thanks David, I didn't see you PR, I'll have a look!
Also @03July where is the java.lang.OutOfMemoryError generated?

03July · Answer 6 · Sat Mar 04 2023 03:36:05 GMT+0800 (China Standard Time)

Thanks David, I didn't see you PR, I'll have a look! Also @03July where is the java.lang.OutOfMemoryError generated?

At the same stack trace.
When deserializing the objectInputStream.readObject() use the value store in the scan state file or the file size to get the size of memory that needs to be allocated. As the value or the file is corrupt, it tries to allocate 4GB which is not possible and useless in this case.

03July · Answer 7 · Sat Mar 04 2023 04:06:14 GMT+0800 (China Standard Time)

Thanks David. The defensive code, seems good for me.
I'm wondering if it would be more optimal to use FileInputStream.available() to check the file size instead of using Context.getFileStreamPath.

Other information:
As a temporary workaround. I delete the scan state file at the start of the app if a crash was reported. So I was expected to get rid of the problem linking to the upgrade of the dependencies which make the file incompatible, but I still have 12% crashes of the daily user count. So I don't understand why this problem is happening now. The conclusion is something corrupts the file and it is not linked to the upgrade.

David G. Young · Answer 8 · Sat Mar 04 2023 13:14:37 GMT+0800 (China Standard Time)

@03July why do you believe that this issue is a “problem linking to the upgrade of the dependencies which make the file incompatible”? Is that just conjecture, or do you have evidence that some upgrade is the cause? Upgrade of what?

I am not sure how you detect if there was a previous crash in order to delete the file, but assuming you can do this effectively, the fact that you still see a 12 percent crash rate in daily users could mean that:

The file sometimes grows to an enormous size during a subsequent app run or becomes corrupt again.
Your app’s daily use rate is low enough as a proportion of the install base that you can get a 12% crash rate of daily users for awhile even if the file delete fix works.

Any thoughts on the above appreciated. I want to be careful we understand the problem so any fix does not make things worse.

David G. Young · Answer 9 · Sat Mar 04 2023 13:30:12 GMT+0800 (China Standard Time)

Regarding using FileInputStream.available(), I think this may not work, as the docs say this about the method: available()
“Returns an estimate of the number of remaining bytes that can be read (or skipped over) from this input stream without blocking by the next invocation of a method for this input stream.”

I am not sure of the implementation but the term “estimate” says to me that the API contract does not guarantee to tell you the total file size — for large files, perhaps it just tells you some max buffer size that could be read in a single shot. Even if the current implementation does return the max file size, it seems like the API docs suggest this could behave differently in a future implementation.

03July · Answer 10 · Sun Mar 05 2023 05:54:39 GMT+0800 (China Standard Time)

@davidgyoung This beacon library is integrated in my app for many years now. And the 2.19.4 version is deployed since oct 2022 without any crashes (only ANR / deadlock from MonitoringRegions). The total crashes rate was only 0.48% before the upgrade.
In my lastest update, I mostly update targetSdkVersion from 30 to 31 and compileSdkVersion from 32 to 33 and update all Google Play Services / Firebase versions to the latest releases. When published on Google Play (Rollout at 1%), I saw millions of this problem (app crashed on start) so I had to halt the rollout. This is why I'm suspecting the dependencies but I cannot affirm it.

Thanks to the low crash rate of the previous version, I decided to use Firebase.crashlytics.didCrashOnPreviousExecution() as a workaround. Of course it can be any type of crashes but it a simple way to avoid the app to be blocked by this problem and continue to run normaly.
As the time goes by, the dialy crash rate is going down to 8% now. The delete fix seems to work but the IllegalStateException still producing 31K events for 19K users and OutOfMemoryError 17K events for 11K users.
With the delete fix, we should not have more events than users, this is why I said something corrupts the file and it is not only linked to the upgrade.

03July · Answer 11 · Sun Mar 05 2023 06:01:24 GMT+0800 (China Standard Time)

@davidgyoung Regarding using FileInputStream.available() you're right it's a very slight optimisation so it is not worth the risk as the documentation is not clear enough for the estimate size (even if we don't need the exact file size but the buffer size won't do). Also the implementation may change in the future is for sure a stopper.

David G. Young · Answer 12 · Sun Mar 05 2023 09:42:53 GMT+0800 (China Standard Time)

Thanks for the answers, @03July. This is being published as 2.19.6-beta3. It should be available on Maven Central in an hour or so

03July · Answer 13 · Mon Mar 06 2023 02:01:09 GMT+0800 (China Standard Time)

@davidgyoung Thanks, but 2.19.6-beta3 is still not available on Maven.

David G. Young · Answer 14 · Mon Mar 06 2023 04:37:44 GMT+0800 (China Standard Time)

Please check now. I was able to build and release an app with the new version from maven central.

03July · Answer 15 · Tue Mar 07 2023 20:06:57 GMT+0800 (China Standard Time)

@davidgyoung Thanks. I successfully deployed 2.19.6-beta3. (Rollout at 1%) and I can still see a few OutOfMemoryError.

03July · Answer 16 · Mon Mar 27 2023 23:57:02 GMT+0800 (China Standard Time)

I just published an update of my app with the same version 2.19.6-beta3 and there is no more crash (not even one) of type:

OutOfMemoryError
NullPointerException (Attempt to invoke virtual method 'boolean java.lang.Class.isPrimitive()' on a null object reference)

With the previous version of my app (same Beacon lib version). I had 7.23% crash rate.

I don't see what really changed in my update (exempt the "Gradle JDK") which I switched back to "Android Studio java home" JetBrains Runtime version 11.0.15 C:\Program Files\Android\Android Studio\jbr.
Before I used an external JDK library in the previous version of my app.