initializationError with androidx.test: runner : 1.5.0
joehoag opened this issue · comments
Description
I was recently forced to upgrade the version of androidx.test:runner that we use from 1.4.0 to 1.5.0. After I did that, I noticed a subtle problem where not all of our tests were running. Looking closer at the logs, I saw things like this:
10-12 09:24:21.946: I/TestRunner(7396): run started: 31 tests
10-12 09:24:21.947: I/TestRunner(7396): started: test_CNone_class_filter_supertest(com.xxx.xxx.xxx.xxx.ClassFilterMobileTests)
10-12 09:24:21.947: I/TestRunner(7396): finished: test_CNone_class_filter_supertest(com.xxx.xxx.xxx.xxx.ClassFilterMobileTests)
10-12 09:24:21.949: I/TestRunner(7396): started: test_C1194420_filter_by_subtitles(com.xxx.xxx.xxx.xxx.ClassFilterMobileTests)
10-12 09:24:21.949: I/TestRunner(7396): finished: test_C1194420_filter_by_subtitles(com.xxx.xxx.xxx.xxx.ClassFilterMobileTests)
10-12 09:24:21.951: I/TestRunner(7396): started: test_C25220_can_filter_by_length(com.xxx.xxx.xxx.xxx.ClassFilterMobileTests)
10-12 09:24:21.951: I/TestRunner(7396): finished: test_C25220_can_filter_by_length(com.xxx.xxx.xxx.xxx.ClassFilterMobileTests)
10-12 09:24:21.952: I/TestRunner(7396): started: initializationError(com.xxx.xxx.xxx.xxx.ProfileMobileTests)
10-12 09:24:21.953: E/TestRunner(7396): failed: initializationError(com.xxx.xxx.xxx.xxx.ProfileMobileTests)
10-12 09:24:21.953: E/TestRunner(7396): ----- begin exception -----
10-12 09:24:21.959: E/TestRunner(7396): java.lang.RuntimeException: Failed to instantiate test runner class androidx.test.internal.runner.junit4.AndroidJUnit4ClassRunner
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.ext.junit.runners.AndroidJUnit4.throwInitializationError(AndroidJUnit4.java:129)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.ext.junit.runners.AndroidJUnit4.loadRunner(AndroidJUnit4.java:121)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.ext.junit.runners.AndroidJUnit4.loadRunner(AndroidJUnit4.java:82)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.ext.junit.runners.AndroidJUnit4.<init>(AndroidJUnit4.java:56)
10-12 09:24:21.959: E/TestRunner(7396): at java.lang.reflect.Constructor.newInstance0(Native Method)
10-12 09:24:21.959: E/TestRunner(7396): at java.lang.reflect.Constructor.newInstance(Constructor.java:343)
10-12 09:24:21.959: E/TestRunner(7396): at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104)
10-12 09:24:21.959: E/TestRunner(7396): at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.junit4.AndroidAnnotatedBuilder.runnerForClass(AndroidAnnotatedBuilder.java:63)
10-12 09:24:21.959: E/TestRunner(7396): at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:70)
10-12 09:24:21.959: E/TestRunner(7396): at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:37)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.AndroidRunnerBuilder.runnerForClass(AndroidRunnerBuilder.java:149)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.AndroidLogOnlyBuilder.runnerForClass(AndroidLogOnlyBuilder.java:93)
10-12 09:24:21.959: E/TestRunner(7396): at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:70)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.DirectTestLoader.doCreateRunner(DirectTestLoader.java:45)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.TestLoader.getRunnersFor(TestLoader.java:64)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.TestRequestBuilder.build(TestRequestBuilder.java:835)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.runner.AndroidJUnitRunner.buildRequest(AndroidJUnitRunner.java:650)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:418)
10-12 09:24:21.959: E/TestRunner(7396): at com.xxx.xxx.xxx.util.xxxTestRunner.onStart(CallistoTestRunner.kt:110)
10-12 09:24:21.959: E/TestRunner(7396): at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2145)
10-12 09:24:21.959: E/TestRunner(7396): Caused by: java.lang.reflect.InvocationTargetException
10-12 09:24:21.959: E/TestRunner(7396): at java.lang.reflect.Constructor.newInstance0(Native Method)
10-12 09:24:21.959: E/TestRunner(7396): at java.lang.reflect.Constructor.newInstance(Constructor.java:343)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.ext.junit.runners.AndroidJUnit4.loadRunner(AndroidJUnit4.java:112)
10-12 09:24:21.959: E/TestRunner(7396): ... 19 more
10-12 09:24:21.959: E/TestRunner(7396): Caused by: java.lang.StackOverflowError: stack size 1041KB
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$ClassTokenizerState.<init>(Unknown Source:0)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$MethodTokenizerState.parse(ClassesArgTokenizer.java:85)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$ClassTokenizerState.parse(ClassesArgTokenizer.java:51)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$MethodTokenizerState.parse(ClassesArgTokenizer.java:85)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$ClassTokenizerState.parse(ClassesArgTokenizer.java:51)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$MethodTokenizerState.parse(ClassesArgTokenizer.java:85)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$ClassTokenizerState.parse(ClassesArgTokenizer.java:51)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$MethodTokenizerState.parse(ClassesArgTokenizer.java:85)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$ClassTokenizerState.parse(ClassesArgTokenizer.java:51)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$MethodTokenizerState.parse(ClassesArgTokenizer.java:85)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$ClassTokenizerState.parse(ClassesArgTokenizer.java:51)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$MethodTokenizerState.parse(ClassesArgTokenizer.java:85)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$ClassTokenizerState.parse(ClassesArgTokenizer.java:51)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$MethodTokenizerState.parse(ClassesArgTokenizer.java:85)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$ClassTokenizerState.parse(ClassesArgTokenizer.java:51)
10-12 09:24:21.959: E/TestRunner(7396): at androidx.test.internal.runner.ClassesArgTokenizer$MethodTokenizerStat
10-12 09:24:21.959: E/TestRunner(7396): ----- end exception -----
10-12 09:24:21.974: W/StackTrimmer(7396): Stack trace too long, trimmed to first 65536 characters.
10-12 09:24:21.975: I/TestRunner(7396): finished: initializationError(com.xxx.xxx.xxx.xxx.ProfileMobileTests)
10-12 09:24:21.975: D/TestDiscoveryListener(7396): JUnit reported com.xxx.xxx.xxx.xxx.ProfileMobileTests#initializationError; discarding as bogus.
10-12 09:24:21.977: I/TestRunner(7396): started: test_C25201_tap_each_fitness_discipline(com.xxx.xxx.xxx.xxx.ClassListMobileTests)
10-12 09:24:21.977: I/TestRunner(7396): finished: test_C25201_tap_each_fitness_discipline(com.xxx.xxx.xxx.xxx.ClassListMobileTests)
10-12 09:24:21.979: I/TestRunner(7396): started: test_C14841925_verify_notes_block_details_in_workout_details(com.xxx.xxx.xxx.xxx.CircuitsMobileTests)
10-12 09:24:21.979: I/TestRunner(7396): finished: test_C14841925_verify_notes_block_details_in_workout_details(com.xxx.xxx.xxx.xxx.CircuitsMobileTests)
This line:
10-12 09:24:21.975: D/TestDiscoveryListener(7396): JUnit reported com.xxx.xxx.xxx.xxx.ProfileMobileTests#initializationError; discarding as bogus.
means that none of my ProfileMobileTests run on that shard. (I am running our tests on Firebase Test Lab, via Flank.)
So, for some reason, the test runner is now considering my ProfileMobileTests to be unworthy of running. Sometimes it excludes other test modules, for the same reason. I don't see anything special about ProfileMobileTests or the other excluded test modules.
My best guess so far is this: We run a lot of tests per shard. For the case above, 31 tests are running on that shard. Is it possible that the StackOverflowError is not a sign of an infinite loop, but rather the parser having so many test cases to get through (recursively) that it runs out of stack space before it completes? It doesn't seem likely, but I'm grasping at straws here.
I'm sorry that I don't have all of the desired information; I couldn't repro the problem "in the small". I think that it requires a lot of tests running in order for it to manifest.
As I said, this all started happening when I upgraded the version of androidx.test:runner that we use from 1.4.0 to 1.5.0.
Can anyone help me to decipher the failure message and stack trace associated with processing ProfileMobileTests and/or suggest next steps?
Steps to Reproduce
Expected Results
All tests run
Actual Results
One test module was excluded and its tests did not run.
AndroidX Test and Android OS Versions
androidx.test:runner:1.5.0 and Android 9
Link to a public git repo demonstrating the problem:
Hello -- can I get any feedback on this issue?
The stack trace is confusing, because ClassesArgTokenizer should only be invoked much earlier in the test run, when the runner is deciding which class to run.
It looks from the stack trace that this AndroidJUnit4 constructor is getting called, which is resulting in the command line arguments getting parsed again.
Normally when running on Android, the other constructor should be called
Are you using a custom instrumentation runner?
That being said a StackOverflowError shouldn't happen regardless, but it will be difficult for me to diagnose that without knowing the exact input that was being sent to ClassesArgTokenizer. Would you be able to use a step debugger to see what input is being passed?
Oh wait, I think my diagnosis was wrong. I was looking at the wrong AndroidJUnit4 class.
ext.junit.AndroidJUnit4 will always reparse the entire command line arguments. That seems wrong...
I am using a custom test runner, CallistoTestRunner, which derives from AndroidJUnitRunner. The crash apparently happens when CallistoTestRunner.onStart() calls into AndroidJUnitRunner.onStart().
And this failure is fairly early in the run, before any actual tests are run.
The whole "started:/finished:" pre-test sequence is something that is new to runner:1.5.0.
I will work on getting some step-debugger info for you...
The whole "started:/finished:" pre-test sequence is something that is new to runner:1.5.0.
Can you explain in more detail this comment? The differences between runner 1.4 and 1.5 should be relatively minor
Can you explain in more detail this comment? The differences between runner 1.4 and 1.5 should be relatively minor
Well, full disclosure, we updated a few things along with the runner. We also made the following related updates:
- androidx.test:rules from 1.4.0 to 1.6.0-alpha01
- androidx.test:monitor from 1.6.1 to 1.7.0-alpha01
- All espresso libs (accessibility, contrib, core, idling-resource, intents, web) from 3.4.0 to 3.6.0-alpha01
Some background: we wanted to upgrade to Espresso 3.5.0, but that release causes screenshots to be automatically taken each time an espresso-match failed. And we didn't like that, so we bumped up to 3.6.0-alpha01 so that we could disable those screenshots.
What I meant was we didn't use to see this sort of pre-test run-through before the tests started:
10-12 09:24:21.946: I/TestRunner(7396): run started: 31 tests
10-12 09:24:21.947: I/TestRunner(7396): started: test_CNone_class_filter_supertest(com.xxx.xxx.xxx.xxx.ClassFilterMobileTests)
10-12 09:24:21.947: I/TestRunner(7396): finished: test_CNone_class_filter_supertest(com.xxx.xxx.xxx.xxx.ClassFilterMobileTests)
10-12 09:24:21.949: I/TestRunner(7396): started: test_C1194420_filter_by_subtitles(com.xxx.xxx.xxx.xxx.ClassFilterMobileTests)
10-12 09:24:21.949: I/TestRunner(7396): finished: test_C1194420_filter_by_subtitles(com.xxx.xxx.xxx.xxx.ClassFilterMobileTests)
10-12 09:24:21.951: I/TestRunner(7396): started: test_C25220_can_filter_by_length(com.xxx.xxx.xxx.xxx.ClassFilterMobileTests)
10-12 09:24:21.951: I/TestRunner(7396): finished: test_C25220_can_filter_by_length(com.xxx.xxx.xxx.xxx.ClassFilterMobileTests)
This is where seemingly random test modules are now being excluded:
10-12 09:24:21.975: I/TestRunner(7396): finished: initializationError(com.xxx.xxx.xxx.xxx.ProfileMobileTests)
10-12 09:24:21.975: D/TestDiscoveryListener(7396): JUnit reported com.xxx.xxx.xxx.xxx.ProfileMobileTests#initializationError; discarding as bogus.
And given that that was coming from TestRunner, I assumed it was a TestRunner issue. But I could be wrong about that.
With this many variables at play and no repro case it will be difficult for me to diagnose exactly what is happening. It definitely seems suspicious for ClassesArgTokenizer to fail in the middle of the test run.
I've uploaded a change which should stop ClassesArgTokenizer from being invoked for every ext.junit.AndroidJUnit4 annotated test class - I'm hoping that will address this issue.
OK, Thanks. I realize that I haven't done a great job of giving you enough info to repro. I'll give your change a try.