Giters

MarathonLabs / marathon

Cross-platform test runner

Home Page:https://docs.marathonlabs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Obj-C tests not executed properly

michaelhuman opened this issue · comments

commented

Hi!
I've met some strange fails with Marathon on my iOS project, so I decided to make brand new proj from Xcode 15.1 / Empty App template / Obj-C lang (it generated 2 template unit tests as well) to demonstrate the problem.
XApp.zip

Congif files:

workers:
  - transport:
      type: local
    devices:
      - type: simulatorProfile
        deviceType: com.apple.CoreSimulator.SimDeviceType.iPhone-15-Pro

name: "XApp"
outputDir: "marathon-output"
debug: true
analyticsTracking: false
bugsnagReporting: false
vendorConfiguration:
  type: "iOS"
  bundle:
    application: XApp.app
    testApplication: XApp.app/PlugIns/XAppTests.xctest
    testType: xctest
uncompletedTestRetryQuota: 0

After running Marathon we have the output:
marathon_output.txt
with 2 failed tests. As I see, the reason of it is that Marathon couldn't find tests and executed 0 tests in each suit.

For example, raw command line invocation:
D 14:40:02.278 [DefaultDispatcher-worker-12 @coroutine#280] <DebugLogPrinter> /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun /tmp/marathon/67F6B08F-4138-4ED7-974B-A85C9CE062E6/marathon.xctestrun "-only-testing:XAppTests/testPerformanceExample" -enableCodeCoverage NO -resultBundlePath /tmp/marathon/67F6B08F-4138-4ED7-974B-A85C9CE062E6/67F6B08F-4138-4ED7-974B-A85C9CE062E6.95e3ff9f-6bce-40df-a2a6-bbad5793b064.xcresult -destination-timeout 30 -destination "platform=iOS simulator,id=67F6B08F-4138-4ED7-974B-A85C9CE062E6"
executed 0 tests (which marked "TEST EXECUTE SUCCEEDED").

To make it with success I injected the additional path "XAppTests/" to the test route like this (see "-only-testing:XAppTests/XAppTests/testPerformanceExample"):
D 14:40:02.278 [DefaultDispatcher-worker-12 @coroutine#280] <DebugLogPrinter> /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun /tmp/marathon/67F6B08F-4138-4ED7-974B-A85C9CE062E6/marathon.xctestrun "-only-testing:XAppTests/XAppTests/testPerformanceExample" -enableCodeCoverage NO -resultBundlePath /tmp/marathon/67F6B08F-4138-4ED7-974B-A85C9CE062E6/67F6B08F-4138-4ED7-974B-A85C9CE062E6.95e3ff9f-6bce-40df-a2a6-bbad5793b064.xcresult -destination-timeout 30 -destination "platform=iOS simulator,id=67F6B08F-4138-4ED7-974B-A85C9CE062E6"
and now we have "Executed 1 test" as expected.

So it looks like Marathon cannot construct the correct path to an obj-c test suit. Or maybe I miss something? Thank you.

commented

Hey @michaelhuman

So the relevant parts of your log are:

I 14:39:58.714 [main @coroutine#2] <com.malinskiy.marathon.Marathon> Scheduling 3 tests
D 14:39:58.715 [main @coroutine#2] <com.malinskiy.marathon.Marathon> XAppTests#testExample, XAppTests#testPerformanceExample, XAppTests#testPerformanceExample

This contains test duplication for some reason, curious how the tests are actually named.

D 14:39:59.669 [AppleSimulatorDevice - execution - local-2 @coroutine#227] <c.m.m.i.bin.xcrun.xcodebuild.Xcodebuild> Running xcrun xcodebuild test-without-building -xctestrun /tmp/marathon/67F6B08F-4138-4ED7-974B-A85C9CE062E6/marathon.xctestrun '-only-testing:XAppTests/testExample' -enableCodeCoverage NO -resultBundlePath /tmp/marathon/67F6B08F-4138-4ED7-974B-A85C9CE062E6/67F6B08F-4138-4ED7-974B-A85C9CE062E6.e17696a1-24e7-4b32-a51d-dff931e3b227.xcresult -destination-timeout 30 -destination 'platform=iOS simulator,id=67F6B08F-4138-4ED7-974B-A85C9CE062E6'

and

D 14:40:00.217 [DefaultDispatcher-worker-8 @coroutine#231] <DebugLogPrinter> Testing started
D 14:40:01.336 [DefaultDispatcher-worker-8 @coroutine#231] <DebugLogPrinter> Test Suite 'Selected tests' started at 2024-02-14 14:40:01.335.
D 14:40:01.337 [DefaultDispatcher-worker-8 @coroutine#231] <DebugLogPrinter> Test Suite 'XAppTests.xctest' started at 2024-02-14 14:40:01.335.
D 14:40:01.338 [DefaultDispatcher-worker-8 @coroutine#231] <DebugLogPrinter> Test Suite 'XAppTests.xctest' passed at 2024-02-14 14:40:01.336.
D 14:40:01.339 [DefaultDispatcher-worker-8 @coroutine#231] <DebugLogPrinter> 	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
D 14:40:01.339 [DefaultDispatcher-worker-8 @coroutine#231] <DebugLogPrinter> Test Suite 'Selected tests' passed at 2024-02-14 14:40:01.336.
D 14:40:01.339 [DefaultDispatcher-worker-8 @coroutine#231] <DebugLogPrinter> 	 Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds

This means that your tests are not executed likely due to invalid test parsing.

For fixing this I need the output of the test parsing commands and the list of actual tests present.

marathon/vendor/vendor-ios/src/main/kotlin/com/malinskiy/marathon/ios/NmTestParser.kt

Line 71 in 6ec57df

val rawObjectiveCTests = device.binaryEnvironment.nm.objectiveCTests(remoteTestBinary)

marathon/vendor/vendor-ios/src/main/kotlin/com/malinskiy/marathon/ios/bin/nm/Nm.kt

Line 39 in 6ec57df

listSymbolsVia(path, "'nm -U \"$path\" | grep \" t \" | cut -d\\ -f3,4 | cut -d \"-\" -f2 | cut -d \"[\" -f2 | cut -d \"]\" -f1 | grep \" test\"'")

Alternatively, you can always try xctestparser

commented

I did a Marathon run with xctestparser enabled (same result as above with NM parser), here is the output:
marathon_output2.txt

D 15:31:04.866 [DefaultDispatcher-worker-11 @coroutine#129] <TestRunProgressParser> Test .XAppTests.testExample finished with result <passed> after 0.001 seconds
D 15:31:05.130 [DefaultDispatcher-worker-4 @coroutine#129] <TestRunProgressParser> Test .XAppTests.testPerformanceExample finished with result <passed> after 0.267 seconds

And the XCTestCase:

#import <XCTest/XCTest.h>

@interface XAppTests : XCTestCase

@end

@implementation XAppTests

- (void)setUp {
    // Put setup code here. This method is called before the invocation of each test method in the class.
}

- (void)tearDown {
    // Put teardown code here. This method is called after the invocation of each test method in the class.
}

- (void)testExample {
    // This is an example of a functional test case.
    // Use XCTAssert and related functions to verify your tests produce the correct results.
}

- (void)testPerformanceExample {
    // This is an example of a performance test case.
    [self measureBlock:^{
        // Put the code you want to measure the time of here.
    }];
}

@end
commented

If the result is the same then I'm sure test filtering is broken for your test bundle as in -only-testing flags are not working as expected. Try to manually run a single test using plain xcodebuild and compare it with the marathon's invocation

commented

Just did NM output:

nm -U XApp.app/PlugIns/XAppTests.xctest/XAppTests | grep " t " | cut -d\  -f3,4 | cut -d "-" -f2 | cut -d "[" -f2 | cut -d "]" -f1 | grep " test" 
XAppTests testExample
XAppTests testPerformanceExample
XAppTests testPerformanceExample
commented

Plain xcodebuild execution has a similar problem:

xcodebuild -xctestrun XApp_XApp_iphonesimulator17.2-arm64.xctestrun \
-only-testing:XAppTests/testExample -destination 'platform=iOS Simulator,name=iPhone 15 Pro' \          
-enableCodeCoverage NO -destination-timeout 30 test-without-building
Command line invocation:
    /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -xctestrun XApp_XApp_iphonesimulator17.2-arm64.xctestrun "-only-testing:XAppTests/testExample" -destination "platform=iOS Simulator,name=iPhone 15 Pro" -enableCodeCoverage NO -destination-timeout 30 test-without-building

User defaults from command line:
    IDEPackageSupportUseBuiltinSCM = YES

--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:iOS Simulator, id:67F6B08F-4138-4ED7-974B-A85C9CE062E6, OS:17.2, name:iPhone 15 Pro }
{ platform:iOS Simulator, id:67F6B08F-4138-4ED7-974B-A85C9CE062E6, OS:17.2, name:iPhone 15 Pro }
Testing started
2024-02-14 16:56:36.428 xcodebuild[3483:48400] [MT] IDETestOperationsObserverDebug: 24.235 elapsed -- Testing started completed.
2024-02-14 16:56:36.428 xcodebuild[3483:48400] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2024-02-14 16:56:36.428 xcodebuild[3483:48400] [MT] IDETestOperationsObserverDebug: 24.235 sec, +24.235 sec -- end

Test session results, code coverage, and logs:
	/Users/mikhail/Library/Developer/Xcode/DerivedData/XApp-cpmefszitiblsffbwyfmrszkpnyq/Logs/Test/Test-XApp-2024.02.14_16-56-12-+0300.xcresult

** TEST EXECUTE SUCCEEDED **

0 tests executed.

And after changing the path of the test ("XAppTests/" added) in the "-only-testing" arg we have success:

xcodebuild -xctestrun XApp_XApp_iphonesimulator17.2-arm64.xctestrun \
-only-testing:XAppTests/XAppTests/testExample -destination 'platform=iOS Simulator,name=iPhone 15 Pro' \
-enableCodeCoverage NO -destination-timeout 30 test-without-building
Command line invocation:
    /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -xctestrun XApp_XApp_iphonesimulator17.2-arm64.xctestrun "-only-testing:XAppTests/XAppTests/testExample" -destination "platform=iOS Simulator,name=iPhone 15 Pro" -enableCodeCoverage NO -destination-timeout 30 test-without-building

User defaults from command line:
    IDEPackageSupportUseBuiltinSCM = YES

--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:iOS Simulator, id:67F6B08F-4138-4ED7-974B-A85C9CE062E6, OS:17.2, name:iPhone 15 Pro }
{ platform:iOS Simulator, id:67F6B08F-4138-4ED7-974B-A85C9CE062E6, OS:17.2, name:iPhone 15 Pro }
Testing started
Test suite 'XAppTests' started on 'Clone 1 of iPhone 15 Pro - XApp (3895)'
Test case '-[XAppTests testExample]' passed on 'Clone 1 of iPhone 15 Pro - XApp (3895)' (0.001 seconds)
2024-02-14 16:59:01.846 xcodebuild[3698:52216] [MT] IDETestOperationsObserverDebug: 24.434 elapsed -- Testing started completed.
2024-02-14 16:59:01.847 xcodebuild[3698:52216] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2024-02-14 16:59:01.847 xcodebuild[3698:52216] [MT] IDETestOperationsObserverDebug: 24.434 sec, +24.434 sec -- end

Test session results, code coverage, and logs:
	/Users/mikhail/Library/Developer/Xcode/DerivedData/XApp-fgkvcemprigawucedeohomvzlrgh/Logs/Test/Test-XApp-2024.02.14_16-58-37-+0300.xcresult

** TEST EXECUTE SUCCEEDED **
commented

The issue here is that the output of the xcodebuild is not printing proper test cases:

Test case '-[XAppTests testExample]' passed on 'Clone 1 of iPhone 15 Pro - XApp (3895)' (0.001 seconds)

when it's actually XAppTests.XAppTests testExample where first XAppTests here is the target name and second XAppTests being the class (see

marathon/vendor/vendor-ios/src/test/resources/fixtures/test_output/success_0.log

Line 13 in 6ec57df

Test Case '-[sample_appUITests.MoreTests testPresentModal]' started.
and other fixtures for examples of output). This fails both nm and xctest parsers since it's just a bug in the xcode itself.

That being said, I've semi-patched this same problem in flutter ecosystem in #879 which was released in 0.9.1. Give it a go

commented

It appeared, I had a previous 0.9.0 release :(. After updating to 0.9.1 I finally could execute obj-c tests with success, but only with the testParserConfiguration: type: "xctest" enabled (nm doesn't work unfortunately).
Thanks for the hint!