Xcode 11.5 (11.4.1) produces incorrect code-coverage for modules built with `wholemodule` optimisation

Radar: FB7723487

After switching to Xcode 11.5, we noticed that code-coverage started to produce unpredictable results to us. The investigation showed that xccov generated unreliable results when a module built with wholemodule option for SWIFT_COMPILATION_MODE. The issue is reproducible for Xcode 11.4.1 as well.

This repository provides a minimal example to reproduce the issue.

TL;DR

So far, it looks like that changes in the Swift 5.2.* Toolchain cause the problem. Perhaps, the compiler changed the way how inlinable functions, extensions and other non-trivial cases are processed during the compilation pipeline but xccov was not updated accordingly.

How does it work

There is a check_coverage.sh script that executes the following steps:

Checkout CocoaLumberjack repository to use it as an example project
Run tests twice with different SWIFT_COMPILATION_MODE options and produce xcresult bundles
Show target coverage for both runs
Show diff between runs compiled with incremental and wholemodule options

Results

Note: all the results below were collected with Xcode 11.5 / Swift 5.2.4 Toolchain. I see the same behaviour for Xcode 11.4.1 as well.

As you can see from the table below, xccov fails to count functions (classes?) properly when a module compiled with wholemodule optimisation. With the wholemodule option, only one file is tracked as covered with unit-tests.

SWIFT_COMPILATION_MODE option	Coverage	Source files	Functions (covered/total)
incremental	53.52%	3	76/142
wholemodule	100.00%	1	37/37

This is a diff produced by xccov diff command between reports produces for incremental and wholemodule:

{
    "addedTargets": [],
    "lineCoverageDelta": {
        "coveredLinesDelta": -39,
        "executableLinesDelta": -105,
        "lineCoverageDelta": 0.46478873239436624
    },
    "removedTargets": [],
    "targetDeltas": [
        {
            "addedFiles": [],
            "fileDeltas": [],
            "lineCoverageDelta": {
                "coveredLinesDelta": -39,
                "executableLinesDelta": -105,
                "lineCoverageDelta": 0.46478873239436624
            },
            "name": "CocoaLumberjackSwift.framework",
            "removedFiles": [
                {
                    "documentLocation": "/Users/dive/radar-xcode-11-5-code-coverage-issue/CocoaLumberjack/Sources/CocoaLumberjackSwift/CocoaLumberjack.swift"
                },
                {
                    "documentLocation": "/Users/dive/radar-xcode-11-5-code-coverage-issue/CocoaLumberjack/Sources/CocoaLumberjackSwift/DDAssert.swift"
                }
            ]
        }
    ]
}

As I can see from the diff, xccov stopped to produce coverage in the following cases:

For global functions
For @inline functions
For static and general extensions

You can check the full diff between incremental and wholemodule results here.

Xcode 11.3.1

Just to be sure, I ran the same script for Xcode 11.3.1. The issue does not exist in this version.

SWIFT_COMPILATION_MODE option	Coverage	Source files	Functions (covered/total)
incremental	53.19%	3	75/141
wholemodule	53.19%	3	75/141

Notes

The script generates coverage for targets only. You can change it by removing --only-targets option from produceCodeCoverage function
The script cleans up the CocoaLumberjack repository after each run and removes the xcresult bundles (check cleanUp function if you want to change it)

dive / radar-xcode-11-5-code-coverage-issue