emanuel-braz / dlcov

dlcov verify dart / flutter code coverage threshold

Home Page:https://pub.dev/packages/dlcov

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Can't run without Flutter SDK

bradyt opened this issue · comments

I have some Dart libraries being tested in CI, without the Flutter SDK, especially so custom Docker images with extra tools, used in the CI, can be modified and built, pushed and pulled faster over slower networks, and take up less size on devices, since available Docker images with Flutter SDK tend to be much larger.

When I try to run dart pub global run dlcov -c 100 on CI, it errors since flutter executable is not available, for flutter test --coverage.

I'm hoping to check coverage in CI without a dependency on Flutter SDK.

@bradyt
I understand you, and I realized that I made a bad choice in adding this dependency, while I was trying to make things as automatic as possible.

Because Flutter only takes into account the tested files, it doesn't consider the files that have 0% of tests, and to solve this problem I had to join three steps:

1 - Generate a dependency file, so that Flutter takes all files into account.
2 - Generate the .lcov file using "flutter test --coverage" (this step depends on the step above)
3 - And finally, check the test coverage.

I can create a test-command parameter. That way it would run all three steps at once, but only if the parameter is given.

Example:
dlcov -c 100 --test-command="flutter test --coverage" : Executes step 1, 2 and 3

or

dlcov -c 100 Run only step 3. The coverage/lcov.info file is assumed to already exist.

If you need to generate the dependency file outside of CI/CD, I can create the prepare command

dlcov prepare Generate dependencies file (This step is important to happen before generating "lcov.info" through flutter test --coverage)

So you can run step 1 and 2, outside the CI/CD, and keep these files in your version control. And just run step 3 dlcov -c 100 in the CI/CD pipeline.

What do you think? would you like to suggest a different solution?

That sounds great. I tested in CI, with your project as subtree, removed the call to flutter test --coverage, ran dart pub global activate -spath ../dlcov, used test_cov to generate the coverage/lcov.info, and can even use other tools to remove .g.dart from lcov file, and use dlcov -c 100 -e '' to get the desired error.

dlcov already removes .g, .freezed and .part by default, but you can override this behavior.

I'm going to make these changes I mentioned and I'd like to advise you to use the dlcov prepare command before running test_cov.
Example: dlcov prepare && test_cov && dlcov -c 100

This way you will have a report more coherent with reality, since the dlcov will prepare the references first.

When I'm done with these tweaks, you can use it this way too:

dlcov -c 100 --lcov-gen="test_cov"
or
dlcov -c 100 --lcov-gen="flutter test --coverage"

I'm not sure the prepare is necessary with test_cov. I just checked with stagehand template, copied a lib file to a second file, and the result of dart pub global run test_cov; genhtml coverage/lcov.info coverage dropped from 100% to 50%.

Oh interesting, test plus test_cov won't detect untested libraries. Your dlcov does.

It may be that Dart/Flutter team simply consider this sort of result as expected behavior. The Flutter skeleton template results in an empty coverage/lcov.info: flutter/flutter#96306.

Hi @bradyt , may you test the new version (4.0.1)?

I removed the Flutter SDK dependency, and you can use this command: dlcov -c 100

If you consider all untested files you can use the parameter include-untested-files in conjunction with lcov-gen
dlcov -c 100 --lcov-gen="test_cov" --include-untested-files=true

You can also generate untested files reference, this way:
dlcov gen-refs
test_cov
dlcov -c 100

It works great, thank you!

I'm now able to use dlcov -c 100 on CI without the Flutter SDK dependency.

I'm not sure what happened with dlcov gen-refs. Locally it works fine, even after git clean -dffx and no coverage/lcov.info, but running locally with act, I have to do mkdir coverage; touch coverage/lcov.info, otherwise I get the following:

FileSystemException: Cannot open file, path = 'coverage/lcov.info' (OS Error: No such file or directory, errno = 2)

I'll try to research this a little more in the near future.

EDIT: I think I typo'd, it must be dlcov -c 100 that can throw that error if the file is not there. Wrote it carefully, CI no longer fails.

Edited, disregard the issue described in previous comment.

Great! So I believe we can close this issue. Thanks for the support!