Incorrect Coverage
hotforfeature opened this issue · comments
The current coverage being generated does not reflect the source. At the moment, since the test files are so simple, it's hard to notice. However, when a more complex class and test spec is introduced, we get errors when the coverage points to lines that don't match up to the source file.
I believe the problem is that rollup-plugin-stanbul is instrumenting the wrong file. I've tried with no luck to try and get either the karma-remap-istanbul or the karma-remap-coverage working to compensate.
I created a fork with a LoggerService class and test case to demonstrate the issue: https://github.com/hotforfeature/ionic2-boilerplate
Clone, npm install, then npm test to get the following error:
Sales Service
✔ should load component
✔ should load component
LoggerService
.create()
✔ should create a Logger with prefix
Finished in 0.061 secs / 0.179 secs
SUMMARY:
✔ 3 tests completed
--------------------|----------|----------|----------|----------|----------------|
File | % Stmts | % Branch | % Funcs | % Lines |Uncovered Lines |
--------------------|----------|----------|----------|----------|----------------|
app/ | 25.4 | 15.38 | 50 | 25.4 | |
logger.service.ts | 25.4 | 15.38 | 50 | 25.4 |... 94,95,96,97 |
pages/contact/ | 83.33 | 80 | 100 | 83.33 | |
contact.ts | 83.33 | 80 | 100 | 83.33 | 15 |
pages/home/ | 85.71 | 80 | 100 | 85.71 | |
home.ts | 85.71 | 80 | 100 | 85.71 | 34 |
--------------------|----------|----------|----------|----------|----------------|
All files | 35.53 | 33.33 | 64.29 | 35.53 | |
--------------------|----------|----------|----------|----------|----------------|
13 10 2016 14:50:46.844:ERROR [coverage]: TypeError: Cannot read property 'text' of undefined
13 10 2016 14:50:46.844:ERROR [coverage]: TypeError: Cannot read property 'text' of undefined
at /Users/andrew.mitchell/Documents/Projects/GitHub/ionic2-boilerplate/node_modules/istanbul/lib/report/html.js:241:45
at Array.forEach (native)
at annotateFunctions (/Users/andrew.mitchell/Documents/Projects/GitHub/ionic2-boilerplate/node_modules/istanbul/lib/report/html.js:224:26)
at HtmlReport.writeDetailPage (/Users/andrew.mitchell/Documents/Projects/GitHub/ionic2-boilerplate/node_modules/istanbul/lib/report/html.js:427:9)
at /Users/andrew.mitchell/Documents/Projects/GitHub/ionic2-boilerplate/node_modules/istanbul/lib/report/html.js:489:26
at SyncFileWriter.writeFile (/Users/andrew.mitchell/Documents/Projects/GitHub/ionic2-boilerplate/node_modules/istanbul/lib/util/file-writer.js:57:9)
at FileWriter.writeFile (/Users/andrew.mitchell/Documents/Projects/GitHub/ionic2-boilerplate/node_modules/istanbul/lib/util/file-writer.js:147:23)
at /Users/andrew.mitchell/Documents/Projects/GitHub/ionic2-boilerplate/node_modules/istanbul/lib/report/html.js:488:24
at Array.forEach (native)
at HtmlReport.writeFiles (/Users/andrew.mitchell/Documents/Projects/GitHub/ionic2-boilerplate/node_modules/istanbul/lib/report/html.js:482:23)
at /Users/andrew.mitchell/Documents/Projects/GitHub/ionic2-boilerplate/node_modules/istanbul/lib/report/html.js:484:22
at Array.forEach (native)
at HtmlReport.writeFiles (/Users/andrew.mitchell/Documents/Projects/GitHub/ionic2-boilerplate/node_modules/istanbul/lib/report/html.js:482:23)
at HtmlReport.writeReport (/Users/andrew.mitchell/Documents/Projects/GitHub/ionic2-boilerplate/node_modules/istanbul/lib/report/html.js:566:14)
at writeReport (/Users/andrew.mitchell/Documents/Projects/GitHub/ionic2-boilerplate/node_modules/karma-coverage/lib/reporter.js:68:16)
at /Users/andrew.mitchell/Documents/Projects/GitHub/ionic2-boilerplate/node_modules/karma-coverage/lib/reporter.js:296:11
logger.service.ts is only 81 lines long. The text summary indicates that the instrumentation was looking at lines beyond the source file, which indicates to me that it was instrumenting a generated file somewhere in the rollup.
Since the line numbers are off, this causes an error in the HTML reporter when it tries to annotate lines from the original source file that are not present.
Note that this is only affecting code coverage. The actual test process itself is working fine, so you can disable code coverage and be able to continue working.
@hotforfeature thank you vary much for your report. It looks related by the order of the plugin (see artberri/rollup-plugin-istanbul#6), but unfortunately typescript complicate things. I have open an issue on rollup-plugin-istanbul, but i will keep trying to resolve it. Maybe same tweak on ts + istanbul plugins can make the magic.
+1
I got the same problem. I tring find the solution by now...I've did a test with a new project using angular-cli that use webpack and the coverage works great :(
I think that missing point is a rollup-remap-istanbul-plugin that wrap https://github.com/SitePen/remap-istanbul.
I will try to make one in the next days, but if it takes too much i will concentrate on other things, like automatic .ipa, that in long term is more useful.
About webpack, have you noticed this PR :) -> ionic-team/ionic-app-scripts#158
@hotforfeature I had the same error with karma-remap-istanbul, switched to karma-remap-coverage as workaround without problems. Did you inline the source maps as suggested?
@duydao tried source inline but not karma-remap-coverege yet. Thank you, will try soon. Do you mind pasting same of your code from karma.config to speed up things? Anyway thank you!
Sure, this is the relevant configuration in karma.conf.js:
reporters: ['mocha', 'coverage', 'remap-coverage'],
coverageReporter: {
type: 'in-memory'
},
remapCoverageReporter: {
'text-summary': null, // to show summary in console
html: './coverage/html'
},@duydao mmm there is something I'm missing in rollupPreprocessor i think. I tried
// base path that will be used to resolve all patterns (e.g. files, exclude)
basePath: '../',
frameworks: ['jasmine'],
plugins: [
require('karma-jasmine'),
require('karma-rollup-plugin'),
require('karma-phantomjs-launcher'),
require('karma-mocha-reporter'),
require('karma-coverage'),
require('karma-remap-coverage')
],
reporters: ['mocha', 'coverage', 'remap-coverage'],
files: ['config/karma-shim.ts'],
preprocessors: {'config/karma-shim.ts': ['rollup', 'coverage']},
rollupPreprocessor: {
context: 'this',
plugins: [
angular({
exclude: 'node_modules/**'
}),
ts({
typescript: require('typescript'),
sourceMap: false,
inlineSourceMap: true
}),
alias({
'@angular/core/testing': path.resolve(__dirname, '../node_modules/@angular/core/testing/index.js'),
'@angular/platform-browser-dynamic/testing': path.resolve(__dirname, '../node_modules/@angular/platform-browser-dynamic/testing/index.js'),
'@angular/compiler/testing': path.resolve(__dirname, '../node_modules/@angular/compiler/testing/index.js'),
'@angular/platform-browser/testing': path.resolve(__dirname, '../node_modules/@angular/platform-browser/testing/index.js')
}),
commonjs(),
nodeResolve({jsnext: true, main: true, browser: true}),
buble()
],
format: 'iife',
// sourceMap: "inline"
},
port: 9876,
colors: true,
logLevel: config.LOG_INFO,
autoWatch: true,
browsers: ['PhantomJS'],
singleRun: true,
coverageReporter: {
type: 'in-memory'
},
remapCoverageReporter: {
'text-summary': null, // to show summary in console
html: './coverage/istanbul'
},
But it's throwing:
16 10 2016 23:37:42.549:WARN [reporter]: SourceMap position not found for trace: undefined
PhantomJS 2.1.1 (Mac OS X 0.0.0) ERROR
SyntaxError: Expected token ']'
at config/karma-shim.ts:9
There seems to be a JS error in the karma.conf.js or in the shim.
Sorry, I don't have the excact config file anymore, we moved to system.js since rollup was terribly slow due to redux (or having lots of module dependencies like rxjs).
We are using a modified karma config of https://github.com/angular/quickstart
Using the above configuration for me does not generate any errors, but it presents 0/0 code code coverage for statements, branches, functions, and lines.
I think I got that error once because ./config.karma-shim.ts was in typescript. I installed istanbul@1.1.0-alpha.1 in the project to get around it.
Last release 0.10.* fixed Istanbul. Also as the ionic team as moved to Webpack i changed the bundling system for unit tests from Rollup to Webpack. This release has finally Typedoc working for documentation. Thank you all, and let me know if you have any issue with the new release.
Thanks @marcoturi, it's working now after I updated my repository to match.
Quick note if anybody finds this issue and is still having problems: it is critical that "istanbul-instrumenter-loader": "0.2.0" is set in package.json. A regular npm install --save-dev istanbul-instrumenter-loader will save a later version beyond 0.2.0 that causes similar Cannot read property 'text' of undefined errors.
@hotforfeature Thanks for the info, i would add another issue i met during the upgrade. If you ancounter any issue about karma not reloading correctly in watch mode, look here.
Finally for tracking purpose, the issue about istanbul-instrumenter-loader is tracked here