Analysis fails at highlighter visitor when analyzing UTF-8 encoded file with BOM
mjdetullio opened this issue · comments
Environment:
- SonarQube 5.3
- Java Properties Plugin 1.5
- Gradle Scanner
I noticed some commits towards 1.6 related to detecting files with a BOM and they got reverted. What is your current line of thinking for handling this problem?
A related problem I noticed is, while properties files by default should be ISO-8859-1, this project uses sonar.sourceEncoding which may not apply to all languages. For example my Java source could be UTF-8 but the properties ISO-8859-1. It's also possible to use a Reader set to UTF-8 to read properties files, so it's impossible to know what encoding should really be used. Best convention would be to stick with the default.
I think that there needs to be a more graceful way to handle these files with a BOM such that they do not fail analysis entirely.
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':sonarqube'.
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:69)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.PostExecutionAnalysisTaskExecuter.execute(PostExecutionAnalysisTaskExecuter.java:35)
at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:64)
at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:52)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:53)
at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:203)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:185)
at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:62)
at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:50)
at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:25)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:110)
at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:37)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
at org.gradle.execution.DefaultBuildExecuter.access$000(DefaultBuildExecuter.java:23)
at org.gradle.execution.DefaultBuildExecuter$1.proceed(DefaultBuildExecuter.java:43)
at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:30)
at org.gradle.initialization.DefaultGradleLauncher$4.run(DefaultGradleLauncher.java:158)
at org.gradle.internal.Factories$1.create(Factories.java:22)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:52)
at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:155)
at org.gradle.initialization.DefaultGradleLauncher.access$200(DefaultGradleLauncher.java:36)
at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:103)
at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:97)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:62)
at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:97)
at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:86)
at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:102)
at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28)
at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:47)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:32)
at org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:77)
at org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:47)
at org.gradle.launcher.exec.DaemonUsageSuggestingBuildActionExecuter.execute(DaemonUsageSuggestingBuildActionExecuter.java:51)
at org.gradle.launcher.exec.DaemonUsageSuggestingBuildActionExecuter.execute(DaemonUsageSuggestingBuildActionExecuter.java:28)
at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:43)
at org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:170)
at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:237)
at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:210)
at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35)
at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:206)
at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169)
at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
at org.gradle.launcher.Main.doAction(Main.java:33)
at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:54)
at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:35)
at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
at org.gradle.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:30)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:129)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)
Caused by: org.sonar.squidbridge.api.AnalysisException: Unable to analyze file: /Users/mjdetullio/myproject/src/main/resources/mybundle_da.properties
at org.sonar.squidbridge.AstScanner.scanFiles(AstScanner.java:127)
at org.sonar.plugins.jproperties.JavaPropertiesSquidSensor.analyse(JavaPropertiesSquidSensor.java:87)
at org.sonar.batch.phases.SensorsExecutor.executeSensor(SensorsExecutor.java:58)
at org.sonar.batch.phases.SensorsExecutor.execute(SensorsExecutor.java:50)
at org.sonar.batch.phases.PhaseExecutor.execute(PhaseExecutor.java:98)
at org.sonar.batch.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:185)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:132)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:117)
at org.sonar.batch.scan.ProjectScanContainer.scan(ProjectScanContainer.java:243)
at org.sonar.batch.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:238)
at org.sonar.batch.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:236)
at org.sonar.batch.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:236)
at org.sonar.batch.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:228)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:132)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:117)
at org.sonar.batch.task.ScanTask.execute(ScanTask.java:55)
at org.sonar.batch.task.TaskContainer.doAfterStart(TaskContainer.java:86)
at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:132)
at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:117)
at org.sonar.batch.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:122)
at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:119)
at org.sonar.runner.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:67)
at org.sonar.runner.impl.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:61)
at com.sun.proxy.$Proxy50.execute(Unknown Source)
at org.sonar.runner.api.EmbeddedRunner.doExecute(EmbeddedRunner.java:274)
at org.sonar.runner.api.EmbeddedRunner.runAnalysis(EmbeddedRunner.java:165)
at org.sonar.runner.api.EmbeddedRunner.runAnalysis(EmbeddedRunner.java:152)
at org.sonarqube.gradle.SonarQubeTask.run(SonarQubeTask.java:84)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:75)
at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.doExecute(AnnotationProcessingTaskFactory.java:226)
at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.execute(AnnotationProcessingTaskFactory.java:219)
at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.execute(AnnotationProcessingTaskFactory.java:208)
at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:585)
at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:568)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:80)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:61)
... 60 more
Caused by: java.lang.IllegalArgumentException: Unable to highlight file [moduleKey=com.mycompany:myproject, relative=src/main/resources/mybundle_da.properties, basedir=/Users/mjdetullio/myproject] from offset 808 to offset 876
at org.sonar.api.batch.sensor.highlighting.internal.DefaultHighlighting.highlight(DefaultHighlighting.java:85)
at org.sonar.batch.source.DefaultHighlightable$DefaultHighlightingBuilder.highlight(DefaultHighlightable.java:79)
at org.sonar.jproperties.ast.visitors.SyntaxHighlighterVisitor.visitNode(SyntaxHighlighterVisitor.java:93)
at com.sonar.sslr.impl.ast.AstWalker.visitNode(AstWalker.java:114)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:85)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.visitChildren(AstWalker.java:99)
at com.sonar.sslr.impl.ast.AstWalker.visit(AstWalker.java:87)
at com.sonar.sslr.impl.ast.AstWalker.walkAndVisit(AstWalker.java:69)
at org.sonar.squidbridge.AstScanner.scanFiles(AstScanner.java:106)
... 95 more
Caused by: java.lang.IllegalArgumentException: 876 is not a valid offset for file [moduleKey=com.mycompany:myproject, relative=src/main/resources/iosenroll_da.properties, basedir=/Users/mjdetullio/myproject]. Max offset is 874
at org.sonar.api.internal.google.common.base.Preconditions.checkArgument(Preconditions.java:148)
at org.sonar.api.batch.fs.internal.DefaultInputFile.newPointer(DefaultInputFile.java:267)
at org.sonar.api.batch.fs.internal.DefaultInputFile.newRange(DefaultInputFile.java:262)
at org.sonar.api.batch.sensor.highlighting.internal.DefaultHighlighting.highlight(DefaultHighlighting.java:83)
... 105 more
As the Java specifications state that properties files must be encoded in ISO-8859-1, I finally decided to not support files encoded in UTF-8 with BOM. I believe that you should re-encode your properties files in ISO-8859-1. What do you think? Next week (when I'm back from vacation), I'll update the README file regarding file encoding.
The project I'm configuring for analysis is relatively large and I do not own the source, so I can't re-encode some 1000 properties files off UTF-8. I did remove the BOM on the ~10 bad files.
If your plan is to only support ISO-8859-1, then I think you should remove any references to sonar.sourceEncoding as well. It may be worthwhile to always decode files using ISO-8859-1 with a CharsetDecoder and create an issue if there are exceptions when decoding. Because it seems easy to run into this problem, I think this should be a good candidate for a rule.
Thanks for your suggestions. I'll try and improve the plugin as soon as I'm back in two weeks. And I'll keep you posted.
@mjdetullio, all decoding is done in ISO-8859-1 now: b31f042
But, it won't remove the issue in some cases: with UTF-8 files starting with a BOM. Indeed, the BOM character is decoded without any error as  in ISO-8859-1. Then the error on syntax highlighting is triggered. I added a note about encoding in the README file: https://github.com/racodond/sonar-jproperties-plugin/blob/master/README.md. I don't really want to start checking for a BOM character before highlighting a file... Any feedback is welcome.
I think breaking sonar analysis due to encoding is just awful. Encoding should really be a rule instead.
@dcsobral: There's no way to know the encoding of a file.
True, but you can distinguish between a valid iso-8859-1 file and a non-valid one, and you can detect BOM for UTF-8/UTF-16. I don't see much of a point in detecting encoding, aside from enabling other rules to be processed. Valid files and UTF-8 would cover most cases, though.
Now, not crashing due to invalid encoding and having invalid encoding be a rule, these are useful things.
I do agree with @dcsobral, the better approach is to have it as sonar rule, instead of abort the whole validation process. Even though I understand your reasons @racodond I do believe you should provide this:
- Direct feedback to the user/developer about the potential issue; Let's say like this: catching the exception, and printing out as a WARN message with the same information you stated on the README file.
I recently got this issue after upgrading from Sonar 5.2 to 5.6 LTS, it actually broke a couple of my Jenkins' jobs and I had to dig through google in order to get the real root cause. The alternative that I'm proposing will provide some sort of backward compatibility instead of forcing the users to either change a bunch of properties files across several projects or (if they don't dig enough the internet in order to get your explanation) just disabling the plugin on sonar.
I will add support of UTF-8 files that will fix this issue. See #35