Feature Request: Summary of Test Scenarios in HTML Report
alpha1592 opened this issue · comments
This is a feature request for HTML Reports.
Currently in the HTML reports, the left side menu bar shows a list of test specifications that were tested. Its red for a failed test spec and green for passed, which is great. However, when you click on the test spec, you get a page that is loaded with all test scenarios that are part of the test spec. If you have a lot of test scenarios, like we do (200+), there is lots of scrolling involved in order to get to failed test scenarios. Also, its hard to keep track of which scenario you are looking at.
It would be nice if we can have either;
- A summary table that lists all the scenarios with their pass/fail info and the ability to click on it to go directly to the section for that scenario or,
- Update the menu bar on the left to become a sub-menu (once we click on the test spec) which shows all the scenarios in the spec with pass/fail as green and red colours.
Both options are shown here...
html-report.zip
@alpha1592 We like to keep the reports as simple as possible so that it is suitable for most users. In this case it looks like a very specific request.
You can always customize the reports to match your needs. html-report supports themes, so you can build your own theme and the plugin will render your report accordingly.
This #131 (comment) should help you in creating your own theme for the reports.
if you have difficulty in creating this, please do reach out here or on the gitter channel for help.
@nehashri Thanks, i will look into creating my own theme.
However, i do believe this is a common use case. I have talked to several testers who would want to be able to get to their test scenarios quicker. When you have 50+ test scenarios in a spec with large number of test failures, you have a lot of stack traces etc being shown on the report, the report gets really long in terms of vertical space. To be able to get to your 40th test scenario out of 50 to investigate the failures, you would be scrolling for days.
I agree that there's a usability issue when navigating reports for large test suites.
We'll look into this soon to see what the best design to solve this problem.
Thank you.
Displaying wrapping the scenario text in as a sub menu is a problem when the scenario is descriptive.
Displaying the summary of the scenarios (as a table) will make a user scroll in case there are a lot of scenarios.
Will these alternative solutions solve your problem?
- Have an option or button to toggle between showing all scenarios or only failed scenarios
- Collapse/summarise successful scenarios and expand only failed scenarios. Each scenario can be expanded or collapsed individually and there will be an option to expand collapse all.
I agree that adding scenarios to the sub-menu might not work especially if the scenario title is too long. However, the Summary Table is something that would give testers a quick view of the pass fail information and would still be relatively less scrolling compared to scrolling through scenarios.
In our case, we are testing SOAP Web Services and we output Request/Response messages to the report. The response messages are really really long sometimes, and so even if there were 5 failures out of 10, in order to get to the 5th scenario to investigate the failure, we would to scroll a lot. instead, if we had a table with a clickable link/anchor to go directly to the 5th scenario, than that would save a lot more effort. You could even have a table that looks similar to the one you have for CSV data rows (when a scenario is run with multiple rows in a CSV). You have a red row for failed and green row for pass. Same Idea and it looks good too.
I also like this idea in conjunction with the summary table noted above.
- Collapse/summarise successful scenarios and expand only failed scenarios. Each scenario can be expanded or collapsed individually and there will be an option to expand collapse all.
Closing this as an old issue. However, any PR's to fix this will be merged.