prmr / DScribe-Research

Data artifact for research on unit test generation for documentation

DScribe-Research

This repository contains the data artifact for our research on unit test generation for documentation.

License and Attribution

The content of this repository is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

To indicate attribution, please link to this repository and cite the following technical report:

Mathieu Nassif, Alexa Hernandez, Ashvitha Sridharan, and Martin P. Robillard. Generating Unit Tests for Documentation. ArXiv preprint arXiv:2005.08750, 12 pages. 18 May 2020.

Structure

• Section 6 - Usefulness Study: this folder contains the configuration and results of our application of DScribe to the Apache Commons IO project, as well as an Excel file detailing all identified exception specification units (ESU). The internal structure is as follows:
  • Apache Commons IO.xlsx: detail of the ESUs
  • agreement-usefulness.xlsx: results of the independent re-annotation phase to assess the subjectivity of the task
  • configs: invocation files
  • templates: template definition file
  • output-tests: generated unit tests
  • output-docs: generated documentation (look for @dscribe Javadoc tags)
  • testUtils: helper methods for the test code
• Section 7 - Comparison Study: this folder contains the sample of unit tests generated by DScribe, EvoSuite, and Randoop, as well as tests from the Apache Commons IO test suite, annotated by external developers for their focus and readability. The internal structure is as follows:
  • Annotation Results.xlsx: raw annotations given by the annotators
  • Annotator {A,B,C,D} tests: tests evaluated by each annotator, sampled from all four test suites and transformed to remove generation artifacts (e.g., serial test names such as test0123)
  • EvoSuite tests: all tests generated by EvoSuite
  • Randoop tests: all tests generated by Randoop
  • Tests generated by DScribe are available in the Section 6 - Usefulness Study folder, and tests from the Commons IO test suite are available at https://github.com/apache/commons-io/tree/commons-io-2.6/src/test/java/org/apache/commons/io
• Section 8 - Validation Study: this folder contains the detailed list of all units of specifications extracted from a subset the unit tests of the Apache Commons Lang, Math, and Configuration projects, as well as the results of the independent re-annotation to assess the subjectivity of the study.
• Section 9 - Limitations Study: this folder contains the final codes attributed to each test in our sample from five open source projects (Freemind, Eclipse, Weka, Tomcat, and Hibernate). The catalog of codes is the result of a preliminary open coding process.

DScribe tool

The implementation of our technique, DScribe, is available in a separate repository, located at https://github.com/prmr/DScribe