This is a dataset of open source Java classes and some metrics on them. Every now and then I make a new version of it using the scripts in this repository. You are welcome to use it in your researches. Each release has a fixed version. By referring to it in your research you avoid ambiguity and guarantees repeatability of your experiments.
This is a more formal explanation of this project: in PDF.
The latest ZIP archive with the dataset is here: cam-2023-10-22.zip (2.19Gb). There are 33 metrics calculated for 862,517 Java classes from 1000 GitHub repositories, including: lines of code (reported by cloc); NCSS; cyclomatic and cognitive complexity (by PMD); Halstead volume, effort, and difficulty; maintainability index; number of attributes, constructors, methods; and others (see PDF).
Previous archives (took me a few days to build each of them, using a pretty big machine):
- cam-2023-10-22.zip (2.19Gb): 1000 repos, 33 metrics, 863K classes
- cam-2023-10-11.zip (3Gb): 959 repos, 29 metrics, 840K classes
- cam-2021-08-04.zip (692Mb): 1000 repos, 15 metrics
- cam-2021-07-08.zip (387Mb): 1000 repos, 11 metrics
If you want to create a new dataset,
just run the following command and the entire dataset will be built in the current directory
(you need to have Docker installed),
where 1000 is the number of repositories to fetch from GitHub
and XXX is
your personal access token:
$ docker run --detach --name=cam --rm --volume "$(pwd):/dataset" \
-e "TOKEN=XXX" -e "TOTAL=1000" -e "TARGET=/dataset" \
yegor256/cam:0.8.1 "make -e >/dataset/make.log 2>&1"This command will create a new Docker container, running in the background.
(run docker ps -a, in order to see it).
If you want to run docker interactively and see all the logs, you can just
disable detached mode
by removing the --detach option from the command.
The dataset will be created in the current directory (may take some time,
maybe a few days!), and a .zip archive will also be there. Docker container
will run in the background: you can safely close the console and come back when the
dataset is ready and the container is deleted.
If the script fails at some point, you can restart it again, without deleting previously created files. The process is incremental --- it will understand where it stopped before.
You can also run it without Docker:
$ make wipe
$ make TOTAL=100Should work, if you have all the dependencies installed, as suggested in the Dockerfile.
In order to analyze just a single repository, do this
(yegor256/tojos as an example):
$ make wipe
$ make REPO=yegor256/tojosYou may want to use this dataset as a basis, with an intend of adding your own metrics on top of it. It should be easy:
- Clone this repo into
cam/directory - Download ZIP archive
- Unpack it to the
cam/dataset/directory - Add a new script to the
cam/metrics/directory (useast.pyas an example) - Delete all other files except yours from the
cam/metrics/directory - Run
makein thecam/directory:sudo make install; make all
The make should understand that a new metric was added. It will apply this new metric
to all .java files, generate new .csv reports, aggregate them with existing
reports (in the cam/dataset/data/ directory),
and then the final .pdf report will also be updated.
Fork repository, make changes, send us a pull request.
We will review your changes and apply them to the master branch shortly,
provided they don't violate our quality standards. To avoid frustration,
before sending us your pull request please run full build:
$ sudo make install
$ make testThis should take a few minutes to complete, without errors.