[Gimie] License detection

[Cyril Matthey-Doret]

Software licenses are a crucial part of code repositories, as they define whether and how the code can be reused.
Licenses are generally provided in the form of a text file, and sometimes as a header of source files. Automatically detecting the presence and type of an OSI approced license in repositories would be very useful.

Objective: Automatically identify and classify the license in a given repository.

Requirements

• Define where to look for a license (LICENSE file, COPYING ?, file headers, ...)
• Assess the usability and usefulness of different license matchers
• [ ] Identify schema.org mapping
• [ ] Write PoC / Example

[Robin Franken]

I have made some progress on this. I would love input on ideas with regards to matching the file containing the license.
I don't want to open each file - this would be too heavy given the runtime of the license matcher. Right now, I'm using a regular expression:

(licens.*)|(reuse)|(copy)

But I think more searchterms can be added to this, I'm also considering adding different languages for license.

With regard to the serialization of the resulting license triples - I am using schema.org/license for the predicate, and spdx ontology for the object. The subject is still a question: What should the object be, to which we attach all our properties? I was thinking something like: "some_repo_related_prefix:[insert_repos_name]". This instance can then be rdf:type'd as a schema.org/CodeRepository.

Check out my branch! (License_scanner)

[Cyril Matthey-Doret]

But I think more searchterms can be added to this, I'm also considering adding different languages for license.

I agree checking LICENS., COPY. and REUSE.* would already be a good start. We can always think about optimizing this later.

With regard to the serialization of the resulting license triples

It will be easier to maintain and test if we separate the code that gets the license (path -> URI) from the code that materializes the triple or serializes the RDF. It's probably easier if the URI is simply kept as an attribute of LicenseMetadata in this part of the code.

What should the object be, to which we attach all our properties?

For remote repositories, I guess it would make sense to use the remote URL ? For local repositories, indeed we likely need to create some prefix... I'm not sure about the requirements for this, but AFAIK it should :

• be globally unique
• yield the same URI if we run gimie twice on the same repo
• yield different URI if we run gimie twice on different repos with the same name

Maybe something along the lines local://repo-name/first-commit-hash could do the trick, but I think this would require some separate discussion.

[Robin Franken]

Just to come back to this point, I think something like this could work:

If it's an online repository, we can use the repository URL as a URI.
Advantages:

• The URI is immediately dereferencable to the actual thing we are trying to describe in triples. For physical objects like a building this is not possible, and you always need some kind of "representation" in the form of a URI, but for an online repository it's pretty elegant.
• No need for uuid's or other unique-making identifiers to be postfixed or prefixed.
• If we use our own URI like www.example.com/myrepository this should be hosted somewhere, and redirect to the online repo. Using the actual online repository URL circumvents this.

Disadvantages:

• I'm not sure how GitHub deals with persistence. If you change the name of your repo, does the old name become available again for others to use? And even if that's not the case, when GitHub/lab/whatever is down or the repo is removed or name is changed, the URI is no longer dereferencable... That would kind of defeat the purpose.

For a local repository, we could probably just do indeed a local://repo-name/ and hash combination, I like this idea, but would maybe advocate truncating such a hash, like 5 characters or something. This would still be plenty robust in our use case I think (collision factor for 6 character hash at 100 000 entries is already like 17%, so adding the reponame as a further "hash" would reduce that number even more. And that is under the assumption of 100k entries. our scenario is probably more likely to stay in the realm of 100's or at most 1000's of tools.
The shorter and more readable the URI, the easier it is to use in queries. This SHOULD not be an important factor in deciding a URI, but seeing as labels and definitions/comments are not always added, having a somewhat readable meaningful URI can be useful in small scale projects like this.

I'll do some experimentation with truncating a hash algorithm and adding it to the make-a-graph part of the script. This kind of URI generation should probably be a method for some class so it can be re-used everytime we want to make a triple.

[Cyril Matthey-Doret]

Thanks ! Sounds good, feel free to create a new issue to define a URI :)

If you change the name of your repo, does the old name become available again for others to use?

AFAIK, the old URL redirects to the new one as long as it is left unused. As names are scoped by user/organization, this is usually not a big problem.

This kind of URI generation should probably be a method for some class so it can be re-used everytime we want to make a triple.

Indeed, that would make sense