chrislo / Similarity

A document similarity calculation library for Ruby. OFFICIAL REPO: https://github.com/bbcrd/Similarity

Similarity

Overview

A Ruby library for calculating the similarity between pieces of text using a Term Frequency-Inverse Document Frequency method.

A bag of words model is used. Terms in the source documents are downcased and punctuation is removed, but stemming is not currently implemented.

This library was written to facilitate the creation of diagrams talked about by Jonathan Stray in his full-text visualization of the Iraq War Logs post. An example of how to generate a Gephi compatible file including labelling of nodes with key words is included in the examples directory.

The library is written in pure-ruby for portability, and has not been rigorously optimised. I’d be interested in any comments, suggestions and patches for how to improve performance.

Usage

require 'similarity'

:

corpus = Corpus.new

:

doc1 = Document.new({:content=>"Broad powers for hacking inquiry"})
doc2 = Document.new({:content=>"UK unemployment level falls again"})
doc3 = Document.new({:content=>"NI riots leads to 26 arrests"})

:

corpus << doc1
corpus << doc2
corpus << doc3

:

similarity = corpus.similarity(doc1, doc2)

For more examples, see the examples directory.

Todo

• Performance improvements
  • Switch to storing document vector spaces in sparse form, using linalg?
• (Optional) stemming of source terms

Contributing

• Fork the project
• Send a pull request
• Don’t touch the .gemspec, I’ll do that when I release a new version

Author

Chris Lowis - BBC R&D