ilongshan / stanford_compression_library

The goal of the library is to help with research in the area of data compression. This is not meant to be fast or efficient implementation, but rather for educational purpose

Stanford Compression Library

The goal of the library is to help with research in the area of data compression. This is not meant to be fast or efficient implementation, but rather for educational purpose

This library is currently being used for the course EE274: Data Compression course, Fall 22 at Stanford University, to augment the lectures and for homeworks:

1. EE274 Lecture materials (slides etc)
2. EE274 Course notes (in progress)
3. The video recordings for lectures can be found as a Youtube Playlist

Compression algorithms

Here is a list of algorithms implemented.

Prefix-free codes

• Huffman codes
• Shannon codes
• Fano codes
• Shannon Fano Elias
• Golomb codes
• Universal integer coder
• Elias Delta code

Entropy coders

• rANS
• tANS
• Typical set coder
• Arithmetic coder
• Context-based Adaptive Arithmetic coder
• Range coder

Universal lossless compressors

• zlib (external)
• zstd (external)
• LZ77
• LZ77 (sliding window version)

NOTE -> the tests in each file should be helpful as a "usage" example of each of the compressors. More details are on the project wiki.

Getting started

• Create conda environment and install required packages:

  conda create --name myenv python=3.8
  conda activate myenv
  

• Clone the repo

  git clone https://github.com/kedartatwawadi/stanford_compression_library.git
  cd stanford_compression_library
  

• Install the scl package

  pip install -e . #install the package in a editable mode
  

• Run unit tests

  To run all tests:

  find scl -name "*.py" -exec py.test -s -v {} +
  

  To run a single test

  py.test -s -v scl/core/data_stream.py
  

Getting started with understanding the library

In-depth information about the library will be in the comments. Tutorials/articles etc will be posted on the wiki page: https://github.com/kedartatwawadi/stanford_compression_library/wiki/Introduction-to-the-Stanford-Compression-Library

How to submit code

Run a formatter before submitting PR

black <dir/file> --line-length 100

Note that the Github actions CI uses black, so the PR will fail if black is not used. (see .github/workflows/black.yml),

Contact

The best way to contact the maintainers is to file an issue with your question.