Speedy37 / speedupdate-rs

Efficient and reliable update system for small to big set of files

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Update system

A very efficient, reliable update system with a static repository.

Efficiency is achieved by:

  • minimizing download size by only downloading required parts,
  • minimizing download request: the update path is often a single big chunk of bytes
  • on the fly decompression/patching
  • on the fly integrity checks
  • resumable update process
  • Unreal Engine 4 pak file patcher (no more multigigabytes diff file)
  • very fast and with good ratio lossless codecs (brotli, zstd, ue4pak)

Reliability is achieved by:

  • enforcing download integrity
  • enforcing file integrity
  • auto repair process if something is wrong
  • rust borrowing rules and typings (no unwrap, no unsafe outside of codecs, ...)

Easy to use

  • static repository with JSON metadata
  • repository management cli tool
  • update test cli tool
  • logging infrastructure (rust log)
  • C binding with update progress reporting

Dependencies

To achieve thoses goals this library depends on:

  • rust: a memory safe programming language
  • future-rs: zero-cost Futures in Rust
  • reqwest: an ergonomic, batteries-included HTTP Client for Rust,
  • native-tls: a wrapper over a platform's native TLS implementation

Supported codecs

  • ue4pak: Unreal Engine 4 pak file direct support for hyper efficient patch files
  • zstd: a good compressor with very fast decompression speed (~ 800MB/s)
  • brotli: a very good compressor with fast decompression speed (~ 300MB/s)
  • lzma: an awesome compressor with slow decompression speed (~ 50MB/s)
  • vcdiff-rs: for vcdiff decoding

Documentation

The documentation of repository format and some internal things is in the doc folder.

Terminology

  • version: a node in the transition graph
  • package: an edge between two versions in the transition graph
  • operation: a way to synchronize a file or directory for a given relative file path
  • repository: a collection of packages and versions forming a transition graph with a pointer to the current version
  • workspace: local files managed and updated by this system

Update logic

The update system only uses 2 threads:

  • one to manages download
  • one to apply changes to the workspace

The update logic is made of multiples stages

  • prepare: find the shortest path of packages to download
  • update: do the actual update
  • recover: if something did go wrong in the update pass, try to fix it

Repository

Here is an overview of repository files content:

Project layout

  • bin/repository: rust cli tool to manage repository (set current version, add version, ...)
  • bin/workspace: rust cli tool to manage workspace (check integrity, update, downgrade, ...)
  • bindings/c: C bindings to integrate workspace functionality to your program
  • src: update system core library

Future improvements

Split big files into smaller chunks so the repair process can only download corrupted chunks.

Add a 4th stage workaround: if something did go wrong again in the update pass, try to fix it by doing something else.

About

Efficient and reliable update system for small to big set of files

License:MIT License


Languages

Language:Rust 99.2%Language:C 0.8%