xxlabaza / log-file

A Java library for working with append-only files

Overview log-file

A Java library for working with append-only files (also known as the log files). The main methods of such files are:

• append - for appending arbitary data;
• load - for fully file read from the begining till the end.

Usage example

import java.nio.file.Paths;

import com.xxlabaza.utils.log.file.Config;
import com.xxlabaza.utils.log.file.LogFile;

import io.appulse.utils.Bytes;
import io.appulse.utils.HexUtil;


// let's instantiate a log file:
Config config = Config.builder()
    .path(Paths.get("./my.log"))
    .blockBufferSizeBytes(32)
    .forceFlush(false)
    .build();

LogFile logFile = new LogFile(config);


// than, write some data to log file
Bytes record = Bytes.resizableArray()
    .write4B(42);

logFile.append(record);


// now, read all records from the file and print them to STDOUT
logFile.load((buffer, position) -> {
  String dump = HexUtil.prettyHexDump(buffer);
  System.out.println(dump);
  return true; // true - continue reading, false - stop reading
});


// flush and close the file
logFile.close();

Under the hood

A log file has the next structure:

log file structure
file header		blocks
version	block size	block 0	...	block n
1 byte	4 bytes	blocks count * block size

File header

File header's description:

version

The file's format version, which tells the features set is used in the file.

block size

The size of a block buffer, in bytes, which is used in the current file.

Block

Each block consist of the records set:

block
record 1	...	record n

The records have the following structure:

record
checksum	type	body length	body
4 bytes	1 byte	2 bytes	body length

checksum

The checksum value, which is calculated from type, body length and body values.

type

One of the record's types:

• FULL - the data is presented entirely in this record;
• FIRST - it is the first chunk of data;
• MIDDLE - one of the middle parts of data;
• LAST - the last piece of data.

body length

The length of the next body section, in bytes.

body

the record's payload.