Eider is not suitable for production usage as is. Further development on this repo will be done only as needed for changes to Aeron Cookbook.
Annotation based flyweight generator built for fast iteration while developing Aeron Cookbook. Intended for messages over Aeron and Aeron IPC when all processes are built and deployed as a single unit. Only suitable for single threaded usage scenarios.
Given a specification object, Eider generates a zero-copy flyweight that can be used to read and write to a buffer with random access. The original specification object is not used at runtime. The generated flyweight has no runtime dependencies beyond Java and the target buffer implementation.
Initial implementation uses Agrona Direct Buffers for read and requires Mutable Direct Buffers for write support. Values are read from and written to the buffer directly - they do not hold any field values internally - and as a result, cannot be used or held like regular objects. Random access writes and reads are supported.
Current features:
- Generate zero-copy flyweights that support fixed length objects
- supports multiple underlying buffers including
UnsafeBuffer
,MutableDirectBuffer
andDirectBuffer
implementations. Object adjusts internally depending on the provided buffer implementation, making it simpler to work with in read only paths such as AeronSubscriptions
andEgressListeners
. - Type support is limited to:
- boolean
- short
- int
- long
- fixed length ASCII strings. Use
WithPadding
variants to write with padding (this is useful for reused flyweights). In all cases, trailing space chars get trimmed on read.
- supports multiple underlying buffers including
- Generates a helper to detect message types in a buffer
- see Aeron Cookbook for sample in use
- Sequence Generator
- Optional repositories
- Note! Repositories hold data within internal buffers and maps. This will cause allocation.
- repositories hold a maximum number of items, and cannot be resized at runtime
appendWithKey
appends an item to the end of the buffer, up to the pre-defined capacitygetByKey
,getByBufferIndex
,getByBufferOffset
,containsKey
andIterator<>
functionalitygetCrc32
useful to support cross process comparison of repository contents (e.g. in a Aeron Cluster determinism check)getOffsetByBufferIndex
andappendByCopyFromBuffer
which can be used for Aeron Cluster snapshot reading and writing.- optional indexed fields. Indexes support updates and transactions. Indexes update synchronously at the time a field write occurs.
- optional Unique indexes on fields
- optional transactional support.
- Optional transactional support on each flyweight. If this is enabled, the flyweight adds
beginTransaction
,commit
androllback
methods. Note, by default reads are dirty; the buffer is only rolled back to the state it was in whenbeginTransaction
was called ifrollback
was called.- Note: this will allocate a buffer of length equal to the flyweight buffer length internally.
- Composite reader/writer
- provides a single, keyed object which contains multiple Eider objects read/written into a single buffer
- Optional repository support
- Reasonable performance
- Performance varies on the object complexity and buffer implementation. JMH test included uses Agrona UnsafeBuffer.
- Complex transactional objects read and write around 1.2 million messages/second (a roundtrip of around 0.82μs)
- Simple non-transactional objects read and write at around 175 million messages/second (a roundtrip of around 5.7ns)
Features not planned for future releases:
- byte[], BigDecimal, char or other type support
- versioning or backwards/forwards compatibility
- support for anything but JVM
- thread safety
- schema validation
- migrations
- mulitple variable length fields
- repeating groups and sub-objects
- Nullable objects with customizable null representations
- Maven Central publishing
The following specification defines an object with an identifier, timestamp, enabled flag and a 9 character CUSIP string. The eiderId here is the message type identifier, and is written to the buffer at a fixed position regardless of message specification. If no predefined value is given, one is automatically assigned and made available as a static field on the generated object. The eider Id can be used to simplify building demuxers.
@EiderSpec(eiderId = 42, name="Sample")
public class SampleSpec
{
private int id;
private long timestamp;
private boolean enabled;
@EiderAttribute(maxLength = 9)
private String cusip;
}
Using the generated object is as simple as:
final Sample read = new Sample();
final Sample write = new Sample();
final EpochClock clock = new SystemEpochClock();
final long now = clock.time();
final int initialOffset = 0;
ExpandableArrayBuffer buffer = new ExpandableArrayBuffer(Sample.BUFFER_LENGTH);
write.setUnderlyingBuffer(buffer, initialOffset);
read.setUnderlyingBuffer(buffer, initialOffset);
write.writeHeader();
write.writeCusip("037833100");
write.writeEnabled(true);
write.writeId(213);
write.writeTimestamp(now);
Assertions.assertTrue(read.validateHeader());
Assertions.assertEquals("037833100", read.readCusip());
Assertions.assertTrue(read.readEnabled());
Assertions.assertEquals(now, read.readTimestamp());
Assertions.assertEquals(213, read.readId());
Flyweights may optionally be transactional. Note that this will allocate a buffer of length Flyweight.BUFFER_LENGTH
when the flyweight is created. This internal transactional buffer will be reused as the flyweight moves around a buffer.
@EiderSpec(transactional = true)
public class SampleTxn
{
@EiderAttribute(key = true)
private int id;
@EiderAttribute(maxLength = 9)
private String cusip;
}
Setting transactional = true
on the @EiderSpec
attribute enables transaction support. Note that reads before commit/rollback are dirty.
final String CUSIP = "037833100";
final SampleTxn read = new SampleTxn();
final SampleTxn write = new SampleTxn();
ExpandableArrayBuffer buffer = new ExpandableArrayBuffer(SampleTxn.BUFFER_LENGTH);
read.setUnderlyingBuffer(buffer, 0);
write.setUnderlyingBuffer(buffer, 0);
write.writeCusip(CUSIP);
Assertions.assertEquals(CUSIP, read.readCusip());
write.beginTransaction();
write.writeCusip("zzzzzzzzz");
//by default dirty reads are supported
Assertions.assertEquals("zzzzzzzzz", read.readCusip());
write.rollback();
Assertions.assertEquals(CUSIP, read.readCusip());
Warnings:
- any call to
setUnderlyingBuffer
will invalidate the transaction buffer. You cannot callbeginTransaction
followed bysetUnderlyingBuffer
followed byrollback
for example - you cannot call
rollback
until you have calledbeginTransaction
- you cannot call
commit
after callingrollback
@EiderSpec(name = "SequenceGenerator")
public class SequenceGeneratorSpec
{
@EiderAttribute(sequence = true)
private int tradeId;
}
One spec can hold more than one sequence. Initialize and get new sequences by:
final SequenceGenerator generator = SequenceGenerator.INSTANCE();
//initialize to 1
generator.initializeTradeId(1);
//get next tradeId
int nextTrade = generator.nextTradeIdSequence();
@EiderRepository(transactional = true)
@EiderSpec
public class SampleSpec
{
@EiderAttribute(key = true)
private int id;
@EiderAttribute(maxLength = 9)
private String cusip;
}
To enable repository generation, just add a @EiderRepository
attribute to a @EiderSpec
object. You will have to set exactly one field to be a key using @EiderAttribute(key = true)
.
The repository generated will allow you to work with fixed size repositories that cannot hold more items than specified.
final SampleSpecRepository repository = SampleSpecRepository.createWithCapacity(2);
SampleSpec flyweight = repository.createWithKey(1);
flyweight.writeCusip("CUSIP0001");
long crc32Initial = repository.getCrc32();
repository.beginTransaction();
flyweight = repository.createWithKey(2);
flyweight.writeCusip("CUSIP0002");
long crc32Final = repository.getCrc32();
flyweight = repository.getByKey(1);
Assertions.assertEquals("CUSIP0001", flyweight.readCusip());
flyweight = repository.getByKey(2);
Assertions.assertEquals("CUSIP0002", flyweight.readCusip());
Assertions.assertNotEquals(crc32Initial, crc32Final);
repository.rollback();
Assertions.assertNull(repository.getByKey(2));
Assertions.assertEquals(crc32Initial, repository.getCrc32());
Repositories also hold an iterator, which allows you to iterate through all elements in the buffer.
Warnings:
- You will not be able to alter the key of a flyweight returned by the repository using
createWithKey
orgetByKey
.
@EiderRepository(transactional = true)
@EiderSpec
public class SampleSpec
{
@EiderAttribute(key = true)
private int id;
@EiderAttribute(maxLength = 9, indexed = true)
private String cusip;
}
Indexes are added when an @EiderRepository
object has @EiderAttribute
with indexed = true
.
final SampleSpecRepository repository = SampleSpecRepository.createWithCapacity(2);
SampleSpec flyweight = repository.createWithKey(1);
flyweight.writeCusip("CUSIP0001");
flyweight = repository.createWithKey(2);
flyweight.writeCusip("CUSIP0002");
List<Integer> allCusip1 = repository.getAllWithIndexCusipValue("CUSIP0001");
flyweight = repository.getByOffset(allCusip1.get(0));
Assertions.assertEquals(1, allCusip1.size());
Assertions.assertEquals("CUSIP0001", flyweight.readCusip());
flyweight.writeCusip("CUSIP0003");
List<Integer> allCusip1NowEmpty = repository.getAllWithIndexCusipValue("CUSIP0001");
List<Integer> allCusip3 = repository.getAllWithIndexCusipValue("CUSIP0003");
Assertions.assertEquals(0, allCusip1NowEmpty.size());
Assertions.assertEquals(1, allCusip3.size());
final EiderObjectRepository sourceRepo = EiderObjectRepository.createWithCapacity(3);
final EiderObjectRepository destRepo = EiderObjectRepository.createWithCapacity(3);
//...write some data to the sourceRepo...
for (int i = 0; i < sourceRepo.getCurrentCount(); i++)
{
int offset = sourceRepo.getOffsetByBufferIndex(i);
destRepo.appendByCopyFromBuffer(sourceRepo.getUnderlyingBuffer(), offset);
}
assertEquals(sourceRepo.getCrc32(), destRepo.getCrc32());
When writing to an Aeron Cluster snapshot, use the getOffsetByBufferIndex
to step through the buffer. When reading from the Aeron Cluster snapshot image, make use of appendByCopyFromBuffer
to append an item into the repository by copying from the provided image.
Composite objects allow you to construct a buffer containing 2..n EiderSpec
objects.
Given the following objects:
@EiderSpec(name = "OrderStatus")
public class OrderStatusSpec
{
private boolean filled;
private long acceptedTimestamp;
private long filledTimestamp;
}
@EiderSpec(name = "Order")
public class OrderSpec
{
@EiderAttribute(maxLength = 14)
private String clOrdId;
@EiderAttribute(maxLength = 1)
private String side;
private long price;
private long quantity;
@EiderAttribute(maxLength = 9)
private String instrument;
}
We can construct a composite by:
@EiderComposite(name = "OrderBookEntry", eiderId = 688)
@EiderRepository
public class OrderBookEntrySpec
{
@EiderAttribute(key = true)
long id;
OrderSpec order;
OrderStatusSpec status;
}
This will produce a composite object that can use an internal or external buffer. For example:
ExpandableDirectByteBuffer buffer = new ExpandableDirectByteBuffer(OrderBookEntry.BUFFER_LENGTH);
OrderBookEntry entry = new OrderBookEntry(buffer, 0);
entry.getOrder().writeClOrdId("Order123");
entry.getOrder().writePrice(12300L);
entry.getOrder().writeQuantity(1_000_000L);
entry.getStatus().writeAcceptedTimestamp(500L);
entry.getStatus().writeFilledTimestamp(800L);
Assertions.assertEquals("Order123", entry.getOrder().readClOrdId());
Assertions.assertEquals(12300L, entry.getOrder().readPrice());
Assertions.assertEquals(1_000_000L, entry.getOrder().readQuantity());
Assertions.assertEquals(500L, entry.getStatus().readAcceptedTimestamp());
Assertions.assertEquals(800L, entry.getStatus().readFilledTimestamp());
short eiderSpecId = EiderHelper.getEiderSpecId(0, buffer);
Assertions.assertEquals(688, eiderSpecId);
Composite Repositories are supported as well. They behave much the same as object repositories.
OrderBookEntryRepository repository = OrderBookEntryRepository.createWithCapacity(10);
OrderBookEntry entry = repository.appendWithKey(1);
...
boolean containsIt = repository.containsKey(1);
...
OrderBookEntry read = repository.getByKey(1);
...
while (repository.allItems().hasNext())
{
OrderBookEntry item = repository.allItems().next();
...
}
It's primarily being used for another project with messages sent over Aeron and Aeron Cluster byte buffers, plus data held within a Replicated State Machine running within Aeron Cluster.
- Java 15
- Gradle 6.7.1
- Agrona 1.9.0