jNVMf is a Java library for remote access to NVMe storage. It implements client side of the NVMe over Fabrics (NVMf) specification. jNVMf has been tested to work with SPDK, Linux and Mellanox offloading targets. Futhermore it supports advanced NVMe/f features like fused operations, incapsule data and dataset management.
Building the source requires Apache Maven and a Java JDK with Java version 8 or higher. jNVMf depends on DiSNI while the java dependecy is automatically picked up from maven central you need to download and compile the native library of DiSNI. Check the DiSNI README for instructions on how to build the native library. To build jNVMf use:
mvn -DskipTests package
or mvn -DskipTests install
To run the unit tests make sure the LD_LIBRARY_PATH contains libdisni (see above) and execute:
mvn test -Prdma -DargLine="-DdestinationAddress=10.100.0.5 -DlocalAddress=10.100.0.2 -Dnqn=nqn.2017-06.io.crail:cnode4420 -Dport=4420"
where destinationAddress
is the target IP address, localAddress
is the local IP of the RDMA device, nqn
the NVMe qualified name of the controller you want to connect to and port
the port of the target.
jNVMf provides a small benchmark tool. Run:
java -cp target/jnvmf-1.0-jar-with-dependencies.jar:target/jnvmf-1.0-tests.jar com.ibm.jnvmf.benchmark.NvmfClientBenchmark --help
to shows the available arguments.
For example:
java -cp target/jnvmf-1.0-jar-with-dependencies.jar:target/jnvmf-1.0-tests.jar com.ibm.jnvmf.benchmark.NvmfClientBenchmark -a 10.100.0.22 -p 4420 -g 4096 -i 3 -m RANDOM -n 10 -nqn nqn.2016-06.io.spdk:cnode1 -o bench.csv -qd 1 -rw read -s 4096 -qs 64 -H -I
- executes a
-m RANDOM
-rw read
test - to a target at
-a 10.100.0.22
,-p 4420
with controllernqn nqn.2016-06.io.spdk:cnode1
- all accesses are aligned to
-g 4096
bytes - statistics are printed every
-i 3
seconds-n 10
times - output is written to
-o bench.csv
in csv format - queue depth
-qd 1
- transfer size
-s 4096
bytes - queue size
-qs 64
entries - histogram
-H
- RDMA inline data
-I
Sample usage of the jNVMf API to perform a read operation:
Nvme nvme = new Nvme();
/* target ip and port we want to connect to */
InetSocketAddress socketAddress = new InetSocketAddress("10.100.0.1", 4420);
/* controller NVMe qualified name */
NvmfTransportId tid = new NvmfTransportId(socketAddress,
new NvmeQualifiedName("nqn.2016-06.io.spdk:cnode1"));
/* connect to controller */
Controller controller = nvme.connect(tid);
/* enable controller */
controller.getControllerConfiguration().setEnable(true);
controller.syncConfiguration();
controller.waitUntilReady();
/* create a io queue pair with submission queue size 64 */
IoQueuePair queuePair = controller.createIoQueuePair(64);
/* allocate and register buffer to be used for transfer */
ByteBuffer buffer = ByteBuffer.allocateDirect(4096);
KeyedNativeBuffer registeredBuffer = queuePair.registerMemory(buffer);
/* create a new read command */
NvmReadCommand command = new NvmReadCommand(queuePair);
/* create a response */
Response<NvmResponseCapsule> response = command.newResponse();
/* set buffer */
NvmIOCommandCapsule commandCapsule = command.getCommandCapsule();
commandCapsule.setSglDescriptor(buffer);
/* set length, LBA and namespace identifier */
NvmIoCommandSqe sqe = commandCapsule.getSubmissionQueueEntry();
sqe.setStartingLba(0);
/* read one block*/
sqe.setNumberOfLogicalBlocks(1);
sqe.setNamespaceIdentifier(new NamespaceIdentifier(1));
volatile boolean done = false;
/* set callback */
command.setCallback(new OperationCallback() {
@Override
public void onStart() {
}
@Override
public void onComplete() {
System.out.println("command completed");
/* check response */
NvmCompletionQueueEntry cqe = response.getResponseCapsule().getCompletionQueueEntry();
StatusCode.Value statusCode = cqe.getStatusCode();
if (!statusCode.equals(GenericStatusCode.getInstance().SUCCESS)) {
System.out.println("Command not successful");
}
done = true;
}
@Override
public void onFailure(RdmaException e) {
System.out.println("There was an RDMA error when executing the command");
done = true;
}
});
command.execute(response);
/* wait until command is done */
while (!done);
PRs are always welcome. Please fork, and make necessary modifications, and let us know.
If you have questions or suggestions, feel free to post at:
https://groups.google.com/forum/#!forum/zrlio-users
or email: zrlio-users@googlegroups.com