Does nvme cli support IO commands with SGL?

[yalanliufio]

Hello

Does nvme cli support IO commands with SGL (Scatter Gather List)? If it is, how to send the commands? could you give an example for read command?

[Keith Busch]

The driver decides when to use sgl or prp. The decision is based on which descriptor list is smaller. The way to get sgl used is send something large-ish that's also physically contiguous. nvme-cli will prefer huge pages for >=512k transfers, so you'll get sgls for that size if you've enabled huge pages and your device supports that mode.

[yalanliufio]

Thanks for the information and it is very helpful. Another question is: If I enable huge pages, how to check IO command is using SGL or PRP? What tools can be used to display all 16 DWORDS of a command? Would like to have DWORD 0 and DATA PTR (DWORD 7, 7, 8, 9) values so I can make sure SGL is used in the command.

[Keith Busch]

No good options, really. The driver's existing trace event happens before the prp/sgl decision occurs, and the rest of the setup functions are inline, so can't easily kprobe them either. This is what I'm coming up with right now:

#!/usr/bin/bpftrace

#include <linux/blk-mq.h>
#include <linux/nvme.h>

struct nvme_iod {
        struct nvme_command     *pcmd;
        u64                     result;
        u8                      genctr;
        u8                      retries;
        u8                      flags;
        u16                     status;
#ifdef CONFIG_NVME_MULTIPATH
        unsigned long           start_time;
#endif
        void *ctrl;
        struct nvme_command cmd;
}

kprobe:nvme_unmap_data
{
        $i = (struct nvme_iod *)(arg1 + sizeof(struct request));
        printf("flags:%x\n", $i->cmd.common.flags);
}

Example output when I have hugepages enabled:

flags:40

Example output when I don't have huge pages enabled:

flags:0

The flags being the important part to distinguish SGL vs PRP usage.

[yalanliufio]

Good morning Keith,

Could you let me know how to print out this flags values? what trace or tools you used to print this value? Thank you!

[Keith Busch]

Could you let me know how to print out this flags values? what trace or tools you used to print this value? Thank you!

The script I provided is already an executable format. You just need bpftrace installed, and then run the script in a terminal while you run your nvme-cli command in another.

Most distro's have an appropriate package for bpftrace (check https://github.com/iovisor/bpftrace/blob/master/INSTALL.md for examples on installing from repository package managers), and your kernel needs kprobes enabled.