Getting "error reading cq (-259)" with fi_write()
sahaludheen opened this issue · comments
When trying to write image buffer with size over 256 bytes to server from client using fi_write() with verbs provider, getting below error.
"error reading cq (-259)".
Anything below 256 bytes is accepted by the api.
corresponding screenshot is attached
error -259 means a completion with error is present in the CQ. You can use fi_cq_readerr() to retrieve the error entry and check for the fields err and prov_errno to help understanding what went wrong.
error -259 means a completion with error is present in the CQ. You can use
fi_cq_readerr()to retrieve the error entry and check for the fieldserrandprov_errnoto help understanding what went wrong.
Tried with
fi_cq_readerr(cq, &comp, 0);
printf("Completion with error: %s (err %d prov_errno %d).\n",fi_strerror(comp.err), comp.err,comp.prov_errno);
The output gives
Completion with error: Input/output error (err 5 prov_errno -1073741536)
Any information on the provider being used and the platform you are using?
ok, I noticed you were using the verbs provider. Is it on windows machine?
@dshinaberry Could you take a look?
256 may be the cut-off for inline data. I would check that the source data for the write has been registered and a valid memory descriptor is being passed in.
In linux also same issue is present. But different err and prov_no
Completion with error: Operation canceled (err 125 prov_errno 5)
Hi @j-xiong, Could you please help us figure this out, currently we are blocked from creating data transfer applications from windows to Linux using verbs. We need at least 3MB to transfer at a time. It will be very helpful if you can provide a solution.
Thanks
@JibinNajeeb @sahaludheen As @shefty mentioned, have you checked that the source buffer is registered and a valid mr_desc is passed to fi_write()?
I think @shefty is on the right track.
Completion with error: Input/output error (err 5 prov_errno -1073741536)
The prov_error is 0xC0000120, which is ND_CANCELED. IIRC this is the error code you get when you exceed the maximum inline data amount.
below are the sample codes we are using for testing. We couldn't still figure out how to transfer 2MB of data using RMA.
Is there something we are missing, we are using mellanox cards
#define BUF_SIZE 5242880 //5MB
static int run(struct perftest_parameters user_param) {
int ret;
float intervel = 16.66;//in msec
int iter = user_param.duration / (float)intervel;
//printf("no:of iteration:%d\n", iter);
if (dst_addr) {
//****
****
****//
//buf_name contains 2mb image data
ret = fi_write(ep, buf_name, len, local_desc_write, fi_addr, remote_addr_write,
remote_key_write, NULL); // error happening
}
else {
read_images(iter);
}
//sync_progress();
return 0;
}
void ft_fill_mr_attr(struct iovec *iov, int iov_count, uint64_t access,
uint64_t key, struct fi_mr_attr *attr)
{
attr->mr_iov = iov;
attr->iov_count = iov_count;
attr->access = access;
attr->offset = 0;
attr->requested_key = key;
attr->context = NULL;
}
static void initialize(void) {
fi_getinfo(FI_VERSION(1, 9), NULL, NULL, 0, hints, &info);
fi_fabric(info->fabric_attr, &fabric, NULL);
fi_domain(fabric, info, &domain, NULL);
fi_endpoint(domain, info, &ep, NULL);
cq_attr.size = 128;
cq_attr.format = FI_CQ_FORMAT_MSG;
fi_cq_open(domain, &cq_attr, &cq, NULL);
fi_ep_bind(ep, &cq->fid, FI_SEND | FI_RECV);
av_attr.type = FI_AV_TABLE;
av_attr.count = 1;
fi_av_open(domain, &av_attr, &av, NULL);
fi_ep_bind(ep, &av->fid, 0);
fi_enable(ep);
struct fi_mr_attr attr = { 0 };
struct iovec iov = { 0 };
int ret;
uint64_t flags;
uint64_t access = FI_WRITE | FI_REMOTE_WRITE;
iov.iov_base = buf_write;
iov.iov_len = BUF_SIZE;
ft_fill_mr_attr(&iov, 1, access, MR_KEY, &attr);
fi_mr_regattr(domain, &attr, 0, &mr_write);
local_desc_write = fi_mr_desc(mr_write);
local_key_write = fi_mr_key(mr_write);
local_addr_write = info->domain_attr->mr_mode & FI_MR_VIRT_ADDR ?
(uintptr_t)buf_write : 0;
exchange_info();
}
int main(int argc, char **argv) {
buf_write = (char*)malloc(sizeof(char)*BUF_SIZE);
hints = fi_allocinfo();
if (!hints)
return EXIT_FAILURE;
dst_addr = argv[optind];
hints->ep_attr->type = FI_EP_RDM;
hints->caps = FI_RMA;
hints->tx_attr->op_flags = FI_DELIVERY_COMPLETE;
hints->domain_attr->mr_mode = FI_MR_ENDPOINT | FI_MR_LOCAL |
FI_MR_PROV_KEY | FI_MR_ALLOCATED | FI_MR_VIRT_ADDR;
hints->fabric_attr->prov_name = "verbs";
init_oob();
initialize();
}
@JibinNajeeb In your code, buf_name is the local buffer for fi_write. You need to register buf_name and get its mr_desc instead of using local_desc_write which is for buf_write.
@j-xiong Thanks for pointing it out. Now the application is working for windows(sender) to linux(receiver). But getting below error when trying from linux(sender) to windows(receiver).
(err 5 prov_errno 10) is EIO, IBV_WC_REM_ACCESS_ERR. This makes me think that something still isn't correct with the memory region registration.
Have you tried running the pingpong utility that comes with libfabric in both directions (windows as client/linux as server and linux as client/windows as server) to make sure that there aren't any configuration or other problems? If you can do this, then we can be pretty confident that the issue has to do with something in your specific code.
If you can use pingpong without problem, then the next thing I'd do is turn the logging up and see if there are any issues that might shed light on the problem. I'd be interested in the log from the windows side in particular to see if it might be invoking unsupported functionality. The Windows port of the verbs API is not fully complete. There are some modes that aren't supported.
@JibinNajeeb In your code, the return value of libfabric API calls were never checked. It is possible the error happened earlier, e.g., when fi_mr_regattr was called. If the memory registration failed on Windows, the remote write key used for fi_write (on Linux) would be invalid, resulting in remote access error.
It is good practice to always check the return code for non-void functions so that error can be caught as close to the cause as possible.
This issue is stale because it has been open 360 days with no activity. Remove stale label or comment, otherwise it will be closed in 7 days.