udmabuf of_reserved_mem_device failed
rockybulwinkle opened this issue · comments
Hello,
I'm trying to set up a udmabuf on a memory device inside the PL side of a ZynqMP at address 0x4_0000_0000. I'm using the Petalinux kernel version 4.19.0-xilinx-v2019.1 with a Debian Stretch rootfs. I'm running into an issue though, of_reserved_mem_device_init is failing, returning -22. Here is the section of my device tree for the reserved memory region and udmabuf:
amba_pl@0 {
#address-cells = <0x2>;
#size-cells = <0x2>;
compatible = "simple-bus";
ranges;
reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;
image_buf0: image_buf@0 {
compatible = "shared-dma-pool";
reusable;
reg = <0x4 0x0 0x0 0x400000>;
label = "image_buf0";
};
};
udmabuf@0 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf0";
size = <0x0 0x400000>;
memory-region = <&image_buf0>;
};
};
Here's what happens when I try to load udmabuf:
rocky@zynq:~/repos/udmabuf$ sudo insmod udmabuf.ko
[ 1076.852053] udmabuf amba_pl@0:udmabuf@0: of_reserved_mem_device_init failed. return=-22
[ 1076.860210] udmabuf amba_pl@0:udmabuf@0: driver installed.
[ 1076.865730] udmabuf: probe of amba_pl@0:udmabuf@0 failed with error -22
I've attached the full kernel log as well. Can you help me find where it is going wrong?
Thank you for the issue
I looked at dmesg.txt but probably reserved-memory is not assigned as CMA.
If reserved-memory is allocated as CMA, you should get a message like this:
[ 0.000000] Reserved memory: created CMA memory pool at 0x30c00000, size 144 MiB
[ 0.000000] OF: reserved mem: initialized node image_buf_1@30c00000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created CMA memory pool at 0x39c00000, size 100 MiB
[ 0.000000] OF: reserved mem: initialized node image_buf_0@39C00000, compatible id shared-dma-pool
[ 0.000000] cma: Reserved 16 MiB at 0x2f000000
It seems that of_reserved_mem_device_init is failing because reserved-memory is not allocated as a CMA.
Is it correct that the reserved-memory node is under the amba_pl @ 0 node?
I posted there my working device tree.
You can look at that.. I was struggling with similar problem. Check first your compatibility nose if you are using udmabuf or the other one u-dma-buf.
If I remember correctly I had the reserved memory node outside of the amba section, only the udma bug inside
Thank you for the issue
I looked at dmesg.txt but probably reserved-memory is not assigned as CMA.
If reserved-memory is allocated as CMA, you should get a message like this:[ 0.000000] Reserved memory: created CMA memory pool at 0x30c00000, size 144 MiB [ 0.000000] OF: reserved mem: initialized node image_buf_1@30c00000, compatible id shared-dma-pool [ 0.000000] Reserved memory: created CMA memory pool at 0x39c00000, size 100 MiB [ 0.000000] OF: reserved mem: initialized node image_buf_0@39C00000, compatible id shared-dma-pool [ 0.000000] cma: Reserved 16 MiB at 0x2f000000It seems that of_reserved_mem_device_init is failing because reserved-memory is not allocated as a CMA.
Is it correct that the reserved-memory node is under the amba_pl @ 0 node?
Thank you for your response! Yes, I put my reserved-memory node under amba_pl@0
I posted there my working device tree.
You can look at that.. I was struggling with similar problem. Check first your compatibility nose if you are using udmabuf or the other one u-dma-buf.
If I remember correctly I had the reserved memory node outside of the amba section, only the udma bug inside
Thanks, I'll give that a try.
I posted there my working device tree.
You can look at that.. I was struggling with similar problem. Check first your compatibility nose if you are using udmabuf or the other one u-dma-buf.
If I remember correctly I had the reserved memory node outside of the amba section, only the udma bug inside
I checked, my kernel is 4.19 and has not backported any udmabuf driver from kernel 5.
I read this article by xilinx: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/18841683/Linux+Reserved+Memory
If I mark my memory as "reusable" the kernel fails to boot. That is, it does not print anything to the serial port after u-boot hands control off to the kernel. But if I mark it as "no-remap", it continues to boot and I see the message in the kernel log:
[ 0.000000] Reserved memory: created DMA memory pool at 0x0000000400000000, size 64 MiB
However when I try to load udmabuf, I see the follow errors:
[ 321.403398] udmabuf: loading out-of-tree module taints kernel.
[ 321.446095] udmabuf udmabuf@0: assigned reserved memory node image_buf@0
[ 321.452817] dma_alloc_coherent() failed. return(0)
[ 321.457607] udmabuf udmabuf@0: driver setup failed. return=-12
[ 321.463583] udmabuf udmabuf@0: driver installed.
[ 321.468230] udmabuf: probe of udmabuf@0 failed with error -12
I believe -12 is ENOMEM. I should note that I had increased both the udmabuf and reserved memory device tree nodes from 4MiB to 64MiB, though I wouldn't think that would cause it to run out of memory.
udmabuf uses dma_alloc_coherent () to allocate memory.
And dma_alloc_coherent () allocates memory from the CMA.
However, when the property of reserved-memory is "no-map", the area reserved by reserved-memory is not treated as CMA.
So it's not surprising that udmabuf fails.
Setting the reserved-memory property to "reusable" will cause Linux to fail to boot, but that problem must be resolved first.
On another note, what are you trying to do?
The area where the address starts from 0x4_0000_0000 is assigned to FPD-PL (HPM0). Do you want to use BRAM on PL as udmabuf?
Thank you for your reply.
We are working on a system that is going to use the PL to connect the PS to a large memory device and make it available over HPM0. It can only support 128 or 256 bit access sizes and does not support hardware coherency. Setting this memory to be cacheable should let it do those large access sizes for its cacheline fills. It seemed like your module would allow us to manage the coherency in software easily. Configuring these low level memory settings is not something I'm experienced in which is why I'm trying to use this module.
As an intermediate step in development, I have connected the PL's DDR4 to HPM0 using MIG in Vivado. I am able to read and write to the memory by mmaping /dev/mem, but it's being treated as device memory and is not being cached.
Thank you for your reply.
Regrettably, udmabuf wasn't supposed to do that.
udmabuf expects to reserve a memory area that Linux manages as system memory.
By the way, does your system manage the PL side memory area as Linux memory?
For example, does the memory node of the device tree look like this:
memory@0 {
device_type = "memory";
reg = <0x0 0x00000000 0x0 0x80000000
0xA 0x00000000 0x0 0x00400000>;
};
reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;
image_buf0: image_buf@0 {
compatible = "shared-dma-pool";
reusable;
reg = <0xA 0x00000000 0x0 0x00400000>;
label = "image_buf0";
};
};
Note: The device tree above has not been confirmed to work.
Thank you for your reply.
Regrettably, udmabuf wasn't supposed to do that.
udmabuf expects to reserve a memory area that Linux manages as system memory.By the way, does your system manage the PL side memory area as Linux memory?
For example, does the memory node of the device tree look like this:memory@0 { device_type = "memory"; reg = <0x0 0x00000000 0x0 0x80000000 0xA 0x00000000 0x0 0x00400000>; }; reserved-memory { #address-cells = <2>; #size-cells = <2>; ranges; image_buf0: image_buf@0 { compatible = "shared-dma-pool"; reusable; reg = <0xA 0x00000000 0x0 0x00400000>; label = "image_buf0"; }; };Note: The device tree above has not been confirmed to work.
We would ultimately like to be able to use the memory as system memory.
I see in your example device tree it is at address 0xA_0000_0000, however I believe HPM0 is mapped to 0x4_0000_0000 (size 4GB) and 0x10_0000_0000 (size 224GB)
I have tried adding this memory to the memory node before but Linux does not seem to acknowledge its existence. See below for my /memory node in the device tree. I see no message about memory at 0x4_0000_0000 in the boot logs and don't see the reg for it in the /proc/device-tree/memory/reg, just the other two memory ranges that were in the stock DTS. However, I do see u-boot report the hash of my device tree, so the correct one is definitely getting loaded.
memory {
device_type = "memory";
reg = <0x0 0x0 0x0 0x7ff00000
0x4 0x0 0x0 0x20000000
0x8 0x0 0x0 0x80000000>;
};
Then I tried adding in the reserved-memory node, shown below. With this the kernel failed to boot at all.
memory {
device_type = "memory";
reg = <0x0 0x0 0x0 0x7ff00000
0x4 0x0 0x0 0x20000000
0x8 0x0 0x0 0x80000000>;
};
reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;
image_buf0: image_buf@0 {
compatible = "shared-dma-pool";
reusable;
reg = <0x4 0x00000000 0x0 0x00400000>;
label = "image_buf0";
};
};
In the previous experiments, I had the memory mapped to 0x4_0000_0000 in my hardware design. In the next build, I'm going to move it to 0x10_0000_0000 and add an ILA to HPM0 to see if it is attempting any access to the memory when the reserved-memory is present.
Thanks again!
It's been a while since I last commented. We've made some progress. Our memory at HPM0 is now registered as system memory and I am seeing it successfully allocate the memory during boot:
[ 0.000000] Reserved memory: created CMA memory pool at 0x0000000400000000, size 256 MiB
[ 0.000000] OF: reserved mem: initialized node image_buf@0, compatible id shared-dma-pool
[ 0.000000] cma: Reserved 256 MiB at 0x0000000010000000
Now, when I insmod u-dma-buf, I don't see it create any entries. Here is the relevant part of the device tree. These are all in the root of the tree:
memory {
device_type = "memory";
reg = <0x0 0x0 0x0 0x20000000 0x4 0x00000000 0x0 0x20000000>;
};
reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;
image_buf0: image_buf@0 {
compatible = "shared-dma-pool";
reusable;
reg = <0x4 0x00000000 0x0 0x10000000>;
label = "image_buf0";
};
};
udmabuf@0 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf0";
size = <0x0 0x400000>;
memory-region = <&image_buf0>;
};
Here I want to say something about the memory ranges there. Instead of going through the PL MIG/DDR4, we are now looping from HPM0 to HP1 (offsetting by 512MB) on the PL. So HPM0 is now a mirror of the PS's DDR4 offset by 512MB. This is why we set the memory size to only 0x2000_0000 in the device tree. I don't think this would affect how u-dma-buf operates though.
Are you using "udmabuf" or "u-dma-buf"?
See Readme.md for the difference between "udmabuf" and "u-dma-buf".
If you are using "u-dma-buf" then you must specify "ikwzm, u-dma-buf" for compatible property.
Thank you, I was using the wrong one. After setting it to u-dma-buf it recognizes the device tree node. Now I'm running into an issue where it is not allocating at the correct address.
I have changed my device tree slightly since the last post because it was not recognizing my 64 bit size correctly (it said invalid size 0). Now it's just "size = <0x400000>;"
root@ps_loopback:~/repos/udmabuf# insmod u-dma-buf.ko
[ 751.388072] u-dma-buf udmabuf@0: assigned reserved memory node image_buf@0
[ 751.397289] u-dma-buf udmabuf0: driver version = 3.0.1
[ 751.402436] u-dma-buf udmabuf0: major number = 241
[ 751.407399] u-dma-buf udmabuf0: minor number = 0
[ 751.412187] u-dma-buf udmabuf0: phys address = 0x0000000001400000
[ 751.418459] u-dma-buf udmabuf0: buffer size = 4194304
[ 751.423766] u-dma-buf udmabuf0: dma device = udmabuf@0
[ 751.429252] u-dma-buf udmabuf0: dma coherent = 0
[ 751.434037] u-dma-buf udmabuf@0: driver installed.
...
root@ps_loopback:/proc/device-tree/reserved-memory/image_buf@0# hexdump reg
0000000 0000 0400 0000 0000 0000 0000 0010 0000
0000010
Thanks again for the help.
I see in your documentation for setting a memory-region, it uses cell size 1. Does it support cell size 2 for 64 bit addressing?
Currently, size, offset, etc. are u32(32bit unsigned).
Please wait a bit longer for u64 support.
Currently, size, offset, etc. are u32(32bit unsigned).
Please wait a bit longer for u64 support.
Does this mean no 64 bit addresses as well, or just for sizing?
The address depends on the architecture. For example, 64bit for arm64.
The size is internally architecture like address, but the device tree size property is 32 bit.
u-dma-buf v3.1.0 allows 64 bit value to be specified for the size property.
If you want to specify a buffer size of 4GiB or more, specify a 64bit value as follows.
A 64-bit value is expressed by arranging two in the order of upper 32 bits and lower 32 bits.
udmabuf@0x00 {
compatible = "ikwzm,u-dma-buf";
size = <0x01 0x00000000>; // size = 0x1_0000_0000
};
The address depends on the architecture. For example, 64bit for arm64.
The size is internally architecture like address, but the device tree size property is 32 bit.
Ok, that's what I thought...
I've updated to 3.1.0. It doesn't seem to be pointing at the right address. Here's my current device tree:
reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;
image_buf0: image_buf@400000000 {
compatible = "shared-dma-pool";
reusable;
reg = <0x4 0x00000000 0x0 0x10000000>;
label = "image_buf0";
};
};
udmabuf@400000000 {
compatible = "ikwzm,u-dma-buf";
device-name = "udmabuf0";
size = <0x400000>;
memory-region = <&image_buf0>;
};
Head of my dmesg (CMA reservations):
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
[ 0.000000] Linux version 4.19.0-xilinx-v2019.2 (rocky@saturn) (gcc version 8.2.0 (GCC)) #8 SMP Thu Jul 9 10:10:15 CDT 2020
[ 0.000000] Machine model: ZynqMP ZCU102 Rev1.0
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: UEFI not found.
[ 0.000000] Reserved memory: created CMA memory pool at 0x0000000400000000, size 256 MiB
[ 0.000000] OF: reserved mem: initialized node image_buf@400000000, compatible id shared-dma-pool
[ 0.000000] cma: Reserved 256 MiB at 0x0000000010000000
[ 0.000000] On node 0 totalpages: 262144
[ 0.000000] DMA32 zone: 1792 pages used for memmap
And messages when I load the module:
root@ps_loopback:~/repos/udmabuf# insmod u-dma-buf.ko
[ 74.324034] u_dma_buf: loading out-of-tree module taints kernel.
[ 74.332094] u-dma-buf udmabuf@400000000: assigned reserved memory node image_buf@400000000
[ 74.342622] u-dma-buf udmabuf0: driver version = 3.1.0
[ 74.347782] u-dma-buf udmabuf0: major number = 242
[ 74.352768] u-dma-buf udmabuf0: minor number = 0
[ 74.357560] u-dma-buf udmabuf0: phys address = 0x0000000001400000
[ 74.363828] u-dma-buf udmabuf0: buffer size = 4194304
[ 74.369142] u-dma-buf udmabuf0: dma device = udmabuf@400000000
[ 74.375316] u-dma-buf udmabuf0: dma coherent = 0
[ 74.380111] u-dma-buf udmabuf@400000000: driver installed.
u-dma-buf still uses the wrong address when I set the CMA region to just be at 0x4_0000_0000.
u-dma-buf v3.2.0-rc.3 has been experimentally released.
Please try specifying dma-mask=<64> or dma-mask=<40> in the device tree.
udmabuf@400000000 {
compatible = "ikwzm,u-dma-buf";
device-name = "udmabuf0";
size = <0x400000>;
dma-mask = <64>;
memory-region = <&image_buf0>;
};
If you do not specify the dma-mask, the dma area is limited to the 32-bit address.
Therefore, the buffer may not be allocated from the memory area specified by memory-region.
This is my guess. I don't have your environment, so I can't confirm.
If it doesn’t bother you, please try.
u-dma-buf v3.2.0-rc.3 has been experimentally released.
Please try specifying dma-mask=<64> or dma-mask=<40> in the device tree.
udmabuf@400000000 { compatible = "ikwzm,u-dma-buf"; device-name = "udmabuf0"; size = <0x400000>; dma-mask = <64>; memory-region = <&image_buf0>; };If you do not specify the dma-mask, the dma area is limited to the 32-bit address.
Therefore, the buffer may not be allocated from the memory area specified by memory-region.
This is my guess. I don't have your environment, so I can't confirm.
If it doesn’t bother you, please try.
I think this did it! I see this when I load the kernel module:
root@zynq:/mnt/foo# insmod u-dma-buf.ko
[ 25.048672] u_dma_buf: loading out-of-tree module taints kernel.
[ 25.055924] u-dma-buf udmabuf@400000000: assigned reserved memory node image_buf@400000000
[ 25.175385] u-dma-buf udmabuf0: driver version = 3.2.0-rc.3
[ 25.180958] u-dma-buf udmabuf0: major number = 243
[ 25.185923] u-dma-buf udmabuf0: minor number = 0
[ 25.190708] u-dma-buf udmabuf0: phys address = 0x0000000400000000
[ 25.196967] u-dma-buf udmabuf0: buffer size = 268435456
[ 25.202452] u-dma-buf udmabuf@400000000: driver installed.
I see lots of cacheline fills to the memory on HPM0 on my ILA. I tested this on both kernel 4.19 and 5.4, both seemed to work correctly.
Thank you so much for your help. I couldn't have gotten this working on my own.
Thank you for trying.
However, I am worried that another problem will occur.
That is, does the kernel panic when running sync_for_device or sync_for_cpu?
If you get a kernel panic, please report it.
Here's the program I'm using to test this. All it does is open up the u-dma-buf device and read and write random addresses and data. I have it set to a small MEM_SIZE so it fits easily in the cache for testing:
https://gist.github.com/rockybulwinkle/e0b8ce174cc576619820e1ffd99570d9
While it's running, I echo 1 to sync_for_cpu and sync_for_device in a separate bash shell.
On the ILA, each time I write 1 to sync_for_cpu I see it do a write, then a bunch of reads. No kernel panic.
Again on the ILA, each time I wrote 1 to sync_for_device, I see it do the same. A write then a bunch of reads. No kernel panic.
I hope this calms your concerns. If this isn't the test you meant I'd be happy to try it again after more clarification.
Thank you for reporting.
I was relieved.
If you don't mind, please close this issue.
Thank you for your cooperation.