One large udmabuf or more smaller ones
rejuce opened this issue · comments
I need about 100MB of vdma buffers.
I wondered... Does it make performance wise a difference if I make one udmabuf with 100mb or 5 with 20mb each?
Thank you for the issue
I don't think there is any performance difference.
I decided to go with two buffers, as the represent two different cmos chips.
I am having trouble though to configure the device tree so that two udmabuf get created (petalinux build runs thought, but I never get the devices under /dev/udmabuf...
If insmod udmabuf with a buffer argument, it is loaded and the buffer is created and working. Just with multiple buffers loaded over device tree I am having trouble
Where would I see why it failed?
Or someone see the mistake in my system-user.dtsi? (very bottom)
I'm sorry I couldn't help you.
I have never used Petalinux.
thank you very much for your work and fast reply.
in the readme you describe, that udmabuf can be described in the device-tree so when using insmod udmabuf.ko, the information about the buffers is taken from the device tree.
how does the device tree need to look like for 2 buffers ? (in your examples its always only one buffer, while you state that up to 9 are supported)
/{
udmabuf@0 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf249";
size = <0x08DCC000>; // 193621200*32 max framebuffers vdma can handle
memory-region = <&image_buf_IMX249>;
};
};
/{
udmabuf@1 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf273";
size = <0x060B0000>; // 145610882*32 max framebuffers vdma can handle
memory-region = <&image_buf_IMX273>;
};
};
would that be correct?
that would give me /dev/udmabuf249 and /dev/udmabuf273 after insmod udmabuf?
The Linux CMA region has alignment constraints.
For MPSoC, the address and size that can be specified in the regs property of reserved-memory must be in units of 0x00400000 (4MiB).
Please refer to this issue #29.
All my adresses and buffer sizes are page 4MiB aligned!
rephrasing the question:
your readme states if a entry like that is present in the device tree:
udmabuf@0x00 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf0";
size = <0x00100000>;
};
after insmod udmabuf.ko the information is taken from there. How would the node for udmabuf1 look like? (if there would be information for udmabuf0 and udmabuf1 in the device tree?)
Would it be like that? or also udmabuf@0x00 and just a differnt device-name?:
udmabuf@0x01 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf1";
size = <0x00100000>;
};
uum...
I'm sorry, but I don't understand what you said.
the readme states:
The maximum number of DMA buffers that can be allocated using insmod is 8 (udmabuf0/1/2/3/4/5/6/7).
so how does the device-tree need to look like to get
udmabuf0
udmabuf1
...
the examples always show only one buffer (udmabuf0)
I'm sorry. The following is correct. The device tree is not required if set with the insmod argument.
The maximum number of DMA buffers that can be allocated using argments of insmod is 8 (udmabuf0/1/2/3/4/5/6/7).
shell$ sudo insmod udmabuf.ko udmabuf0=0x00010000 udmabuf1=0x00010000
[ 76.977400] udmabuf: loading out-of-tree module taints kernel.
[ 76.980213] udmabuf udmabuf.0: DMA mask not set
[ 76.985196] udmabuf udmabuf0: driver version = 1.4.6
[ 76.985211] udmabuf udmabuf0: major number = 241
[ 76.985215] udmabuf udmabuf0: minor number = 0
[ 76.985220] udmabuf udmabuf0: phys address = 0x00000000700a0000
[ 76.985224] udmabuf udmabuf0: buffer size = 65536
[ 76.985229] udmabuf udmabuf0: dma device = udmabuf.0
[ 76.985233] udmabuf udmabuf0: dma coherent = 0
[ 76.985238] udmabuf udmabuf.0: driver installed.
[ 76.985906] udmabuf udmabuf.1: DMA mask not set
[ 76.990465] udmabuf udmabuf1: driver version = 1.4.6
[ 76.990477] udmabuf udmabuf1: major number = 241
[ 76.990481] udmabuf udmabuf1: minor number = 1
[ 76.990485] udmabuf udmabuf1: phys address = 0x00000000700b0000
[ 76.990490] udmabuf udmabuf1: buffer size = 65536
[ 76.990494] udmabuf udmabuf1: dma device = udmabuf.1
[ 76.990499] udmabuf udmabuf1: dma coherent = 0
[ 76.990503] udmabuf udmabuf.1: driver installed.If you want to reserve two udmabuf in the device tree, describe as follows.
udmabuf@0x00 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf0";
size = <0x00100000>;
};
udmabuf@0x01 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf1";
size = <0x00100000>;
};
thank you for your reply. i am still trrying to figure out why it is not working.
Here is my device tree now: I made sure now everything, sizes and adresses ist 4 MiB aligned.
my current device tree:
reserved-memory {
#address-cells = <1>;
#size-cells = <1>;
#sizranges;
ranges;
image_buf_IMX273: image_buf_0@39C00000 {
compatible = "shared-dma-pool";
reusable;
reg = <0x39C00000 0x06400000>;
label = "image_buf_IMX273";
};
image_buf_IMX249: image_buf_1@30c00000 {
compatible = "shared-dma-pool";
reusable;
reg = <0x30c00000 0x09000000>;
label = "image_buf_IMX249";
};
};
udmabuf@0x00 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf0";
size = <0x09000000>; // 1936*2*1200*32 max framebuffers vdma can handle rounded up to nxt 4MiB
memory-region = <&image_buf_IMX249>;
dma-coherent;
};
udmabuf@0x01 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf1";
size = <0x06400000>; // 1456*1088*2*32 max framebuffers vdma can handle rounded up to nxt 4MiB
memory-region = <&image_buf_IMX273>;
dma-coherent;
};
I can see now that linux creates my reserved-memory shared-dma-pool areas on bootup /dmesg
[ 0.000000] Reserved memory: created CMA memory pool at 0x30c00000, size 144 MiB
[ 0.000000] OF: reserved mem: initialized node image_buf_1@30c00000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created CMA memory pool at 0x39c00000, size 100 MiB
[ 0.000000] OF: reserved mem: initialized node image_buf_0@39C00000, compatible id shared-dma-pool
[ 0.000000] cma: Reserved 16 MiB at 0x2f000000
unfortunately after insmod u-dma-buf.ko ... i dont get any nodes under /dev
(i would expect udmabuf0 and 1) ... I dont see any errors either.
[ 528.357715] u_dma_buf: loading out-of-tree module taints kernel.
is all that dmesg shows about u_dma_buf....no other text that buffers are created or erros.
if I try to load the module manually (without using the device tree specification)
insmod u-dma-buf.ko udmabuf0=100000 in only can enter sizes up to the 16MB reserved cma (larger than 16mib fails with -12 ). there is no way to force to use a certain reserved-memory region that way, right?
Do you see whats going wrong? Why do i not get any devices under /dev? What am I doing wrong? How can I find out whats going wrong as I dont see any errors?
Thank you so much for your help.
Add the udmabuf node below the bus node.
For example, for ZYBO-Z & on FPGA-SoC-Linux, add it under the "amba" node as follows:
amba {
compatible = "simple-bus";
#address-cells = <0x1>;
#size-cells = <0x1>;
interrupt-parent = <0x4>;
ranges;
phandle = <0xc>;
udmabuf@0x00 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf0";
size = <0x00100000>;
};
udmabuf@0x01 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf1";
size = <0x00100000>;
};
};
The node name of the bus depends on the system.
Please specify the bus name of your system.
im am using zynq 7000, so its amba
as you told me i inserted the udmabuf nodes under amba...unfortunately the behaviour stays exactly the same as described before
amba {
...
///here comes a lot of other amba peripherals
...
udmabuf@0x00 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf0";
size = <0x9000000>;
memory-region = <0x8>;
dma-coherent;
};
udmabuf@0x01 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf1";
size = <0x6400000>;
memory-region = <0x9>;
dma-coherent;
};
};
Are you using "udmabuf" or "u-dma-buf"?
See Readme.md for the difference between "udmabuf" and "u-dma-buf".
If you are using "u-dma-buf" then you must specify "ikwzm, u-dma-buf" for compatible property.
I tried it with ZYBO-Z7 + Linux Kernel 4.19.57 + udmabuf.
The device tree is as follows.
reserved-memory {
#address-cells = <0x1>;
#size-cells = <0x1>;
ranges;
zynq-dma-unusable-area {
reg = <0x0 0x80000>;
};
image_buf_IMX273: image_buf@39C00000 {
compatible = "shared-dma-pool";
reusable;
reg = <0x39C00000 0x06400000>;
label = "image_buf_IMX273";
};
image_buf_IMX249: image_buf@30C00000 {
compatible = "shared-dma-pool";
reusable;
reg = <0x30C00000 0x09000000>;
label = "image_buf_IMX249";
};
};
udmabuf@249 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf249";
size = <0x08000000>;
memory-region = <&image_buf_IMX249>;
};
udmabuf@273 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf273";
size = <0x06400000>;
memory-region = <&image_buf_IMX273>;
};
The result looks like this:
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.19.57-armv7-fpga (ichiro@Jabberwock) (gcc version 7.4.0 (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04)) #0 SMP PREEMPT Wed Jul 10 11:33:57 DST 2019
[ 0.000000] CPU: ARMv7 Processor [413fc090] revision 0 (ARMv7), cr=18c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] OF: fdt: Machine model: Digilent Zybo Z7 board
[ 0.000000] earlycon: cdns0 at MMIO 0xe0001000 (options '115200n8')
[ 0.000000] bootconsole [cdns0] enabled
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] Reserved memory: created CMA memory pool at 0x30c00000, size 144 MiB
[ 0.000000] OF: reserved mem: initialized node image_buf@30C00000, compatible id shared-dma-pool
[ 0.000000] Reserved memory: created CMA memory pool at 0x39c00000, size 100 MiB
[ 0.000000] OF: reserved mem: initialized node image_buf@39C00000, compatible id shared-dma-pool
[ 0.000000] cma: Reserved 16 MiB at 0x2f000000
[ 0.000000] random: get_random_bytes called from start_kernel+0x80/0x408 with crng_init=0
[ 0.000000] percpu: Embedded 16 pages/cpu s36172 r8192 d21172 u65536
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 260608
[ 0.000000] Kernel command line: console=ttyPS0,115200 root=/dev/mmcblk0p2 rw rootwait earlycon
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 760932K/1048576K available (7168K kernel code, 298K rwdata, 1904K rodata, 1024K init, 135K bss, 21404K reserved, 266240K cma-reserved, 12288K highmem)
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB)
[ 0.000000] fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
[ 0.000000] vmalloc : 0xf0800000 - 0xff800000 ( 240 MB)
[ 0.000000] lowmem : 0xc0000000 - 0xf0000000 ( 768 MB)
[ 0.000000] pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
[ 0.000000] modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
[ 0.000000] .text : 0x(ptrval) - 0x(ptrval) (8160 kB)
[ 0.000000] .init : 0x(ptrval) - 0x(ptrval) (1024 kB)
[ 0.000000] .data : 0x(ptrval) - 0x(ptrval) ( 299 kB)
[ 0.000000] .bss : 0x(ptrval) - 0x(ptrval) ( 136 kB)
[ 4.727808] udmabuf: loading out-of-tree module taints kernel.
[ 4.734096] udmabuf: loading out-of-tree module taints kernel.
[ 4.777068] udmabuf udmabuf@249: assigned reserved memory node image_buf@30C00000
[ 5.426893] udmabuf udmabuf249: driver version = 1.4.6
[ 5.432076] udmabuf udmabuf249: major number = 244
[ 5.437039] udmabuf udmabuf249: minor number = 0
[ 5.441910] udmabuf udmabuf249: phys address = 0x30c00000
[ 5.447481] udmabuf udmabuf249: buffer size = 134217728
[ 5.453037] udmabuf udmabuf249: dma device = udmabuf@249
[ 5.458756] udmabuf udmabuf249: dma coherent = 0
[ 5.463548] udmabuf udmabuf@249: driver installed.
[ 5.469897] udmabuf udmabuf@273: assigned reserved memory node image_buf@39C00000
[ 5.865279] vmap allocation for size 104861696 failed: use vmalloc=<size> to increase size
[ 5.899494] dma_alloc_coherent() failed. return(0)
[ 5.904682] udmabuf udmabuf@273: driver setup failed. return=-12
[ 5.912011] udmabuf udmabuf@273: driver installed.
[ 5.916813] udmabuf: probe of udmabuf@273 failed with error -12
Unfortunately it failed because of Linux Kernel limitations.
On my system (arm32 + linux kernel 4.19.57), the area that can be secured by vmalloc is 240MB, as the following log shows.
[ 0.000000] Virtual kernel memory layout:
[ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB)
[ 0.000000] fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
[ 0.000000] vmalloc : 0xf0800000 - 0xff800000 ( 240 MB)
[ 0.000000] lowmem : 0xc0000000 - 0xf0000000 ( 768 MB)
[ 0.000000] pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
[ 0.000000] modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
[ 0.000000] .text : 0x(ptrval) - 0x(ptrval) (8160 kB)
[ 0.000000] .init : 0x(ptrval) - 0x(ptrval) (1024 kB)
[ 0.000000] .data : 0x(ptrval) - 0x(ptrval) ( 299 kB)
[ 0.000000] .bss : 0x(ptrval) - 0x(ptrval) ( 136 kB)
The buffer size of the two udmabuf is over 240MB, so the virtual address cannot be allocated and it fails.
I overlooked that udma-buf to change the device compatibility entry compared to udmabuf
And thank you for the hint with vmalloc.. I see the same 240MB
If I try 100MiB + 150 MiB both fail
Id I try 60MiB + 150 MiB the 150MiB fails, the 60 MiB works. So even both allocs are smaller than the 240 the 150 seem to be a problem.
If I use 80 MiB for both, both work...
Do you know by any chance if there is a limit per single allocation?
Otherwise I will try out how far I can go and have to compromise on the frame buffers...
At any rate thank you very much for your work!!
Hello, thank you for the module. Setting it up with your guide was perfect !
I have the same problem as @rejuce, I try to allocate 2 buffers of 307200 bits (38400 bytes), but I am not sure which is the correct way.
Am I supposed to give the size in bits
size = <0x0004B000>;
or bytes
size = <0x00009600>;
And how would they go in my dti to declare 2 buffers? I tried the code below:
udmabuf@0x00 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf0";
size = <0x0004B000>;
};
udmabuf@0x01 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf1";
size = <0x0004B000>;
};
But only udmabuf0 loads correctly, while the other one fails. Any suggestions?
Thank you for your work
Gilles
I have the same problem as @rejuce, I try to allocate 2 buffers of 307200 bits (38400 bytes), but I am not sure which is the correct way.
Am I supposed to give the size in bits
size = <0x0004B000>;
or bytes
size = <0x00009600>;
Readme.md of udmabuf has the following description.
### `size`
The `size` property is used to set the capacity of DMA buffer in bytes.
The `size` property is mandatory.And how would they go in my dti to declare 2 buffers? I tried the code below:
udmabuf@0x00 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf0";
size = <0x0004B000>;
};
udmabuf@0x01 {
compatible = "ikwzm,udmabuf-0.10.a";
device-name = "udmabuf1";
size = <0x0004B000>;
};But only udmabuf0 loads correctly, while the other one fails. Any suggestions?
I cannot answer because there is not enough information.
Make sure that you place them within the amba section in your device tree. When it was just there it did not work
Also make sure that you have compatible = "ikwzm,udmabuf-0.10.a"; right.
There are two!! Repositories... The other one is u-dma-buf... Make sure you put here the right one depending on which kernel you built (udmabuf, or u-dma-buf) there is a note about it in the readme, but I totally overlooked it at first
@Gilleslenaerts007 I also had a similar issue where one of the udma buffers would load but not the other.
Hello, thank you for the module. Setting it up with your guide was perfect ! I have the same problem as @rejuce, I try to allocate 2 buffers of 307200 bits (38400 bytes), but I am not sure which is the correct way. Am I supposed to give the size in bits size = <0x0004B000>; or bytes size = <0x00009600>;
And how would they go in my dti to declare 2 buffers? I tried the code below: udmabuf@0x00 { compatible = "ikwzm,udmabuf-0.10.a"; device-name = "udmabuf0"; size = <0x0004B000>; }; udmabuf@0x01 { compatible = "ikwzm,udmabuf-0.10.a"; device-name = "udmabuf1"; size = <0x0004B000>; };
But only udmabuf0 loads correctly, while the other one fails. Any suggestions?
Thank you for your work Gilles
I needed to change the minor-number I believe this is due to some issues with loading the drivers back to back, but not sure. Nevertheless it worked for me.
Working system-user.dtsi:
dma_buf_rx: dma_buf0@0x20400000 {
compatible = "shared-dma-pool";
reusable;
reg = <0x0 0x20400000 0x0 0x00400000>;
label = "dma_buf_rx";
};
dma_buf_tx: dma_buf1@0x20c00000 {
compatible = "shared-dma-pool";
reusable;
reg = <0x0 0x20c00000 0x0 0x00400000>;
label = "dma_buf_tx";
};
};
udmabuf_rx@0x20400000 {
compatible = "ikwzm,u-dma-buf";
device-name = "udmabuf_rx";
minor-number = <1>;
size = < 0x0 0x00400000>; // 4MiB KB
memory-region = <&dma_buf_rx>;
};
udmabuf_tx@0x20c00000 {
compatible = "ikwzm,u-dma-buf";
device-name = "udmabuf_tx";
minor-number = <2>;
size = < 0x0 0x00400000>; // 4MiB KB
memory-region = <&dma_buf_tx>;
};