Zynq UltraScale+ (ARM Cortex-A53)

Question

Zynq UltraScale+ (ARM Cortex-A53)

GiuseppeDiGuglielmo opened this issue 7 years ago · comments

On a Xilinx Zynq UltraScale+ (CPU ARM Cortex-A53), I am running the Linux kernel 4.9-xilinx-v2017.2.

I know that both OS and CPU are not listed as supported, but --- hopefully --- the porting is easy.

For cross-compiling, I modified the Makefile:
ARCH := arm64
CROSS_COMPILE ?= aarch64-linux-gnu-

When I insmod the kernel driver, I get the following error:
# insmod udmabuf.ko udmabuf0=1024
[ 8554.453499] dma_alloc_coherent() failed
[ 8554.457374] udmabuf: couldn't create udmabuf0 driver

Can I fix the problem?

KAWAZOME Ichiro · Answer 1 · Sat Sep 23 2017 15:04:02 GMT+0800 (China Standard Time)

Thank you for the issue.

Unfortunately, at the moment, I do not have ARM 64 so I have not confirmed its behavior.

However, there is another hope from other people that you want udmabuf to work on ARM 64. I would like to follow hope as much as possible. Please wait a little more.

Also, if you have any information, please let me know.

KAWAZOME Ichiro · Answer 2 · Fri Sep 29 2017 07:36:14 GMT+0800 (China Standard Time)

I attempted to fix the problem that dma_alloc_coherent () fails on the ARM64 architecture.

Unfortunately, since I am not yet ready for Zynq UltraScale +, I can not confirm whether compilation and execution succeed. Please wait a little more.

Maybe it will be successful if you build with it. please try it.

The branch is arm64-develop. I have not merged to the master branch yet.

shell$ git checkout arm64-develop

GDG · Answer 3 · Fri Sep 29 2017 11:16:29 GMT+0800 (China Standard Time)

Thank you for spending time on this issue.

I try to compile the code on the branch.

For ARM32, it works correctly (I insmod the module without problems).

For ARM64, it fails at compilation time:
/udmabuf.git/udmabuf.c: In function ‘udmabuf_driver_file_mmap’:
udmabuf.git/udmabuf.c:444:37: error: implicit declaration of function ‘pgprot_dmacoherent’ [-Werror=implicit-function-declaration]
vma->vm_page_prot = pgprot_dmacoherent(vma->vm_page_prot);

The function pgprot_dmacoherent is only for ARM:
arch/arm/include/asm/pgtable-nommu.h

I will try to figure out what you are trying to do and see if I can complete the porting. If you have comments about the differences between ARM32 and ARM64 implementations, please let me know. Finally, if you like to try remotely an ARM64 (ZynqMP), please let me know.

KAWAZOME Ichiro · Answer 4 · Fri Sep 29 2017 11:51:47 GMT+0800 (China Standard Time)

Thank you for trying.

The compile error that reported me is my mistake.

Although it is an urgent action, I pushed the modified source.

I think that I must study more about this subject. In ARM and ARM64, the bus configuration and DMA mapping are different.

It seems that it will take some more time to fundamentally solve.

GDG · Answer 5 · Sat Sep 30 2017 03:32:26 GMT+0800 (China Standard Time)

I saw your latest changes.

In the meantime, given this post: http://permalink.gmane.org/gmane.linux.kernel.commits.head/433710, I locally modified your code as the following snippet and it worked (I have still to properly verify it).

            case SYNC_DMACOHERENT :
                vma->vm_flags    |= VM_IO;
                vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); //pgprot_dmacoherent(vma->vm_page_prot);
                break;

GDG · Answer 6 · Tue Oct 03 2017 05:02:20 GMT+0800 (China Standard Time)

On ARM64, I am trying to allocate a 16MB contiguous buffer, but I get the following error:

insmod udmabuf.ko udmabuf0=16777216
CHRDEV "udmabuf" major number 229 goes below the dynamic allocation range
dma_alloc_coherent() failed
udmabuf: couldn't create udmabuf0 driver

Please, notice that I am using the latest revision (9092aff).

Am I doing anything wrong?

That should be possible because I have enabled the CMA module in the kernel and I have reserved (16MB). At kernel boot time I have the following log:

[    0.000000] cma: dma_contiguous_reserve(limit 100000000)
[    0.000000] cma: dma_contiguous_reserve: reserving 16 MiB for global area
[    0.000000] cma: cma_declare_contiguous(size 0x0000000001000000, base 0x0000000000000000, limit 0x0000000100000000 alignment 0x0000000000000000)
[    0.000000] cma: Reserved 16 MiB at 0x000000007f000000
[    0.000000] On node 0 totalpages: 1048576
[    0.000000]   DMA zone: 7168 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 524288 pages, LIFO batch:31
[    0.000000]   Normal zone: 7168 pages used for memmap
[    0.000000]   Normal zone: 524288 pages, LIFO batch:31
...
[    0.000000] Kernel command line: earlycon clk_ignore_unused root=/dev/mmcblk0p2 rw rootwait cma=16MB uio_pdrv_genirq.of_id=generic-uio
...
[    0.000000] Memory: 4032140K/4194304K available (10172K kernel code, 1074K rwdata, 3048K rodata, 896K init, 403K bss, 145780K reserved, 16384K cma-reserved)

GDG · Answer 7 · Tue Oct 03 2017 08:21:27 GMT+0800 (China Standard Time)

Please, note also that the same code compiled for ARM32 works:

root@zc702:/boot# insmod udmabuf.ko udmabuf0=1228800 udmabuf1=1228800
udmabuf: loading out-of-tree module taints kernel.
udmabuf udmabuf0: driver installed
udmabuf udmabuf0: major number   = 244
udmabuf udmabuf0: minor number   = 0
udmabuf udmabuf0: phys address   = 0x3f100000
udmabuf udmabuf0: buffer size    = 1228800
udmabuf udmabuf1: driver installed
udmabuf udmabuf1: major number   = 244
udmabuf udmabuf1: minor number   = 1
udmabuf udmabuf1: phys address   = 0x3f300000
udmabuf udmabuf1: buffer size    = 1228800

while on ARM64, has problems:

root@zcu102:/boot# insmod udmabuf.ko udmabuf1=1228800 udmabuf0=1228800
[ 3778.974058] udmabuf udmabuf0: driver installed
[ 3778.978459] udmabuf udmabuf0: major number   = 234
[ 3778.983203] udmabuf udmabuf0: minor number   = 0
[ 3778.987805] udmabuf udmabuf0: phys address   = 0x000000007fd00000
[ 3778.993876] udmabuf udmabuf0: buffer size    = 1228800
[ 3778.999162] dma_alloc_coherent() failed
[ 3779.003016] udmabuf: couldn't create udmabuf1 driver

I really appreciate your effort.

KAWAZOME Ichiro · Answer 8 · Tue Oct 03 2017 08:27:38 GMT+0800 (China Standard Time)

Thank you for trying.

The CMA buffer area used by udmabuf is shared with other devices. If other embedded devices have already allocated buffers from the CMA, the capacity available for that udmabuf will decrease.

If you want to reserve 16 Mbytes with udmabuf, you will have to increase the capacity of CMA.

The capacity that can be secured by CMA can be specified by the parameter at the time of starting the kernel. For example, if devicetree is specified as follows, the maximum number of bytes of CMA buffer can be changed to 128 Mbytes

chosen {
        bootargs = "console=ttyPS0,115200n8 consoleblank=0 root=/dev/mmcblk0p2 rw rootwait earlyprintk cma=128M";
        linux,stdout-path = "/amba@0/serial@e0001000";
    };

Alternatively, when specifying parameters with u-boot, specify the capacity of CMA as follows.

    setenv bootargs console=ttyPS0,115200 root=/dev/mmcblk0p2 rw rootwait uio_pdrv_genirq.of_id=generic-uio cma=128M

KAWAZOME Ichiro · Answer 9 · Tue Oct 03 2017 08:52:01 GMT+0800 (China Standard Time)

while on ARM64, has problems:

root@zcu102:/boot# insmod udmabuf.ko udmabuf1=1228800 udmabuf0=1228800
[ 3778.974058] udmabuf udmabuf0: driver installed
[ 3778.978459] udmabuf udmabuf0: major number   = 234
[ 3778.983203] udmabuf udmabuf0: minor number   = 0
[ 3778.987805] udmabuf udmabuf0: phys address   = 0x000000007fd00000
[ 3778.993876] udmabuf udmabuf0: buffer size    = 1228800
[ 3778.999162] dma_alloc_coherent() failed
[ 3779.003016] udmabuf: couldn't create udmabuf1 driver

udmabuf succeeded in securing the first buffer, but the second failed?
This may be a difficult problem.

Thank you for valuable information.

KAWAZOME Ichiro · Answer 10 · Sun Jan 07 2018 10:06:05 GMT+0800 (China Standard Time)

I have released udmabuf v0.9.0 corresponding to ARM64.

I am confirmed to work with UltraZed Starter Kit.
http://zedboard.org/product/ultrazed-eg-starter-kit

I hope I can be of any help to you.

KAWAZOME Ichiro · Answer 11 · Tue Jan 09 2018 14:01:16 GMT+0800 (China Standard Time)

I have released udmabuf v0.9.0 corresponding to ARM64.
This issue closes once.
Thank you for your cooperation.

giovanig · Answer 12 · Tue Mar 20 2018 22:44:35 GMT+0800 (China Standard Time)

Hello,
I cloned the repo using the arm64-develop and compiled the code using arm64 as ARCH and aarch64-linux-gnu- as CROSS_COMPILE flags.

When I load the module, got the same error in this thread, using the Zynq Ultrascale+ A53 and Kernel 4.14.5:

insmod udmabuf.ko udmabuf0=1000
[ 749.926651] dma_alloc_coherent() failed
[ 749.930545] udmabuf: couldn't create udmabuf0 driver

Am I missing something?

KAWAZOME Ichiro · Answer 13 · Tue Mar 20 2018 23:07:21 GMT+0800 (China Standard Time)

Thank you for trying.

Have you tried v1.1.0?

I confirmed the operation of udmabuf in the following environment.

https://github.com/ikwzm/ZynqMP-FPGA-Linux

If you have not tried it yet, please try it.

giovanig · Answer 14 · Tue Mar 20 2018 23:28:05 GMT+0800 (China Standard Time)

Thanks for your answer. I made a try now:

git clone --depth=1 -b v1.1.0 https://github.com/ikwzm/udmabuf

Set ARCH and CROSS_COMPILE in Makefile and type make. The output was the following:

Building modules, stage 2.
MODPOST 1 modules
WARNING: modpost: Found 1 section mismatch(es).
To see full details build your kernel with:
'make CONFIG_DEBUG_SECTION_MISMATCH=y'
CC X/udmabuf/udmabuf.mod.o
LD [M] X/udmabuf/udmabuf.ko

Then when I tried to load the module, got the same error:

insmod udmabuf.ko udmabuf0=1000
[ 1613.567900] dma_alloc_coherent() failed
[ 1613.572128] udmabuf: couldn't create udmabuf0 driver

The kernel log has the following output:

Mar 20 15:25:12 linaro-gnome kernel: [ 1613.567894] dma_set_mask(DMA_BIT_MASK(32)) failed
Mar 20 15:25:12 linaro-gnome kernel: [ 1613.567900] dma_alloc_coherent() failed
Mar 20 15:25:12 linaro-gnome kernel: [ 1613.572128] udmabuf: couldn't create udmabuf0 driver

KAWAZOME Ichiro · Answer 15 · Wed Mar 21 2018 01:00:43 GMT+0800 (China Standard Time)

uum... I can not understand...

It may not work on Linux Kernel 4.14. It is running on Linux Kernel 4.9.

Can you allocate buffers from udmabuf using the device tree?

KAWAZOME Ichiro · Answer 16 · Thu Mar 22 2018 13:13:07 GMT+0800 (China Standard Time)

I am sorry to have kept you waiting.
I am in trouble because I can not build Linux Kernel 4.14 that runs with ZynqMP.
Although it is tentative, I made udmabuf v1.2.0-rc1.
If you have plenty of time, please try trying to run udmabuf v1.2.0-rc1 on Linux Kernel 4.14.

Min Ma · Answer 17 · Thu May 10 2018 02:05:06 GMT+0800 (China Standard Time)

Just to let you know. The udmabuf v1.2.0-rc1 can work with Linux Kernel 4.14 + ZynqMP.

But I see this warning in compile time:
WARNING: /home/minm/udmabuf-1.2.0-rc1/udmabuf.o(.init.text+0x1b8): Section mismatch in reference from the function init_module() to the variable .exit.text:$x
a variable __exit $x.
This is often seen when error handling in the init function
uses functionality in the exit path.
The fix is often to remove the __exit annotation of
$x so it may be used outside an exit section.

KAWAZOME Ichiro · Answer 18 · Thu May 10 2018 10:32:53 GMT+0800 (China Standard Time)

Thank you for your reply.
I fixed it immediately.
Although it is tentative, I made udmabuf v1.2.0-rc2 (eace167)