setting vma->vm_page_prot before dma_mmap_coherent makes no sense

[Tingqian Li]

Hello, ikwzm, great work!
But I got a question, from the source of dma_mmap_coherent(), it seems setting vma->vm_page_prot for coherent dma buffer make no sense because kernel will chose the right page_prot for us, right?

https://elixir.bootlin.com/linux/latest/source/kernel/dma/mapping.c#L162

Thanks!

[KAWAZOME Ichiro]

Thank you for the issue.

The line of source code you have shown is:

	vma->vm_page_prot = arch_dma_mmap_pgprot(dev, vma->vm_page_prot, attrs);

The value of vma-> vm_page_prot is put in the second argument of the arch_dma_mmap_pgprot function. The value of vma-> vm_page_prot obtained by this function is based on vma-> vm_page_port set in the second argument. So I think that it is not to make no sense.

Do you have any thoughts about this?

[Tingqian Li]

well, further dive into it, you can see it on arm64 platform, it will be writecombine anyway (in case there is no coherent caches interconnect), even the input prot is not.
https://elixir.bootlin.com/linux/latest/source/arch/arm64/mm/dma-mapping.c#L36

so we cannot override it simply, right?

My point is, since "coherent" DMA mapping API is designed so that no explicit cache flushing is needed, maybe you shouldn't do cache flushing on such DMA mapping in the beginning? see:

https://www.kernel.org/doc/Documentation/DMA-API-HOWTO.txt

I'm not sure, what do you think?

Maybe your design would be more meaningful if you use streaming DMA mapping instead of coherent mapping?

[KAWAZOME Ichiro]

uum ...
There are many things I do not understand.
My code looks like this:

static int udmabuf_device_file_mmap(struct file *file, struct vm_area_struct* vma)
{
    struct udmabuf_device_data* this = file->private_data;

    if (vma->vm_pgoff + vma_pages(vma) > (this->alloc_size >> PAGE_SHIFT))
        return -EINVAL;

    if ((file->f_flags & O_SYNC) | (this->sync_mode & SYNC_ALWAYS)) {
        switch (this->sync_mode & SYNC_MODE_MASK) {
            case SYNC_MODE_NONCACHED : 
                vma->vm_flags    |= VM_IO;
                vma->vm_page_prot = _PGPROT_NONCACHED(vma->vm_page_prot);
                break;
            case SYNC_MODE_WRITECOMBINE : 
                vma->vm_flags    |= VM_IO;
                vma->vm_page_prot = _PGPROT_WRITECOMBINE(vma->vm_page_prot);
                break;
            case SYNC_MODE_DMACOHERENT :
                vma->vm_flags    |= VM_IO;
                vma->vm_page_prot = _PGPROT_DMACOHERENT(vma->vm_page_prot);
                break;
            default :
                break;
        }
    }
    vma->vm_private_data = this;

#if (USE_VMA_FAULT == 1)
    {
        unsigned long page_frame_num = (this->phys_addr >> PAGE_SHIFT) + vma->vm_pgoff;
        if (pfn_valid(page_frame_num)) {
            vma->vm_ops = &udmabuf_device_vm_ops;
            udmabuf_device_vma_open(vma);
            return 0;
        }
    }
#endif

    return dma_mmap_coherent(this->dma_dev, vma, this->virt_addr, this->phys_addr, this->alloc_size);
}

In my code, dma_mmap_coherent () is not used in most cases. Instead, we use the Udmabuf Device VM Area Operations to mmap().
The reason for doing this is that, at the beginning of development (Linux Kernel 3.11 ~), using dma_mmap_coherent () somehow could not control the cache.
Perhaps the current dma_mmap_coherent () may not be able to control the cache. The behavior of dma_mmap_coherent () is not clear to me either.

At the moment, the control of the cache by Udmabuf Device VM Area Operations works well.

[Tingqian Li]

Oh, yes, I didn't notice the default value of USE_VMA_FAULT is 1, that makes sense now, Thanks for clarification!

I see your design now, very interesting, you allocate with "coherent" DMA mapping, but make a workaround to allow mapping into user-space in non-coherent way and allow user app to do cache sync.

OK, this kind of hack is fine, but if you check the code in dma_alloc_coherent() you can see the designer is doing its best to mapping the buffer into kernel with the most accurate page_prot satisfying "coherent" and also with as best CPU access performance as possible (it checks if platform support system-level cache coherent or if the device DMA can maintain CPU cache coherency by it's own like ARM ACE protocol, if so it will use cached prot, otherwise, writecombines is used), so giving the choice to user is not what dma_alloc_coherent() trying to do.

Anyway, your hack is smart, Thanks again for your answer!

[Tingqian Li]

Last comment, maybe change dma_alloc_coherent() with dma_direct_alloc_pages() and eliminate dma_map_coherent() to avoid confusion?

[KAWAZOME Ichiro]

dma_direct_alloc_pages () has been implemented since Linux Kernel 5.0? I have not yet been familiar with the dma mapping mechanism since Linux Kernel 5.0.
I will consider from now on.

[Tingqian Li]

Thanks！Great work! :)