What is the difference between dma_mmap_coherent and remap_pfn_range?

Question

Currently, I am using an example driver to learn from, and from which I have based my own custom driver around. The mmap code is very nearly identical, save for the fact that I allow the user to manage their own requested size and base my memory allocation around that and the fact that I automatically create the char device within /dev.

To explain the context, for my use case, I'd like to narrow out an issue that I'm having. dma_mmap_coherent testably works when using kmalloc'd memory, but when I have a reserved physical address region that I want to use remap_pfn_range with it quietly appears to work, and dmesg doesn't report any errors, but when I go to read, no matter what I've written there it always returns 0xff bytes. This is true whether I use the iowrite & ioread in kernel land after ioremap'ing the memory or trying to write in userland using a small mmap'ing userland test.

I've done as much research on the topic as I can I think. All I can find for documentation of remap_pfn_range is the kernel.org page, and some kernel gmain mailing list archives on remap_pfn_range replacing remap_page_range. As for dma_mmap_coherent, I was able to find a little bit more, including a presentation from the linux archives.

Ultimately there has to be a difference; there seems to be so many different ways to map kernel memory into user land. The particular question I have is: what is the difference between dma_mmap_coherent and remap_pfn_range?

Edit it might be nice to provide a general overview of the ways to map kernel memory into userland in general, covering how different apis would be used in a kernel driver mmap callback.

Answer 1

dma_mmap_coherent() is defined in dma-mapping.h as a wrapper around dma_mmap_attrs(). dma_mmap_attrs() tries to see if a set of dma_mmap_ops is associated with the device (struct device *dev) you are operating with, if not it calls dma_common_mmap() which eventually leads to a call to remap_pfn_range(), after setting the page protection as non-cacheable (see dma_common_mmap() in dma-mapping.c).

As to a general overview of mmap'ing kernel memory to user space works, the following is my quick and simple way of mmap'ing DMA buffers from user space :

1. Allocate a Buffer via an IOCTL and designate a buffer ID for each buffer with some flags:

   /* A copy-from-user call needs to be done before in the IOCTL */
   static int my_ioctl_alloc(struct my_struct *info, struct alloc_info *alloc)
   {
   
           ...
           info->buf->kvaddr = dma_alloc_coherent(dev, alloc->size, info->buf->phyaddr, GFP_KERNEL);
           info->buf->buf_id = alloc->buf_id;
           ...
   }
   

2. Define an mmap file ops :

   static const struct file_operations my_fops = {
           .open = my_open,
           .close = my_close,
           .mmap = my_mmap,
           .unlocked_ioctl = my_ioctl,
   };
   

   Do not forget to register the my_fops struct somewhere in your driver's probe function.

3. Implement mmap file ops :

    static int my_mmap(struct file *fptr, struct vm_area_struct *vma)
    {
            ...
            desc_id = vma->vm_pgoff;
            buf = find_buf_by_id(alloc, desc_id);
            vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
            ret = remap_pfn_range(vma, vma->vm_start, buf->phyaddr >> PAGE_SHIFT, vma->vm_end - vma->vm_start, vma->vm_page_prot);
            if (ret) {
                 /* Error Handle */
            }
            return 0;
    }
   

With this your kernel driver should have the minimum to allocate and mmap buffers. Freeing the buffers is an exercise for the bonus points!

In the application, you would open() the file and get a valid file descriptor fd, call the allocate IOCTL and set the buffer ID before performing a copy-to-kernel. In the mmap, you would give the buffer ID via the offset parameter :

      mmap(NULL, buf_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, buffer_id << PAGE_SHIFT);

PAGE_SHIFT is an architecture dependent compile time MACRO fixed in the kernel. Hope this helps.

This is not checkpatch.pl compliant code, nor is this the best practice, but it's one way I know how to do this. Comments/improvements/suggestions welcome!

See Linux Device Drivers - Chapter 15: Memory Mapping and DMA for the textbook examples and good background information for the interested reader.