Reading from USB device and writing to physical address

Question

I have a USB device that outputs data of size of one byte, and I want to pass these bytes to FPGA component that exists on AXI bridge, FPGA and CPU are on the same chip... it's SoC FPGA Altera Cyclone V. CPU is ARM Cortex-A9. Kernel version 3.7.0.

There is a software that reads from the USB device and writes to a dump file... it works just fine. I tried to use mmap() to map the FPGA address to the virtual space and write to it from the userspace. When doing so... after say a minute, the kernel seem to crash.

I wrote a driver for my FPGA component and I passed the driver path to that software as a file, so that it writes to it, and eventually to my FPGA component, but the same result... kernel crashes again after a random time.

I also wrote a simple program that reads bytes from a local file and pass it to FPGA... this works fine either ways (using mmap() or driver module), the file passes through to the FPGA with no problems at all no matter how big is the file.

So the problem is when passing from USB device to FPGA, either using mmap() or a driver module.

Here is a sample crash message:

  Internal error: Oops - undefined instruction: 0 [#1] SMP ARM
  Modules linked in: ipv6
  CPU: 1    Not tainted  (3.7.0 #106)
  PC is at scheduler_ipi+0x8/0x4c
  LR is at handle_IPI+0x10c/0x19c
  pc : [<800521a0>]    lr : [<800140d4>]    psr: 80000193
  sp : bf87ff58  ip : 8056acc8  fp : 00000000
  r10: 00000000  r9 : 413fc090  r8 : 00000001
  r7 : 00000000  r6 : bf87e000  r5 : 80535018  r4 : 8053eec0
  r3 : 8056ac80  r2 : bf87ff58  r1 : 00000482  r0 : 00000481
  Flags: Nzcv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
  Control: 10c5387d  Table: 3f0c404a  DAC: 00000015
  Process swapper/1 (pid: 0, stack limit = 0xbf87e240)
  Stack: (0xbf87ff58 to 0xbf880000)
  ff40:                                                       00000000 800140d4
  ff60: fffec10c 8053e418 bf87ff90 fffec100 8000f6e0 8000851c 8000f708 8000f70c
  ff80: 60000013 ffffffff bf87ffc4 8000e180 00000000 00000000 00000001 00000000
  ffa0: bf87e000 80565688 803ddfb0 80541fc8 8000f6e0 413fc090 00000000 00000000
  ffc0: 8053e9b8 bf87ffd8 8000f708 8000f70c 60000013 ffffffff 00000020 8000f894
  ffe0: 3f86c06a 00000015 10c0387d 805658d8 0000406a 003d1ee8 31ca2085 5c1021c3
  Code: eaffffad 80564700 e92d4800 e1a0200d (4c4c9b50)
  ---[ end trace 9e492cde975c41f9 ]---

Other crash messages start like:

  Unable to handle kernel paging request at virtual address 2a7a4390
  Internal error: Oops - bad syscall: ebcffb [#1] SMP ARM
  pgd = bf318000
  [2a7a4390] *pgd=00000000

And:

 Internal error: Oops - undefined instruction: 0 [#2] SMP ARM
 Modules linked in: ipv6
 CPU: 1    Tainted: G      D       (3.7.0 #106)

Here is the full crash messages.

I noticed that all the crash messages I get intersect with the PC and LR locations, but actually I don't have previous experience with Linux kernel. I found similar error messages online but none of the proposed solutions worked for me.

Source Code:

This is function is called whenever a new buffer of bytes arrives from USB:

static void rtlsdr_callback(unsigned char *buf, uint32_t len, void *ctx)
{
    if (ctx) {
        if (do_exit)
            return;

        if ((bytes_to_read > 0) && (bytes_to_read < len)) {
            len = bytes_to_read;
            do_exit = 1;
            rtlsdr_cancel_async(dev);
        }

/*      if (fwrite(buf, 1, len, (FILE*)ctx) != len) {
            fprintf(stderr, "Short write, samples lost, exiting!\n");
            rtlsdr_cancel_async(dev);
        }
*/
        if (fm_receiver_addr == NULL)
        {
            virtual_base = mmap(NULL, HPS2FPGA_SPAN, PROT_WRITE, MAP_PRIVATE, fd, HPS2FPGA_BASE);
            if (virtual_base == MAP_FAILED)
            {
                perror("mmap");
                close(fd);
                exit(1);
            }

            fm_receiver_addr = (unsigned char*)(virtual_base + FM_DEMOD_OFFSET);
        }

        int i, j;
        for (i = 0; i < len; i++)
        {
            *fm_receiver_addr = buf[i];
            for (j = 0; j < 150; j++);
        }

        if (bytes_to_read > 0)
            bytes_to_read -= len;
    }
}

You see I commented fwrite() function (it's used by the original code to write to files) and replaced it with my code that writes to my FPGA component: *fm_receiver_addr = buf[i];. Before that I check the address to see if it's valid and obtain another address if it's not.

For the other way, the driver module, I wrote this code:

#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/device.h>
#include <linux/platform_device.h>
#include <linux/uaccess.h>
#include <linux/ioport.h>
#include <linux/io.h>

#define HPS2FPGA_BASE       0xC0000000
#define HPS2FPGA_SPAN       PAGE_SIZE

void* fm_demod_addr;
int i;

// Get a driver entry in Sysfs
static struct device_driver fm_demod_driver = 
{
    .name = "fm-demodulator",   // Name of the driver
    .bus = &platform_bus_type,  // Which bus does the device exist
};

// Function that is used when we read from the file in /sys, but we won't use it
ssize_t fm_demod_read(struct device_driver* drv, char* buf)
{ return 0; }

// Function that is called when we write to the file in /sys
ssize_t fm_demod_write_sample(struct device_driver* drv, const char* buf, size_t count)
{
    if (buf == NULL)
    {
        pr_err("Error! String must not be NULL!\n");
        return -EINVAL;
    }

    for (i = 0; i < count; i++)
    {
        iowrite8(buf[i], fm_demod_addr);
    }

    return count;
}

// Set our module's pointers and set permissions mode
static DRIVER_ATTR(fm_demod, S_IWUSR, fm_demod_read, fm_demod_write_sample);

// Set module information
MODULE_LICENSE("Dual BSD/GPL");
MODULE_AUTHOR("Siraj Muhammad <[email protected]>");
MODULE_DESCRIPTION("Driver for FPGA component 'FM Demodulator'");

static int __init fm_demod_init(void)
{
    int ret;
    struct resource* res;

    // Register driver in kernel
    ret = driver_register(&fm_demod_driver);
    if (ret < 0)
        return ret;

    // Create file system in /sys
    ret = driver_create_file(&fm_demod_driver, &driver_attr_fm_demod);
    if (ret < 0)
    {
        driver_unregister(&fm_demod_driver);
        return ret;
    }

    // Request exclusive access to the memory region we want to write to
    res = request_mem_region(HPS2FPGA_BASE, HPS2FPGA_SPAN, "fm-demodulator");
    if (res == NULL)
    {
        driver_remove_file(&fm_demod_driver, &driver_attr_fm_demod);
        driver_unregister(&fm_demod_driver);
        return -EBUSY;
    }

    // Map the address into virtual memory
    fm_demod_addr = ioremap(HPS2FPGA_BASE, HPS2FPGA_SPAN);
    if (fm_demod_addr == NULL)
    {
        driver_remove_file(&fm_demod_driver, &driver_attr_fm_demod);
        driver_unregister(&fm_demod_driver);
        release_mem_region(HPS2FPGA_BASE, HPS2FPGA_SPAN);
        return -EFAULT;
    }

    return 0;
}

static void __exit fm_demod_exit(void)
{
    // Remove file system from /sys
    driver_remove_file(&fm_demod_driver, &driver_attr_fm_demod);
    // Unregister the driver
    driver_unregister(&fm_demod_driver);
    // Release requested memory
    release_mem_region(HPS2FPGA_BASE, HPS2FPGA_SPAN);
    // Un-map address
    iounmap(fm_demod_addr);

}

module_init(fm_demod_init);
module_exit(fm_demod_exit);

And I revert the userspace code to its original state, and pass the driver path: /sys/bus/platform/drivers/fm-demodulator/fm_demod to the userspace app to write to it.

Any thought about it?

Answer 1

  Internal error: Oops - undefined instruction: 0 [#1] SMP ARM
  PC is at scheduler_ipi+0x8/0x4c
  LR is at handle_IPI+0x10c/0x19c
  pc : [<800521a0>]    lr : [<800140d4>]    psr: 80000193
  [snip]
  Code: eaffffad 80564700 e92d4800 e1a0200d (4c4c9b50)
  ---[ end trace 9e492cde975c41f9 ]---

No one can probably absolutely know the answer. Note: undefined instruction!

The PC is at scheduler_ipi+0x8/0x4c, this is hardcore ARM-Linux scheduling; an inter-processor interrupt. You can disassemble the 'Code:' part to help,

   0:   eaffffad        b       0xfffffebc
   4:   80564700        subshi  r4, r6, r0, lsl #14
   8:   e92d4800        push    {fp, lr}
   c:   e1a0200d        mov     r2, sp
  10:   4c4c9b50        mcrrmi  11, 5, r9, ip, cr0

The crash is at the instruction mcrrmi and this appears to be non-sense. If you disassemble sched/core.o you will see the instruction sequence, but I bet that the '4c4c9b50' value is corrupt. Ie, this is not the code the compiler generated.

So the problem is when passing from USB device to FPGA, either using mmap() or a driver module.

I will use a zen move and think a little. The USB device use DMA? Your FPGA is probably also some how in control of the ARM/AXI bus. I would at least consider the possibility that the FPGA is corrupting a bus cycle and perhaps flipping address bits and causing a phycial write to kernel code space. This can happen when you use an innocent by-stander like a DMA peripheral. The ARM CPU will use cache and burst everything.

Things to check,

1. The code address in (brackets) is reported as the compiler produced. If not, hardware has probably corrupted things. It is hard for Linux code to do this as the kernel code pages are typically R/O.

2. You should also produce disassembler for any code and see what register is in effect. For instance, the (4c4c9b50) code can be found with,

   printf '\x50\x9b\x4c\x4c' > arm.bin
   objdump -marm -b binary -D arm.bin

You can just objdump vmlinux to find the assembler for the scheduler_ipi routine and then determine what a pointer might be. For instance, if this_rq() was in R9 and R9 is bogus, then you have a clue.

If the code is corrupt, you need a bus analyzer and/or some routine to monitor the location and report whenever it changes to try and locate the source of corruption.