How to Verify Atomic Writes?

Question

I have searched diligently (both within the S[O|F|U] network and elsewhere) and believe this to be an uncommon question. I am working with an Atmel AT91SAM9263-EK development board (ARM926EJ-S core, ARMv5 instruction set) running Debian Linux 2.6.28-4. I am writing using (I believe) the tty driver to talk to an RS-485 serial controller. I need to ensure that writes and reads are atomic. Several lines of source code (listed below the end of this post relative to the kernel source installation directory) either imply or implicitly state this.

Is there any way I can verify that writing/reading to/from this device is actually an atomic operation? Or, is the /dev/ttyXX device considered a FIFO and the argument ends there? It seems not enough to simply trust that the code is enforcing this claim it makes - as recently as February of this year freebsd was demonstrated to lack atomic writes for small lines. Yes I realize that freebsd is not exactly the same as Linux, but my point is that it doesn't hurt to be carefully sure. All I can think of is to keep sending data and look for a permutation - I was hoping for something a little more scientific and, ideally, deterministic. Unfortunately, I remember precisely nothing from my concurrent programming classes in the college days of yore. I would thoroughly appreciate a slap or a shove in the right direction. Thank you in advance should you choose to reply.

Kind regards,

Jayce

drivers/char/tty_io.c:1087

void tty_write_message(struct tty_struct *tty, char *msg)
{
    lock_kernel();
    if (tty) {
        mutex_lock(&tty->atomic_write_lock);
        if (tty->ops->write && !test_bit(TTY_CLOSING, &tty->flags))
            tty->ops->write(tty, msg, strlen(msg));
        tty_write_unlock(tty);
    }
    unlock_kernel();
    return;
}

arch/arm/include/asm/bitops.h:37

static inline void ____atomic_set_bit(unsigned int bit, volatile unsigned long *p)
{
    unsigned long flags;
    unsigned long mask = 1UL << (bit & 31);

    p += bit >> 5;

    raw_local_irq_save(flags);
    *p |= mask;
    raw_local_irq_restore(flags);
}

drivers/serial/serial_core.c:2376

static int
uart_write(struct tty_struct *tty, const unsigned char *buf, int count)
{
    struct uart_state *state = tty->driver_data;
    struct uart_port *port;
    struct circ_buf *circ;
    unsigned long flags;
    int c, ret = 0;

    /*
     * This means you called this function _after_ the port was
     * closed.  No cookie for you.
     */
    if (!state || !state->info) {
        WARN_ON(1);
        return -EL3HLT;
    }

    port = state->port;
    circ = &state->info->xmit;

    if (!circ->buf)
        return 0;

    spin_lock_irqsave(&port->lock, flags);
    while (1) {
        c = CIRC_SPACE_TO_END(circ->head, circ->tail, UART_XMIT_SIZE);
        if (count < c)
            c = count;
        if (c <= 0)
            break;
        memcpy(circ->buf + circ->head, buf, c);
        circ->head = (circ->head + c) & (UART_XMIT_SIZE - 1);
        buf += c;
        count -= c;
        ret += c;
    }
    spin_unlock_irqrestore(&port->lock, flags);

    uart_start(tty);
    return ret;
}

Also, from the man write(3) documentation:

An attempt to write to a pipe or FIFO has several major characteristics:

• Atomic/non-atomic: A write is atomic if the whole amount written in one operation is not interleaved with data from any other process. This is useful when there are multiple writers sending data to a single reader. Applications need to know how large a write request can be expected to be performed atomically. This maximum is called {PIPE_BUF}. This volume of IEEE Std 1003.1-2001 does not say whether write requests for more than {PIPE_BUF} bytes are atomic, but requires that writes of {PIPE_BUF} or fewer bytes shall be atomic.

Answer 1

I think that, technically, devices are not FIFOs, so it's not at all clear that the guarantees you quote are supposed to apply.

Are you concerned about partial writes and reads within a process, or are you actually reading and/or writing the same device from different processes? Assuming the latter, you might be better off implementing a proxy process of some sort. The proxy owns the device exclusively and performs all reads and writes, thus avoiding the multi-process atomicity problem entirely.

In short, I advise not attempting to verify that "reading/writing from this device is actually an atomic operation". It will be difficult to do with confidence, and leave you with an application that is subject to subtle failures if a later version of linux (or different o/s altogether) fails to implement atomicity the way you need.

Answer 2

I think PIPE_BUF is the right thing. Now, writes of less than PIPE_BUF bytes may not be atomic, but if they aren't it's an OS bug. I suppose you could ask here if an OS has known bugs. But really, if it has a bug like that, it just ought to be immediately fixed.

If you want to write more than PIPE_BUF atomically, I think you're out of luck. I don't think there is any way outside of application coordination and cooperation to make sure that writes of larger sizes happen atomically.

One solution to this problem is to put your own process in front of the device and make sure everybody who wants to write to the device contacts the process and sends the data to it instead. Then you can do whatever makes sense for your application in terms of atomicity guarantees.