PlainDatagramSocketImpl (IOException: Operation not permitted)

Question

I'm experiencing the following problem with DatagramSockets:

java.io.IOException: Operation is not permitted.
    at java.net.PlainDatagramSocketImpl.send(Native Method)
    at java.net.DatagramSocket.send(DatagramSocket.java:693)

The exception occurs randomly and I can't really see a pattern. That makes it harder for me to debug this.

Nevertheless, I suspect that it occurs more often when I'm sending a lot of data.

I'm having multiple threads sending over this socket but this shouldn't be a problems since I read Java Sockets would be thread-safe.

Can someone tell me when and under which conditions such an exception can be thrown?

Here is my basic network code:

package de.oompf.netwrk;

import java.io.IOException;
import java.net.DatagramPacket;
import java.net.DatagramSocket;
import java.net.SocketException;

class Server implements Runnable {

    private final EventBus bus;
    private final Thread serverThread;
    private final DatagramSocket socket;

    Server(EventBus bus) throws SocketException {
    this.bus = bus;
    serverThread = new Thread(this, "Server Thread");
    socket = getBoundSocket();
    socket.setSoTimeout(2400);
    }

    private static DatagramSocket getBoundSocket() throws SocketException {
    for (int port : Configuration.getPortList()) {
        try {
            return new DatagramSocket(port);
        } catch (SocketException e) {
        }
    }
    return new DatagramSocket(0);
    }

    int getPort() {
    return socket.getLocalPort();
    }

    void start() {
    bus.subscribe(this);
    serverThread.start();
    }

    void stop() {
    serverThread.interrupt();
    socket.close();
    }

    @Override
    public void run() {
    DatagramPacket p = new DatagramPacket(new byte[4096], 4096);
    while (!serverThread.isInterrupted()) {
        try {
            socket.receive(p);
            bus.publish(new IncomingPacket(p.getData(), p.getLength(), p.getAddress(), p.getPort()));
        } catch (IOException e) {
            if (socket.isClosed()) {
                break;
            }
        }
    }
    }

    void send(OutgoingPacket p) {
    try {
        if (p.getData()[0] == 0x03) {
        }
        socket.send(new DatagramPacket(p.getData(), p.getData().length, p.getSocketAddress()));
    } catch (IOException e) {
        if (socket.isClosed()) {
            serverThread.interrupt();
        } else {
            e.printStackTrace();
        }
    }
    }
}

There are a lot of classes working behind this. I'm just going to post a few lines where my stack trace ends.

private void handleBootstrapRequest(IncomingPacket p) {
if (p.getLength() == 21) {
    byte[] requestNodeBytes = new byte[20];
    System.arraycopy(p.getData(), 1, requestNodeBytes, 0, 20);
    try {
        Node requestNode = new Node(requestNodeBytes);
        if (needsRelay(requestNode)) {
            byte[] forwardPacket = new byte[47];
            forwardPacket[0] = 0x07;
            System.arraycopy(requestNode.getBytes(), 0, forwardPacket, 1, 20);
            System.arraycopy(me.getBytes(), 0, forwardPacket, 21, 20);
            System.arraycopy(p.getAddress().getAddress(), 0, forwardPacket, 41, 4);
            System.arraycopy(ByteBuffer.allocate(2).putShort((short) (p.getPort() - Short.MAX_VALUE)).array(), 0, forwardPacket, 45, 2);
            /* Will send a packet (doing some routing first) */
            relay(forwardPacket, requestNode);
        } else {
            List<Neighbour> references = routing.getClosest(requestNode, 7);
            byte[] answerPacket = new byte[2 + references.size() * 26];
            answerPacket[0] = 0x06;
            answerPacket[1] = (byte) references.size();
            for (int i = 0; i < references.size(); i++) {
                Neighbour n = references.get(i);
                System.arraycopy(n.getBytes(), 0, answerPacket, 2 + i * 26, 20);
                System.arraycopy(n.getAddress().getAddress().getAddress(), 0, answerPacket, 22 + i * 26, 4);
                System.arraycopy(ByteBuffer.allocate(2).putShort((short) (n.getAddress().getPort() - Short.MAX_VALUE)).array(), 0, answerPacket, 26 + i * 26, 2);
            }
            /* That's where my stack trace ends and where the packet gets onto an event bus (100% working properly) */
            bus.publish(new OutgoingPacket(answerPacket, p.getSocketAddress()));
        }

        byte[] quickResponse = new byte[21];
        quickResponse[0] = 0x02;
        System.arraycopy(me.getBytes(), 0, quickResponse, 1, 20);

        /* see last comment */
        bus.publish(new OutgoingPacket(quickResponse, p.getSocketAddress()));
    } catch (InvalidNodeException e) {
    }
}
}

As I said, it's possible that multiple outgoing packets are on the event bus when multiple packet handlers are invoked by my packet handler pools.

Answer 1

Just ran into the same issue a few years later.

In my case the failure was due to conntrack filling up, which isn't specific to Java and has been reported elsewhere (1, 2). One simple way around the problem is to make all udp traffic bypass conntrack:

iptables -I PREROUTING -t raw -p udp -j NOTRACK
iptables -I OUTPUT -t raw -p udp -j NOTRACK

As you can see below, the raw PREROUTING and OUTPUT chains are right before conntrack:

If you don't want to bypass conntrack completely, you can instead tweak its configuration in sysctl to allow more connections.