gsm network card port mapping with udp sockets in python

Question

I have a python server listening for udp packets and responding back to the sender. In the clients I am creating connections back to the server in the same port. The clients work over gsm network and their ip's are not public so there is some port mapping done by the network, so the server can contact them back, I assume.

What happens is that sometimes the port mapping must be getting some problems, because the opened socket is still being able to communicate client->server but not the reverse.

It got to a point where I have two clients with the same pair (ip,address). In the server, when I receive messages from them, using sock.recvfrom, I know clearly that the messages are from two different clients taking into account their content, but the address is the same.

I've put a nc -l on the server and run nc on each client and its clear that both gsm cards are sharing the same external ip, so the last one to open the port, gets its mapping and the older one is working, assuming that it still has the mapping (without having it anymore).

I didn't make the original code I'm using so I can't really explain why a specific given port is used for the clients reaching out the server, but it seems to me that even with random ports the problem would persist because the network is remapping the port after some time (I imagine it should be after some event like gsm signal loss).

Anyone has any idea of what can I do to avoid this situation?

Answer 1

Dynamic NAT (Name Address Translation)

In Dynamic NAT, the internal side must initiate the connection and the NAT'ing device is assigning a state from an internal packet's 5 tuple:

{protocol, internal_ip, internal_port, foreign_ip, foreign_port}

to build a state entry that (assuming only one external IP) must match a less discriminate amount of external traffic using the remaining 4 of the 5 tuple:

 {protocol, _, internal_port, foreign_ip, foreign_port}

Looking at the external 4 tuple, and your protocol/configuration, you may find that there can never be more than one state entry for this service on each foreign NAT device without a conflict.

Additionally, since UDP has no handshake, every new packet from the client side can equally initiate a new state. So a problem like you are describing would really stem from conflicting existing state and not a lack of state as it could for a TCP connection that remained idle past its Dynamic NAT entry.

Details like whether the NAT device discriminates on all 4 possible parts of the tuple for external traffic and how long a state entry lasts are usually configurable. How it handles overlapping states can vary (most recent created, most recent used, blocking of new conflicting states) but usually isn't configurable. Still, assuming a diverse group of NAT devices (i.e. there is not one dominant telco with a single configuration,) you will get the good average result if you ensure there are at least multiple non-conflicting external 4-tuples available to your clients behind well configured NAT devices.

• If the clients use DNS to get the server, then round-robin on as many IPs as you can use. Every time you double the IPs the server could have in the tuple, you have twice as many non-conflicting states on every NAT device.

• At every opportunity where you can change half of clients to send to a different server port or choose a different client port you are multiplying the number of non-conflicting states as well, except in terribly configured NAT devices. Since ports aren't scarce this is much more practical than getting a block (or regional blocks) of IPs.

• A last possibility is to add complexity to your application level protocol. For example, having the server only send a new port # on the default port. I would be careful that you don't end up slowly reimplementing an existing service such as STUN.

caveat for random ports

If the clients have a set port that is not the same as the server, then some predecessor may have encountered significant filtering (wherever there is dynamic NAT there are all the prerequisites for stateful filtering, so NAT without filtering is rare.) In this case it may be best not to use random ports but maybe a chunk of a thousand or so pretty high in the address space. If the client has the same port as the server then the predecessor was just as likely confused or lazy in selecting "ports that match" and it may be fine to start with random high port binds to minimize collisions, then have clients fallback to the best tested when they can't otherwise connect. In either case you would have to look for places where the fixed clients port has become relied upon.

Less Likely Possibilities

1. If your server supports multiple client Apps or the client App is threaded, forks, etc it is also possible that these are really two sendmsgs from separate contexts on the same device. A 4 tuple in UDP is not necessarily exclusive as it is in TCP, but you most likely know enough about your systems and protocol to exclude this possibility.

2. If your clients are being directly allocated real addresses and continue to use them past their reallocation, I'd say either the OS is unaware and there should be a lot going wrong on these clients besides your service, or the OS is aware but has done something insufficient like mark them deprecated but accessible to your clients if they hold on to bound ports. I'm a little skeptical of this since it would require significant over-engineering in client's network management and under-engineering in the telco networks, so I think a small number of clients would show up this way while you would have other harder to describe bugs due to lost traffic, etc. Still I would consider:

   • Encourage clients not to reuse configured sockets or any of the get* lookups used that may be dependent on a local address.