Ping via linux routing table not working [OR] how is this expected?

Question

Long question in short:

Ping over r1-r4-r2 path works using 10.0.1.* or 10.0.2.* IP addresses, but fails if we alter the path to r1-r3-r2 using 1.0.0.* or 1.0.1.* IP addresses for the exactly same packets (except for the fact that packets' src and dst IP fields are changed from 10.* to 1.* and vice-versa at s1 and s2 respectively). Why?

---

Question in detail:

I have a small topology as below

 h1 -- s1 -- r1 -- r4 -- r2 -- s2 -- h2
              \         /
               \       /
                \     /
                  r3

The s's are OpenvSwitch instances while r's are Ubuntu 16 Linux machines.

IP Addresses are:

h1-eth0 - 10.0.1.10/24
s1      - 10.0.1.50/24
h2-eth0 - 10.0.2.10/24
s2      - 10.0.2.50/24
r1-eth0 - 10.0.1.1/24
r1-eth1 - 10.0.11.2/24
r1-eth2 - 10.0.12.2/24
r2-eth0 - 10.0.2.1/24
r2-eth1 - 10.0.13.1/24
r2-eth2 - 10.0.5.1/24
r3-eth0 - 10.0.12.1/24
r3-eth1 - 10.0.5.2/24
r4-eth0 - 10.0.11.1/24
r4-eth1 - 10.0.13.2/24

As you can see, there are two similar paths between r1 and r2. I add the following static entries.

r1

sudo ip route add 10.0.2.0/24 via 10.0.11.1

r2

sudo ip route add 10.0.1.0/24 via 10.0.13.2

r4

sudo ip route add 10.0.1.0/24 via 10.0.11.2
sudo ip route add 10.0.2.0/24 via 10.0.13.1

The ping between h1 and h2 works as expected. Now, since the switches are OVS (and thus are OpenFlow-enabled) I install entries in s1 to map the destination IPs to a different subnet.

i.e. the IP 10.0.1.10 would be mapped to 1.0.0.10 while the IP 10.0.2.10 would be mapped to 1.0.1.10 when such a packet is received at s1, while the destination IPs would be mapped back to original at s2.

(I have checked that these entries are indeed correct and are working as expected. Also I have added this entry only to match ICMP packets). Similar procedure would be done when ping reply is sent by h1.

Along with these, I install static routes in the routers to route these IPs.

r1

sudo ip route add 1.0.0.0/24 via 10.0.1.50
sudo ip route add 1.0.1.0/24 via 10.0.12.1

r2

sudo ip route add 1.0.0.0/24 via 10.0.5.2
sudo ip route add 1.0.1.0/24 via 10.0.2.50

r3

sudo ip route add 1.0.0.0/24 via 10.0.12.2
sudo ip route add 1.0.1.0/24 via 10.0.5.1

Now if I ping h1 from h2, the packet starts with destination IP 10.0.1.10, which is mapped to 1.0.0.10 at s2, r2 routes this and sends it to r3, r3 routes it and sends to r1. But r1, even after receiving the packet at one interface and having the matching entry in the Linux routing table does not route and forward packet.

Even ip route get outputs the correct port to which the packet should be forwarded. There are no firewall entries in ip tables as well.

---

Some additional information:

• If I change the newly added routing entries to use the original path of r1-r4-r2 (i.e., we route on this path with mapped ip's) , it behaves as expected and the ping works as expected.

• Alternatively, if I change the old routing entries for 10.0.2.0/24 in r1 and
  10.0.1.0/24 in r2 (which now ideally don't even have to be matched by the new packets as their Destination IPs are in 1.0.0.* range or 1.0.1.* only) to use the new path r1-r3-r4 along with this mapped-IP packets, the ping between r2 and r1 works as expected.

Details that may be required:

The final routing tables are as follows:

r1

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.11.1       0.0.0.0         UG    0      0        0 eth1
1.0.0.0         10.0.1.10       255.255.255.0   UG    0      0        0 eth0
1.0.1.0         10.0.12.1       255.255.255.0   UG    0      0        0 eth2
10.0.1.0        0.0.0.0         255.255.255.0   U     1      0        0 eth0
10.0.2.0        10.0.11.1       255.255.255.0   UG    0      0        0 eth1
10.0.11.0       0.0.0.0         255.255.255.0   U     1      0        0 eth1
10.0.12.0       0.0.0.0         255.255.255.0   U     1      0        0 eth2

r2

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.13.2       0.0.0.0         UG    0      0        0 eth1
1.0.0.0         10.0.5.2        255.255.255.0   UG    0      0        0 eth1
1.0.1.0         10.0.2.50       255.255.255.0   UG    0      0        0 eth0
10.0.1.0        10.0.13.2       255.255.255.0   UG    0      0        0 eth1
10.0.2.0        0.0.0.0         255.255.255.0   U     1      0        0 eth0
10.0.5.0        0.0.0.0         255.255.255.0   U     1      0        0 eth2
10.0.13.0       0.0.0.0         255.255.255.0   U     1      0        0 eth1

r3

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.0.5.1        0.0.0.0         UG    0      0        0 eth1
1.0.0.0         10.0.12.2       255.255.255.0   UG    0      0        0 eth0
1.0.1.0         10.0.5.1        255.255.255.0   UG    0      0        0 eth1
10.0.1.0        10.0.12.2       255.255.255.0   UG    0      0        0 eth0
10.0.2.0        10.0.5.1        255.255.255.0   U     1      0       0 eth1
10.0.5.0        0.0.0.0         255.255.255.0   U     1      0        0 eth1
10.0.12.0       0.0.0.0         255.255.255.0   U     1      0        0 eth0

r4

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 eth4
1.0.0.0         10.0.11.2       255.255.255.0   UG    0      0        0 eth0
1.0.1.0         10.0.13.1       255.255.255.0   UG    0      0        0 eth1
10.0.1.0        10.0.11.2       255.255.255.0   UG    0      0        0 eth0
10.0.2.0        10.0.13.1       255.255.255.0   UG    0      0        0 eth1
10.0.11.0       0.0.0.0         255.255.255.0   U     1      0        0 eth0
10.0.13.0       0.0.0.0         255.255.255.0   U     1      0        0 eth1
192.168.0.0     0.0.0.0         255.255.255.0   U     1      0        0 eth4

Note: 192.168.0.* is a subnet connected to outside Internet.

What do you think is the problem ? I am completely baffled looking at this problem.

Answer 1

The behavior of Linux routing here was as expected.

The flag for reverse path filter
i.e. /proc/sys/net/ipv4/conf/<interfacename>/rp_filter was turned ON (by setting value to 1) by default.

Reverse path filter is provided as a security feature of Linux kernel. A common example is private IP space escaping onto the Internet. If you have an interface with a route of 195.96.96.0/24 to it, you do not expect packets from 212.64.94.1 to arrive there. So kernel drops such a packet if the flag is set to 1.

More formally,

Reverse path filtering is a mechanism adopted by the Linux kernel to check whether the source IP address of the packet that is been received is routable.

So in other words, when a machine with reverse path filtering enabled receives a packet, the machine will first check whether the source of the received packet is reachable through the interface it came in.

• If it is routable through the interface which it came, then the machine will accept the packet.
• If it is not routable through the interface, which it came, then the machine will drop that packet.

Latest kernels provide one more option value of 2. This option is slightly more liberal in terms of accepting traffic.

If the received packet's source address is routable through any of the interfaces on the machine, the machine will accept the packet.

Answer 2

To make that work, use this on all machines:

# for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do
>  echo 0 > $i 
> done

Or make the following entry in your /etc/sysctl.conf

net.ipv4.conf.all.rp_filter = 0

rp_filter will filter packets in the three modes: 0 disabled, 1 strict and 2 loose.

Example

Client A - 192.168.2.10 - connected to router via eth0

Router
   eth0   - 192.168.2.150
    routes - 192.168.2.0/24
   eth1   - 10.42.43.1
    routes - 10.42.43.0/24

Note: No default route

Client C - 10.42.43.50 - connected to router via eth1

With this setup and rp_filter on the router set to “loose mode” (2) a packet on eth0 from 1.2.3.4 to 10.42.43.50 will be blocked.

With rp_filter on the router set to “strict mode” (1) a packet on eth0 from source address 10.42.43.2 will be blocked.

When set to “disabled” (0) both packets would go through.

Answer 3

First off your topology details is incomplete you are missing r3 and r4 details but they can be inferred.

Instead of trying to troubleshoot your issue I'm just going to try to explain what needs to happen. However it would be much easier if you just used a routing protocol like OSPF which is designed to make this easy so you don't have to do it by hand.

Each device that is routing needs to know how to get to every other subnet if its to be accessible. So this means you can either add in default routes (ie routes that match 0.0.0.0/0) or you can enter in each subnet with corresponding next-ip into each router (see below). Usually you do not need to add routes for subnets that are connected (IE you have an ip on that router in that subnet)

R1 routes

10.0.13.0/24 -> 10.0.11.1
10.0.5.0/24 -> 10.0.11.1
10.0.2.0/24 -> 10.0.11.1

R2 routes

10.0.1.0/24 -> 10.0.13.2
10.0.12.0/24 -> 10.0.13.2
10.0.11.0/24 -> 10.0.13.2

R3 routes

10.0.1.0/24 -> 10.0.12.2
10.0.11.0/24 -> 10.0.12.2
10.0.13.0/24 -> 10.0.5.1
10.0.2.0/24 -> 10.0.5.1

R4 Routes

10.0.1.0/24 -> 10.0.11.2
10.0.12.0/24 -> 10.0.11.2
10.0.2.0/24 -> 10.0.13.1
10.0.5.0/24 -> 10.0.13.1

For devices H1, S1, H2, and S2 they should have a default route that points to the gateway 10.0.1.1 and 10.0.2.1.