How to use a second Elastic Network Interface on the same subnet

Question

When I spin up an Amazon EC2 CentOS 7 server in, say, availability zone us-east-1a, the server is automatically assigned a primary private IP address on eth0, such as 172.31.8.244/20 and a gateway of 172.31.0.1. If I then attach a second interface on eth1, I can specify the address, which needs to be within the 172.31.0.0/20 subnet (or one will be assigned to me automatically within that subnet). Eth1 will have the same gateway as eth0. Let's say I am assigned 172.31.12.121/20. I use the same security group on both eth0 and eth1, which allows SSH only in and everything out.

The problem is that when I try to SSH to eth0 from a different server, it works fine. But when I try to SSH to eth1 I get a timeout. ip addr and ip route show that both interfaces are up and have the correct routes. I can even SSH locally to eth1 and the /var/log/secure log shows the correct entries as when I SSH to eth0 bound to eth1. What do I need to do to be able to SSH to either interface from a different server?

Answer 1

The problem is asymmetric routing. A request to eth1 comes in eth1 and goes out eth0. The reply coming out on eth0 has a different IP address than in the request, and so it is dropped on the client side. The solution is to set up rules that allow responses to route through eth1.

First, make sure you have created an AMI of your server, because if you enter the wrong thing in following steps, you may lose all connectivity to the server and be unable to do anything but reboot it from the Amazon console web page.

Start off by setting the default route for each interface in separate tables:

ip route add default via 172.31.0.1 dev eth0 tab 1
ip route add default via 172.31.0.1 dev eth1 tab 2

To check those were properly added use:

ip route show table 1
ip route show table 2

Now you need to add rules that say to use the different tables depending on the source IP address:

ip rule add from 172.31.8.244/32 tab 1
ip rule add from 172.31.12.121/32 tab 2

You can check all of the rules with:

ip rule

You should now be able to connect to either IP address from a client machine. You can also use the bind option of SSH to connect from either interface on this server to a client machine:

ssh centos@client_ip_address -i mykey.pem  (uses the default, eth0)
ssh -b 172.31.12.121 centos@client_ip_address -i mykey.pem  (uses eth1)
ssh -b 172.31.8.244 centos@client_ip_address -i mykey.pem  (uses eth0)

You can use both interfaces to connect to other EC2 servers in the same availability zone and for any interface that has a Public IP assigned to it, you can connect to the outside world or to other EC2 servers in the same VPC, even if they are in different availability zones.

But what if you want to connect to other EC2 servers that are in the same VPC but different availability zones? In other words, servers in the same data center. The problem is that the Private IP address is masked at 20 bits, which confines you to one availability zone. So for datacenter us-east-1 you have:

us-east-1a:  172.31.0.0/20
us-east-1b:  172.31.16.0/20
us-east-1d:  172.31.48.0/20
us-east-1e:  172.31.32.0/20

To connect across availability zones in one VPC and in one datacenter you need a 16-bit mask. ip addr will show:

inet 172.31.12.121/20 brd 172.31.31.255 scope dynamic eth1

If losf -n | egrep 172.31.12.121 shows you that this address is not in use you can add the new mask and delete the old. Note that the broadcast address has to change at the same time the mask changes:

ip addr add 172.31.12.121/16 dev eth1 brd 172.31.255.255
ip addr del 172.31.12.121/20 dev eth1

Now you should be able to connect from an EC2 server in availability zone A to another host in availability zone B, so long as they are in the same VPC, even if they do not have Public IP addresses.

Troubleshooting:

If you are having problems, try resetting both interfaces, which will remove any manual twiddling you have done. First copy /etc/sysconfig/network-scripts/ifcfg-eth0 to /etc/sysconfig/network-scripts/ifcfg-eth1, editing the second file to change the DEVICE from eth0 to eth1. Then add a line to /etc/sysconfig/network which says GATEWAYDEV=eth0. Finally, run /etc/init.d/network restart (no, it should not disconnect you). Then start over with the above commands.