Weird TCP issue with Amazon S3 [closed]

Question

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here.

More info: https://stackoverflow.com/faq

I am having a weird TCP issue that makes no sense at all. I am tearing my hair out.

When downloading a file from Amazon S3 (and only Amazon S3- every other site works fine) some percentage of the time the connections dies. This only happens from our servers in Seattle with a web server behind a NAT router. Connecting directly from the router works fine. From our servers here in Victoria everything works fine, and we can't replicate this at all.

• Web server in Seattle (tried 10 different servers) -> SNAT router (2 different routers; old kernels and new kernels) -> Amazon S3 = doesn't work ~10% of the time
• Web server in Seattle -> SNAT router -> any other website = works
• Router box -> Amazon S3 = works
• Web server in Victoria -> SNAT router in Victoria -> Amazon S3 = works

Things I've tried:

• Disabling window scaling
• Lowering the window size
• upgrading the router to the newest kernel
• Disabling the firewall entirely

In the failure case Amazon (or something along the way) sends us a 868 length packet instead of the expected 1500 byte packet. The server upon seeing the packets replies with a RST packet (error has occurred) and the connection is torn down.

Any help would be greatly appreciated!

Below are two TCP dumps-

------------ Begin Error Case TCP Dump -------------
tcpdump -r /tmp/s3doesntwork-died 
reading from file /tmp/s3doesntwork-died, link-type EN10MB (Ethernet)
19:58:42.542189 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: S 193799772:193799772(0) win 5840 <mss 1460,sackOK,timestamp 760821159 0,nop,wscale 5>
19:58:42.544115 IP 189-81.amazon.com.www > locum.sparklit.com.39491: S 3148664267:3148664267(0) ack 193799773 win 8190 <mss 1460,nop,wscale 6,nop,nop,sackOK>
19:58:42.545176 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 1 win 183
19:58:42.545184 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: P 1:212(211) ack 1 win 183
19:58:42.548113 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . ack 212 win 916
19:58:42.558108 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 1:1461(1460) ack 212 win 916
19:58:42.558117 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 1461:2921(1460) ack 212 win 916
19:58:42.558123 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 2921:4381(1460) ack 212 win 916
19:58:42.558128 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 4381:5841(1460) ack 212 win 916
19:58:42.559108 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 5841:7301(1460) ack 212 win 916
19:58:42.559118 IP 189-81.amazon.com.www > locum.sparklit.com.39491: . 7301:8129(828) ack 212 win 916
19:58:42.559138 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: R 193799984:193799984(0) win 0
19:58:42.559169 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 1461 win 274
19:58:42.559176 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 2921 win 365
19:58:42.559180 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 4381 win 457
19:58:42.559188 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 5841 win 548
19:58:42.560167 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: . ack 7301 win 639
19:58:45.308618 IP locum.sparklit.com.39491 > 189-81.amazon.com.www: F 212:212(0) ack 7301 win 639
19:58:45.310512 IP 189-81.amazon.com.www > locum.sparklit.com.39491: R 3148671568:3148671568(0) win 8201

Update: Traceroute to Amazon

1  64.34.33.195 (64.34.33.195)  0.795 ms  0.775 ms  0.762 ms
2  six01-sea4.amazon.com (206.81.80.147)  0.746 ms  0.732 ms  0.715 ms
3  72.21.222.183 (72.21.222.183)  2.657 ms  2.651 ms  2.638 ms
4  72.21.222.179 (72.21.222.179)  2.618 ms  2.604 ms  2.591 ms
5  * * *
6  * * *
7  * * *

Update: The offending incoming packet that causes the RST to be sent

19:58:42.559118 IP (tos 0x0, ttl 57, id 18905, offset 0, flags [DF], proto TCP (6), length 868) 189-81.amazon.com.www >     locum.sparklit.com.39491: ., cksum 0x9f6e (correct), 7301:8129(828) ack 212 win 916
0x0000:  4500 0364 49d9 4000 3906 05d7 cfab bd51  [email protected]
    0x0010:  4022 21c5 0050 9a43 bbac ea50 0b8d 2730  @"!..P.C...P..'0
    0x0020:  5010 0394 9f6e 0000 7c75 a901 cd57 0718  P....n..|u...W..
    0x0030:  a786 e954 4160 3734 f563 5029 e7ad 48a7  ...TA`74.cP)..H.
    0x0040:  34c0 b11b 75a2 a341 c1e6 8aab a03c 31ee  4...u..A.....<1.
    0x0050:  1496 c9ef df22 aadc b87e 8431 fc2a dcd6  ....."...~.1.*..
    0x0060:  e72d 8cf7 aa92 5b12 4923 3f51 50cd 5195  .-....[.I#?QP.Q.
    0x0070:  7910 6ce4 0fc0 d63c f115 276b b7e7 5bf7  y.l....<..'k..[.
    0x0080:  508e d8fa d655 c5b3 1638 3cd6 6cd1 198c  P....U...8<.l...
    0x0090:  1c7f 1b7e 59a4 4370 9c87 523d 0ae2 adb4  ...~Y.Cp..R=....
    0x00a0:  4d8e 7ad5 7954 c6ac 79e3 9e05 4148 6c97  M.z.yT..y...AHl.
    0x00b0:  b711 f262 47fa 363f 9d52 bcd4 a58e 177f  ...bG.6?.R......
    0x00c0:  b33b 1033 b530 7351 d8eb 29a9 dbcf 2c6b  .;.3.0sQ..)...,k
    0x00d0:  b161 99e0 3e67 192b 8c9f a735 89ad f886  .a..>g.+...5....
    0x00e0:  4d3a ff00 462e b0ad 8dec 8f04 9bdb 9121  M:..F..........!
    0x00f0:  4263 0ac4 81c9 18e7 ae6a 1a65 d8db d3ee  Bc.......j.e....
    0x0100:  3722 b608 cf24 1182 2ba1 b39b 728a f29b  7"...$..+...r...
    0x0110:  df1e db68 9af5 b69a e51e 5923 2ed0 29fd  ...h......Y#..).
    0x0120:  f0e4 6303 f8bb f4ae ef44 d5e3 bc67 8f70  ..c......D...g.p
    0x0130:  5b88 8299 a30d 931e e1c0 38fc 6bca a917  [.........8.k...
    0x0140:  7b9d 8a2d 2bb3 5ef7 9e0d 625e c59c d6dc  {..-+.^...b^....
    0x0150:  e778 cd67 5c26 ecd2 a64b 6737 345b 643d  .x.g\&...Kg74[d=
    0x0160:  6a3d b57e f22f 98d5 455e b5da b546 0c60  j=.~./..E^...F.`
    0x0170:  5c51 5281 8a69 4e69 888c af15 0b0a b240  \QR..iNi.......@
    0x0180:  c540 c334 0101 1935 04b5 6996 a091 7ad0  [email protected].
    0x0190:  2676 ba0d 8bbc 30cc 3385 c7f2 a835 ab19  &v....0.3....5..
    0x01a0:  2e0b 003a d75b e17b 453a 03b6 070a a7f4  ...:.[.{E:......
    0x01b0:  a6cf 671e 7270 6b59 ae5e 5f43 28eb 73cd  ..g.rpkY.^_C(.s.
    0x01c0:  53c2 8f34 9c8e 3e95 d268 fe0f 4864 4661  S..4..>..h..HdFa
    0x01d0:  9008 2702 b75e 782d 81e0 0fc2 b3ae bc51  ..'..^x-.......Q
    0x01e0:  0c00 e080 477a 9bb6 3b1f 08fc 7efd 943e  ....Gz..;...~..>
    0x01f0:  26eb ff00 163c 4bad 695e 1b97 51d2 2f6e  &....<K.i^..Q./n
    0x0200:  4cb6 f35b dca3 7c98 1d46 ec8f a62b c675  L..[..|..F...+.u
    0x0210:  4fd9 dfc7 1a4b b2df e812 da63 fe7e 0b2f  O....K.....c.~./
    0x0220:  eb8a fd38 bff1 d055 386f c735 cf5f f8fa  ...8...U8o.5._..
    0x0230:  7705 44ac 54f6 2723 f2ae d862 2a25 6b1c  w.D.T.'#...b*%k.
    0x0240:  f2a5 06ee 7e6b a7c1 bd79 f2a2 2b1d c3b1  ....~k...y..+...
    0x0250:  ba5c d759 e03f 849a e787 3564 d6af 2d6d  .\.Y.?....5d..-m
    0x0260:  db4e b7fd d48f 0ca1 8866 1c0e dcd7 db37  .N.......f.....7
    0x0270:  b2d9 eb2e 4dd6 9767 704f 5692 d949 fcf1  ....M..gpOV..I..
    0x0280:  53da 7c3c d0fc 45a3 ea3a 4cf6 3f67 b379  S.|<..E..:L.?g.y
    0x0290:  a197 cbb5 7311 0dcf 20d6 8f11 2d99 2a94  ....s.......-.*.
    0x02a0:  7a1e 69e1 9fda 77c5 7e0d d3ad 74f8 eeac  z.i...w.~...t...
    0x02b0:  e6b3 b641 1c71 df5b 6085 1db7 a907 f3cd  ...A.q.[`.......
    0x02c0:  7a3e 81fb 6b4c 42ff 0069 6831 4ebd dec6  z>..kLB..ih1N...
    0x02d0:  e837 fe3a c335 3d9f ec75 e01d 576b 5c47  .7.:.5=..u..Wk\G
    0x02e0:  ab12 c3a7 f683 8fe9 5a29 fb07 7c31 3f32  ........Z)..|1?2
    0x02f0:  dbea 81fd 4ea7 2e7f 98ae 37ec 99aa 53e8  ....N.....7...S.
    0x0300:  cb5a 97ed cfe0 ad1f 4d6b 99f4 bd56 79c1  .Z......Mk...Vy.
    0x0310:  ff00 8f48 d151 c8ef 8627 15f4 1681 acc5  ...H.Q...'......
    0x0320:  e21d 0b4e d56d e378 adef add2 e238 e5c6  ...N.m.x.....8..
    0x0330:  f556 1900 e38c d781 d97e c3bf 0ce0 71f6  .V.......~....q.
    0x0340:  9b1d 42ea 31fc 0fa9 cf8f fd0a be80 d2b4  ..B.1...........
    0x0350:  f834 8d3a d2c2 d13c ab4b 5896 1850 9270  .4.:...<.KX..P.p
    0x0360:  8a30 073e                                .0.>

Answer 1

So we finally figured it out-

It turns out that our router was injecting the RST packets because Netfilter declared the packet invalid. There is a setting that makes netfitler more liberal so I tried enabling it (inet.ipv4.netfilter.ip_conntrack_tcp_be_liberal=1) and the connection is no longer torn down.

However, I still haven't figured out why netfilter declared that packet as invalid. Enabling "ip_conntrack_log_invalid" doesn't cause anything to be printed in the log.