Indy FTP, large files and NAT routers

Question

I have been using Indy to transfers files via FTP for years now but have not been able to find a satisfactory solution for the following problem.

When a user is uploading a large file, behind a router, sometimes the following happens: the file is uploaded OK, but in the mean time the command channel gets disconnected because of a timeout. Normally this doesn't happens with a direct connection to the server, because the server "knows" that a transfer is being taking place on the data channel. Some routers are not aware of this, though and the command channel is closed.

Many programs send a NOOP command periodically to keep the command channel alive even if this is not part of the standard FTP specification. My question: how do I do that? Do I send the NOOP command in the OnWork event? Does this cause any collateral damage in some way, like, do I need to process some response? How do I best solve this problem?

Answer 1

We use several approaches to handle this: (1) Enable TCP/IP keepalives on the control channel during transfers, and (2) recover gracefully after the connection drops, (3) support resuming broken transfers.

A lot of FTP clients will send NOOPs while everything is idle, but I don't know if any that send them during a data transfer because you would need to handle the responses in that case, and many servers won't send them back until the data is finished transferring.

1. Indy 10.5.8 (Delphi XE2) supports TCP/IP keep alives natively. Just use TIdFTP's NATKeepAlive property.

   For previous releases, assign the OnDataChannelCreate/OnDataChannelDestroy events:

   const
     KeepAliveIdle = 2 * SecsPerMin;
     KeepAliveInterval = 2 * SecsPerMin;
     IOC_VENDOR = $18000000;
     SIO_KEEPALIVE_VALS = DWORD(IOC_IN or IOC_VENDOR or 4);
   
   type
     tcp_keepalive = record
       onoff: u_long;
       keepalivetime: u_long;
       keepaliveinterval: u_long;
     end;
   
   procedure TFtpConnection.DataChannelCreated(Sender: TObject;
     ADataChannel: TIdTCPConnection);
   var
     Socket: TIdSocketHandle;
     ka: tcp_keepalive;
     Bytes: DWORD;
   begin
     // Enable/disable TCP/IP keepalives.  They're very small (40-byte) packages
     // and will be sent every KeepAliveInterval seconds after the connection has
     // been idle for KeepAliveIdle seconds.  In Win9x/NT4 the idle and timeout
     // values are system wide and have to be set in the registry;  the default is
     // idle = 2 hours, interval = 1 second.
     Socket := (FIdFTP.IOHandler as TIdIOHandlerSocket).Binding;
     if Win32MajorVersion >= 5 then begin
       ka.onoff := 1;
       ka.keepalivetime := KeepAliveIdle * MSecsPerSec;
       ka.keepaliveinterval := KeepAliveInterval * MSecsPerSec;
       WSAIoctl(Socket.Handle, SIO_KEEPALIVE_VALS, @ka, SizeOf(ka), nil, 0, @Bytes,
         nil, nil)
     end
     else
       Socket.SetSockOpt(Id_SOL_SOCKET, Id_SO_KEEPALIVE, Id_SO_True)
   end;
   
   procedure TFtpConnection.DataChannelDestroy(ASender: TObject;
     ADataChannel: TIdTCPConnection);
   var
     Socket: TIdSocketHandle;
   begin
     Socket := (FIdFTP.IOHandler as TIdIOHandlerSocket).Binding;
     Socket.SetSockOpt(Id_SOL_SOCKET, Id_SO_KEEPALIVE, Id_SO_False)
   end;
   

2. To recover gracefully if the file has been successfully transferred, just reconnect at the end and do a SIZE or LIST to get the file size. If they match then the file was successfully transferred and you don't need to do anything else. If the server supports it you can also send an XCRC command to get the CRC value, so you can compare it to the local file.

3. If you want to be really robust, you can also check TIdFTP.CanResume. If it's set the server supports the REST command, so you can pick up transferring where you left off, by passing true to the AResume parameter of TIdFTP.Get/Put.