Attempting to upload to FTP: System.Net.WebException: System error

Question

I have an API that takes in XML and ultimately uploads files based on information in the XML. The uploads are on a schedule (also from XML), and I have tested everything surrounding it and know it works.

I am getting an error about 40% of the time on the first file that I attempt to upload in each time cycle (time cycle = 45 minutes for some files, 30 minutes for others).

Here is my code for the upload:

try {
    LoggerFTP.Log("Uploading file: " + filename, false);

    // Create the request.
    FtpWebRequest request = (FtpWebRequest)WebRequest.Create(appSettingsFTP.ftpUrl + @"/" + filename);
    request.Method = WebRequestMethods.Ftp.UploadFile;
    request.Timeout = 6000000; //set to 100 minutes
    //request.Timeout = -1; //set to infinite

    // Add the login credentials.
    request.Credentials = new NetworkCredential(appSettingsFTP.ftpLogin, appSettingsFTP.ftpPassword);

    // Grab the file contents.
    StreamReader sourceStream = new StreamReader(appSettingsFTP.uploadFileDirectory + filename);
    byte[] fileContents = Encoding.UTF8.GetBytes(sourceStream.ReadToEnd());
    sourceStream.Close();
    request.ContentLength = fileContents.Length;

    // Copy the file contents to the outgoing stream.
    Stream requestStream = request.GetRequestStream();
    requestStream.Write(fileContents, 0, fileContents.Length);
    requestStream.Close();

    FtpWebResponse response = (FtpWebResponse)request.GetResponse();
    //Logger.Log(filename.ToString() + " " + "Upload Complete, Status: " + response.StatusCode + " " + response.StatusDescription, false);
    //Took response.StatusDescription out because it appears to be creating extra line feeds.
    LoggerFTP.Log(filename.ToString() + " " + "Upload Complete, Status: " + response.StatusCode, false);
}
catch (Exception ex) {
    LoggerFTP.Log(ex.ToString(), false);
}

I have researched the issue and saw something online about it potentially being a speed thing. Like, there is a timeout. But I have my timeout set to 100 minutes for my FtpWebRequest, so it can't possibly be that? I don't know. This is also running as a service so it is hard to test this aspect of the code.

Here is the exception that is getting logged in my logger (e.ToString):

System.Net.WebException: System error. ---> System.Net.InternalException: System error.
   at System.Net.PooledStream.PrePush(Object expectedOwner)
   at System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)
   at System.Net.FtpWebRequest.FinishRequestStage(RequestStage stage)
   at System.Net.FtpWebRequest.SyncRequestCallback(Object obj)
   at System.Net.FtpWebRequest.RequestCallback(Object obj)
   at System.Net.CommandStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.Stream.Dispose()
   at System.Net.ConnectionPool.Destroy(PooledStream pooledStream)
   at System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)
   at System.Net.FtpWebRequest.AttemptedRecovery(Exception e)
   at System.Net.FtpWebRequest.SubmitRequest(Boolean async)
   --- End of inner exception stack trace ---
   at System.Net.FtpWebRequest.GetRequestStream()
   at CPMainSpringAPIExportsSC.UploadFTP.FTPUploadMethod(String viewname, String filename)

Answer 1

I am getting exactly the same stack trace in an SSIS package attempting to FTP over SSL. It works great without SSL, but as soon as I enable SSL, it blows up.

    System.Net.WebException: System error. --->
    System.Net.InternalException: System error.    at
    System.Net.PooledStream.PrePush(Object expectedOwner)    at
    System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)    at
    System.Net.FtpWebRequest.FinishRequestStage(RequestStage stage)    at
    System.Net.FtpWebRequest.SyncRequestCallback(Object obj)    at
    System.IO.Stream.Close()    at
    System.Net.ConnectionPool.Destroy(PooledStream pooledStream)    at
    System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse)    at
    System.Net.FtpWebRequest.AttemptedRecovery(Exception e)    at
    System.Net.FtpWebRequest.SubmitRequest(Boolean async)
    --- End of inner exception stack trace ---    at
    System.Net.FtpWebRequest.CheckError()    at
    System.Net.FtpWebRequest.GetRequestStream()    at
    ST_0ff7348de65a468bb358ab0206e3721f.ScriptMain.Main() in c:\Users\Stephens\AppData\Local\Temp\Vsta\e664c8a71bb647ff9e9dc6ac32d7b615\ScriptMain.cs:line 155 at 
    System.Net.FtpWebRequest.CheckError()    at
    System.Net.FtpWebRequest.GetRequestStream()    at
    ST_0ff7348de65a468bb358ab0206e3721f.ScriptMain.Main() in c:\Users\Stephens\AppData\Local\Temp\Vsta\e664c8a71bb647ff9e9dc6ac32d7b615\ScriptMain.cs:line 155

Because the error is so generic, I decided to look at the .NET source to see if I can catch a clue about what is breaking. If you go here:

http://labs.developerfusion.co.uk/SourceViewer/browse.aspx?assembly=SSCLI&namespace=System.Net#{%22pageClientState%22%3A%22type-2844%2Ccsharp%22}

and skip down to line 281, you will see the definition for internal void PrePush(object expectedOwner) which is what is executing when the exception happens. Here is what it looks like:

    internal void PrePush(object expectedOwner)
    {
        lock (this) {
            //3 // The following tests are retail assertions of things we can't allow to happen.
            if (null == expectedOwner) {
                if (null != m_Owner && null != m_Owner.Target)
                    throw new InternalException();
                // new unpooled object has an owner
            }
            else {
                if (null == m_Owner || m_Owner.Target != expectedOwner)
                    throw new InternalException();
                // unpooled object has incorrect owner
            }

            m_PooledCount++;

            if (1 != m_PooledCount)
                throw new InternalException();
            // pushing object onto stack a second time
            if (null != m_Owner)
                m_Owner.Target = null;
        }
    }

Eventually I discovered that FtpWebRequest only supports explicit FTP (port 21) and not implicit FTP (port 990). This was definitively stated here:

Does .NET FtpWebRequest Support both Implicit (FTPS) and explicit (FTPES)?

Anyway, in my case, it was a firewall issue. Originally we configured for implicit FTP, which was ports 989, 990, 49152-65535 (per the vendor's tech staff). I checked with my network guy and we opened up ports 20, 21, 989, 990 and 40000-655535 for explicit and things worked like a champ afterwards.

However, in your case, you appear to have some connection pool excitement going on. There is a good post on this subject here:

How to improve the Performance of FtpWebRequest?

You might want to take a look at mucking around with your connection pool set up and see if you can make some progress. Hope this helps!

Regards,

Stuart