The underlying connection was closed: An unexpected error occurred on a receive

Question

I'm here because I have a problem while downloading some files through ftp protocol. It's weird because it occurs occasionally and even for the same single file.

Just a precision: I'm downloading very large files (from 500 Mo to 30Go)

Here are the kind of Exceptions returned by my function : (sorry it's in french)

System.Net.WebException: La connexion sous-jacente a été fermée : Une erreur inattendue s'est produite lors de la réception. à System.Net.FtpWebRequest.CheckError() à System.Net.FtpWebRequest.SyncRequestCallback(Object obj) à System.IO.Stream.Close() à System.Net.ConnectionPool.Destroy(PooledStream pooledStream) à System.Net.ConnectionPool.PutConnection(PooledStream pooledStream, Object owningObject, Int32 creationTimeout, Boolean canReuse) à System.Net.FtpWebRequest.FinishRequestStage(RequestStage stage) à System.Net.FtpWebRequest.SyncRequestCallback(Object obj) à System.Net.CommandStream.Abort(Exception e) à System.Net.CommandStream.CheckContinuePipeline() à System.Net.FtpDataStream.System.Net.ICloseEx.CloseEx(CloseExState closeState) à System.Net.FtpDataStream.Dispose(Boolean disposing) à System.IO.Stream.Close() à UtilityClasses.FTP.Download(String srcDirectoryPath, String file, String destDirectoryPath)

Here is the code used to download :

the download methods :

public Dictionary<string, object> Download(string srcDirectoryPath, string file, string destDirectoryPath, int attemptLimitNb, int delay)
    {
        int attemptNb = 0;
        bool downloadFailed;
        Dictionary<string, object> result = new Dictionary<string,object>();

        do
        { 
            attemptNb++;
            result = Download(srcDirectoryPath, file, destDirectoryPath);
            downloadFailed = result["downloadfailed"] != null;
            if (downloadFailed) Thread.Sleep((int)(1000 * delay));
        }
        while (downloadFailed && attemptNb < attemptLimitNb);
        return result;
    }

public Dictionary<string, object> Download(string srcDirectoryPath, string file, string destDirectoryPath)
    {
        Exception downloadFailed = null;
        Dictionary<string, object> result = new Dictionary<string, object>();
        bool fileFound = false;

        try
        {
            if (destDirectoryPath == null || !Directory.Exists(destDirectoryPath)) throw new Exception("Download destination path does not exist");
            if (file != null && file != "")
            {
                if (file.Contains("/"))
                {
                    throw new Exception("Invalid file name. Impossible to download");
                }

                Uri serverUri;
                if (srcDirectoryPath == null || srcDirectoryPath == "")
                {
                    serverUri = new Uri("ftp://" + this.Server + "/" + file);
                }
                else if (Regex.IsMatch(srcDirectoryPath, "^/.*$") || Regex.IsMatch(srcDirectoryPath, "^.*/$"))
                {
                    throw new Exception("Path must not start and end with '/'");
                }
                else
                {
                    serverUri = new Uri("ftp://" + this.Server + "/" + srcDirectoryPath + "/" + file);
                }

                if (serverUri.Scheme != Uri.UriSchemeFtp) throw new Exception("server URI Scheme does not match  FTP URI Scheme");

                if (Exists(srcDirectoryPath, file))
                {
                    fileFound = true;

                    FtpWebRequest downloadRequest = (FtpWebRequest)FtpWebRequest.Create(serverUri);
                    downloadRequest.Credentials = new NetworkCredential(UserName, Password);
                    downloadRequest.KeepAlive = false;
                    downloadRequest.Method = WebRequestMethods.Ftp.DownloadFile;
                    FtpWebResponse response = (FtpWebResponse)downloadRequest.GetResponse();

                    Stream responseStream = response.GetResponseStream();
                    FileStream fileStream = new FileStream(Path.Combine(destDirectoryPath, file), FileMode.Create);
                    byte[] buffer = new byte[2000];
                    int read = 0;
                    try
                    {
                        do
                        {
                            read = responseStream.Read(buffer, 0, buffer.Length);
                            fileStream.Write(buffer, 0, read);
                            fileStream.Flush();
                        }
                        while (read != 0);
                    }
                    catch (Exception e)
                    {
                        fileStream.Close();
                        responseStream.Close();
                        response.Close();
                        throw e;
                    }
                    fileStream.Close();
                    responseStream.Close();
                    response.Close();
                }
            }
        }
        catch (WebException webExcptn)
        {
            downloadFailed = webExcptn;
        }
        finally
        {
            result.Add("filefound", fileFound);
            result.Add("downloadfailed", downloadFailed);
        }

        return result;
    }

the Exists method :

public bool Exists(string srcPath, string elementName)
    {
        if (elementName == null || elementName == "")
        {
            return false;
        }

        Uri serverUri;
        bool res = false;

        if (srcPath == null || srcPath == "")
        {
            serverUri = new Uri("ftp://" + this.Server);
        }
        else if (Regex.IsMatch(srcPath, "^/.*$") || Regex.IsMatch(srcPath, "^.*/$"))
        {
            throw new Exception("Path must not start and end with '/'");
        }
        else
        {
            serverUri = new Uri("ftp://" + this.Server + "/" + srcPath);

        }
        if (serverUri.Scheme != Uri.UriSchemeFtp) throw new Exception("server URI Scheme does not match  FTP URI Scheme");

        FtpWebRequest listingRequest = (FtpWebRequest)FtpWebRequest.Create(serverUri);
        listingRequest.Credentials = new NetworkCredential(UserName, Password);
        listingRequest.KeepAlive = false;
        listingRequest.Method = WebRequestMethods.Ftp.ListDirectory;
        FtpWebResponse response = (FtpWebResponse)listingRequest.GetResponse();

        Stream responseStream = response.GetResponseStream();
        StreamReader streamReader = new StreamReader(responseStream);
        string ftpElementName;
        do
        {
            ftpElementName = Path.GetFileName(streamReader.ReadLine());
            if (ftpElementName == null) break;
            else
            {
                string pattern = "^" + elementName.Replace("[", "\\[").Replace("]", "\\]").Replace("+", "[+]").Replace(".", "[.]") + "$";
                if (Regex.IsMatch(ftpElementName, pattern, RegexOptions.IgnoreCase))
                {
                    res = true;
                }
            }
        }
        while (ftpElementName != null && !res);
        streamReader.Close();
        responseStream.Close();
        response.Close();

        return res;
    }

Maybe it's a timeout problem, but i don't really know. I searched for a long time for an answer but without success. Maybe some of you will have a solution.

///

EDIT : Some progress :

I have tested my code in debug mode with VS and in fact the Exception above is the consequence of a previous one. (I Couldn't know that because I only wrote the last Exception returned in a log file)

Here is the original exception :

Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

The second exception is caused by this part of the Download method code :

catch (Exception e)
{
    fileStream.Close();
    responseStream.Close();  // <<<<<<<<<<<<<<
    response.Close();
    throw e;
}

I keep on my investigations but it seems that the "timeout pb" hypothesis is the most consistent. I will try with a large timeout value tonight.

Answer 1

Just want to strengthen ScottE's diagnosis, and be more specific. Timeout is most likely the issue.

Either .Net implementation of FtpWebRequest is erroneous or the MSDN document has a typo, the default value of FtpWebRequest.Timeout is not -1 (Infinite). It is 100000 (100 seconds).

In addition there is another timeout issue. A few tests have shown that responseStream always has a timeout value of 300000 (300 seconds). I do not know how this value is assigned. Anyways, this value needs to be modified to accommodate large files.

In summary, the solution is to set FtpWebRequest.Timeout and Stream.Timeout to a sufficiently large value.

Answer 2

Here's a good thread of things to try:

http://social.msdn.microsoft.com/Forums/en/ncl/thread/47634ec2-4d40-4d3f-b075-8cc92bfa2b24

Increasing the timeout is probably a good idea in the least.