Delphi- downloading files from the web with sockets

Question

I have tried Synapse, Indy and ICS and I am not satisfied with them. I want to download multiple files in multiple parts at the same time, support resuming, gzip-encoded files, cookies, logging to websites using POST and so on. So I think I will just write in with sockets. But I see there is a plenty of sockets available in Delphi: TTcpClient, TRawSocket, TCGIRequest, TClientSocket and so on. They are all badly documented- very hard to find usage examples. I tried using TTcpClient but sometimes the program freezes and then hits a timeout and I have no idea why. Looks like a problem while waiting for responses. This is surely not a server problem because I test on localhost. What is the best socket to use to work with HTTP protocol? Something easy to use?

I want to use in both Delphi 7 and XE2. I don't want to use anything WinAPI like, so I don't have to deal with PChars and other non-Delphi things.

I am thinking of something like, either:

1) Does exactly what I want- downloads multiple pieces with progressbars of many files at the same file

OR

2) Something like telnet- so I just write HTTP commands as strings and I get arrays of bytes in return which I can either transform into strings or save into tstreams.

Answer 1

I cannot comment on the other libraries, but Indy supports everything you are asking for.

The TIdHTTP component can download files using byte ranges, if the server supports it. You can use the TIdHTTP.Head() method to check if the server return a Accept-Ranges: bytes header for a given URL. If it does, then you can set the TIdHTTP.Request.ContentRangeStart and TIdHTTP.Request.ContentRangeEnd properties to the desired byte offets as needed when downloading the data from that URL. To download multiple parts of a resource at the same time, simply run multiple threads, each with its own TIdHTTP component, in parallel. You will have to then either download multiple pieces to separate temp files and then merged them together into the final file when finished, or else create a single pre-sized file and then open multiple TFileStream objects to it at the desired starting offsets for TIdHTTP to write to.

TIdHTTP supports cookies. You can either assign a TIdCookieManager component to the TIdHTTP.CookieManager property, or you can leave it unassigned and TIdHTTP will create a TIdCookieManager object internally. Either way, also set the TIdHTTP.AllowCookies property to True.

Website login be accomplished one of two different ways, depending on whether the server uses HTTP-based authentication or WebForm-based authentication:

1. For HTTP authentication, add desired IdAuthentication... units to your uses clause, like IdAuthentacationDigest and IdAuthenticationNTLM, or the IdAllAuthentications unt, to register individual authentication classes with TIdHTTP, and then set the TIdHTTP.Request.UserName and TIdHTTP.Request.Password properties as needed. If the WebServer asks for authentication during a request, TIdHTTP will pick the appropriate class and use it to login with the specified credentials. If the server rejects the credentials, the TIdHTTP.OnAuthorization event will be fired to give you a chance to change the credentials and try again.

2. For WebForm authentication, there is a TIdMultipartFormDataStream class that can be passed to the TIdHTTP.Post() method for posting a multipart/form-data formatted POST request.

TIdHTTP in Indy 10 supports gzip-files and deflate-compressed files (TIdHTTP in Iny 9 and earlier does not support compression). You can assign a TIdZLibCompressorBase-derived component, such as TIdCompressorZLib, to the TIdHTTP.Compressor property, and TIdHTTP will handle all of the details for you, including sending an appropriate Accept-Encoding request header, and checking the server's Transfer-Encoding response header.

TIdHTTP has OnWorkBegin, OnWork, and OnWorkEnd events that can be used for progress bar and such. The OnWorkBegin event has an AWorkCountMax parameter that specifies the total number of bytes being transferred if known, otherwise it will be 0 (HTTP responses that use a Content-Length header will set the AWorkCountMax accordingly, but HTTP responses that use a Transfer-Encoding: chunked header will not). The OnWork event has an AWorkCount parameter that specifies how many bytes have been transferred so far, regardless of the AWorkCounMax value.