I am trying to use the .NET WebRequest/WebResponse classes to access the Twitter streaming API here "http://stream.twitter.com/spritzer.json"
.
I need to be able to open the connection and read data incrementally from the open connection.
Currently, when I call WebRequest.GetResponse
method, it blocks until the entire response is downloaded. I know there is a BeginGetResponse
method, but this will just do the same thing on a background thread. I need to get access to the response stream while the download is still happening. This just does not seem possible to me with these classes.
There is a specific comment about this in the Twitter documentation:
"Please note that some HTTP client libraries only return the response body after the connection has been closed by the server. These clients will not work for accessing the Streaming API. You must use an HTTP client that will return response data incrementally. Most robust HTTP client libraries will provide this functionality. The Apache HttpClient will handle this use case, for example."
They point to the Appache HttpClient, but that doesn't help much because I need to use .NET.
Any ideas whether this is possible with WebRequest/WebResponse
, or do I have to go for lower level networking classes? Maybe there are other libraries that will allow me to do this?
Thx Allen
HTTP Streaming is a push-style data transfer technique that allows a web server to continuously send data to a client over a single HTTP connection that remains open indefinitely.
Streaming APIs are used to read data in real-time from the web for consumers for precise, up-to-date results. For example, they are typically used by social media platforms to deliver media content such as audio and data. Typically Social Networks tend to use WebSocket, which is a subset of Streaming APIS.
I ended up using a TcpClient, which works fine. Would still be interested to know if this is possible with WebRequest/WebResponse though. Here is my code in case anybody is interested:
using (TcpClient client = new TcpClient())
{
string requestString = "GET /spritzer.json HTTP/1.1\r\n";
requestString += "Authorization: " + token + "\r\n";
requestString += "Host: stream.twitter.com\r\n";
requestString += "Connection: keep-alive\r\n";
requestString += "\r\n";
client.Connect("stream.twitter.com", 80);
using (NetworkStream stream = client.GetStream())
{
// Send the request.
StreamWriter writer = new StreamWriter(stream);
writer.Write(requestString);
writer.Flush();
// Process the response.
StreamReader rdr = new StreamReader(stream);
while (!rdr.EndOfStream)
{
Console.WriteLine(rdr.ReadLine());
}
}
}
BeginGetResponse is the method you need. It allows you to read the response stream incrementally:
class Program
{
static void Main(string[] args)
{
WebRequest request = WebRequest.Create("http://stream.twitter.com/spritzer.json");
request.Credentials = new NetworkCredential("username", "password");
request.BeginGetResponse(ar =>
{
var req = (WebRequest)ar.AsyncState;
// TODO: Add exception handling: EndGetResponse could throw
using (var response = req.EndGetResponse(ar))
using (var reader = new StreamReader(response.GetResponseStream()))
{
// This loop goes as long as twitter is streaming
while (!reader.EndOfStream)
{
Console.WriteLine(reader.ReadLine());
}
}
}, request);
// Press Enter to stop program
Console.ReadLine();
}
}
Or if you feel more comfortable with WebClient (I personnally prefer it over WebRequest):
using (var client = new WebClient())
{
client.Credentials = new NetworkCredential("username", "password");
client.OpenReadCompleted += (sender, e) =>
{
using (var reader = new StreamReader(e.Result))
{
while (!reader.EndOfStream)
{
Console.WriteLine(reader.ReadLine());
}
}
};
client.OpenReadAsync(new Uri("http://stream.twitter.com/spritzer.json"));
}
Console.ReadLine();
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With