I'm doing some web scraping using the following methods
This is successful for most websites.
var content = new FormUrlEncodedContent(new[]
{
new KeyValuePair<string, string>("v1", "value1"),
new KeyValuePair<string, string>("v2", "value2"),
new KeyValuePair<string, string>("v3", "value3"),
});
var response = await client.PostAsync("http://url.com", content);
html = await response.Content.ReadAsStringAsync();
Sometimes certain website needs this way to get response.
var url = "http://url.com?v1=value1&v2=value2&v3=value3";
var response = await client.PostAsync(url, null);
html = await response.Content.ReadAsStringAsync();
And sometimes needs this to get response.
var query = "v1=value1&v2=value2&v3=value3";
var content = new ByteArrayContent(Encoding.UTF8.GetBytes(query));
var response = await client.PostAsync("http://url.com", content);
html = await response.Content.ReadAsStringAsync();
I really have no idea what is the difference.
If you use FormUrlEncodedContent
your parameters will be sent in request body and formatted as query string.
POST http://url.com/ HTTP/1.1
Host: url.com
Content-Length: 29
Expect: 100-continue
Connection: Keep-Alive
v1=value1&v2=value2&v3=value3
In the second case parameters will be sent as query part of URL.
POST http://url.com/?v1=value1&v2=value2&v3=value3 HTTP/1.1
Host: url.com
Content-Length: 0
In the third case you sent a content of query
in the request body.
In your sample it has the same effect as in the first case, but you made formatting by hand.
The URL (query string) has a (browser dependant) size limit. The body of a POST request doesn't have this limit. So you would use the body to send a file.
On the other hand, a URL can be used in a link.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With