Menu
Is there a limit to the size when making HTTP GET requests in Node.js? And if so, how can I change this?
var url = "..." // very long, ~50'000 chars http.get(url, function (res) { res.pipe(fs.createWriteStream("file.txt")); }); Gives me this:
<html><body><h1>400 Bad request</h1> Your browser sent an invalid request. </body></html> Same thing using wget in PowerShell or Bash works perfectly fine.
$url = "..." wget -outf file.txt $url 
486
The official documentation specifies a maximum length of 2048 characters for the <loc> element, which is used to submit URLs: URL of the page. This URL must begin with the protocol (e.g. “http”) and end with a trailing slash if required by the web server. This value must not exceed 2,048 characters.
Although officially there is no limit specified by RFC 2616, many security protocols and recommendations state that maxQueryStrings on a server should be set to a maximum character limit of 1024. While the entire URL, including the querystring, should be set to a max of 2048 characters.
As is, node. js can process upwards of 1000 requests per second and speed limited only to the speed of your network card. Note that it's 1000 requests per second not clients connected simultaneously. It can handle the 10000 simultaneous clients without issue.
Node can hold up to 1.5GB in memory at one time, but no more.
There are many ways to perform an HTTP GET request in Node.js, depending on the abstraction level you want to use. The simplest way to perform an HTTP request using Node.js is to use the Axios library: However, Axios requires the use of a 3rd party library.
Google Chrome allows the maximum length of the URL to be of the size 2MB (2048 characters). In Firefox the length of the URL can be unlimited but practically after 65,536 characters the location bar no longer displays the URL.
In get-request.js, add the following code. In the above code, you start by first assigning the URL that will receive the request to the URL variable. Next, you are calling the http.get () method and passing in the URL stated and a callback function that will handle the response from the URL.
There is a built-in request size limit enforced by Node. Requested headers + URI should not be more than 80 kb.
As it is defined in http_parser.h#L55:
/* Maximium header size allowed */ #define HTTP_MAX_HEADER_SIZE (80*1024) Asuming that UTF-8 character can be between 1 and 4 bytes, the size of a string with 50 000 characters would be from 50 000 to 200 000 bytes, or from ~48kb to 195kb.
121
In newer versions of NodeJS (v12.6.0+), the header request size limit is 8kb by default, and this also impacts the response header size. To use header sizes above 8KB, you can use the switch --max-http-header-size 65535 (or whatever size you need). This is described in-depth here.
23
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
