How to read & handle POST requests with Netcat

Question

I want to make a simple HTTP server with Bash & netcat, and have a problem with reading POST requests entirely - the last line is always missing.

The server is started like this:

netcat -l -p 8080 -e ./ncserver.sh

The ncserver.sh, reduced to bare minimum which displays the problem:

#!/bin/bash

while read INPUT; do
    echo "Req line: $INPUT" >&2
done;

I use Postman to make requests to the server, and expect the script to dump entire request data to stderr. The request is a simple JSON:

{
    name: "Eugene",
    age: 34
}

Update: Raw postman request data:

POST /foo/bar HTTP/1.1
Host: localhost:8080
Content-Type: application/json
Cache-Control: no-cache
Postman-Token: 7f5a57a7-1664-e79c-2242-7ba6b638e260

{
    name: "Eugene",
    age: 34
}

In the server's output I get all the headers and request body which is missing the last line, the }. The same holds for other content types, such as multipart/form-data - last line is always missing.

Note: if I add an empty line after JSON, I can see the } in terminal output.

I tried using cat /dev/stdin instead of read but get No such device or address.

The question probably can be generalized as how do I read HTTP requests in correct binary form in Bash?

Answer 1

Try as below.

#!/bin/bash

while IFS= read -r INPUT || [ "$INPUT" ]; do
    echo "Req line: $INPUT" >&2
done;   

Point to notice.

IFS= is to preserve any indentation.
-r is to prevent backslash interpretation.
Check for INPUT: When read reaches the end-of-file instead of end-of-line it assigns variable but exits. hence, check if we have anything in INPUT.

Answer 2

The newline behavior is your friend; The best way to do this would be to read the headers (which end at the first empty line), and use the Content-Length header of the POST request to read the body.

For instance,

### Create the response FIFO
rm -f response
mkfifo response

# read the headers of the request
while read line; do
    echo $line

    # break the loop when an empty line is encountered
    [ -z "$trline" ] && break
    
    # read the content length header so we know how big the body is
    CONTENT_LENGTH_REGEX='Content-Length:\s(.*?)'

    [[ "$trline" =~ $CONTENT_LENGTH_REGEX ]] &&
      CONTENT_LENGTH=`echo $trline | sed -E "s/$CONTENT_LENGTH_REGEX/\1/"`
  done

# if a content-lenth is passed, read the body
if [ ! -z "$CONTENT_LENGTH" ]; then
  BODY_REGEX='(.*?)=(.*?)'

  ## Read the remaining request body
read -n$CONTENT_LENGTH -t1 body

This assumes that the server includes a named pipe called response through which responses are sent back.

In other words, the server is started with:

while true; do
    cat response | nc -lvN 3000 | handleRequest
done

See this bash web server tutorial and the docs for sed for more (better) info.