Weird HTTP problem/mistake with Lisp

Question

I'm attempting to learn a little more about handling sockets and network connections in SBCL; so I wrote a simple wrapper for HTTP. Thus far, it merely makes a stream and performs a request to ultimately get the header data and page content of a website.

Until now, it has worked at somewhat decently. Nothing to brag home about, but it at least worked.

I have come across a strange problem, however; I keep getting "400 Bad Request" errors.

At first, I was somewhat leery about how I was processing the HTTP requests (more or less passing a request string as a function argument), then I made a function that formats a query string with all the parts I need and returns it for use later... but I still get errors.

What's even more odd is that the errors don't happen every time. If I try the script on a page like Google, I get a "200 Ok" return value... but at other times on other sites, I'll get "400 Bad Request".

I'm certain its a problem with my code, but I'll be damned if I know exactly what is causing it.

Here is the code that I am working with:

(use-package :sb-bsd-sockets)

(defun read-buf-nonblock (buffer stream)
  (let ((eof (gensym)))
    (do ((i 0 (1+ i))
         (c (read-char stream nil eof)
            (read-char-no-hang stream nil eof)))
        ((or (>= i (length buffer)) (not c) (eq c eof)) i)
      (setf (elt buffer i) c))))

(defun http-connect (host &optional (port 80))
"Create I/O stream to given host on a specified port"
  (let ((socket (make-instance 'inet-socket
                   :type :stream
                   :protocol :tcp)))
    (socket-connect
     socket (car (host-ent-addresses (get-host-by-name host))) port)
    (let ((stream (socket-make-stream socket
                    :input t
                    :output t
                    :buffering :none)))
      stream)))

(defun http-request (stream request &optional (buffer 1024))
"Perform HTTP request on a specified stream"
  (format stream "~a~%~%" request )
  (let ((data (make-string buffer)))
    (setf data (subseq data 0
               (read-buf-nonblock data
                      stream)))
    (princ data)
    (> (length data) 0)))

(defun request (host request)
"formated HTTP request"
  (format nil "~a HTTP/1.0 Host: ~a" request host))

(defun get-page (host &optional (request "GET /"))
"simple demo to get content of a page"
  (let ((stream (http-connect host)))
    (http-request stream (request host request)))

Answer 1

A few things. First, to your concern about the 400 errors you are getting back, a few possibilities come to mind:

• "Host:" isn't actually a valid header field in HTTP/1.0, and depending on how fascist the web server you are contacting is about standards, it would reject this as a bad request based on the protocol you claim to be speaking.
• You need a CRLF between your Request-line and each of the header lines.
• It is possible that your (request) function is returning something for the Request-URI field -- you substitute in the value of request as the contents of this part of the Request-line -- that is bogus in one way or another (badly escaped characters, etc.). Seeing what it is outputting might help out some.

Some other more general pointer to help you along your way:

• (read-buf-nonblock) is very confusing. Where is the symbol 'c' defined? Why is 'eof' (gensym)ed and then not assigned any value? It looks very much like a byte-by-byte copy taken straight out of an imperative program, and plopped into Lisp. It looks like what you have reimplemented here is (read-sequence). Go look here in the Common Lisp Hyperspec, and see if this is what you need. The other half of this is to set your socket you created to be non-blocking. This is pretty easy, even though the SBCL documentation is almost silent on the topic. Use this:

  (socket-make-stream socket :input t :output t :buffering :none :timeout 0)

• The last (let) form of (http-connect) isn't necessary. Just evaluate

  (socket-make-stream socket :input t :output t :buffering :none)

without the let, and http-connect should still return the right value.

• In (http-request)...

Replace:

 (format stream "~a~%~%" request )
 (let ((data (make-string buffer)))
 (setf data (subseq data 0
            (read-buf-nonblock data
                               stream)))
 (princ data)
 (> (length data) 0)))

with

(format stream "~a~%~%" request )
(let ((data (read-buf-nonblock stream)))
    (princ data)
    (> (length data) 0)))

and make (read-buf-nonblock) return the string of data, rather that having it assign within the function. So where you have buffer being assigned, create a variable buffer within and then return it. What you are doing is called relying on "side-effects," and tends to produce more errors and harder to find errors. Use it only when you have to, especially in a language that makes it easy not to depend on them.

• I mostly like the the way get-page is defined. It feels very much in the functional programming paradigm. However, you should either change the name of the (request) function, or the variable request. Having both in there is confusing.

Yikes, hands hurt. But hopefully this helps. Done typing. :-)

Answer 2

Here's a possibility:

HTTP/1.0 defines the sequence CR LF as the end-of-line marker.

The ~% format directive is generating a #\Newline (LF on most platforms, though see CLHS).

Some sites may be tolerant of the missing CR, others not so much.