go block vs thread in core.async

Question

From http://martintrojer.github.io/clojure/2013/07/07/coreasync-and-blocking-io/ :

To get a bit more concrete let's see what happens when we try to issue some HTTP GET request using core.async. Let's start with the naive solution, using blocking IO via clj-http.

(defn blocking-get [url]
  (clj-http.client/get url))


(time
   (def data
     (let [c (chan)
           res (atom [])]
       ;; fetch em all
       (doseq [i (range 10 100)]
         (go (>! c (blocking-get (format "http://fssnip.net/%d" i)))))
       ;; gather results
       (doseq [_ (range 10 100)]
         (swap! res conj (<!! c)))
       @res
       )))

Here we're trying to fetch 90 code snippets (in parallel) using go blocks (and blocking IO). This took a long time, and that's because the go block threads are "hogged" by the long running IO operations. The situation can be improved by switching the go blocks to normal threads.

(time
   (def data-thread
     (let [c (chan)
           res (atom [])]
       ;; fetch em all
       (doseq [i (range 10 100)]
         (thread (>!! c (blocking-get (format "http://fssnip.net/%d" i)))))
       ;; gather results
       (doseq [_ (range 10 100)]
         (swap! res conj (<!! c)))
       @res
       )))

What does it mean that "go block threads are hogged by the long running IO operations"?

Answer 1

Go blocks are intended to be a sort of light-weight cooperative threads; they provide thread-like behaviour with less overhead than full JVM threads by using a few threads in a pool and switching go blocks when they park - for instance, when waiting on a channel using <!. The thread-switching cannot work when you call a method in the block that blocks the JVM thread, so you quickly run out of JVM threads. Most standard Java (and Clojure) IO operations will block the current thread when waiting.