cURL through NSTask not terminating if a pipe is present

Question

I am trying to read the contents of a URL synchronously for a simple command-line batch script in Swift. I am using cURL for simplicity's sake - I know I could use NSURLSession if I had to. I am also building this with swift build using the open-source version of Swift on OSX.

The problem is that on certain URLs, the NSTask never terminates, if stdout has been redirected to a pipe.

// This will hang, and when terminated with Ctrl-C reports "(23) Failed writing body"
import Foundation
let task = NSTask()
let pipe = NSPipe()
task.launchPath = "/usr/bin/curl"
task.arguments = ["http://trove.nla.gov.au/newspaper/page/21704647"]
task.standardOutput = pipe
task.launch()
task.waitUntilExit()

However, if you remove the pipe, or change the URL, the task succeeds.

// This will succeed - no pipe
import Foundation
let task = NSTask()
task.launchPath = "/usr/bin/curl"
task.arguments = ["http://trove.nla.gov.au/newspaper/page/21704647"]
task.launch()
task.waitUntilExit()

// This will succeed - different URL
import Foundation
let task = NSTask()
let pipe = NSPipe()
task.launchPath = "/usr/bin/curl"
task.arguments = ["http://trove.nla.gov.au/newspaper/page/21704646"]
task.standardOutput = pipe
task.launch()
task2.waitUntilExit()

Running any of the examples directly using curl from Terminal succeeds, so there is something about the interaction with NSTask, when retrieving from that specific URL (and a few others), and when a pipe is present, that is causing cURL to fail.

Answer 1

Expanding a little on @Hod's answer: The standard output of the launched process is redirected to a pipe, but your program never reads from the other pipe end. A pipe has a limited buffer, see for example How big is the pipe buffer? which explains that the pipe buffer size on macOS is (at most) 64KB.

If the pipe buffer is full then the launched process cannot write on it anymore. If the process uses blocking I/O then a write() to the pipe will block until until at least one byte can be written. That does never happen in your case, so the process hangs and does not terminate.

The problem can occur only if the amount written to standard output exceeds the pipe buffer size, which explains why it happens only with certain URLs and not with others.

As a solution, you can read from the pipe, e.g. with

let data = pipe.fileHandleForReading.readDataToEndOfFile()

before waiting for the process to terminate. Another option is to use asynchronous reading, e.g. with the code from Real time NSTask output to NSTextView with Swift:

pipe.fileHandleForReading.readabilityHandler = { fh in
    let data = fh.availableData
    // process data ...
}

That would also allow to read both standard output and standard error from a process via pipes without blocking.

Answer 2

Both curl and NSPipe buffer data. Based on the error you're getting when you ctrl-c out (which indicates curl couldn't write the expected amount of data), you've got a bad interaction between these.

Try adding the -N option to curl to prevent it from buffering its output.

curl can also output progress. I don't think that's causing a problem, but you might add -s to only get the data just in case.