Understanding Python HTTP streaming

Question

I'm struggling to access a streaming API using Python and Requests.

What the API says: "We’ve enabled a streaming endpoint to for requesting both quote and trade data utilizing a persistent HTTP socket connection. Streaming data from the API consists of making an Authenticated HTTP request and leaving the HTTP socket open to continually receive data."

How I've been trying to access the data:

s = requests.Session()
def streaming(symbols):
    url = 'https://stream.tradeking.com/v1/market/quotes.json'
    payload = {'symbols': ','.join(symbols)}
    return s.get(url, params=payload, stream=True)  
r = streaming(['AAPL', 'GOOG'])

The Requests docs here show two things of interest: Use a generator/iterator for use with chunked data, passed in the data field. For streaming data, it suggests using code such as:

for line in r.iter_lines():
    print(line)

Neither seem to work, although I've no idea what to put in the generator function, since the example is unclear. Using r.iter_lines(), I get the output: "b'{"status":"connected"}{"status":disconnected"}'"

I can access the headers, and the response is HTTP 200, but can't get valid data, or find clear examples on how to access streaming HTTP data in python. Any help would be appreciated. The API recommends using Jetty for Java to keep the stream open, but I'm not sure how to do this in Python.

Headers: {'connection': 'keep-alive', 'content-type': 'application/json', 'x-powered-by': 'Express', 'transfer-encoding': 'chunked'}

Answer 1

As verbsintransit has stated, you need to solve your authentication problems, your streaming problems however can be fixed by using this example:

s = requests.Session()

def streaming(symbols):
    payload = {'symbols': ','.join(symbols)}
    headers = {'connection': 'keep-alive', 'content-type': 'application/json', 'x-powered-by': 'Express', 'transfer-encoding': 'chunked'}
    req = requests.Request("GET",'https://stream.tradeking.com/v1/market/quotes.json',
                           headers=headers,
                           params=payload).prepare()

    resp = s.send(req, stream=True)

    for line in resp.iter_lines():
        if line:
            yield line


def read_stream():

    for line in streaming(['AAPL', 'GOOG']):
        print line


read_stream()

The if line: condition is checking if the line is an actual message or just a connection keep-alive.

Answer 2

Not sure if you figured this out, but TradeKing doesn't put newlines in between their JSON blobs. You thus have to use iter_content to get it byte by byte, append that byte to a buffer, try to decode the buffer, on success clear the buffer and yield the resultant object. :(