Reading JSON files from curl in Python

Question

Let's say I have a command in Python that looks like

command = 'curl ...etc" > result.json'
subprocess.call(command, shell = True)
file = open("result.json").read()

What it does right now is GET from the place I curl and store the result in result.json, then I open it to read it. I wonder if there's way I can read it directly without storing to local first?

Answer 1

You can use the stdlib (json & urllib2) to avoid the use of external commands:

import json
import urllib2
url = "http://httpbin.org/get"
response = urllib2.urlopen(url)
data = response.read()
values = json.loads(data)

But I'd recommend to use requests to simplify your code. Here a sample from the docs:

import requests
r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
r.status_code
200
r.headers['content-type']
'application/json; charset=utf8'
r.encoding
'utf-8'
r.text
u'{"type":"User"...'
r.json()
{u'private_gists': 419, u'total_private_repos': 77, ...}

Python 3 Update

Please consider that in Python3 urllib2 does not exists anymore, you should use urllib which comes in the standard library

req = urllib.request.Request(url)
response = urllib.request.urlopen(req)
data = response.read()
values = json.loads(data)    

Answer 2

python3 use curl to request data
For example, change the curl to python in the below.

$ curl -XPOST http://httpbin.org/post -H "Content-Type:application/json" -d '{"attribute":"value"}'

intstall urllib3 using $ python -m pip install urllib3

import urllib3
import json
data = {'attribute': 'value'}
encoded_data = json.dumps(data).encode('utf-8')
r = http.request(
    'POST',
    'http://httpbin.org/post',
    body=encoded_data,
    headers={'Content-Type': 'application/json'}
)
json.loads(r.data.decode('utf-8'))['json']

print result

{'attribute': 'value'}

ref: https://urllib3.readthedocs.io/en/latest/user-guide.html