Reading an JSON file from S3 using Python boto3

Question

I kept following JSON in S3 bucket 'test'

{   'Details' : "Something"  } 

I am using following code to read this JSON and printing the key 'Details'

s3 = boto3.resource('s3',                     aws_access_key_id=<access_key>,                     aws_secret_access_key=<secret_key>                     ) content_object = s3.Object('test', 'sample_json.txt') file_content = content_object.get()['Body'].read().decode('utf-8') json_content = json.loads(repr(file_content)) print(json_content['Details']) 

And i am getting error as 'string indices must be integers' I don't want to download the file from S3 and then reading..

Answer 1

As mentioned in the comments above, repr has to be removed and the json file has to use double quotes for attributes. Using this file on aws/s3:

{   "Details" : "Something" } 

and the following Python code, it works:

import boto3 import json  s3 = boto3.resource('s3')  content_object = s3.Object('test', 'sample_json.txt') file_content = content_object.get()['Body'].read().decode('utf-8') json_content = json.loads(file_content) print(json_content['Details']) # >> Something 

Answer 2

The following worked for me.

# read_s3.py from boto3 import client BUCKET = 'MY_S3_BUCKET_NAME' FILE_TO_READ = 'FOLDER_NAME/my_file.json' client = client('s3',                  aws_access_key_id='MY_AWS_KEY_ID',                  aws_secret_access_key='MY_AWS_SECRET_ACCESS_KEY'                 ) result = client.get_object(Bucket=BUCKET, Key=FILE_TO_READ)  text = result["Body"].read().decode() print(text['Details']) # Use your desired JSON Key for your value  

It is not good idea to hard code the AWS Id & Secret Keys directly. For best practices, you can consider either of the followings:

(1) Read your AWS credentials from a json file (aws_cred.json) stored in your local storage:

from json import load from boto3 import client ... credentials = load(open('local_fold/aws_cred.json')) client = client('s3',                  aws_access_key_id=credentials['MY_AWS_KEY_ID'],                  aws_secret_access_key=credentials['MY_AWS_SECRET_ACCESS_KEY']                 ) 

(2) Read from your environment variable (my preferred option for deployment):

import os client = boto3.client('s3',                        aws_access_key_id=os.environ['MY_AWS_KEY_ID'],                        aws_secret_access_key=os.environ['MY_AWS_SECRET_ACCESS_KEY']                      ) 

Let's prepare a shell script (set_env.sh) for setting the environment variables and add our python script (read_s3.py) as follows:

# set_env.sh export MY_AWS_KEY_ID='YOUR_AWS_ACCESS_KEY_ID' export MY_AWS_SECRET_ACCESS_KEY='YOUR_AWS_SECRET_ACCESS_KEY' # execute the python file containing your code as stated above that reads from s3 python read_s3.py # will execute the python script to read from s3 

Now execute the shell script in a terminal as follows:

sh set_env.sh