mock boto3 response for downloading file from S3

Question

I've got code that downloads a file from an S3 bucket using boto3.

# foo.py
def dl(src_f, dest_f):
  s3 = boto3.resource('s3')
  s3.Bucket('mybucket').download_file(src_f, dest_f)

I'd now like to write a unit test for dl() using pytest and by mocking the interaction with AWS using the stubber available in botocore.

@pytest.fixture
def s3_client():
    yield boto3.client("s3")

from foo import dl
def test_dl(s3_client):

  with Stubber(s3_client) as stubber:
    params = {"Bucket": ANY, "Key": ANY}
    response = {"Body": "lorem"}
    stubber.add_response(SOME_OBJ, response, params)
    dl('bucket_file.txt', 'tmp/bucket_file.txt')
    assert os.path.isfile('tmp/bucket_file.txt')

I'm not sure about the right approach for this. How do I add bucket_file.txt to the stubbed reponse? What object do I need to add_response() to (shown as SOME_OBJ)?

Answer 1

Have you considered using moto3?
Your code could look the same way as it is right now:

# foo.py
def dl(src_f, dest_f):
  s3 = boto3.resource('s3')
  s3.Bucket('mybucket').download_file(src_f, dest_f)

and the test:

import boto3
import os
from moto import mock_s3

@mock_s3
def test_dl():
    s3 = boto3.client('s3', region_name='us-east-1')
    # We need to create the bucket since this is all in Moto's 'virtual' AWS account
    s3.create_bucket(Bucket='mybucket')
    
    s3.put_object(Bucket='mybucket', Key= 'bucket_file.txt', Body='')
    dl('bucket_file.txt', 'bucket_file.txt')

    assert os.path.isfile('bucket_file.txt')

The intention of the code becomes a bit more obvious since you simply work with s3 as usual, except for there is no real s3 behind the method calls.