Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Finding items between 2 dates using boto3 and dynamodb scan

I am in the process of moving my NodeJS backend over to Python3.6.

In NodeJS I was able to use the aws SDK to get items between two dates like so :

```

var now = moment().tz("Pacific/Auckland").format()
var yesterday = moment().add(-24,"Hours").tz("Pacific/Auckland").format()
var docClient = new AWS.DynamoDB.DocumentClient();
  var params = {
    TableName : process.env.TableName,
    FilterExpression : '#ts between :val1 and :val2',
    ExpressionAttributeNames: {
        "#ts": "timeStamp",
    },
    ExpressionAttributeValues: {
      ':val1': yesterday,
      ':val2': now
    }
  }

```

And this works fine, however the between function in boto3 does not seem to do the same thing.

This is the current code I am using to try get something similar:

```

import boto3
import time
from datetime import datetime, timedelta
from boto3.dynamodb.conditions import Key, Attr

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('table_name')

now = int(datetime.timestamp(datetime.now()))
three_hours_ago =  int(datetime.timestamp(datetime.now() - timedelta(hours=3)))

fe = Key('timeStamp').between(three_hours_ago,now);
response = table.scan(
                FilterExpression=fe
            )
print(response)

```

This returns no items, I am sure that there are items between these two dates.

Any help would be much appreciated.

like image 855
NeuroWinter Avatar asked Mar 18 '18 03:03

NeuroWinter


People also ask

Does DynamoDB scan return all items?

A Scan operation in Amazon DynamoDB reads every item in a table or a secondary index. By default, a Scan operation returns all of the data attributes for every item in the table or index. You can use the ProjectionExpression parameter so that Scan only returns some of the attributes, rather than all of them.

How does boto3 connect to DynamoDB?

Connecting AWS resources to the python environment requires a boto3 package. Creating a dynamo DB client is a connection instance that lets us connect with our dynamo DB service. We need to specify region_name , aws_access_key_id , aws_secret_access_key in order to connect with our dynamoDb service.

Which is faster scan or query in DynamoDB?

For faster response times, design your tables and indexes so that your applications can use Query instead of Scan . (For tables, you can also consider using the GetItem and BatchGetItem APIs.)


2 Answers

I managed to resolve this using the following code:

now             = datetime.datetime.now()
three_hours_ago = now - datetime.timedelta(hours=3)
#  TODO make the db UTC 
now             = now.strftime('%FT%T+13:00')
three_hours_ago = three_hours_ago.strftime('%FT%T+13:00')

fe       = Key('timeStamp').between(three_hours_ago,now);
response = table.scan(
                FilterExpression=fe
            )
like image 174
NeuroWinter Avatar answered Oct 08 '22 18:10

NeuroWinter


A possible solution could be:

attr = boto3.dynamodb.conditions.Attr('timeStamp')

response = table.scan(
                FilterExpression=attr.between(first_date,second_date)
            )

However, please note that the date should be formatted as YYYY-MM-DD (ISO 8601)

like image 31
ispravnic Avatar answered Oct 08 '22 18:10

ispravnic