AWS Timestream: Unable to ingest records into AWS Timestream

Question

As you all know, AWS Timestream was made generally available in the last week.

Since then, I have been trying to experiment with it and understanding how it models and stores the data.

I am facing an issue in ingesting records into Timestream. I have certain records dated 23rd April 2020. On trying to insert these records into a Timestream table, I get the RecordRejected error. According to this link, a record is rejected if it has the same dimension, timestamp or if the timestamp is beyond the retention period of the memory store of the table.

I have set the retention period of the memory store of my table to 12 months. According to the documentation: any records having a timestamp beyond 12 months would be rejected. However, the above mentioned record gets rejected despite having a timestamp within 12 months from now.

On investigating further, I have noticed that, records with today's date (5th Oct 2020) get ingested successfully, however, records with a date 30 days before do not get ingested, i.e. 5th September 2020. To ensure this, I have also tried inserting a record with the date 6th Sept and a few more days between today's date and 5th Sept. All these are getting inserted successfully.

Could somebody explain why I am not able to insert records having a timestamp within the retention period of the memory store? It only allows me to insert records that are at the most 30 days old. I would also like to know if there is a way we could insert historical data directly into the magnetic store. The memory store retention period may not be sufficient for my use case and I may need to insert data that is 2 years old or more. I understand this is not a classic use case of timestream, but I am still curious to know.

I am stuck on this issue and would really appreciate some help.

Thank you in advance.

Answer 1

I had a very similar issue, and for me it turned out that I had to set the Memory Store Retention Period to 8766 hours - which is slightly MORE than one year. I've no clue why that is, and why it works, but it worked for me importing older data.

PS: I'm pretty sure it's a bug in timestream PPS: I've found the value by using the "default" set in the aws console. No other value worked for me.