Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How does Amazon RDS backup/snapshot actually work?

I am an Amazon RDS customer and am experiencing daily amazon RDS write latency spikes, corresponding roughly to the backup window. I will also see spikes at the end of a snapshot (case in point: running a snapshot takes appx 1 hour, and in the final 5 minutes, write latency spikes). I am running a multi-AZ m1.large deployment.

Is there anyone on Stack who can explain how Amazon RDS backup is actually working? I've read the Amazon RDS docs, and as far as I can tell, Amazon RDS is not behaving according to spec. Specifically, these backup/snapshot operations should be hitting my replica, and therefore not causing any downtime/performance hit, or so I thought.

I can distill my problem into six questions:

  • What is technically happening during a snapshot and a backup, and how are they different? (If you answer this question, please tell me if you are able to empirically confirm your answer, or are simply quoting me documentation).
  • Is a spike in write latency to be expected during the backup window on a multi-AZ deployment?
  • Is a spike in write latency to be expected at the end of a snapshot on a multi-AZ deployment?
  • Would my write latency spike be even higher if I was not multi-AZ ?
  • Architecturally, would I be able to avoid these write latency spikes if I rolled my own database running on two m1.large EC2 instances?
  • Are there any configurations I can use that would avoid these write latency spikes while still hosting my DB with RDS, or am I effectively at the mercy of Amazon?

Bonus Question: where and how do you host your mysql database?

I can say that I have been generally happy with RDS except for these daily write latency issues. I love the built-in database monitoring and it was fairly simple to setup and get going.

Thanks!

amazon RDS write latency

like image 305
esilver Avatar asked Mar 09 '11 17:03

esilver


People also ask

What is the difference between RDS backup and snapshot?

First backup is a full backup, while the others are incremental. When a DB instance is deleted, a final DB snapshot can be created upon deletion; which can be used to restore the deleted DB instance at a later date. RDS retains the final user-created DB snapshot along with all other manually created DB snapshots.

Does AWS backup use snapshots?

Q: How does AWS Backup work with other AWS services that have backup capabilities? Today, several AWS services offer backup features that help you protect your data, such as S3 replication, EBS snapshots, RDS snapshots, Amazon FSx backups, Amazon DynamoDB backups, and AWS Storage Gateway snapshots.

Does taking RDS snapshot affect performance?

Automated Snapshots These are the daily snapshots that are created automatically by your RDS instance. They have a "Snapshot Type" of "automated". Performance is never impacted when automated snapshots are being created.

Where does Amazon RDS store DB snapshots and automated backups?

Amazon RDS DB snapshots and automated backups are stored in S3. You can use the AWS Management Console, the ModifyDBInstance API, or the modify-db-instance command to manage the period of time your automated backups are retained by modifying the RetentionPeriod parameter.


2 Answers

We also run several RDS instances, in addition to MySQL on some machines that we manage ourselves. I can't comment specifically, as I'm not an Amazon engineer, but several things I've learned that might explain what you're seeing:

  • Although Amazon does not share the backend details 100%, we strongly suspect that they are using their EBS system to back RDS databases.

  • This article helps explain EBS limitations and snapshot functionality http://blog.rightscale.com/2008/08/20/amazon-ebs-explained/ Again, while it's not explicit, it would make sense for Amazon to be using this infrastructure to provide RDS services.

  • Typically, a MySQL backup, in contrast to a snapshot, involves using a tool like mysqldump to create a file of SQL statements that will then reproduce the database. The database does not need to be frozen to do this. With an EBS backend, the best practice is to freeze the database (pause all transactions) while you are snapshotting to avoid data corruption.

  • The spikes you're seeing at the ends of the backup window. If replication is paused by Amazon during the snapshot of your replica, the replica would then need to "catch up" on transactions when the snapshot was complete. This would cause a latency spike.

  • Replication across a multi-AZ deployment is inherently slower then a single AZ deployment. The price you pay for better redundancy.

like image 69
Joshua Avatar answered Oct 21 '22 10:10

Joshua


Amazon revealed the basic architecture that they use in Multi AZ deployments. This may help people to take decisions

https://aws.amazon.com/blogs/database/amazon-rds-under-the-hood-multi-az/

like image 44
Anurag Kale Avatar answered Oct 21 '22 10:10

Anurag Kale