How to backup Tarantool memtx properly?

Question

Setup:

tarantool instance, using memtx/vinyl storage. box.cfg.snapshot_period (how often memtx takes disk snapshots) set to 14400 (every 4 hours).

Users do their valuable commits into DB, almost permanently.

Backup system:

I do my backups every 15 minutes by calling box.backup.start() and copying listed files to backup storage with some external script.

Recently I found that checksum of copied files changes only after snapshot is taken (every 4 hours).

I've checked if any of *.xlog (that's where memtx puts all information on commit, as I know) files are included in backup files - and they are not.

The things I want to made clear:

1. Is there any way to backup xlog files? Would it be enough to just copy them with the rest of file list? Or they could be changed and become inconsistent during copying process between box.backup.start() and box.backup.stop()?

2. Is there any parameter for box.backup.start() to backup all information, stored in memtx if backup happens between snapshots?

I see the possible workaround by taking a snapshot before every backup, but I want to make sure there's no way to backup xlog files before implementing it.

Answer 1

OK, box.backup.start() wasn't designed for this scenario - it lists the files from the checkpoint, but not WAL files created after the checkpoint. Looks like you're using backup for a scenario in which we usually set up replication.

As a workaround, you could simply rsync() your en

Answer 2

The things I want to made clear: ...

Well. For 1 - 2. I have a suggestion: just copy files using rsync or/and scp as well as any other external tools after checkpoint reached.

Means there are no chance to make it possible, and you can't have behaviour which you have described by using box.backup.* functions.

I prefer use rsync and not only for Tarantool. For instance, sometime for having the best experience with archive logs (I wished to have a point of recovery) for PosgreSQL I have been using rsync as well.