How can I restore Cassandra snapshots?

Question

I'm building a backup and restore process for a Cassandra database so that it's ready when I need it, and so that I understand the details in order to build something that will work for production. I'm following Datastax's instructions here:

http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_backup_restore_c.html.

As a start, I'm seeding the database on a dev box then attempting to make the backup/restore work. Here's the backup script:

#!/bin/bash

cd /opt/apache-cassandra-2.0.9
./bin/nodetool clearsnapshot -t after_seeding makeyourcase
./bin/nodetool snapshot -t after_seeding makeyourcase

cd /var/lib/
tar czf after_seeding.tgz cassandra/data/makeyourcase/*/snapshots/after_seeding

Yes, tar is not the most efficient way, perhaps, but I'm just trying to get something working right now. I've checked the tar, and all the files are there.

Once the database is backed up, I shut down Cassandra and my app, then rm -rf /var/lib/cassandra/ to simulate a complete loss.

Now to restore the database. Restoration "Method 2" from http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_backup_snapshot_restore_t.html is more compatible with my schema-creation component than Method 1.

So, Method 2/Step 1, "Recreate the schema": Restart Cassandra, then my app. The app is built to re-recreate the schema on startup when necessary. Once it's up, there's a working Cassandra node with a schema for the app, but no data.

Method 2/Step 2 "Restore the snapshot": They give three alternatives, the first of which is to use sstableloader, documented at http://www.datastax.com/documentation/cassandra/2.0/cassandra/tools/toolsBulkloader_t.html. The folder structure that the loader requires is nothing like the folder structure created by the snapshot tool, so everything has to be moved into place. Before going to all that trouble, I'll just try it out on one table:

>./bin/sstableloader makeyourcase/users
Error: Could not find or load main class org.apache.cassandra.tools.BulkLoader

Hmmm, well, that's not going to work. BulkLoader is in ./lib/apache-cassandra-2.0.9.jar, but the loader doesn't seem to be set up to work out of the box. Rather than debug the tool, let's move on to the second alternative, copying the snapshot directory into the makeyourcase/users/snapshots/ directory. This should be easy, since we're throwing the snapshot directory right back where it came from, so tar xzf after_seeding.tgz should do the trick:

cd /var/lib/
tar xzf after_seeding.tgz
chmod -R u+rwx cassandra/data/makeyourcase

and that puts the snapshot directories back under their respective 'snapshots' directories, and a refresh should restore the data:

cd /opt/apache-cassandra-2.0.9
./bin/nodetool refresh -- makeyourcase users

This runs without complaint. Note that you have to run this for each and every table, so you have to generate the list of tables first. But, before we do that, note that there's something interesting in the Cassandra logs:

INFO 14:32:26,319 Loading new SSTables for makeyourcase/users...
INFO 14:32:26,326 No new SSTables were found for makeyourcase/users

So, we put the snapshot back, but Cassandra didn't find it. I also tried moving the snapshot directory under the existing SSTables directory, and copying the old SSTable files into the existing directory, with the same error in the log. Cassandra doesn't log where it expects to find them, just that it can't find them. The docs say to put them into a directory named data/keyspace/table_name-UUID, but there is no such directory. There is one named data/makeyourcase/users/snapshots/1408820504987-users/, but putting the snapshot dir there, or the individual files, didn't work.

The third alternative, the "Node restart method" doesn't look suitable for a multi-node production environment, so I didn't try that.

Edit:

Just to make this perfectly explicit for the next person, here are the preliminary, working backup and restore scripts that apply the accepted answer.

myc_backup.sh:

#!/bin/bash

cd ~/bootstrap/apache-cassandra-2.0.9
./bin/nodetool clearsnapshot -t after_seeding makeyourcase
./bin/nodetool snapshot -t after_seeding makeyourcase

cd /var/lib/
tar czf after_seeding.tgz cassandra/data/makeyourcase/*/snapshots/after_seeding

myc_restore.sh:

#!/bin/bash

cd /var/lib/
tar xzf after_seeding.tgz
chmod -R u+rwx cassandra/data/makeyourcase

cd ~/bootstrap/apache-cassandra-2.0.9
TABLE_LIST=`./bin/nodetool cfstats makeyourcase | grep "Table: " | sed -e 's+^.*: ++'`
for TABLE in $TABLE_LIST; do
    echo "Restore table ${TABLE}"
    cd /var/lib/cassandra/data/makeyourcase/${TABLE}
    if [ -d "snapshots/after_seeding" ]; then
        cp snapshots/after_seeding/* .
        cd ~/bootstrap/apache-cassandra-2.0.9
        ./bin/nodetool refresh -- makeyourcase ${TABLE}
        cd /var/lib/cassandra/data/makeyourcase/${TABLE}
        rm -rf snapshots/after_seeding
        echo "    Table ${TABLE} restored."
    else
        echo "    >>> Nothing to restore."
    fi
done

Answer 1

Added more details:

You can run the snapshot for your particular keyspace using:

$ nodetool snapshot <mykeyspace> -t <SnapshotDirectoryName>

This will create the snapshot files inside the snapshots directory in data.

When you delete your data, make sure you don't delete the snapshots folder or you will not be able to restore it (unless you are moving it to another location / machine.)

$ pwd
/var/lib/cassandra/data/mykeyspace/mytable
$ ls
mykeyspace-mytable-jb-2-CompressionInfo.db mykeyspace-mytable-jb-2-Statistics.db
mykeyspace-mytable-jb-2-Data.db mykeyspace-mytable-jb-2-Filter.db mykeyspace-mytable-jb-2-Index.db
mykeyspace-mytable-jb-2-Summary.db mykeyspace-mytable-jb-2-TOC.txt snapshots


$ rm *
rm: cannot remove `snapshots': Is a directory

Once you are ready to restore, copy back the snapshot data into the keyspace/table directory (one for each table):

$ pwd
/var/lib/cassandra/data/mykeyspace/mytable
$ sudo cp snapshots/<SnapshotDirectoryName>/* .

You mentioned:

and that puts the snapshot directories back under their respective 'snapshots' directories, and a refresh >should restore the data:

I think the issue is that you are restoring the Snapshot data into the snapshot directory. It should go right in the table directory. Everything else seems right, let me know.

Answer 2

The docs say to put them into a directory named data/keyspace/table_name-UUID, but there is no such directory.

You don't have this UUID directory because you are using cassandra 2.0 and this UUID thing started with cassandra 2.2