Drop keyspace hangs

Question

Problem: drop keyspace MyKeyspace; hangs.

Environment: This is an Ubuntu 12.04 64bit in virtualbox, running a single Cassandra instance (on a development machine).

Cassandra is 1.1.6:

myuser@myhost:~$ /usr/bin/nodetool -h localhost version
ReleaseVersion: 1.1.6

Plenty of free disk space:

myuser@myhost:~$ df
Filesystem                1K-blocks    Used Available Use% Mounted on
/dev/mapper/myhost-root   100232772 3100308  92112816   4% /
udev                        1016760       4   1016756   1% /dev
tmpfs                        410340     268    410072   1% /run
none                           5120       0      5120   0% /run/lock
none                        1025844       0   1025844   0% /run/shm
/dev/sda1                    233191   24999    195751  12% /boot

Machine is idle:

myuser@myhost:~$ uptime
 21:24:50 up  3:46,  2 users,  load average: 0.06, 0.04, 0.05

How I got there:

The machine was running another db, all fine for long time. Now I created a new keyspace MyKeyspace, and run a Java program to import data (using titan graph, but that shouldn't matter). After a couple thousand of records imported (a couple of MB only) the import program did not make progress anymore, and logged 6 times:

418455 [RetryService : myhost(192.168.1.241):9160] INFO  com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor  - Reactivating myhost

Then my program (titan graph actually) gave up with:

com.thinkaurelius.titan.diskstorage.TemporaryStorageException: Temporary failure in storage backend

During all this time I was connected to /usr/bin/cassandra-cli and could successfully execute show keyspaces;.

Then I decided to start over; drop the keyspace. That's where it hangs now, for hours. It doesn't respond to ctrl-c either. Meanwhile I'm able to log in by ssh, connect with cassandra-cli, and run show keyspaces;. The keyspace is still there. Also, my Java app can access that data store, but it's read only. Reading succeeds, but writes fail. It's just a timeout I get from the titan graph library when writing:

com.thinkaurelius.titan.core.TitanException: ID renewal thread on partition [2] did not complete in time. [60007 ms]

Any commands I could run to see what's going on? Should I report a bug?

Answer 1

If you have auto_snapshot enabled in cassandra.yaml (it's enabled by default), then Cassandra will take a snapshot before dropping the keyspace. If you don't have JNA set up properly, this can sometimes cause problems, so I would check that first.