Get All Keys From a LevelDB Database

Question

I'm writing a script to collect the hashes of all bitcoin blocks. The program bitcoind, if a certain setting is changed, stores metadata for all blocks in a LevelDB database. The key of each set of metadata is the block's hash, which is typically used as an identifier for it. Essentially, I'm trying to get a specific part of the metadata (the transaction IDs) out of each block. The script I'm writing is in Haskell, although I could always do a shell command if necessary. To put my problem in general terms, I'm not sure if the most simple way to do this is to find all block hashes (keys) and then call bitcoind to get the metadata for each of them. If there's any way to simply get every value from a LevelDB database directly, that would work as well. What's the most simple and efficient way to do this?

Answer 1

I don't know exactly how this can be done with haskell, but the most efficient way to do it with C++ is to use the leveldb::Iterator and iterate over the entire key range:

0x00000000 -> 0xFFFFFFFF (32-bit keys)
0x0000000000000000 -> 0xFFFFFFFFFFFFFFFF (64-bit keys)

From the example:

leveldb::Slice start = ...; // 0x00000000
leveldb::Slice end = ...; // 0xFFFFFFFFFFFFFFFF

The example shows that when you're iterating the keys and values can be loaded:

leveldb::Slice key = it->key();
leveldb::Slice value = it->value();

Since you don't care about the value you can just skip the part where you query the it->value(), just collect the keys you need (in this case all of them) and you should be good. I wouldn't worry about the performance hit of unnecessarily loading up the values.

However, one big thing to note here: you can't do this while bitcoind is running!!!

The reason why this is the case is because there can be only one live instance of the level DB open at any given time and bitcoind most likely already holds the db instance open, trying to open another instance (even just for reading) will result in an error. You can get snapshots of the database and iterate the snapshots tho. A snapshot allows you to guarantee a minimum downtime for bitcoind.

Finally, if you can't afford any downtime on your bitcoind client, then you might need to write a wrapper around either bitcoind or leveldb. You would have to introduce a level of abstraction somewhere.

Wrapper around bitcoind:

1. Shuts down bitcoind while you're making a snapshot.
2. Buffers any requests to bitcoind until you're done with the snapshot.
3. Starts bitcoind again.

Wrapper around leveldb:

The wrapper around level db is a little more difficult, you have to maintain control of the database instance and you'd have to actually go in and modify the bitcoind code, then build bitcoind and run it.