DynamoDB Local Terrible Performance

Question

DynamoDB local is taking 100+ ms to perform a single put operation against my table. The docs say that throughput is ignored for local dbs, and is only limited by the speed of the hard disk/computer.

Compared to mongodb my write throughput is 100x slower than it should be. Is there something I can do to speed writes to DynamoDB local up?

I will try batch puts, but the problem still remains. At this point, it's going to take me years to input my (rather large) test data.

I'm using Clojure and Faraday as my client api, but have confirmed that is not the bottleneck.

What I tried

I've implemented batches at 25 per, which slowed down the total progress by about a factor of 25 :). So, even with batches, I'm getting a write speed of about 120 ms per item.

Using Mongo, even with conservative WriteConcern/ACKNOWLEDGED flag, results in about 250 microseconds per item (~500x faster), and that's without even needing to send batches. So it's not my harddrive or OS that's the problem.

Answer 1

It's been a while since you asked your question but I still want to give some background information.

MongoDB is so fast because it buffers the updates in memory before writing them to disk. You can read more about this in the official MongoDB FAQ here

DynamoDBLocal on the other hand is quite slow because it uses a SQLite database behind the scenes (You can check this by opening the *.db file in the DynamoDBLocal directory with a SQLite Browser) and SQLite has very slow write speeds compared to other databases. More info here.

DynamoDBLocal is just a basic tool for local development that wraps a subset of the DynamoDB API around a simple SQLite database. The real DynamoDB, of course, does not rely on SQLite and is much faster.

Answer 2

This is an old question, but is still quite relevant - according to Stackoverflow it has been viewed over 2,000 times over the years. As FLXN already explained in his answer, DynamoDB Local was meant to be a tool for developing applications - it wasn't meant to be a real installable replacement for DynamoDB. It is neither efficient enough, nor highly-available or durable enough, to serve as a real database.

But since recently, an alternative is available if you want a real (highly-available and efficient) installable database with DynamoDB API compatibility: ScyllaDB recently announced that the open-source Scylla database now has DynamoDB API support. This support is not yet perfect and still missing a few DynamoDB features, but compatibility is improving quickly and you can check out this document for the current status of Scylla's compatibility with the DynamoDB API.

Full disclosure: I'm a ScyllaDB employee, and one of the developers of Alternator - Scylla's support for the DynamoDB API.