Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is it possible to boost up views/indexes creation in couchdb by multiple cores?

Tags:

couchdb

We have just converted a database of app run-time logs from MySQL to CouchDB. There are around 3M docs converted, occupy around 7GB.

Then we scripted out our first view (with reduce function) and did a query to invoke the first run of that view, make out a view creation/generation. It should take some time and I just did a 'top' on that machine to see process/cpu load stuff. I just noticed that there are 2 heavy process:

beam.smp 
couchjs

It's interesting that they seems to occupy one core, even there are 4 cores on that machine. Is it possible to make couchdb make use of multiple cores while creating/updating views?

like image 644
Drake Guan Avatar asked Feb 18 '11 11:02

Drake Guan


People also ask

How can I speed up my CouchDB?

Using faster disks, striped RAID arrays and modern file systems can all speed up your CouchDB deployment.

What is the maximum number of views a document can have in CouchDB?

There is no hard limit on the number of views. There are a few things I would recommend though: First, split up your views among many design documents. My first thought is 1 per user, but you could probably sub-divide them further depending on how many views you actually have.

What is the maximum number of views a document can have?

There is a showing view limit when you create Public view. The limit is 50 by default.


1 Answers

Currently this is not supported. My guess is that good multicore support will add complexity to CouchDB.

So far, CouchDB always chooses simplicity instead of features. (Did you know that CouchDB is only 18,000 lines of code? Compare with 1 million in MySQL or 100,000 in SQLite.)

When a design document view index is building, it uses one couchjs process. If you have multiple design documents, then those will run in parallel. CouchDB will spawn one couchjs process per design document, and the OS will spread them across multiple cores.

However, if you really need performance, BigCouch is Apache CouchDB-compatible and does support parallelized view index building using an oversharded architecture.

like image 138
JasonSmith Avatar answered Oct 01 '22 22:10

JasonSmith