Menu
I have to migrate 5 million records from PostgreSQL to MongoDb.
I tried using mongify for the same but as it runs on ruby and I am not at all acquainted with ruby i couldn't solve the errors posed by it.
So, I tried writing a code myself in node.js that would first convert PostgreSQL data into JSON and then insert that JSON into mongoDb.
But, this failed as it ate a lot of RAM and not more than 13000 records could be migrated.
Then I thought of writing code in Java because of its garbage collector. It works fine in terms of RAM utilization but the speed is very slow (around 10000 records/hour). At this rate it would take me days to migrate my data.
So, Is there a more efficient and faster way of doing this? Would a python program be faster than the Java program? Or is there any other ready-made tool available for doing the same?
My system configuration is : OS - Windows 7 (64 bit), RAM - 4GB, i3 processor
983
Export the data from your PostgreSQL databases by piping the result of an SQL query into a COPY command, outputting the result either as JSON or TSV. Restructure the data to fit your MongoDB schema by using mongoimport (or as an alternative: use bulkWrite operations to load the data).
If you are looking for a distributed database for modern transactional and analytical applications that are working with rapidly changing, multi-structured data, then MongoDB is the way to go. If a SQL database fits your needs, then Postgres is a great choice.
MongoDB Performance. Various benchmarks have shown that PostgreSQL outperforms MongoDB for data warehousing and data analysis workloads. But in comparing JSON operations between PostgreSQL and MongoDB, there are benchmarks that show an advantage for both databases.
Seems like I am late to the party. However, this might come in handy to somebody, someday!!!!
The following python-based migration framework should come in handy.
https://github.com/datawrangl3r/pg2mongo
Answering to your performance, the migration of each JSON object will be dynamic and there shouldn't be any memory lock issues when you use the above framework.
Hope it helps!!
199
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
