Importing 60,000 nodes

Question

I'm using Table Wizard + Migrate module to import nodes into my Drupal installation.

I need to import around 60,000 questions / answers (they are both nodes) and I thought it would have been an easy task.

However, the migrate process imports 4 nodes per minute, and it would take approximately 11 days to finish the importing.

I was wondering if I can make it faster by importing directly in mysql. But I actually need to create 60,000 nodes. I guess Drupal is going to store additional information in other tables... and it is not that safe.

what do you suggest me to do ? Wait 10 days ? Thanks

Answer 1

Table migrate should be orders of magnitude faster than that.

Are you using pathauto?

If yes, try disabling the pathauto module, often causes big performance problems on import.

Second, if disabling pathauto doesn't work, turn off all non-essential modules you may have running - some modules do crazy stuff. Eliminate other modules as the sources of the problem.

Third, is MySQL db log turned on? That can have a big performance impact - not the level you are talking about, but its something to consider.

Third, install xdebug, and tail your mysql log to see exactly whats happening.

What is your PHP memory limit?

Do you have plenty of disk space left?

Answer 2

If you're not doing it, you should use drush to migrate the nodes in batches. You could even write a shell script for it, if you want it automated. Using the command line should lower the time it takes to import the nodes a lot. With a script, you can make it an automated task that you don't have to worry about.

One thing I want to note though, 4 nodes per minute is very low. I once needed to import some nodes from a CSV file, using migrate etc. I needed to import 300 nodes, with location, 4-5 CCK fields and I did it in a matter of seconds. So if you only import 4 nodes per minute, you either have extremely complex nodes, or something fishy is going on.

What are the specs of the computer you are using for this? Where's the import source located?