Using eventlet with SQLAlchemy

Question

I'm using eventlet to build a simple website crawler starting from this example. I would like to use SQLAlchemy to store web pages' content and metadata for further processing. It's possible to use SQLAlchemy along with eventlet? Could anyone provide a simple example?

Answer 1

The OpenStack Compute (Nova) project uses SQLAlchemy and eventlet, you might be interested in their approach.

Answer 2

Sorry for late reply.

It would really depend on database connection library you use.

• For C extension library, like MySQLdb, Eventlet has db_pool module as used in OpenStack example. Basically, it creates a pool of OS threads to execute blocking DB operations. So you can't really have a lot of parallel queries, but most probably your database would not survive them anyway, so that's not an issue.
• For pure Python library, like myconnpy, pg8000, then just call eventlet.monkey_patch().
• As a special case, Eventlet supports monkey patching for psycopg2. It is a C extension but it is awesome enough to provide IO hooks, so you get best of both worlds. Again, simple call to monkey_patch() would do the trick.