Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

asyncore callbacks launching threads... ok to do?

I'm unfamiliar with asyncore, and have very limited knowledge of asynchronous programming except for a few intro to twisted tutorials.

I am most familiar with threads and use them in all my apps. One particular app uses a couchdb database as its interface. This involves longpolling the db looking for changes and updates. The module I use for couchdb is couchdbkit. It uses an asyncore loop to watch for these changes and send them to a callback.

So, I figure from this callback is where I launch my worker threads. It seems a bit crude to mix asynchronous and threaded programming. I really like couchdbkit, but would rather not introduce issues into my program.

So, my question is, is it safe to fire threads from an async callback?

Here's some code...

def dispatch(change):
    global jobs, db_url # jobs is my queue
    db = Database(db_url)
    work_order = db.get(change['id']) # change is an id to the document that changed. 
                                  # i need to get the actual document (workorder)

    worker = Worker(work_order, db) # fire the thread
    jobs.append(worker)
    worker.start()
    return


main()
.
.
.

consumer.wait(cb=dispatch, since=update_seq, timeout=10000) #wait constains the asyncloop.

Update:

After looking over this more, I have an additional question for the couchdbkit gurus. There will potentially be hundreds of threads using the database. As you can see in my code sample, I am instantiating a couchdbkit.Database object per thread. I think this may be wasteful. So, is it ok for a single database object to be used globally among the threads?

like image 809
sbartell Avatar asked Jan 16 '11 07:01

sbartell


1 Answers

Wouldn't this create a new thread every time the server returns a new document? I would guess that you are best creating a pool of worker threads before you call anything on the server, and just add a job to the queue that these threads are reading their work from in the dispatch method.

But there is no reason why mixing threading and asynchronous programming should be considered dangerous.

like image 133
Andrew Skirrow Avatar answered Nov 14 '22 05:11

Andrew Skirrow