Would twisted be a good choice for building a multi-threaded server?

Question

I need to pull from hundreds of pop3 email accounts, and i want to build a robust server to do this.

Would twisted be a good choice for this type of project?

Right now a simple prototype would be to pull from a single pop3 account, then it would pull from many but it would be a serialized process.

I want to create a server that has multiple threads so it can do things at the same time.

Answer 1

Twisted is an event-driven networking framework written in Python. It builds heavily on asynchronous and non-blocking features and is best conceived to develop networking applications that utilizes these. It has thread support for use cases where you can not provide for asynchronous non-blocking I/O. This is based on the fact that most of time is spent waiting in network I/O operations.

The two model that exploits this is threading model where you create multiple threads, each accomplishing a single task or a single process that uses non-blocking I/O to accomplish multiple task in a single process by interleaving multiple tasks. Twisted is very suitable for the second model.

Non-Blocking model

+--------------------------+
|task1 | wait period | comp|
+--------------------------+
       +--------------------------+
       |task2 | wait period | comp|
       +--------------------------+

You can develop a very robust server with Twisted and it has POP3 / IMAP support.

There is an example of how to build pop3 client with twisted.

Answer 2

Considering that the majority of your POP3 activity is going to be network I/O, this is where Twisted excels. You're not really threading so much as performing event-based asynchronous socket operations, which is the crowning glory of Twisted.

So, yes, Twisted would be a good choice for this type of project. It can do client and server operations equally well and it is almost trivial to spin up a new async TCP client and it already has a POP3 TCP Client available by default.