Menu
I want to code a Server which handles Websocket Clients while doing mysql selects via sqlalchemy and scraping several Websites on the same time (scrapy). The received data has to be calculated, saved to the db and then send to the websocket Clients.
My question ist how can this be done in Python from the logical point of view. How do I need to set up the code structure and what modules are the best solution for this job? At the moment I'm convinced of using twisted with threads in which the scrape and select stuff is running. But can this be done an easier way? I only find simple twisted examples but obviously this seems to be a more complex job. Are there similar examples? How do I start?
580
The communication can be sent either way at any time during the lifetime of the WebSocket connection. The client and the server are continuously connected — data can be sent to the clients all the time, without any need to request it. It's fairly easy to work with WebSockets in Python.
In order to keep the session alive, the client could keep sending a heart-beat or a ping to the server. The retry decorator, every time there is WebSocketConnectionClosedException, reconnects to the WebSocket server in a delay of 2 seconds, keeping the session alive.
To secure the transport, you should use a secure protocol like https:// instead of the unsecure https:// protocol. For WebSockets, you should use the wss:// (TLS-encrypted WebSockets) protocol instead of the unsecure ws:// protocol.. Using the latter method leaves the connection susceptible to third-party interference.
WebSocket Client with PythonCreate a new File “client.py” and import the packages as we did in our server code. Now let's create a Python asynchronous function (also called coroutine). async def test(): We will use the connect function from the WebSockets module to build a WebSocket client connection.
Cyclone, a Twisted-based 'network toolkit', based on/similar to facebook/friendfeed's Tornado server, contains support for WebSockets: https://github.com/fiorix/cyclone/blob/master/cyclone/web.py#L908
Here's example code:
Here's an example of using txwebsocket:
You may have a problem using SQLAlchemy with Twisted; from what I have read, they do not work well together (source). Are you married to SQLA, or would another, more compatible OR/M suffice?
Some twisted-friendly OR/Ms include Storm (a fork) and Twistar, and you can always fall back on Twisted's core db abstraction library twisted.enterprise.adbapi. There are also async-friendly db libraries for other products, such as txMySQL, txMongo, and txRedis, and paisley (couchdb).
You could conceivably use both Cyclone (or txwebsockets) and Scrapy as child services of the same MultiService, running on different ports, but packaged within the same Application instance. The services may communicate, either through the parent service or some RPC mechanism (like JSONRPC, Perspective Broker, AMP, XML-RPC (2) etc), or you can just write to the db from the scrapy service and read from it using websockets. Redis would be great for this IMO.
176
Ideally you'll want to avoid writing your own WebSockets server, but since you're running Twisted, you might not be able to do that: there are several WebSockets implementations (see this search on PyPI). Unfortunately none of them are Twisted-based [Edit see @JP-Calderone's comment below.]
Twisted should drive the master server, so you probably want to begin with writing something that can be run via twistd (see here if your'e new to this). The WebSocket implementation mentioned by @JP-Calderone and Scrapy are both Twisted -based so they should be reasonable trivial to drive from your master Twisted-based server. SQLAlchemy will be more difficult, I've commented on this before in this question.
20
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
