Sending messages from other languages to an IPython kernel

Question

Does anyone have any experience of communicating with IPython kernels from outside of Python?

If I were trying to send messages from a Python app to an IPython kernel, I'd use the zmq.kernelmanager API. As it is, I'll obviously need to write my own kernel manager in another language, but I can't find the information that I'm looking for about the low-level messaging protocols.

Is there an official spec or a 'cheat sheet' that documents the structure of the actual messages that get sent over 0MQ? This page describes a higher-level protocol than what I'm looking for... Will I have to manually pull apart the implementation to find what I want?

Answer 1

This is a document that desparately needs to exist, but the implementation of the wire protocol is implemented in a single object, so it shouldn't be too hard to grok from there. The message spec doc you linked covers the application-level content of each field, but not how it's actually serialized over zeromq. Assuming you have a mesage, as described in that doc, the wire format is pretty simple. It is a multipart zeromq message of at least six parts:

• The leading message parts are the zeromq routing identities (zero-to-many)
• These are followed by a delimiter message, with the bytes <IDS|MSG>
• The hmac digest of the message (an empty string '' if authentication is disabled)
• header
• parent_header
• metadata
• content

header, parent_header, metadata, and content are all described in the messaging doc - these are dictionaries, and serialized to bytes with whatever serialization is currently used. The default in IPython is utf8-encoded JSON, but arbitrary serialization is allowed (msgpack being the most common non-default). Not yet described in the docs is the digest, used for authentication. This is an MD5 HMAC Digest of the message. The key for the digest is found in the key field of the connection file. The 'message' used by the HMAC digest is the concatenation of the bytes of the serialized header, parent_header, metadata, and content, in the same order sent over the wire.

You can disable message signing by specifying the config value

Session.key = ''

to the IPython-related parts of your code, in which case the digest field will always be an empty string ''. I would recommend doing this while getting started, so you can work out the more interesting parts of the implementation first.

Here are a sample execute request and its reply actually sent by IPython.

The request:

[
  <IDS|MSG>
  6ea6b213262402cc1ad3c1d3e342a9f6
  {"date":"2013-04-27T23:22:13.522049","username":"minrk","session":"5b03b89a-93c9-4113-bb85-17ba57233711","msg_id":"c6d0f85e-fc25-4f1e-84e1-3d706b615393","msg_type":"execute_request"}
  {}
  {}
  {"user_variables":[],"code":"1\n","silent":false,"allow_stdin":true,"store_history":true,"user_expressions":{}}
]

and its reply:

[
  5b03b89a-93c9-4113-bb85-17ba57233711
  <IDS|MSG>
  47d1052f6e8f333d18480938ca91719b
  {"date":"2013-04-27T23:22:13.528239","username":"kernel","session":"d7eb303b-d2d0-4723-aef2-738545a8da11","msg_id":"9ed1d332-398c-4132-b203-1e7bf8fed712","msg_type":"execute_reply"}
  {"date":"2013-04-27T23:22:13.522049","username":"minrk","session":"5b03b89a-93c9-4113-bb85-17ba57233711","msg_id":"c6d0f85e-fc25-4f1e-84e1-3d706b615393","msg_type":"execute_request"}
  {"dependencies_met":true,"engine":"645fb29f-37ab-40c9-bc01-b7fbfe3c2112","status":"ok","started":"2013-04-27T23:22:13.524114"}
  {"status":"ok","execution_count":2,"user_variables":{},"payload":[],"user_expressions":{}}
]