Bidirectional flow between D3.js frontend and Python Backend? / Interactive graphs in a website

Question

So this is somewhat similar to

What's easiest way to get Python script output on the web? and Matplotlib: interactive plot on a web server

but none of those used d3.js and I don't think they achieve the same level of interactive-ness.

So I am a newcomer to d3.js and frankly have no clue where I should start in this instance.

Project Flow:

1. Ask user name on the front end. Send this to backend python
2. get graph data (object + edges) for a depth of x where x is distance away from a starting node by Python calls to 3rd party website API
3. run some machine learning on the data (python)
4. display the graph + some numbers (maybe in the bottom right corner) in d3.js loop:
5. Have an interactive d3.js graph so that if I click a node, it will redo the calculation with that as the starting node
6. have it so that d3.js shows part of the text (first 10 chars?) of each node and then shows the full text once you mouse over.

(7) Bonus updating embeddable picture? Like those profile signatures?

Questions: How would I achieve this bidirectional communication? Where should I start? Is there a better way I can do this?

Sidenote: I'm planning to do this project using Google App Engine, I don't know if that lets you use it as a 'vm' though.

Thanks!

Edit: This looks a lot like Python at backend JS at frontend integration, but I don't know where to even start and I think they suggest more art-oriented tools as a result of his project.

Answer 1

It sounds like you need to write a web service in Python that consumes and returns JSON. I'm not sure if the data is originating on the client or the server, so for this example, I'm assuming it comes from the client.

1. POST your graph data in JSON format to your webservice
2. Do processing server-side
3. return JSON representing the results of the processing to the client
4. Render it with d3.js
5. User interaction on the client creates a new or modified data structure
6. Go back to step 1

Maybe your server-side web service method looks like this (this is pseudo-code based on web.py):

import simplejson

class ml(object):

    def POST(self,data=None):
        # turn the JSON string into a Python data structure
        d = simplejson.loads(data)
        # do stuff
        results = alter_data(d)
        web.header('Content-Type','application/json')
        # send JSON back to the client
        return simplejson.dumps(results)

And your client-side code might look something like this (again, just a rough sketch). This example assumes that you're using jQuery to do your AJAX stuff.

function render(data,textStatus) {
    d3
      .select('svg')
      .selectAll('circle')
      .data(data)
      .enter()
      .append('circle')
      .on('click',function() {
          $.getJSON(
              'your_webservice_url',
              // POST an updated data structure to your service
              { 'data' : alter_data(data) },
              // Call the render method again when new data is received
              render);
       });
}

d3.js' enter and exit methods make updating the visual representation of your data very easy. I recommend reading the documentation for those.

This example might give you some ideas about how to represent your graph data with JSON, and how to render it.

Check out the on method for an idea about how to trigger a POST to your web service when a node is clicked.

I'm going to skip the really specific stuff, like displaying text with d3.js. I'm sure you can figure out how to do that by perusing the d3.js documentation.