Akka, Scalatra and Web state questions [closed]

Question

This is a two part question, firstly more of a design question than how to implement it, and secondly some implementation concerns with Akka.

I'm using Scalatra to build a REST service end point that when called will pull images from several sources, manipulate them and return them. This is potentially quite a long running process, and will likely be longer then is acceptable for a single http request/response cycle.

My thoughts on this is that when the call is made I will kick off a bunch of akka Actors to pull the image resources, and have them hand the result to Image processing Actors to scale etc. The initial request itself would return instantly some kind of Processing Id that can be used to make subsequent polling calls to another end point that will return the results as they are processed, with a flag to determine if there are more results available to let the client know to stop polling.

My questions, are as follows:

1. Does this make sense from a design point of view?
2. If I use this approach then each subsequent request to retrieve the processed images would have to have some kind of state awareness to know what images the client already has received. How would you manage this state?
3. I've never looked at this, but would a long running comet style HTTP request make more sense than polling in this situation?

Implementation Part

Assuming I do end up with a design similar to above, I'm very new to Scalatra & Akka (or any Actor paradigm) and I have a few questions.

Ok, first one is a Scala/Scalatra specific question I think. Ok, I looked at the Scalatra docs on akka http://www.scalatra.org/guides/async/akka.html, and in this example they setup the applications bootstrap as follows:

 // Get a handle to an ActorSystem and a reference to one of your actors
  val system = ActorSystem()
  val myActor = system.actorOf(Props[MyActor])

  // In the init method, mount your servlets with references to the system
  // and/or ActorRefs, as necessary.
  override def init(context: ServletContext) {
    context.mount(new PageRetriever(system), "/*")
    context.mount(new MyActorApp(system, myActor), "/actors/*")
  }

I assume that the bootstraping of scalatra happens once on application startup, so does system.actorOf(Props[MyActor]) create a single instance or one per request?

Secondly, say my MyActor class (or a more sensible name) did the following:

class MyActor extends Actor {
  def receive = {
    //Find the [ImageSearchingActor] actor in akka registry and send search message
    case initiateSearchRequest: InitiateSearchRequestMessage => TODO

    //Find free [ImageProcessingActors] in akka registry and send each Image url to process  
    case imageInformationFound : ImageInformationFoundMessage => TODO

    //Persist the result to a cache, or data store with the ProcessingId that all message will pass  
    case imageProcessed : ImageProcessedMessage =>  TODO
  }
}

Now, in this case multiple there are multiple places I will be retrieving images and so several Actors will be grabbing this data. When they find appropriate images several Actors will be used to process the images. If I go with my design then I need to flag somewhere that for a given ProcessingId there are no more processed images avaliable. This means I need to know when ALL image searching and image processing actors have finished for a given ProcessingId. How do I do this?

So that was a lot of question, info to consume I hope it made sense.

Cheers. Chris.

Answer 1

Quite a few questions here. Quick answers:

1. Yes, I think so.

2. You probably want to use one Actor class as an aggregator, master, or coordinator, which kicks off image retrieval and image processing Actor classes. The aggregator then contains the logic for when your overall computation can be considered complete. There are quite a few examples of this sort of thing floating around the internet, if you want concrete examples to match what you're doing, make sure that you look at Akka 2.1.x examples, as that's what you probably need if you're using the current Scalatra codebase.

3. It could be a good idea. Scalatra has an integration with the Atmosphere framework which makes it pretty easy to do websocket/comet long-polling. You can read about in the docs. Whether it makes sense for your application depends on a lot of factors, but it's worth looking at. In my experience socket programming can sometimes be amazing and sometimes be more hassle than it's worth - if you're going through a lot of proxies, for instance, websockets have problems unless SSL is in use. It is is a beautiful thing, though, to see information get pushed into the client as soon as it's available.

To the other question, about ActorSystem creation:

In Scalatra, your controllers are instantiated on a per-request basis; the ActorSystem in your example code only gets initialized once. Whenever a request comes in the door, a new controller instance is created, with a reference to the ActorSystem.

ActorSystem creation is a heavyweight operation, and should happen as few times as you can get away with.