Non-blocking queue of HTTP POST requests with persistence

Question

Before we develop our custom solution, I'm looking for some kind of library, which provides:

Non-blocking queue of HTTP requests

with these attributes:

• Persisting requests to avoid it's loss in case of:
  • network connectivity interruption
  • application quit, forced GC on background app
  • etc..
• Possibility of putting out all these fields:
  • Address
  • Headers
  • POST data

So please, is there anything usable right know, what could save us whole day on developing this?

Right now we don't need any callbacks on completed request and neither saving result data, as there won't be such.

Answer 1

In my humble opinion, a good and straightforward solution would be to develop your own layer (which shouldn't be so complicated) using a sophisticated framework for connection handling, such as Netty https://netty.io/ , together with a sophisticated framework for asynchronous processing, such as Akka http://akka.io/

Let's first look inside Netty support for http at http://static.netty.io/3.5/guide/#architecture.8 :

4.3. HTTP Implementation

HTTP is definitely the most popular protocol in the Internet. There are already a number of HTTP implementations such as a Servlet container. Then why does Netty have HTTP on top of its core?

Netty's HTTP support is very different from the existing HTTP libraries. It gives you complete control over how HTTP messages are exchanged at a low level. Because it is basically the combination of an HTTP codec and HTTP message classes, there is no restriction such as an enforced thread model. That is, you can write your own HTTP client or server that works exactly the way you want. You have full control over everything that's in the HTTP specification, including the thread model, connection life cycle, and chunked encoding.

And now let's dig inside Akka. Akka is a framework which provides an excellent abstraction on the top of Java concurrent API, and it comes with API in Java or Scala.

• It provides you a clear way to structure your application as a hierarchy of actors:
  • Actors communicate through message passing, using immutable message so that you have not to care about thread-safety
  • Actors messages are stored in message boxes, which can be durable
  • Actors are responsible for supervising their children
  • Actors can be run on one or more JVM and can communicate using a wide numbers of protocols
• It provides a lightweight abstraction for asynchronous processing , Future, which is easier to use then Java Futures.
• It provides other fancy stuff such as Event Bus, ZeroMQ adapter, Remoting support, Dataflow concurrency, Scheduler

---

Once you become familiar with the two frameworks, it turns out that what you need can easily be coded through them.

In fact, what you need is an http proxy coded in Netty, that upon a request receival sends immediately a message to an Akka Actor of type FSM (http://doc.akka.io/docs/akka/2.0.2/java/fsm.html) which using a durable mailbox (http://doc.akka.io/docs/akka/2.0.2/modules/durable-mailbox.html )

Answer 2

Here is a link to open-source library that was a Master Thesis of a student at Czech Technical University in Prague. It is very large and powerful library and mainly focuses on location. The good thing about it, though, is that it omitted the headers and other -ish that REST has.

It is the latest fork and hopefully it will give you at least inspiration for "own" solution.