Most elegant way to implement Pipe and Filter Pattern

Question

I want to create a Pipe and Filter based data handler that should manipulate incoming data sets like so, but not neccessarily limited to:

source pipe(could be a data set from a db) <-sink-source-> filter(add an additional field) <-sink-source-> filter(manipulate some more data / remove ie nullify data set)

I have an idea how such architecture will look like in C/C++. But given all the goodies that come with C++11's functional aspects, I hope this task can be done in an elegant way so that it is able to:

• easily scale and use it in a multithreaded environment (e.g. by performing filter tasks as lambda funcions, thus probably avoiding at least some thread handling)
• easily add and remove filters at runtime
• handle input streams lazily

Answer 1

There is a draft in the upcoming C++14 standard that covers this area:

C++ Pipelines - ISO/IEC JTC1 SC22 WG21 N3534 = 2013-03-15

And here is an implementation for it:

code.google.com/p/google-concurrency-library/source/browse/include/pipeline.h

Answer 2

What you are describing is some sort of streaming architecture or pipeline architecture. Standard C++ doesn't have anything this specific, but it provides you, as the library author, the necessary primitives in the language and standard library in order to build out such an architecture. Simply create classes and interfaces as per usual object-oriented programming and then instantiate them and connect them in a pipeline as per your needs.

For example you may have a Source interface and a Sink interface - and a Filter abstract class that implements both Source and Sink, as well as a Pipe class that implements both Source and Sink and just passes the data straight through. This is just one of many ways to name and organize such a framework.