Has anyone used dataflow programming in a real project with a mainstream language?

Question

I am looking at using some Dataflow programming techniques in a clojure program but I am having difficulty in finding much information from projects using Java, C#, or other mainstream languages that have used such techniques in the real world. I would be grateful to hear if anyone has any expereinces they could share regarding this.

Answer 1

Here, we are! We've made... (quotation is from one of my older post):

We've designed and implemented a DF server for our automation project (dispatcher, component iterface, a bunch of components, DF language, DF compiler, UI). It is written in bare C++, and runs on several Unix-like systems (Linux x86, MIPS, avr32 etc., Mac OSX). It lacks several features, e.g. sophisticated flow control, complex thread control (there is only a not too advanced component for it), so it is just a prototype, even it works. We're now working on a full-featured server. We've learnt lot during implementing and using the prototype.

Also, we'll make a visual editor some day.

There're dataflow systems wich don't even mention dataflow approach:

• SynthEdit: http://www.synthedit.com/ - It's an audio related framework and component set for creating VST plugins
• TinyOS: http://www.tinyos.net/ - It's an embedded operating system/framework
• Digital synthetisers/samplers are dataflow systems, programmed - supposedly - in C or some parts in Assembly, check my answer to another post about some examples.
• Quartz Composer, a graphic magic tool for Mac,
• Blender has dataflow subsystem for image composing.

Writing a dataflow system is not rocket science. Here's my older post about the basics of dataflow framework.

The term dataflow is wide. There are realtime synchronous dataflow systems, like synthetisers and samplers, there are asynchronous ones, like our home aut. system (the system is in idle unless the user presses a button or a timer runs out), and there're even different architectures, like spreadsheets or make.

Wanna reading more about dataflow programming? Read J. Paul Morrison's site and book.

Answer 2

Pervasive DataRush is a framework for parallel dataflow programming for any JVM language, including Clojure.

Pervasive DataRush uses a dataflow architecture. The architecture implements a program that executes as a graph of computation nodes interconnected by dataflow queues. The nodes use the queues to share data. As the data is streaming, only data required by any active operation needs to be in memory at any given time, allowing very large data sets to be analyzed. Besides offering the potential for scaling to problems larger than available memory, dataflow graphs exploit multiple forms of parallelism.

Customers are using DataRush for big data analytics and data preparation (ETL).