Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

R job job queueing / workload management

I have been spending some time on Google looking for a queuing / load balancer package for R.

What I am trying to achieve:

  • executing multiple independant from eachother R functions from remote front ends
  • taking advantage of 2 dual-core servers as R backend

Knowing that:

  • each functions is usually processed in 10 to 30 seconds
  • every 5 min on average (but it can at the same time) a set of 8-15 functions to be executed is being sent to the backend (queued for processing: first in first out). The 5 min is an average, several sets can be sent at the same time as well
  • the 2x2 R instance would already be running, with the required packages loaded, they are always the same, so no need to re-load them all the time
  • input amount of data being transfered is very low: 50k max

There is no code parallelization subject here (snow, snowfall foreach, condor and other traditionnal cluster solutions)

Would you know a good package/tool designed for R which could help ?

Thanks a lot !

like image 634
Sam Avatar asked May 18 '11 13:05

Sam


1 Answers

This sounds like a reasonable context for using RApache, which can instantiate several R instances and necessary packages.

like image 137
Iterator Avatar answered Nov 09 '22 21:11

Iterator