10 millions threads in clojure

Question

I am working on an agent-based simulation with approx. 10 million agents.

The agents will be doing nothing for days/weeks at a time, and when they do have to do something, it is not cpu intensive (should take < 1ms).

Is spawning 10 million threads a reasonable thing to try to do? I have read up on Thread/sleep and future which seem promising.

Any advice or suggested reading on such a project would be greatly appreciated.

Answer 1

If you use Clojure's built in agents, these run in a thread pool and will consume only a small amount of ram. events can be dispatched to these agents in various ways depending on your needs. It is fine to have 10M agents in a Clojure program. It may also be worth using one of the async programing libraries such as core.async and have 10M go blocks instead of agents.

Answer 2

Do those agents have significantly different behavior? Or can their behavior be derived from a few models and further configured with data? If you answer no to both questions you might want to consider using asynchronous queues as the main building block of your design.

You should definitely take Charles Duffy's advice seriously and find out more about how what Thread/sleep does and how threads work in general. But I think the issue with the design is not so much technical here as much as relating building blocks of a program with real world objects to a fault.

If you can provide some more details of the nature of what you're trying to accomplish, I might be able to edit this answer and provide a more concrete answer.