I need to run a function int f(int i)
with 10_000 parameters and it takes around 1sec to execute due to I/O time.
In a language like Python, I can use threads (or async/await
, I know, but I'll talk about it later) to parallelize this task.
If I want to always have 10 running threads, and to split the task between them, I can use ThreadingPool :
def f(p):
x = [...]
return x
p = ThreadPool()
xs = p.map(f, range(10_000))
But how does it work ? If I want to implement a similar thing with, let's say NodeJS and f = http("www.google.com", callback)
, where should I begin ? What's the algorithms for this kind of problem ?
Again, I'd like to get 10 requests at the same time, and when one is finished the next one should start.
queue = ["www.google.com", "www.facebook.com"]
var f = function(url) {
http.get(url, (e) => {
const newUrl = queue.pop();
f(newUrl);
});
};
for (var i = 0; i < 10; i++) {
f(queue.pop());
}
In Node. js there are two types of threads: one Event Loop (aka the main loop, main thread, event thread, etc.), and a pool of k Workers in a Worker Pool (aka the threadpool). If a thread is taking a long time to execute a callback (Event Loop) or a task (Worker), we call it "blocked".
Worker Threads in Node. js is useful for performing heavy JavaScript tasks. With the help of threads, Worker makes it easy to run javascript codes in parallel making it much faster and efficient. We can do heavy tasks without even disturbing the main thread.
No, the answer doesn't have to be multi-threaded.
Node. js is Single Threaded, i.e. it executes the code in a single sequence or direction. At a given time, only a single task/ call is executed. Asynchronous and Single-Threaded: Execution doesn't wait for the current request to complete and moves to the next request/call.
Thread pool. In computer programming, a thread pool is a software design pattern for achieving concurrency of execution in a computer program. Often also called a replicated workers or worker-crew model, a thread pool maintains multiple threads waiting for tasks to be allocated for concurrent execution by the supervising program.
In case of thread pool, a group of fixed size threads are created. A thread from the thread pool is pulled out and assigned a job by the service provider. After completion of the job, thread is contained in the thread pool again.
ThreadPoolExecutor class allows to set the core and maximum pool size.The runnables that are run by a particular thread are executed sequentially. Thread Pool Initialization with size = 3 threads. Task Queue = 5 Runnable Objects
If the tasks are very contrasting then it makes sense to use different thread pools for different types of tasks so as to tune them properly. You can restrict maximum number of threads that can run in JVM, reducing chances of JVM running out of memory.
Reimplementation of that Bluebird function I linked to:
const mapWithConcurrency = async (values, concurrency, fn) => {
let i = 0;
let results = values.map(() => null);
const work = async () => {
while (i < values.length) {
const current = i++;
results[current] = await fn(values[current]);
}
};
await Promise.all(Array.from({length: concurrency}, work));
return results;
};
mapWithConcurrency(Array.from({length: 30 * 15}, (_, i) => i), 10, async i => {
const el = document.body.appendChild(document.createElement('i'));
el.style.left = 5 * (i % 30) + 'px';
el.style.top = 5 * (i / 30 | 0) + 'px';
await new Promise(resolve => { setTimeout(resolve, Math.random() * 500); });
el.style.background = 'black';
return 2 * i;
}).then(results => {
console.log(results.length, results.every((x, i) => x === 2 * i));
});
i {
background: grey;
transition: background 0.3s ease-out;
position: absolute;
width: 5px;
height: 5px;
}
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With