Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

In Perl, how can I wait for threads to end in parallel?

I have a Perl script that launches 2 threads,one for each processor. I need it to wait for a thread to end, if one thread ends a new one is spawned. It seems that the join method blocks the rest of the program, therefore the second thread can't end until everything the first thread does is done which sort of defeats its purpose.

I tried the is_joinable method but that doesn't seem to do it either.

Here is some of my code :

use threads;
use threads::shared;

@file_list = @ARGV;      #Our file list
$nofiles = $#file_list + 1; #Real number of files 
$currfile = 1;     #Current number of file to process

my %MSG : shared;              #shared hash

$thr0 = threads->new(\&process, shift(@file_list));
$currfile++;
$thr1 = threads->new(\&process, shift(@file_list));
$currfile++;

while(1){
 if ($thr0->is_joinable()) {
  $thr0->join;
        #check if there are files left to process
  if($currfile <= $nofiles){ 
   $thr0 = threads->new(\&process, shift(@file_list));
   $currfile++;
  }
 }

 if ($thr1->is_joinable()) {
  $thr1->join;
        #check if there are files left to process
  if($currfile <= $nofiles){
   $thr1 = threads->new(\&process, shift(@file_list));
   $currfile++;
  }
 }
}

sub process{
       print "Opening $currfile of $nofiles\n";
       #do some stuff
       if(some condition){
               lock(%MSG);
               #write stuff to hash
       }
       print "Closing $currfile of $nofiles\n";
}

The output of this is :

Opening 1 of 4
Opening 2 of 4
Closing 1 of 4
Opening 3 of 4
Closing 3 of 4
Opening 4 of 4
Closing 2 of 4
Closing 4 of 4
like image 420
Pmarcoen Avatar asked Mar 16 '10 10:03

Pmarcoen


People also ask

Do threads really run in parallel?

Concurrency and Parallelism In the same multithreaded process in a shared-memory multiprocessor environment, each thread in the process can run concurrently on a separate processor, resulting in parallel execution, which is true simultaneous execution.

How do I use multithreading in Perl?

Creating Threads Like any other module, you need to tell Perl that you want to use it; use threads; imports all the pieces you need to create basic threads. The create() method takes a reference to a subroutine and creates a new thread that starts executing in the referenced subroutine.

Can we run two threads simultaneously?

Within a process or program, we can run multiple threads concurrently to improve the performance. Threads, unlike heavyweight process, are lightweight and run inside a single process – they share the same address space, the resources allocated and the environment of that process.

Does Perl support multithreading?

Perl can do asynchronous programming with modules like IO::Async or Coro, but it's single threaded. You can compile Perl with threads, which provide multi-threaded computing.


2 Answers

First off, a few comments on the code itself. You need to make sure you have:

use strict;
use warnings;

at the beginning of every script. Second:

@file_list = @ARGV;      #Our file list
$nofiles = $#file_list + 1; #Real number of files 

is unnecessary as an array in scalar context evaluates to the number of elements in the array. That is:

$nofiles = @ARGV;

would correctly give you the number of files in @ARGV regardless of the value of $[.

Finally, the script can be made much simpler by partitioning the list of files before starting the threads:

use strict; use warnings;

use threads;
use threads::shared;

my @threads = (
    threads->new(\&process, @ARGV[0 .. @ARGV/2]),
    threads->new(\&process, @ARGV[@ARGV/2 + 1 .. @ARGV - 1]),
);

$_->join for @threads;

sub process {
    my @files = @_;
    warn "called with @files\n";
    for my $file ( @files ) {
        warn "opening '$file'\n";
        sleep rand 3;
        warn "closing '$file'\n";
    }
}

Output:

C:\Temp> thr 1 2 3 4 5
called with 1 2 3
opening '1'
called with 4 5
opening '4'
closing '4'
opening '5'
closing '1'
opening '2'
closing '5'
closing '2'
opening '3'
closing '3'

Alternatively, you can let the threads move on to the next task as they are done:

use strict; use warnings;

use threads;
use threads::shared;

my $current :shared;
$current = 0;

my @threads = map { threads->new(\&process, $_) } 1 .. 2;
$_->join for @threads;

sub process {
    my ($thr) = @_;
    warn "thread $thr stared\n";

    while ( 1 ) {
        my $file;
        {
            lock $current;
            return unless $current < @ARGV;
            $file = $ARGV[$current];
            ++ $current;
        }
        warn "$thr: opening '$file'\n";
        sleep rand 5;
        warn "$thr: closing '$file'\n";
    }
}

Output:

C:\Temp> thr 1 2 3 4 5
thread 1 stared
1: opening '1'
1: closing '1'
1: opening '2'
thread 2 stared
2: opening '3'
2: closing '3'
2: opening '4'
1: closing '2'
1: opening '5'
1: closing '5'
2: closing '4'
like image 195
Sinan Ünür Avatar answered Oct 13 '22 04:10

Sinan Ünür


I think you need to move the code that pulls the next file from the list into the threads themselves.

So every thread would not just process one file, but continue to process until the list is empty.

This way, you also save on the overhead of creating new threads all the time.

Your main thread will then join both of them.

Of course, this requires synchronization on the list (so that they do not pull the same data). Alternately, you could split the list into two (one for each thread), but that might result in an unlucky distribution.

(PS: No Perl God, just a humble monk)

like image 33
Thilo Avatar answered Oct 13 '22 03:10

Thilo