Upgrade all modules installed by local::lib

Question

I've been using local::lib to handle the installation of Perl modules on a server so I can get the right versions for some development work without polluting the system installation.

However, the system administrator has recently upgraded Perl from 5.16 to 5.18 and I'm now getting errors relating to binary modules, e.g.

perl -e 'use Scalar::Util'
Perl API version v5.16.0 of List::Util does not match v5.18.0 at /usr/lib64/perl5/5.18.2/XSLoader.pm line 92.
Compilation failed in require at /home/paul/perl5/lib/perl5/x86_64-linux/Scalar/Util.pm line 11.
Compilation failed in require at -e line 1.
BEGIN failed--compilation aborted at -e line 1.

My understanding is that I can fix the problem by forcing local::lib to rebuild all of its modules, but I can't find anything in the documentation which tells me how to do this, or even how to get a list of all the modules that have been installed using local::lib (if I had that in a 'one module per line' text file, I could easily write a Bash script to process it).

Is this possible, or do I have to remove the ~/perl5 directory and reinstall all the modules from scratch (possibly missing some as I can't remember them all)?

Answer 1

If you are using cpanm you can force it to rebuild modules in your local::lib location by using the -L and --reinstall switches:

list_modules | cpanm -L ~/perl5 --reinstall

where list_modules is a script that feeds the names of modules to cpanm (there's also an -f option to cpanm). This script could be like @ikegami's above or something like cpan-outdated (which only lists outdated modules however). Here is a hackish attempt that mostly worked for me recently (note: ikegami's is probably better) - when it fails or the script gives cpanm a module name it doesn't recognize, cpanm keeps going and doesn't seem to break anything (but make backups):

cd $PERL_LOCAL_LIB_ROOT  
perl -MFile::Find -MConfig -E'
      find { 
        wanted => sub { 
                       $mod = $_ if /\.pm\z/; 
                       $mod =~ s/lib\/perl5\/auto\/.*//g;
                       $mod =~ s/lib\/perl5\/\Q$Config{archname}\E\/.*//g;
                       $mod =~ s/lib\/perl5\///g;
                       push @mods, $mod unless $mod =~ /^$/;
                      },no_chdir=>1 
           },"lib/perl5"; @modhash{@mods}=(); say for sort keys %modhash '

By changing where and what File::Find finds with wanted() you can feed a different list of modules to cpanm. It would be nice if cpan or cpanm had an internal _method or -switch that allowed you to force the rebuilding of local::lib installed modules that use XS. Is there such a thing?

It was cpanm, carton (and seeing the node.js tool npm in action) that inspired me to do a lot more local::lib based installs. Now the CORE bundled "CPAN client" that ships with perl (cpan) seems to be getting more automagical and easier to use as well. I really like local::lib since it allows you to use the system perl but manage your own module stack without system level privileges. However, it can be easier overall to manage changes and upgrades if you use perlbrew to run a "non system" ("local") perl. Of course you can do this and still have local::lib, carton etc. manage a directory or application specific stack of modules.

1). Another way to do an upgrade is to use perllocal to generate a list of your currently installed modules (NB I'm not entirely sure if perllocal.pod is kept in a reliable/useful state). [Edit: In fact perllocal keeps a history of your installed modules rather than a list of those currently installed. You'll want to filter this or you will end up reinstalling the entire series of module versions one after the other!). This perlmonks node shows how to clean up your perllocal.pod: http://www.perlmonks.org/?node_id=483020. I prefer to keep the history.]

To parse perllocal.pod for input to cpanm, search through the file and saving the matches in an array, then split them by two's to create a hash from the array elements (key,value,key,value). More recent installations and versions numbers are lower in the file, so you can create a hash with module names as keys and have the values updated by later entries:

 perl -ne 'push @arr, grep {defined}
 (/\A=head2.*:\s+C<Module>\s+L<(.*)\||.*C<VERSION:\s(.*)>\Z/msx); }{
 %h = map{ split/,/,$_,2 } @arr; print "$_\@$h{$_}\n" for keys %h' perllocal.pod

(NB: this doesn't error check - entries in perllocal.pod occasionally lack VERSION data and other oddities so beware.)

2). But this method pales in comparison to the shell script mentioned by ilmari in #perl-help on IRC. It uses jq - a commandline utility you really need to have (you'll want it more after this). If you have used cpanm to install your modules it will have created install.json files. You can leverage that to make list to feed to cpanm for reinstalling your current set of modules:

 find ~/perl5/ -name install.json -exec jq '.name + "@" + .version' {} +

Very fast, very simple and you can combine it with the cpanm method of using curl to self-install in order to rebuild your modules.

3). If you use perlbrew to manage your perl installations you can easily copy and reinstall all your modules from one perl version to another.

HTH!

Answer 2

Update: For some time now, INSTALL_BASE has been producing a better directory structure that avoids this problem for new installs.

And that's why the perl Makefile.PL INSTALL_BASE=... convention (and the corresponding one for Build.PL) used by install::lib sucks.

Removing (or renaming the directory so you have backup) is the easiest solution. You can find out what you had installed by looking for .pm files.

cd ~
mv perl5{,16}
cd perl516/lib/perl5
find -name '*.pm' | xargs perl -MConfig -E'
   for (@ARGV) {
      s!^\./!!;
      s!^5\.\d+\.\d+/!!;
      s!^x86_64-linux/!!;
      s!^auto/!!;
      s!\.pm\z!!;
      s!/!::!g;
      say;
   }
' | xargs cpan

(Do a dry run — one without the trailing | xargs cpan — first.)

Note that if you don't want to be at the mercy of your admin's upgrades, you can use perlbrew to easily install a whole build of Perl in your home dir.