What does "PmmREFCNT_dec: REFCNT decremented below 0" mean?

Question

I must have made perl very upset because it is giving me an error message which is not documented in perldiag:

PmmREFCNT_dec: REFCNT decremented below 0 for 53a6930!.

Depending on its mood, this is sometimes followed by:

*** glibc detected *** /usr/bin/perl: double free or corruption (!prev): 0x0000000004e58a60 *

...or more plainly by:

Segmentation fault

This, clearly, is fatal, but I have also tested it to be trappable. When using Try::Tiny, I will always catch the error at the same place, but when not using it, some more instructions are performed before the crash occurs. Also, even though my module is entirely determinist and I'm quite sure all dependencies are as well, the error doesn't occur all the times.

Unfortunately, the module that is giving me this problem is huge, with lots of dependencies, and I have not been able to replicate the problem on a smaller example. Hence, I cannot ask for help debugging it, but if someone familiar with Perl internals knows under what kind of circumstances this error occurs, that may help me (or whomever else will ever see this message) find the source of the problem and/or a workaround.

In case it is useful, the general idea is this. I have two classes, let's call them Thing and SetOfThings. SetOfThings has an attribute which is a set of Thing instances. Both class also have an explode method which does something like this:

# SetOfThings
sub explode {
    my $self = shift;
    my $new  = dclone $self;
    delete $new->{'some_attribute'};
    $new->set_of_things( map { $_->explode } $self->constraints );
    return $new;
}

# Thing
sub explode {
    my $self = shift;
    return  map { new Thing( do_something_fancy ) } keys %$self;
}

The error usually seems to occur when calling SetOfThings::explode or when calling SetOfThings::set_of_things as a getter.

Edit: Backtrace

I don't believe I am competent enough to interpret it, but I have obtained a backtrace from gdb:

#0  0x00007ffff70a6094 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff70a76a8 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff70aab1c in free () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007ffff7b0869b in Perl_hv_undef_flags () from /usr/lib/libperl.so.5.14
#4  0x00007ffff7b1ae66 in Perl_sv_clear () from /usr/lib/libperl.so.5.14
#5  0x00007ffff7b1b292 in Perl_sv_free2 () from /usr/lib/libperl.so.5.14
#6  0x00007ffff7b04bc3 in Perl_hv_free_ent () from /usr/lib/libperl.so.5.14
#7  0x00007ffff7b04e6e in ?? () from /usr/lib/libperl.so.5.14
#8  0x00007ffff7b08683 in Perl_hv_undef_flags () from /usr/lib/libperl.so.5.14
#9  0x00007ffff7b1ae66 in Perl_sv_clear () from /usr/lib/libperl.so.5.14
#10 0x00007ffff7b1b292 in Perl_sv_free2 () from /usr/lib/libperl.so.5.14
#11 0x00007ffff7b42cef in Perl_leave_scope () from /usr/lib/libperl.so.5.14
#12 0x00007ffff7b11112 in Perl_pp_leave () from /usr/lib/libperl.so.5.14
#13 0x00007ffff7b0bce6 in Perl_runops_standard () from /usr/lib/libperl.so.5.14
#14 0x00007ffff7aad815 in perl_run () from /usr/lib/libperl.so.5.14
#15 0x0000000000400f89 in main ()

Edit 2: Valgrind backtrace

This is what I got from running valgrind. Although I'm still not sure what is going on, at least now I know whom to blame. :-)

==27226== Invalid free() / delete / delete[] / realloc()
==27226==    at 0x4C27D4E: free (vg_replace_malloc.c:427)
==27226==    by 0xA138F42: PmmREFCNT_dec (in /usr/lib/perl5/auto/XML/LibXML/LibXML.so)
==27226==    by 0xA11D3FA: XS_XML__LibXML__Node_DESTROY (in /usr/lib/perl5/auto/XML/LibXML/LibXML.so)
==27226==    by 0x4EE770B: Perl_pp_entersub (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4E7AB90: Perl_call_sv (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4EEDBD8: Perl_sv_clear (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4EEE291: Perl_sv_free2 (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4ED7BC2: Perl_hv_free_ent (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4ED7E6D: ??? (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4EDB682: Perl_hv_undef_flags (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4EEDE65: Perl_sv_clear (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4EEE291: Perl_sv_free2 (in /usr/lib/libperl.so.5.14.2)
==27226==  Address 0x17d0b710 is 0 bytes inside a block of size 32 free'd
==27226==    at 0x4C27D4E: free (vg_replace_malloc.c:427)
==27226==    by 0xA138F42: PmmREFCNT_dec (in /usr/lib/perl5/auto/XML/LibXML/LibXML.so)
==27226==    by 0xA11D3FA: XS_XML__LibXML__Node_DESTROY (in /usr/lib/perl5/auto/XML/LibXML/LibXML.so)
==27226==    by 0x4EE770B: Perl_pp_entersub (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4E7AB90: Perl_call_sv (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4EEDBD8: Perl_sv_clear (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4EEE291: Perl_sv_free2 (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4ED7BC2: Perl_hv_free_ent (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4EDA919: Perl_hv_common (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4F0EEC7: Perl_pp_delete (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4EDECE5: Perl_runops_standard (in /usr/lib/libperl.so.5.14.2)
==27226==    by 0x4E80814: perl_run (in /usr/lib/libperl.so.5.14.2)

Answer 1

Quoting ikegami's comment because I couldn't formulate it better:

It's a bug in Perl or in an XS module. A variable should be freed when its reference count reaches zero, but something attempted to decrement a variable's reference count when it was already zero.

As the output of valgrind shows, in this specific instance, the problem is in XML::LibXML.

I reckon that updating XML::LibXML, as suggested by Sinan Ünür as soon as the problem will be understood and fixed is the way to go. Unfortunately, updating from 2.0001 (Debian stable version) to 2.0116 (CPAN version) did not fix it.

What did fix the problem, finally, was to modify SetOfThings::explode so that it creates a new instance and copies the attributes it needs rather than cloning the current instance and deleting the attributes is doesn't need:

sub explode {
    my $self = shift;
    my $new  = __PACKAGE__->new;
    $new->some_attribute('whatever');
    $new->set_of_things( map { $_->explode } $self->constraints );
    return $new;
}

One of the attributes of the SetOfThings object that was cloned then deleted was a DOM, which clearly XML::LibXML did not appreciate. Thanks to this knowledge and the comments posted, I was finally able to reproduce my problem in a very small script and post a bug report:

#!/usr/bin/perl

use strict;
use warnings;

use Clone 'clone';
use XML::LibXML;

my $dom1 = new XML::LibXML::Document;
my $dom2 = clone $dom1;

As pointed out by ikegami, cloning the Perl variable doesn't copy the underlying C structure used by the library. But XML::LibXML does provide a cloneNode method, therefore changing the last line to

my $dom2 = $dom1->cloneNode(1)

gives the desired result.