Menu
EDIT: The issue below was produced on stock Perl 5.16.3 on an AWS ec2 micro instance. It was ultimately solved by installing and switching to Perl 5.24.3 with perlbrew.
LWP::UserAgent seems to be causing some kind of memory leak when saving content.
use v5.10;
use strict;
use warnings;
use IO::Socket::SSL;
use LWP::UserAgent;
use Memory::Usage;
use Time::HiRes 'usleep';
my $ua = LWP::UserAgent->new;
my $mu = Memory::Usage->new;
$ua->ssl_opts(SSL_ocsp_mode => SSL_OCSP_NO_STAPLE); ## disable OCSP stapling
for (1..50) {
$mu->record("before request: $_");
$ua->get('http://www.wikipedia.org/wiki/Special:Random', ':content_file' => 'temp.html');
usleep 50000;
}
$mu->dump();
Output:
time vsz ( diff) rss ( diff) shared ( diff) code ( diff) data ( diff)
0 213964 ( 213964) 25112 ( 25112) 7532 ( 7532) 8 ( 8) 17976 ( 17976) before request: 1
0 225744 ( 11780) 31428 ( 6316) 8120 ( 588) 8 ( 0) 23580 ( 5604) before request: 2
0 226824 ( 1080) 32500 ( 1072) 8120 ( 0) 8 ( 0) 24660 ( 1080) before request: 3
0 227920 ( 1096) 33596 ( 1096) 8120 ( 0) 8 ( 0) 25756 ( 1096) before request: 4
0 229016 ( 1096) 34692 ( 1096) 8120 ( 0) 8 ( 0) 26852 ( 1096) before request: 5
1 230112 ( 1096) 35784 ( 1092) 8120 ( 0) 8 ( 0) 27948 ( 1096) before request: 6
1 231204 ( 1092) 36880 ( 1096) 8120 ( 0) 8 ( 0) 29040 ( 1092) before request: 7
1 232296 ( 1092) 37968 ( 1088) 8120 ( 0) 8 ( 0) 30132 ( 1092) before request: 8
1 233388 ( 1092) 39060 ( 1092) 8120 ( 0) 8 ( 0) 31224 ( 1092) before request: 9
1 234484 ( 1096) 40152 ( 1092) 8120 ( 0) 8 ( 0) 32320 ( 1096) before request: 10
1 235572 ( 1088) 41248 ( 1096) 8120 ( 0) 8 ( 0) 33408 ( 1088) before request: 11
2 236668 ( 1096) 42344 ( 1096) 8120 ( 0) 8 ( 0) 34504 ( 1096) before request: 12
2 237764 ( 1096) 43440 ( 1096) 8120 ( 0) 8 ( 0) 35600 ( 1096) before request: 13
2 238864 ( 1100) 44536 ( 1096) 8120 ( 0) 8 ( 0) 36700 ( 1100) before request: 14
2 239956 ( 1092) 45632 ( 1096) 8120 ( 0) 8 ( 0) 37792 ( 1092) before request: 15
2 241044 ( 1088) 46720 ( 1088) 8120 ( 0) 8 ( 0) 38880 ( 1088) before request: 16
3 242140 ( 1096) 47816 ( 1096) 8120 ( 0) 8 ( 0) 39976 ( 1096) before request: 17
3 243240 ( 1100) 48916 ( 1100) 8120 ( 0) 8 ( 0) 41076 ( 1100) before request: 18
3 244336 ( 1096) 50008 ( 1092) 8120 ( 0) 8 ( 0) 42172 ( 1096) before request: 19
3 245428 ( 1092) 51096 ( 1088) 8120 ( 0) 8 ( 0) 43264 ( 1092) before request: 20
3 246516 ( 1088) 52196 ( 1100) 8120 ( 0) 8 ( 0) 44352 ( 1088) before request: 21
4 247612 ( 1096) 53288 ( 1092) 8120 ( 0) 8 ( 0) 45448 ( 1096) before request: 22
4 248704 ( 1092) 54380 ( 1092) 8120 ( 0) 8 ( 0) 46540 ( 1092) before request: 23
4 249796 ( 1092) 55476 ( 1096) 8120 ( 0) 8 ( 0) 47632 ( 1092) before request: 24
4 250892 ( 1096) 56568 ( 1092) 8120 ( 0) 8 ( 0) 48728 ( 1096) before request: 25
4 251988 ( 1096) 57660 ( 1092) 8120 ( 0) 8 ( 0) 49824 ( 1096) before request: 26
4 253076 ( 1088) 58752 ( 1092) 8120 ( 0) 8 ( 0) 50912 ( 1088) before request: 27
5 254168 ( 1092) 59852 ( 1100) 8120 ( 0) 8 ( 0) 52004 ( 1092) before request: 28
5 255272 ( 1104) 60940 ( 1088) 8120 ( 0) 8 ( 0) 53108 ( 1104) before request: 29
5 256360 ( 1088) 62036 ( 1096) 8120 ( 0) 8 ( 0) 54196 ( 1088) before request: 30
5 257456 ( 1096) 63132 ( 1096) 8120 ( 0) 8 ( 0) 55292 ( 1096) before request: 31
5 258556 ( 1100) 64236 ( 1104) 8120 ( 0) 8 ( 0) 56392 ( 1100) before request: 32
6 259652 ( 1096) 65324 ( 1088) 8120 ( 0) 8 ( 0) 57488 ( 1096) before request: 33
6 260740 ( 1088) 66416 ( 1092) 8120 ( 0) 8 ( 0) 58576 ( 1088) before request: 34
6 261832 ( 1092) 67504 ( 1088) 8120 ( 0) 8 ( 0) 59668 ( 1092) before request: 35
6 262928 ( 1096) 68600 ( 1096) 8120 ( 0) 8 ( 0) 60764 ( 1096) before request: 36
6 264016 ( 1088) 69692 ( 1092) 8120 ( 0) 8 ( 0) 61852 ( 1088) before request: 37
7 265120 ( 1104) 70796 ( 1104) 8120 ( 0) 8 ( 0) 62956 ( 1104) before request: 38
7 266212 ( 1092) 71888 ( 1092) 8120 ( 0) 8 ( 0) 64048 ( 1092) before request: 39
7 267304 ( 1092) 72980 ( 1092) 8120 ( 0) 8 ( 0) 65140 ( 1092) before request: 40
7 268396 ( 1092) 74072 ( 1092) 8120 ( 0) 8 ( 0) 66232 ( 1092) before request: 41
7 269484 ( 1088) 75168 ( 1096) 8120 ( 0) 8 ( 0) 67320 ( 1088) before request: 42
7 270584 ( 1100) 76260 ( 1092) 8120 ( 0) 8 ( 0) 68420 ( 1100) before request: 43
8 271676 ( 1092) 77356 ( 1096) 8120 ( 0) 8 ( 0) 69512 ( 1092) before request: 44
8 272776 ( 1100) 78448 ( 1092) 8120 ( 0) 8 ( 0) 70612 ( 1100) before request: 45
8 273868 ( 1092) 79540 ( 1092) 8120 ( 0) 8 ( 0) 71704 ( 1092) before request: 46
8 274964 ( 1096) 80632 ( 1092) 8120 ( 0) 8 ( 0) 72800 ( 1096) before request: 47
8 276052 ( 1088) 81728 ( 1096) 8120 ( 0) 8 ( 0) 73888 ( 1088) before request: 48
9 277144 ( 1092) 82816 ( 1088) 8120 ( 0) 8 ( 0) 74980 ( 1092) before request: 49
9 278232 ( 1088) 83912 ( 1096) 8120 ( 0) 8 ( 0) 76068 ( 1088) before request: 50
Perl version is 5.16.3 ; Modules versions below using cpan -D
CPAN: Module::CoreList loaded ok (v5.20170923)
(no description)
S/SU/SULLR/IO-Socket-SSL-2.056.tar.gz
/home/ec2-user/perl5/lib/perl5/IO/Socket/SSL.pm
Installed: 2.056
CPAN: 2.056 up to date
Steffen Ullrich (SULLR)
[email protected]
...
CPAN: Module::CoreList loaded ok (v5.20170923)
(no description)
M/MI/MIKEM/Net-SSLeay-1.85.tar.gz
/home/ec2-user/perl5/lib/perl5/x86_64-linux-thread-multi/Net/SSLeay.pm
Installed: 1.85
CPAN: 1.85 up to date
Mike McCauley (MIKEM)
[email protected]
...
CPAN: Module::CoreList loaded ok (v5.20170923)
(no description)
O/OA/OALDERS/libwww-perl-6.33.tar.gz
/home/ec2-user/perl5/lib/perl5/LWP/UserAgent.pm
Installed: 6.33
CPAN: 6.33 up to date
Olaf Alders (OALDERS)
[email protected]
...
CPAN: Module::CoreList loaded ok (v5.20170923)
(no description)
O/OA/OALDERS/LWP-Protocol-https-6.07.tar.gz
/home/ec2-user/perl5/lib/perl5/LWP/Protocol/https.pm
Installed: 6.07
CPAN: 6.07 up to date
Olaf Alders (OALDERS)
[email protected]
606
There is a memory leak in the OCSP handling which is relevant if the target site uses OCSP stapling (only in this case is revocation checking done by default). A first leak in this area was fixed in Net::SSLeay1 1.83. But I've found a second leak which is not fixed yet although a patch is available (since a few minutes). See issue 125273 for details of the problem and the patch.
It is also possible to work around this problem by disabling OCSP stapling:
use IO::Socket::SSL;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts(SSL_ocsp_mode => SSL_OCSP_NO_STAPLE); ## disable OCSP stapling
$ua->get('http://www.wikipedia.org', ':content_file' => 'save.txt') for 0..500;
1 Net::SSLeay is the underlying library for IO::Socket::SSL which itself provides SSL functionality to LWP.
140
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
