strange memory leak with python + paramiko

Question

I have an (apparent) memory leak in a python script that I can't quite explain (the resident memory just keeps growing). It started off with about 6MB resident, I left it running overnight and it had gotten to over 200MB (I did that to rule out a sawtooth memory usage pattern due to gc). I've condensed it down to this script:

import sys
import time
import paramiko

def update():
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    try:
        ssh.connect(hostname='localhost')
    finally:
        ssh.close()

def main():
    while(True):
        update()
        time.sleep(0.001)

if __name__ == '__main__':
    sys.exit(main())

I thought the problem might be that I keep instantiating a new SSHClient and they somehow weren't getting thrown out, but this version leaks memory even faster!

import sys
import time
import paramiko

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
def update():
    global ssh
    try:
        ssh.connect(hostname='localhost')
    finally:
        ssh.close()

def main():
    while(True):
        update()
        time.sleep(0.001)

if __name__ == '__main__':
    sys.exit(main())

If anyone could shed some light on this, or if I'm just being dumb and someone can point out why I'd be most appreciative. Thanks

Answer 1

I managed to reproduce. When investigating, i found that most probably, the leak is connected to libssl, because there is growing allocation

Before:

35aaa53000-35aaa5b000 rw-p 00053000 00:11 3360939                        /usr/lib64/libssl.so.1.0.0j
...
7f4530000000-7f453013b000 rw-p 00000000 00:00 0 
Size:               1260 kB
Rss:                1012 kB
Pss:                1012 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:      1012 kB
Referenced:         1012 kB
Anonymous:          1012 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
7f453013b000-7f4534000000 ---p 00000000 00:00 0 
Size:              64276 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB

After some time:

35aaa53000-35aaa5b000 rw-p 00053000 00:11 3360939                        /usr/lib64/libssl.so.1.0.0j
...
7f4530000000-7f4530250000 rw-p 00000000 00:00 0 
Size:               2368 kB
Rss:                2120 kB
Pss:                2120 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:      2120 kB
Referenced:         2120 kB
Anonymous:          2120 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
7f4530250000-7f4534000000 ---p 00000000 00:00 0 
Size:              63168 kB
Rss:                   0 kB
Pss:                   0 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:            0 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB

Seems like bug in libssl or paramiko itself, because the gc.garbage is empty and len(gc.get_objects()) is constant, meaning there are no unbreakable cycles and no new python objects (using your first version).

BTW, you can run gc.collect() each iteration to avoid the sawtooth.

Answer 2

Seems at least one other user has run into this recently on Paramiko's IRC; I created a ticket to track it here: https://github.com/paramiko/paramiko/issues/182 - sadly I've no wisdom to share myself as this level of problem is not one I have experience troubleshooting.