Segmentation Fault in Aerospike Python Client

Question

I am using Aerospike Python CLient to put and get some key-value pair:

aerospike_dca_client.put(key,bin)
...
...
bls_key_a = (AEROSPIKE_NAMESPACE,SET_NAME,bls_key)
hp_bid_str_a = aerospike_dca_client.get(bls_key_a)[3];
#Terminates at the line above

However after the put, my program terminates with SEGFAULT during Get command. Following is the stacktrace I have got from GDB:

Program received signal SIGSEGV, Segmentation fault.
as_shm_node_get (cluster=0x353bd90508, ns=0x7fffffffc851 "test", digest=0x7fffffffc8e1 "/\231!\221h\223\240\021KX\377\357O\346u\214q\356\267#\177", write=false, replica=AS_POLICY_REPLICA_MASTER)
    at src/main/aerospike/as_shm_cluster.c:431
    431 src/main/aerospike/as_shm_cluster.c: No such file or directory.
        in src/main/aerospike/as_shm_cluster.c
        Missing separate debuginfos, use: debuginfo-install blas-3.2.1-4.el6.x86_64
        (gdb) bt
#0  as_shm_node_get (cluster=0x353bd90508, ns=0x7fffffffc851 "test", digest=0x7fffffffc8e1 "/\231!\221h\223\240\021KX\377\357O\346u\214q\356\267#\177", write=false, replica=AS_POLICY_REPLICA_MASTER)
            at src/main/aerospike/as_shm_cluster.c:431
#1  0x00007fffdf219ce5 in as_node_get (cluster=0x353bd90508, err=0x7fffffffc920, cn=0x7fffffffc7c0, command=0x7fffffffc740 "\002\003", command_len=82, timeout_ms=1000, retry=1,
                parse_results_fn=0x7fffdf21a97b <as_command_parse_result>, parse_results_data=0x7fffffffc7e0) at src/include/aerospike/as_cluster.h:562
#2  as_command_execute (cluster=0x353bd90508, err=0x7fffffffc920, cn=0x7fffffffc7c0, command=0x7fffffffc740 "\002\003", command_len=82, timeout_ms=1000, retry=1,
                    parse_results_fn=0x7fffdf21a97b <as_command_parse_result>, parse_results_data=0x7fffffffc7e0) at src/main/aerospike/as_command.c:435
#3  0x00007fffdf211a67 in aerospike_key_get (as=0x17dead0, err=0x7fffffffc920, policy=0x17dfbb0, key=<value optimized out>, rec=0x7fffffffc848) at src/main/aerospike/aerospike_key.c:114
#4  0x00007fffdf1e4295 in AerospikeClient_Get_Invoke (self=0x720d30, py_key=0x7fffb0b05730, py_policy=<value optimized out>) at src/main/client/get.c:96
#5  0x00007fffdf1e44a0 in AerospikeClient_Get (self=0x720d30, args=<value optimized out>, kwds=<value optimized out>) at src/main/client/get.c:174
#6  0x00007ffff7d12f24 in call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4033
#7  PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#8  0x00007ffff7d1399e in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4119
#9  call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#10 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#11 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7ffff7b0ee30, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=1, kws=0x7ffff7bc6068, kwcount=0,
                        defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3265
#12 0x00007ffff7c927c8 in function_call (func=0x7fffdf1936e0, arg=0x7fffb0af9450, kw=0x7fffde2dfd70) at Objects/funcobject.c:526
#13 0x00007ffff7c631a3 in PyObject_Call (func=0x7fffdf1936e0, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:2529
#14 0x00007ffff7d11746 in ext_do_call (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4346
#15 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2718
#16 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7fffe041f6b0, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=1, kws=0x1e142b0, kwcount=0, defs=0x0,
                            defcount=0, closure=0x7fffde39bbe0) at Python/ceval.c:3265
#17 0x00007ffff7d12c52 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4129
#18 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#19 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#20 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7fffe042b1b0, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=4, kws=0x7fffcbc517e0, kwcount=0,
                                defs=0x7fffdffa8428, defcount=1, closure=0x0) at Python/ceval.c:3265
#21 0x00007ffff7d12c52 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4129
#22 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#23 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#24 0x00007ffff7d1399e in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4119
#25 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#26 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#27 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7fffe041f730, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=2, kws=0x1d6c958, kwcount=0, defs=0x0,
                                    defcount=0, closure=0x0) at Python/ceval.c:3265
#28 0x00007ffff7d12c52 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4129
#29 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#30 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#31 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7ffff1ac6030, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=1, kws=0x7fffde5b85b8, kwcount=1,
                                        defs=0x7fffdf18bfa8, defcount=1, closure=0x0) at Python/ceval.c:3265
#32 0x00007ffff7d12c52 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4129
#33 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#34 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#35 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7ffff1ac6130, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=0, kws=0x662d48, kwcount=1,
                                            defs=0x7fffdf18bfe8, defcount=1, closure=0x0) at Python/ceval.c:3265
#36 0x00007ffff7d12c52 in fast_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4129
#37 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4054
#38 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2679
#39 0x00007ffff7d14a9e in PyEval_EvalCodeEx (co=0x7ffff1ac6230, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0,
                                                defcount=0, closure=0x0) at Python/ceval.c:3265
                                                ---Type <return> to continue, or q <return> to quit---
#40 0x00007ffff7d14bb2 in PyEval_EvalCode (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>) at Python/ceval.c:673
#41 0x00007ffff7d34850 in run_mod (fp=0x65f7b0, filename=<value optimized out>, start=<value optimized out>, globals=0x7ffff7b9b168, locals=0x7ffff7b9b168, closeit=1, flags=0x7fffffffe0e0)
                                                    at Python/pythonrun.c:1377
#42 PyRun_FileExFlags (fp=0x65f7b0, filename=<value optimized out>, start=<value optimized out>, globals=0x7ffff7b9b168, locals=0x7ffff7b9b168, closeit=1, flags=0x7fffffffe0e0) at Python/pythonrun.c:1363
#43 0x00007ffff7d34a2f in PyRun_SimpleFileExFlags (fp=0x65f7b0, filename=0x7fffffffe4d9 "bidlandscape_sf.py", closeit=1, flags=0x7fffffffe0e0) at Python/pythonrun.c:955
#44 0x00007ffff7d4a194 in Py_Main (argc=<value optimized out>, argv=<value optimized out>) at Modules/main.c:640
#45 0x000000353ba1ed5d in __libc_start_main (main=0x400710 <main>, argc=7, ubp_av=0x7fffffffe208, init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>,
                                                        stack_end=0x7fffffffe1f8) at libc-start.c:226
#46 0x0000000000400649 in _start ()

It looks like aerospike_key_get calls some subroutine in file src/main/aerospike/as_shm_cluster.c, which is absent, hence segmentation fault. I checked my $LD_LIBRARY_PATH and it does contain directory containing aerospike.so.

1. Why this particular file is missing? Is it somewhere else other than aerospike.so?
2. Why segmentation fault. It should be gracefully handled in Python client

Edit:

    Metadata-Version: 1.1
    Name: aerospike
    Version: 2.0.1
    Summary: Aerospike Client Library for Python
    Home-page: http://aerospike.com
    Author: Aerospike, Inc.
    Author-email: [email protected]
    License: Apache Software License
    Location: /home/hdfs/graphlab/lib/python2.7/site-packages
    Requires:
    Classifiers:
      License :: OSI Approved :: Apache Software License
      Operating System :: POSIX :: Linux
      Operating System :: MacOS :: MacOS X
      Programming Language :: Python :: 2.6
      Programming Language :: Python :: 2.7
      Programming Language :: Python :: 3.5
      Programming Language :: Python :: Implementation :: CPython
      Topic :: Database
    Files:
      ../../../aerospike
      ../../../aerospike/lua/aerospike.lua
      ../../../aerospike/lua/as.lua
      ../../../aerospike/lua/stream_ops.lua
      ../../../aerospike/usr-lua
      aerospike-2.0.1-py2.7.egg-info
      aerospike-2.0.1-py2.7.egg-info/PKG-INFO
      aerospike-2.0.1-py2.7.egg-info/SOURCES.txt
      aerospike-2.0.1-py2.7.egg-info/dependency_links.txt
      aerospike-2.0.1-py2.7.egg-info/eager_resources.txt
      aerospike-2.0.1-py2.7.egg-info/not-zip-safe
      aerospike-2.0.1-py2.7.egg-info/top_level.txt
      aerospike.so

echo $LD_LIBRARY_PATH
/home/hdfs/graphlab/lib/python2.7/site-packages/:/usr/local/lib:/home/hdfs/python-2.7.8-install/lib/:/home/hdfs/python-3.4.2-install/lib/

Answer 1

This has already been reported as issues 112 and 115 in the aerospike/aerospike-client-python repo on GitHub, and a posting on the discussion forum.

In your case, the 'workaround' is to use the client correctly. Do not make calls to the database after you close the connection to it explicitly.

If you need this bug to be fixed, you can either fork the project (it is open-sourced under the Apache license) and fix it yourself, or wait for the project's maintainers to fix it themselves based on the existing issues.