JNA link issue while starting cassandra RHEL 6.5

Question

I am trying to setup cassandra in my RHEL 6.5 server. When i start cassandara, i get an ERROR related to JNA. The exception says class not found. However, I see in the logs that the jna jar is added to the classpath. I tried using both apache-cassandra-3.0.0 and apache-cassandra-2.2.3, i am getting the same exception in both. I find that the jna jar is available in $CASSANDRA_HOME/lib and also in /usr/share/java. The jna jar version installed is 4.0.0. Any help is appreciated. Following is the startup logs -

INFO 05:57:57 Classpath: /home/cassandra-new/apache-cassandra-2.2.3/conf:/home/cassandra-new/apache-cassandra-2.2.3/build/classes/main:/home/cassandra-new/apache-cassandra-2.2.3/build/classes/thrift:/home/cassandra-new/apache-cassandra-2.2.3/lib/airline-0.6.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/antlr-runtime-3.5.2.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/apache-cassandra-2.2.3.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/apache-cassandra-clientutil-2.2.3.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/apache-cassandra-thrift-2.2.3.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/cassandra-driver-core-2.2.0-rc2-SNAPSHOT-20150617-shaded.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/commons-cli-1.1.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/commons-codec-1.2.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/commons-lang3-3.1.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/commons-math3-3.2.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/compress-lzf-0.8.4.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/concurrentlinkedhashmap-lru-1.4.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/crc32ex-0.1.1.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/disruptor-3.0.1.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/ecj-4.4.2.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/guava-16.0.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/high-scale-lib-1.0.6.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/jackson-core-asl-1.9.2.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/jackson-mapper-asl-1.9.2.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/jamm-0.3.0.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/javax.inject.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/jbcrypt-0.3m.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/jcl-over-slf4j-1.7.7.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/jna-4.0.0.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/joda-time-2.4.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/json-simple-1.1.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/libthrift-0.9.2.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/log4j-over-slf4j-1.7.7.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/logback-classic-1.1.3.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/logback-core-1.1.3.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/lz4-1.3.0.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/metrics-core-3.1.0.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/metrics-logback-3.1.0.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/netty-all-4.0.23.Final.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/ohc-core-0.3.4.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/ohc-core-j8-0.3.4.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/reporter-config3-3.0.0.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/reporter-config-base-3.0.0.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/sigar-1.6.4.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/slf4j-api-1.7.7.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/snakeyaml-1.11.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/snappy-java-1.1.1.7.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/ST4-4.0.8.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/stream-2.5.2.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/super-csv-2.1.0.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/thrift-server-0.3.7.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/jsr223//.jar:/home/cassandra-new/apache-cassandra-2.2.3/lib/jamm-0.3.0.jar WARN 05:57:57 JNA link failure, one or more native method will be unavailable. WARN 05:57:57 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info. INFO 05:57:57 Initializing SIGAR library WARN 05:57:57 Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : false, nproc limit adequate? : true INFO 05:57:58 Initializing system.sstable_activity INFO 05:57:58 Initializing system.hints INFO 05:57:58 Initializing system.compaction_history INFO 05:57:58 Initializing system.peers INFO 05:57:58 Initializing system.schema_columnfamilies INFO 05:57:59 Initializing system.schema_functions INFO 05:57:59 Initializing system.IndexInfo INFO 05:57:59 Initializing system.schema_columns INFO 05:57:59 Initializing system.schema_triggers INFO 05:57:59 Initializing system.local INFO 05:57:59 Initializing system.schema_usertypes INFO 05:57:59 Initializing system.batchlog INFO 05:57:59 Initializing system.available_ranges INFO 05:57:59 Initializing system.schema_aggregates INFO 05:57:59 Initializing system.paxos INFO 05:57:59 Initializing system.peer_events INFO 05:57:59 Initializing system.size_estimates INFO 05:57:59 Initializing system.compactions_in_progress INFO 05:57:59 Initializing system.schema_keyspaces INFO 05:57:59 Initializing system.range_xfers ERROR 05:57:59 Exception in thread Thread[MemtableFlushWriter:1,5,main] java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native at org.apache.cassandra.utils.memory.MemoryUtil.allocate(MemoryUtil.java:82) ~[apache-cassandra-2.2.3.jar:2.2.3] at org.apache.cassandra.io.util.Memory.(Memory.java:74) ~[apache-cassandra-2.2.3.jar:2.2.3] at org.apache.cassandra.io.util.SafeMemory.(SafeMemory.java:32) ~[apache-cassandra-2.2.3.jar:2.2.3] at org.apache.cassandra.io.compress.CompressionMetadata$Writer.(CompressionMetadata.java:274) ~[apache-cassandra-2.2.3.jar:2.2.3] at org.apache.cassandra.io.compress.CompressionMetadata$Writer.open(CompressionMetadata.java:288) ~[apache-cassandra-2.2.3.jar:2.2.3] at org.apache.cassandra.io.compress.CompressedSequentialWriter.(CompressedSequentialWriter.java:73) ~[apache-cassandra-2.2.3.jar:2.2.3] at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:168) ~[apache-cassandra-2.2.3.jar:2.2.3] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigTableWriter.java:75) ~[apache-cassandra-2.2.3.jar:2.2.3] at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:107) ~[apache-cassandra-2.2.3.jar:2.2.3] at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:84) ~[apache-cassandra-2.2.3.jar:2.2.3] at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:424) ~[apache-cassandra-2.2.3.jar:2.2.3] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:367) ~[apache-cassandra-2.2.3.jar:2.2.3] at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:352) ~[apache-cassandra-2.2.3.jar:2.2.3] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.3.jar:2.2.3] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1134) ~[apache-cassandra-2.2.3.jar:2.2.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_65] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]

Answer 1

I went through the code in CLibrary.java and found following code where the exception is caught -

catch (UnsatisfiedLinkError e)
{
    logger.warn("JNA link failure, one or more native method will be unavailable.");
    logger.trace("JNA link failure details: {}", e.getMessage());
}

I restarted cassandra by changing the log-level in conf/logback.xml to TRACE, to print that extra detail -

<logger name="org.apache.cassandra" level="TRACE"/>

I could now see the real issue -

/tmp/jna-3506402/jna6068045839690239595.tmp: failed to map segment from shared object: Operation not permitted

This issue is caused due to noexec flag on the /tmp folder.

I then decided to change the tmp folder by changing tmpdir using option:
-Djava.io.tmpdir=/home/cassandra/tmp That fixed the issue.

I added the options in cassandra-env.sh file. Added following statement -

JVM_OPTS="$JVM_OPTS -Djava.io.tmpdir=/home/cassandra/tmp"