datastax opscenter agents not installing

Question

I'm running my own version of cassandra on 127.0.1.1. I changed the rpc_address and also the address to 127.0.1.1.

When I'm starting Opscenter I'm prompted to install agents and I'm doing so by the recommended Fix now option. But when I try to install it asks me for some Node ssh credentials. I don't know what this means.

What are the correct credentials to input here?

I tried adding a new user on Linux with root permissions and tried with that username too but it doesn't work. I manually tried to run the install_agent.sh too but it doesn't work.

Am I missing something?

EDIT:

address.yaml

stomp_interface: 127.0.1.1
agent_rpc_interface: "127.0.1.1"
cassandra-conf: /home/$username/Dropbox/Work/ITNow/olderVersions/cassandra2.11/apache-cassandra-2.1.12/conf/cassandra.yaml
stomp_port: 61620
jmx_host: 127.0.1.1
jmx_port: 7199

cassandra.yaml

http://textuploader.com/57ivn

EDIT2:

for username and password I'm using my datastax credentials. For the Private key, I'm using the key in /etc/ssh/ssh_host_rsa_key.pub as mentioned by @apesa. But still it can't install the agents.

Is my address.yaml file correct?

EDIT 3:

Like you can see in the following image, when thought Opscenter is not connected to the agents, I can still see the keyspaces and tables in cassandra.

For sudo netstat -p | grep 127.0.1.1 I get no output.

xyz@ubuntu$ ps -ef | grep datastax-agent
xyz@ubuntu$ ps -ef | grep cassandra

the output is this and this respectively. In short, yes I can see jars and I guess the environment variables too.

How did I start cassandra, you ask?

sudo ./bin/cassandra from the folder of cassandra. After that I'm using spark streaming to stream data from kafka to cassandra. But it shouldn't matter here.

Even if it is not connecting to the agent the opscenter should still show the cassandra instance. Do you see your test_Cluster? Yes! I do!

Finally, nodetool status gives me this:

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  127.0.1.1  58.94 MB   256     ?       367dd1c6-291d-4e57-a2b6-e04d74154a6f  rack1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

What I believe:

I am not sure, but maybe my address.yaml is not correct? I have updated my address.yaml above, maybe I need to change something? Changing the address.yaml gives me different outputs when I run ./datastax-agent. Thanks.

EDIT4:

Tried everything what @apesa suggested and here's the detailed outputs of the scripts. I still have the same problem.

version:

Opscenter: 5.2.4
datastax-agent: 5.2.4
Cassandra: 2.1.12

Output of sudo netstat -p | grep :9042

tcp        0      0 localhost:38155         ganguly:9042            ESTABLISHED 15907/python2.7 
tcp        0      0 localhost:38154         ganguly:9042            ESTABLISHED 15907/python2.7 
tcp        0      0 localhost:38153         ganguly:9042            ESTABLISHED 15907/python2.7 
tcp6       0      0 localhost:38176         ganguly:9042            ESTABLISHED 15951/java      
tcp6       0      0 localhost:38179         ganguly:9042            ESTABLISHED 15951/java      
tcp6       0      0 ganguly:9042            localhost:38154         ESTABLISHED 15720/java      
tcp6       0      0 ganguly:9042            localhost:38153         ESTABLISHED 15720/java      
tcp6       0      0 ganguly:9042            localhost:38179         ESTABLISHED 15720/java      
tcp6       0      0 ganguly:9042            localhost:38176         ESTABLISHED 15720/java      
tcp6       0      0 ganguly:9042            localhost:38155         ESTABLISHED 15720/java  

Output of sudo netstat -p | grep :7199

blank

Output of sudo netstat -p | grep :61620

blank

This is the console log on running sudo ./datastax-agent.

In short, this is the error line:

ERROR [clojure-agent-send-off-pool-0] 2016-02-04 11:33:26,274 Can't \
connect to Cassandra (All host(s) tried for query failed (tried:\
 /127.0.0.1:9042 \
(com.datastax.driver.core.TransportException: \
[/127.0.0.1:9042] Cannot connect))), retrying soon.

But I don't know why it's trying to connect to cassandra on 127.0.0.1 since cassandra is running on 127.0.1.1. What did I do wrong here?

Answer 1

There are a couple ways to install the DataStax agent from OpsCenter. Based on your IP Addr of 127.0.1.1 you must be running a Debian version of Linux. I know it says "optional" but I have never been able to run without a ssh private key. In which case you should install OpenSSH-server using the following command

xyz@ubuntu$ sudo apt-get install openssh-server

Then follow these instructions for setting up a SSH key that you will then use when providing your credentials to the "fix agent" dialog box. The username/password in Datastax for the DSE side is datastax/datastax unless you changed it. That is what you should put in for username / password and then also copy/paste your new SSH key inside the private key box. The ssh key will be found in

/etc/ssh/ssh_host_rsa_key.pub

Once you have all the credentials together go ahead and run the Install Agents tool. If it still fails to complete you should check the datastax-agent folder where you installed DSE and verify there is a jar file for the agent.. It should be something like

datastax-agent-5.2.4-standalone.jar

You should also see the folder structure for the Agent as well, bin, conf... etc directories should all be there. If everything is in place and your cassandra.yaml file is correct you should be able to get the agent running.

You can also do this manually by moving the correct directories and jar files in place and then restarting the agent(s).

EDIT:

After looking at your latest agent.log file I would say this. The broken pipe error in Java refers to a connection that is no longer up or unavailable. Where it is failing in the agent.log is where it is trying to communicate with the opscenter. One of your connections is not up or is mis configured. You also have another log for cassandra that is usually in /var/log/cassandra depending on how you installed cassandra. Comb through that and make sure it does not have errors in it during start up, warnings are fine but errors usually point to the problem.

From this point on you have the the following log files. Note the file locations may be different than yours depending on how you installed everything.

/var/log/cassandra
/var/log/datastax-agent
/var/log/OpsCenter

and you have the following commands to help sort out what is actually running and on what port is it bound. Take a look below, what do these commands tell you?

xyz@ubuntu$ sudo netstat -p | grep 127.0.1.1

You should see at least the following ports bound to your 127.0.1.1 addr: 9042, 8888, 61620 and several others.

Also look at the output of these two commands. If you don't get anything back try running them as sudo. The output will depend on how you started the jars

xyz@ubuntu$ ps -ef | grep datastax-agent
xyz@ubuntu$ ps -ef | grep cassandra

You should see all the loaded jar files as well as the environment variables that were set when starting cassandra and / or datastax-agent.

You might edit your post and explain how you started cassandra. ie as a service or from the command line?

Also, from OpsCenter you should be able to see the test_cluster at 127.0.1.1:8888. If your prompted to fix the agents just exit out of those dialog boxes until you're at the main screen showing the datacenter. Even if it is not connecting to the agent the opscenter should still show the cassandra instance. Do you see your test_Cluster?

Lastly, what do you get when you type this

xyz@ubuntu$ nodetool status

EDIT 2:

Based on your last edit it looks as if everything but the agent is up and running. So says nodetool, which is good. Also since you can see Keyspaces in OpsCenter you are definitely connected to a running cassandra instance.

So looking further it appears your datastax-agent is still not connecting. In looking at your address.yaml file I can see a couple possible issues. This is a simple standalone cassandra cluster with a single IP address of 127.0.1.1. So that 127.0.1.1 address is going to be used in all the config where an IP addr or host is involved. I would make the following changes to your address.yaml file.

stomp_interface: 127.0.1.1
cassandra-conf: /home/$username/Dropbox/Work/ITNow/olderVersions/cassandra2.11/apache-cassandra-2.1.12/conf/cassandra.yaml
stomp_port: 61620

Leave the JMX details out since they are not needed and will be sent by the opscenterd once connected. The agent_rpc_interface is also not needed in this install.

There is also cassandra-env.sh script that is found in the cassandra/conf dir. this script is executed when cassandra starts. look for the following line near the bottom of the script and uncomment it and add your host IP 127.0.1.1 in place of <public name>

# jmx: metrics and administration interface
#
# add this if you're having trouble connecting:
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=127.0.1.1"

Lastly, there is the cassandra utility documented HERE You can pass several cli parameters to the cassandra shell when initializing. This particular one below is useful when you want to start/restart a cassandra cluster after changing the IP or hostname. You will need to restart cassandra once you have made these changes. In fact you shoudl restart everything in this order.. Cassandra, datastax-agent and then opscenter.

xyz@ubuntu$ bin/cassandra -Dcassandra.load_ring_state=false

Once you complete the changes and try starting the agent and it still fails, you are going to need more debugging details. Netstat returned false because of what we piped to the grep utility. try the following as the ports are probably bound to your hostname since your using 127.0.1.1. Run it for the following ports :9042, :7199 and :61620. You're looking for your hostname bound to all of these ports while cassandra is running.

xyz@ubuntu$ sudo netstat -p | grep :9042

Please also edit your question and add the versions of OpsCenter and Datastax-Agent your running. That matters.