Is there a compatibility matrix for Hadoop components?

Question

I wonder if there is a compatibility matrix for the various Hadoop components of the eco-system ?

Each Hadoop upgrade has big compatibility impact, e.g:

• Apache Spark 2.4 does not support Hadoop v3,
• Hadoop does not support Java 9 and 10,
• and so on...

I know that vendors such as Hortonworks publish components lists with each version of their distribution, but this is not meant for the public in large because this includes patched components.

Does one have to go through all the bug trackers over at Jira for each tool to find out about compatibility problems ?

Answer 1

One of the key things that a company like Cloudera/Hortonworks does is taking all the open source projects that make up Hadoop and making sure that they work well together. Both from a functional perspective as well as security a lot of testing and tweaking is done to ensure that everything together forms a proper release.

Now that you have some insight how much effort goes into the release of just one distribution with comparatively strong focus on recent versions, you might understand that there will not be a general overview of 'how everything works with everything' beyond these distributions.

Full disclosure: I am an employee of Cloudera, but even without this I would still recommend you to work with a distribution where possible