What is the difference between tensorflow on spark with the default distributed tensorflow 1.0?

Question

I am trying to install tensorflow on spark onto the server, as I was told by my boss because he thought it would be easy to use. But I also learnt the default distributed tensorflow on the tensorflow website. Can any expert tell me the difference between these two choice of distribution? Will spark automatically assign the parameter server or workers?

Thanks in advance.

Answer 1

I finally installed TensorflowOnSpark(TFOS) on the server and compared it with the default distributed Tensorflow(TF). And my conclusion is:

Pros:

1. TFOS is more automatic. I don’t need to define which node in the cluster as the PS node. I also don’t need to upload the same code to all the nodes.
2. I don’t need to input the command line on each node to start the training.
3. The code change for running on TFOS is not much.

Cons:

1. Sometime, two worker nodes will be automatically assigned to the same GPU and core (K80 with two cores). And it will cause out of memory problem.
2. You need input a long list of configuration on command line before running.
3. You cannot specify which node to be PS node.

If I am wrong somewhere, please correct me.