Connect to spark through a SOCKS proxy

Question

TL;DR How can I connect a local driver to a spark cluster through a SOCKS-proxy.

We have an onsite spark cluster that is behind a firewall that blocks most ports. We have ssh access, so I can create a SOCKS proxy with ssh -D 7777 ....

It works fine for browsing the web-UI's when my browser uses the proxy, but I do not know how to make a local driver use the it.

So far I have this, which obviously is not configuring any proxies:

val sconf = new SparkConf()
  .setMaster("spark://masterserver:7077")
  .setAppName("MySpark")
new SparkContext(sconf)

Which logs these messages 16 times before throwing an exception.

15/01/20 14:43:34 INFO Remoting: Starting remoting
15/01/20 14:43:34 ERROR NettyTransport: failed to bind to server-name/ip.ip.ip.ip:0, shutting down Netty transport
15/01/20 14:43:34 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
15/01/20 14:43:34 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1.
15/01/20 14:43:34 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
15/01/20 14:43:34 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.

Answer 1

Your best shot may be to forward a local port to remote 7077, and then setMaster("spark://localhost:nnnn") where nnnn is the local port you have forwarded.

To do this use ssh -L (instead of -D). I cannot guarantee that this will work, or if it works, that it will continue to work, but at least it will spare you using an actual proxy for this one port. Things that might break it, are mostly secondary connections that the initial connection might trigger. I didn't test this yet, but unless there are secondary connections, in principle it should work.

Also, this doesn't answer the TL;DR-version of your question, but since you have SSH-access, it's more likely to work.