noVNC to x11vnc SSL connection

Question

I am trying to use an SSL connection with x11vnc (VNC server) and noVNC (VNC client). Whenever I try to connect, I get the error "Unsupported Security Types: 19,18" from noVNC and "SSL: ssl_helper[2957]: exit case 2 (ssl_init failed) SSL: accept_openssl: cookie from ssl_helper[2957] FAILED. 0" from x11vnc. If I turn off SSL for x11vnc, the client is able to connect with no problem.

I do realize that x11vnc does come packaged with a java based applet viewer. However, I'm more interested in the HTML5 based noVNC.

Both x11vnc and noVNC both run on the same machine, using the commands:

x11vnc: x11vnc -forever -shared -unixpw_cmd [cmd] -ssl [pem]

noVNC: ./utils/websockify --ssl-only --cert=[pem] --web=./ 6080 localhost:5900

Note: Both are pointing to the same pem. I use websockify instead of noVNC's launch.sh to have more options such as --ssl-only.

The encrypt option for noVNC seems to have no effect on connecting to the server (the results are the same whether it's on or off).

I am most concerned about a secure connection. As of now, it does not seem like turning on noVNC's encrypt option does much if x11vnc's SSL must be turned off (the encrypt option makes noVNC use wss:// instead of ws://). If this does create a secure connection, please let me know. Else, how can I get noVNC and x11vnc working with SSL?

Answer 1

First some clarifications:

noVNC and websockify are actually separate projects:

• websockify is a generic proxy/bridge that allows WebSocket connections (e.g. from a browser) to connect to raw TCP socket services (e.g. a VNC server).
• noVNC is the HTML5 VNC client.

If the VNC server supported WebSocket connections then websockify would not be needed. The only VNC server that currently supports direct WebSocket connections is this fork of libvncserver. websockify is included in noVNC since most VNC servers do not yet support WebSocket clients, but websockify is a separate project.

You are dealing with two different network connections each of which has separate encryption options:

1. noVNC (browser) to websockify - using WebSocket protocol
2. websockify to x11vnc (VNC server) - direct TCP socket connection

The WebSocket protocol supports unencrypted connections (ws://) and SSL/TLS encrypted connections (wss://).

The RFB protocol (Remote Frame Buffer) used in VNC has the ability to upgrade during the initialization to use an encrypted connection. There are several encryption methods supported such as TLS (security type 18), VeNCrypt (security type 19).

Now to your question:

When you pass -ssl PEM to x11vnc this enabled RFB/VNC encryption. noVNC does not support RFB/VNC encryption. Javascript is not fast enough to do encryption/decryption fast enough to be usable for noVNC. There is some discussion about adding a crypto API to Javascript which would enable noVNC to support this type of encryption.

When you enabled encryption in noVNC you are enabling WebSocket encryption (wss://). This encrypts the connection between the browser and websockify. As long as the connection between websockify and the VNC server are over a trusted network (e.g. running both on the same server) and use WebSocket encryption in noVNC then no unencrypted data will be exposed. However, if you run websockify on the same system as your browser and the VNC server is remote then the VNC traffic from you client to the server system will not be encrypted (unless noVNC gets RFB/VNC VeNCrypt encryption support in the future).

The output from websockify will indicate whether the WebSocket connection is encrypted or unencrypted.