x11vnc encryption: ssl works better with the realVNC client than ssh tunneling
Posted by jpluimers on 2020/05/08
“Unencrypted connection” “This connection is unencrypted. Would you like to continue?”
When you run realVNC to an x11vnc server, even over an ssh tunneled connection, it will produce errors like the screenshots on the right (from an Android device) and below (from a Mac).
Before I had the realVNC client on the Mac, the Android message totally put me on the wrong foot. I tried searching x11vnc encryption, of which almost all results – especially the Google Search abstracts – will talk about ssh tunneling. So tried to setup the client to use the SSH endpoint, but it refused because it doesn’t talk SSH.
So then I installed a desktop realVNC client (in this case on my Mac) and got this message:
“Unencrypted connection” “The connection to this VNC Server will not be encrypted”
Then it occurred to me that maybe the VNC server itself could do encryption as well and would not need an SSH tunnel after all. And it does even in the first hit:
x11vnc command line options
So below is a very long quote from them on which I’ve based further searching and found these links and steps:
- [WayBack] x11vnc command line options
- [WayBack] x11vnc: a VNC server for real X displays – Q-53: How can I tunnel my connection to x11vnc via an encrypted SSL channel using an external tool like stunnel?
- [WayBack] x11vnc: a VNC server for real X displays – Q-54: Does x11vnc have built-in SSL tunneling?
- [WayBack] x11vnc SSL Certificates and Key Management
- [WayBack] Remote Access with VNC | Reference | openSUSE Leap 42.3: 4.5 Encrypted VNC Communication
- [WayBack] GitHub – LibVNC/x11vnc: a VNC server for real X displays: This is x11vnc with its development continued by LibVNC and the GitHub community. While 0.9.13 was the last release by the original author Karl Runge, 0.9.14 was the first community-based release here on GitHub.
Basically the old command-line was this:
x11vnc -shared -forever -usepw -display :0 -ultrafilexfer
And the new one this:
x11vnc -shared -forever -usepw -display :0 -ultrafilexfer -ssl -bg
Alternatively: use SSH capable VNC viewers
These viewers are capable of SSH:
- [WayBack] bVNC: Secure VNC Viewer – Android Apps on Google Play
- [WayBack] ssvnc: SSL/SSH VNC viewer (but this has not been updated in years).
[WayBack] x11vnc command line options
-ssl [pem] Use the openssl library (www.openssl.org) to provide a
built-in encrypted SSL/TLS tunnel between VNC viewers
and x11vnc. This requires libssl support to be
compiled into x11vnc at build time. If x11vnc is not
built with libssl support it will exit immediately when
-ssl is prescribed. See the -stunnel option below for
an alternative.
The VNC Viewer-side needs to support SSL/TLS as well.
See this URL and also the discussion below for
ideas on how to enable SSL support for the viewer:
http://www.karlrunge.com/x11vnc/faq.html#faq-ssl-tun
nel-viewers . x11vnc provides an SSL enabled Java
viewer applet in the classes/ssl directory (-http or
-httpdir options.) The SSVNC viewer package supports
SSL tunnels too.
If the VNC Viewer supports VeNCrypt or ANONTLS (vino's
encryption mode) they are also supported by the -ssl
mode (see the -vencrypt and -anontls options for more
info; use -sslonly to disable both of them.)
Use "-ssl /path/to/mycert.pem" to specify an SSL
certificate file in PEM format to use to identify and
provide a key for this server. See openssl(1) for more
info about PEMs and the -sslGenCert and "-ssl SAVE"
options below for how to create them.
The connecting VNC viewer SSL tunnel can (at its option)
authenticate this server if it has the public key part
of the certificate (or a common certificate authority,
CA, is a more sophisticated way to verify this server's
cert, see -sslGenCA below). This authentication is
done to prevent Man-In-The-Middle attacks. Otherwise,
if the VNC viewer simply accepts this server's key
WITHOUT verification, the traffic is protected from
passive sniffing on the network, but *NOT* from
Man-In-The-Middle attacks. There are hacker tools
like dsniff/webmitm and cain that implement SSL
Man-In-The-Middle attacks.
If [pem] is empty or the string "SAVE" then the
openssl(1) command must be available to generate the
certificate the first time. A self-signed certificate
is generated (see -sslGenCA and -sslGenCert for use
of a Certificate Authority.) It will be saved to the
file ~/.vnc/certs/server.pem. On subsequent calls if
that file already exists it will be used directly.
Use "SAVE_NOPROMPT" to avoid being prompted to
protect the generated key with a passphrase. However in
-inetd and -bg modes there will be no prompting for a
passphrase in either case.
If [pem] is "SAVE_PROMPT" the server.pem certificate
will be created based on your answers to its prompts for
all info such as OrganizationalName, CommonName, etc.
Use "SAVE-<string>" and "SAVE_PROMPT-<string>"
to refer to the file ~/.vnc/certs/server-<string>.pem
instead (it will be generated if it does not already
exist). E.g. "SAVE-charlie" will store to the file
~/.vnc/certs/server-charlie.pem
Examples: x11vnc -ssl SAVE -display :0 ...
x11vnc -ssl SAVE-someother -display :0 ...
If [pem] is "TMP" and the openssl(1) utility
command exists in PATH, then a temporary, self-signed
certificate will be generated for this session. If
openssl(1) cannot be used to generate a temporary
certificate x11vnc exits immediately. The temporary
cert will be discarded when x11vnc exits.
If successful in using openssl(1) to generate a
temporary certificate in "SAVE" or "TMP" creation
modes, the public part of it will be displayed to stderr
(e.g. one could copy it to the client-side to provide
authentication of the server to VNC viewers.)
NOTE: In "TMP" mode, unless you safely copy the
public part of the temporary Cert to the viewer for
authenticate *every time* (unlikely...), then only
passive sniffing attacks are prevented and you are
still open to Man-In-The-Middle attacks. This is
why the default "SAVE" mode is preferred (and more
sophisticated CA mode too). Only with saved keys AND
the VNC viewer authenticating them (via the public
certificate), are Man-In-The-Middle attacks prevented.
If [pem] is "ANON" then the Diffie-Hellman anonymous
key exchange method is used. In this mode there
are *no* SSL certificates and so it is not possible
to authenticate either the VNC server or VNC client.
Thus only passive network sniffing attacks are avoided:
the "ANON" method is susceptible to Man-In-The-Middle
attacks. "ANON" is not recommended; instead use
a SSL PEM you created or the default "SAVE" method.
See -ssldir below to use a directory besides the
default ~/.vnc/certs
If your x11vnc binary was not compiled with OpenSSL
library support, use of the -ssl option will induce an
immediate failure and exit. For such binaries, consider
using the -stunnel option for SSL encrypted connections.
Misc Info: In temporary cert creation mode "TMP", set
the env. var. X11VNC_SHOW_TMP_PEM=1 to have x11vnc print
out the entire certificate, including the PRIVATE KEY
part, to stderr. There are better ways to get/save this
info. See "SAVE" above and "-sslGenCert" below.
–jeroen
The Wiert Corner - irregular stream of stuff 
Leave a comment