Accessing data on a server with a self-signed certificate

Accessing encrypted data on the web is relatively simple. Even the somewhat more complicated API using HttpURLConnection is relatively straightforward:

(Examples in Kotlin)

val url = URL(urlString) 
val conn = url.openConnection() as HttpURLConnection 
conn.requestMethod = "GET" 
conn.connect() 
val data = conn.inputStream.bufferedReader().use { it.readText() }
...
conn.close()

This is even true for encrypted communications via https. The system simply adds another layer doing the encryption and decryption and you can use it the same way you would do with unencrypted traffic.

But there is more happening in the background that can go wrong.

When the encrypted channel is being establish, the server sends a certificate. This certificate contains among others:

• the host name of the server
• the public key of the server
• the period for which the key is valid
• and usually some signatures from well know certificate authorities (CA)

If the host name in the certificate does not match the one in the URL or the key has expired or is not yet valid Java/Kotlin will raise an exception and refuse to connect.

The problematic part with self-signed certificates is that they don't carry a signature from a well-known CA because they are “self-signed”. Well-known in this context means that the certificate of the CA is present in the Java keystore. A standard connect request will fail.

The way around this is to create a keystore with the self-signed certificate in it and tell Kotlin/Java to use it.

How you do you get this certificate?

If you use a server with a self-signed certificate, chances are that you have installed it yourself. The certificate can be found in that installation.
Or you open the connection in Firefox. The certificate info page lets you export the cert. A certificate looks similar to this:

-----BEGIN CERTIFICATE----- 
MIIEfjCCA2agAwIBAgIIGhqMkdYVlLwwDQYJKoZIhvcNAQELBQAwSTELMAkGA1UE 
cm5ld ... 
...
fv23f6eTYPc1c4zq7kuTC4Uz385+ZisEV+o0+g3pevavfYL+BeOuNAyd1muvP7Ej 
vIE= 
-----END CERTIFICATE-----

Which is essentially a base64 encoded version with a special header and footer.
You could then store that file on your filesystem or in a ressource file on Android, and tell Kotlin to use it instead of the standard keystore (that's what TlsTest.setCertSocketFactory(cert) will do for you) .

But there is a nicer way. As the certificate is sent from the server when the connection is establish, why can't we use it?

The answer is, we can. However when using the standard functions there is a “chicken and egg” problem: You can access the certificate when the connection is established but you need the certificate to make the connection in the first place...

The solve the class shown below implements a “self-signed certificate friendly” TrustManager. The Trust Manager is responsible to check the certificate validity (expiration) and the chain of trust (back to the certificate authorities). This special Trust Manager calls the functions of the original Trust Manager with one exception.  If the chain of trust has a length of 1 (which is true for self-signed certificates) it forgoes checking the chain of trust.

It uses this connection to obtain the certificate. If also provides a method to get some user readable information, so that the user can decide to trust it or not.

A third method can be used to install a keystore with the obtained self-signed certificate before opening the data connection.

Here is a typical use case:

val (cert, except) = TlsTest.testConnection("https://xxxxxxx")
if (except != null) { .. abort, there was a fatal error ... }

print(TlsTest.certInfo(cert))   // possibly asking the user for confirmation
...
val conn = java.net.URL("https://xxxxxx/foo").openConnection() as HttpURLConnection
TlsTest.setCertSocketFactory(conn, cert)
conn.connect()
...
val data = conn.inputStream.bufferedReader().use { it.readText() }

There is one edge case this class does not cover. During its executing no “real” data is being  transferred. Its all done during the establishment phase. If you try to securely connect to a server that only “speaks” http, this mismatch only becomes apparent when data is sent, which then causes a “javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?".

Here is the class – available as a Gist on Github



P.S. I'm neither a security expert nor a proficient Kotlin programmer. If you spot errors or can suggest improvements, let me know.