(Examples in Kotlin)
val conn = url.openConnection() as HttpURLConnection
conn.requestMethod = "GET"
conn.connect()
val data = conn.inputStream.bufferedReader().use { it.readText() }
...
conn.close()
This is even true for encrypted communications via https. The system simply adds another layer doing the encryption and decryption and you can use it the same way you would do with unencrypted traffic.
But there is more happening in the background that can go wrong.
When the encrypted channel is being establish, the server sends a certificate. This certificate contains among others:
- the host name of the server
- the public key of the server
- the period for which the key is valid
- and usually some signatures from well know certificate authorities (CA)
If the host name in the certificate does not match the one in the URL or the key has expired or is not yet valid Java/Kotlin will raise an exception and refuse to connect.
The problematic part with self-signed certificates is that they don't carry a signature from a well-known CA because they are “self-signed”. Well-known in this context means that the certificate of the CA is present in the Java keystore. A standard connect request will fail.
The way around this is to create a keystore with the self-signed certificate in it and tell Kotlin/Java to use it.
How you do you get this certificate?
If you use a server with a self-signed certificate, chances are that you have installed it yourself. The certificate can be found in that installation.
Or you open the connection in Firefox. The certificate info page lets you export the cert. A certificate looks similar to this:
-----BEGIN CERTIFICATE-----
MIIEfjCCA2agAwIBAgIIGhqMkdYVlLwwDQYJKoZIhvcNAQELBQAwSTELMAkGA1UE
cm5ld ...
...
fv23f6eTYPc1c4zq7kuTC4Uz385+ZisEV+o0+g3pevavfYL+BeOuNAyd1muvP7Ej
vIE=
-----END CERTIFICATE-----
Which is essentially a base64 encoded version with a special header and footer.
You could then store that file on your filesystem or in a ressource file on Android, and tell Kotlin to use it instead of the standard keystore (that's what TlsTest.setCertSocketFactory(cert) will do for you) .
But there is a nicer way. As the certificate is sent from the server when the connection is establish, why can't we use it?
The answer is, we can. However when using the standard functions there is a “chicken and egg” problem: You can access the certificate when the connection is established but you need the certificate to make the connection in the first place...
The solve the class shown below implements a “self-signed certificate friendly” TrustManager. The Trust Manager is responsible to check the certificate validity (expiration) and the chain of trust (back to the certificate authorities). This special Trust Manager calls the functions of the original Trust Manager with one exception. If the chain of trust has a length of 1 (which is true for self-signed certificates) it forgoes checking the chain of trust.
It uses this connection to obtain the certificate. If also provides a method to get some user readable information, so that the user can decide to trust it or not.
A third method can be used to install a keystore with the obtained self-signed certificate before opening the data connection.
Here is a typical use case:
val (cert, except) = TlsTest.testConnection("https://xxxxxxx")
if (except != null) { .. abort, there was a fatal error ... }
print(TlsTest.certInfo(cert)) // possibly asking the user for confirmation
...
val conn = java.net.URL("https://xxxxxx/foo").openConnection() as HttpURLConnection
TlsTest.setCertSocketFactory(conn, cert)
conn.connect()
...
val data = conn.inputStream.bufferedReader().use { it.readText() }
There is one edge case this class does not cover. During its executing no “real” data is being transferred. Its all done during the establishment phase. If you try to securely connect to a server that only “speaks” http, this mismatch only becomes apparent when data is sent, which then causes a “javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?".
Here is the class – available as a Gist on Github
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package de.mstrecke.util | |
| import java.net.HttpURLConnection | |
| import java.net.URL | |
| import java.security.KeyStore | |
| import java.security.KeyStoreException | |
| import java.security.MessageDigest | |
| import java.security.NoSuchAlgorithmException | |
| import java.security.cert.CertificateException | |
| import java.security.cert.CertificateFactory | |
| import java.security.cert.X509Certificate | |
| import javax.net.ssl.* | |
| /** | |
| * Functions to access data on a server that uses a self-signed certificate | |
| * | |
| * Usage: | |
| * | |
| * // "cert" is the certificate to be used, if any | |
| * // "except" is a fatal exception (no connection, etc.), if any | |
| * | |
| * val (cert, except) = TlsTest.testConnection("https://xxxxxxx") | |
| * if (except != null) { .. abort, there was a fatal error ... } | |
| * | |
| * print(TlsTest.certInfo(cert)) // possibly asking the user for confirmation | |
| * ... | |
| * val conn = java.net.URL("https://xxxxxx/foo").openConnection() as HttpURLConnection | |
| * TlsTest.setCertSocketFactory(conn, cert) | |
| * conn.connect() | |
| * ... | |
| * val data = conn.inputStream.bufferedReader().use { it.readText() } | |
| * | |
| * Note: | |
| * This routine will notice if you try a non-TLS connection on a TLS server. | |
| * However, if you try a TLS connection on a non-TLS server, an exception will not thrown | |
| * until later when the first real data is being exchanged: | |
| * "javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?" | |
| */ | |
| class TlsTest { | |
| /** | |
| * Self-signed certificate friendly TrustManager | |
| * | |
| * Uses the functions of a default TrustManager, except | |
| * - the cert chain is not checked if (and only if) it has a length of 1 | |
| * | |
| * Note: | |
| * - expiration dates are checked | |
| * - the host name is checked | |
| * | |
| * Based on https://stackoverflow.com/questions/35545126/an-unsafe-implementation-of-the-interface-x509trustmanager-from-google/35571883#35571883 | |
| */ | |
| private class SelfSignedFriendlyTrustManager | |
| /** | |
| * Constructor for SelfSignedFriendlyTrustManager. | |
| */ | |
| @Throws(NoSuchAlgorithmException::class, KeyStoreException::class) | |
| constructor(keystore: KeyStore) : X509TrustManager { | |
| val standardTrustManager: X509TrustManager | |
| init { | |
| val factory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm()) | |
| factory.init(keystore) | |
| val trustmanagers = factory.trustManagers | |
| if (trustmanagers.isEmpty()) { | |
| throw NoSuchAlgorithmException("no trust manager found") | |
| } | |
| standardTrustManager = trustmanagers[0] as X509TrustManager | |
| } | |
| @Throws(CertificateException::class) | |
| override fun checkClientTrusted(certificates: Array<X509Certificate>, authType: String) { | |
| standardTrustManager.checkClientTrusted(certificates, authType) | |
| } | |
| @Throws(CertificateException::class) | |
| override fun checkServerTrusted(certificates: Array<X509Certificate>?, authType: String) { | |
| if (certificates?.size == 1) { | |
| // only if cert exists and trust chain length == 1 | |
| // check only validity (valid until/not before), disregard trust chain (the "friendly" part | |
| certificates[0].checkValidity() | |
| } else { | |
| standardTrustManager.checkServerTrusted(certificates, authType) | |
| } | |
| } | |
| override fun getAcceptedIssuers(): Array<X509Certificate> { | |
| return this.standardTrustManager.acceptedIssuers | |
| } | |
| } | |
| companion object { | |
| /** | |
| * SocketFactory using the SelfSignedFriendlyTrustManager | |
| * | |
| * @return TrustManager that accepts all self-signed certificates | |
| * @note Host name must match, expiration dates are checked | |
| * | |
| * Used only once to obtain the self-signed certificate | |
| */ | |
| private fun createSelfSignedFriendlySocketFactory(): SSLSocketFactory { | |
| val keyStoreType = KeyStore.getDefaultType() | |
| val keyStore = KeyStore.getInstance(keyStoreType) | |
| keyStore.load(null, null) | |
| // create trustManager with self-signed friendly TrustManager | |
| val trustManagers: Array<out TrustManager> = arrayOf(SelfSignedFriendlyTrustManager(keyStore)) | |
| val context = SSLContext.getInstance("TLS") | |
| context.init(null, trustManagers, null) | |
| return context.socketFactory | |
| } | |
| /** | |
| * Create SocketFactory that only excepts connection secured with the specified certificate | |
| * | |
| * @property cert_PEM certificate in PEM format | |
| * @return SSLSocketFactory that only accepts the cert_PEM certificate | |
| */ | |
| private fun createSingleCASocketFactory(cert_PEM: String): SSLSocketFactory { | |
| val cf = CertificateFactory.getInstance("X.509") | |
| val caInput = cert_PEM.byteInputStream() | |
| val ca = caInput.use { // evaluates to null in case of errors | |
| cf.generateCertificate(it) | |
| } | |
| // Create a KeyStore containing our trusted CAs | |
| val keyStoreType = KeyStore.getDefaultType() | |
| val keyStore = KeyStore.getInstance(keyStoreType) | |
| keyStore.load(null, null) | |
| keyStore.setCertificateEntry("ca", ca) | |
| // Create a TrustManager that trusts the CAs in our KeyStore | |
| val tmfAlgorithm = TrustManagerFactory.getDefaultAlgorithm() | |
| val tmf = TrustManagerFactory.getInstance(tmfAlgorithm) | |
| tmf.init(keyStore) | |
| // Create an SSLContext that uses our TrustManager | |
| val context = SSLContext.getInstance("TLS") | |
| context.init(null, tmf.trustManagers, null) | |
| return context.socketFactory | |
| } | |
| /** | |
| * Set sslSocketFactory of the HttpsURLConnection to accept only this cert | |
| * | |
| * @property conn Http(s)URLConnection opened by calling function | |
| * @property cert_PEM certificate to use or null | |
| */ | |
| fun setCertSocketFactory(conn: HttpURLConnection, cert_PEM: String?) { | |
| if (cert_PEM == null) return | |
| if (conn !is HttpsURLConnection) return // Http_s_URLConnection !! | |
| conn.sslSocketFactory = createSingleCASocketFactory(cert_PEM = cert_PEM) | |
| } | |
| /** | |
| * Test connection to URL | |
| * | |
| * @property urlString URL to test | |
| * @property cert_PEM certificate to use or null | |
| * @return server certificate (if we need it) or null / fatal exception (host no found, etc.) or null | |
| */ | |
| fun testConnection(urlString: String, cert_PEM: String? = null): Pair<String?, Exception?> { | |
| val conn = URL(urlString).openConnection() as HttpURLConnection | |
| val isHttps = conn is HttpsURLConnection | |
| conn.requestMethod = "HEAD" | |
| try { | |
| if (isHttps) { | |
| cert_PEM?.let { | |
| (conn as HttpsURLConnection).sslSocketFactory = createSingleCASocketFactory(it) | |
| } | |
| } | |
| conn.connect() | |
| // Try to read - it should return an empty string | |
| // ... or throw a SocketException if we tried http on a https port | |
| // and yes, the compiler will complain that the result is not used anywhere else | |
| val dummy = conn.inputStream.bufferedReader().use { it.readText() } | |
| conn.disconnect() | |
| // If we are here, everything worked, i.e. connection is: | |
| // HTTP, HTTPS with 'normal' cert (validated via the normal trust chain), or HTTPS with supplied cert_PEM | |
| if (isHttps) { | |
| return Pair(cert_PEM, null) | |
| } | |
| return Pair(null, null) | |
| } catch (e: javax.net.ssl.SSLHandshakeException) { | |
| // If we are here the connection is HTTPS | |
| // - without a supplied cert, i.e. we have to download it, or | |
| // - with a supplied cert, i.e. the old one didn't work anymore => try without a cert, and if that fails download the new one | |
| if (cert_PEM != null) { | |
| try { | |
| // Let's check first if we can now access the URL via the normal chain of trust | |
| val conn2 = URL(urlString).openConnection() as HttpsURLConnection | |
| conn2.connect() | |
| conn2.disconnect() | |
| // no error => we don't need a specific cert | |
| return Pair(null, null) | |
| } | |
| catch (e: javax.net.ssl.SSLHandshakeException) { | |
| } | |
| } | |
| // Try again with the self-signed friendly SocketFactory | |
| val conn3 = URL(urlString).openConnection() as HttpsURLConnection | |
| conn3.sslSocketFactory = createSelfSignedFriendlySocketFactory() | |
| try { | |
| conn3.connect() | |
| } catch (e: Exception) { | |
| return Pair(null, e) // e.g. Hostname ... was not verified | |
| } | |
| // If we are here, we got a HTTPS connection to a server with | |
| // a self-signed cert | |
| return try { | |
| // return the certificate in PEM form | |
| // | |
| // outside Android: java.util.Base64.getMimeEncoder().encodeToString(conn3.serverCertificates[0].encoded) + | |
| // Android: android.util.Base64.encodeToString(conn3.serverCertificates[0].encoded, android.util.Base64.DEFAULT) + | |
| Pair("-----BEGIN CERTIFICATE-----\n" + | |
| android.util.Base64.encodeToString(conn3.serverCertificates[0].encoded, android.util.Base64.DEFAULT) + | |
| "-----END CERTIFICATE-----\n", null) | |
| } catch (e: Exception) { | |
| return Pair(null, e) | |
| } finally { | |
| conn3.disconnect() | |
| } | |
| } catch (e: Exception) { | |
| // any other error from first attempt | |
| return Pair(null, e) | |
| } | |
| } | |
| /** | |
| * Get info string for cert_PEM (subject line and SHA256 checksum | |
| * | |
| * @property cert_PEM certificate in PEM format | |
| * @note Could be used in a dialog to confirm the use of the certificate | |
| */ | |
| fun certInfo(cert_PEM: String?): String { | |
| if (cert_PEM == null) { | |
| return "No cert" | |
| } | |
| val cf = CertificateFactory.getInstance("X.509") | |
| val caInput = cert_PEM.byteInputStream() | |
| val ca = caInput.use { | |
| cf.generateCertificate(it) | |
| } | |
| if (ca == null) return "Error" // this shouldn't happen | |
| val md = MessageDigest.getInstance("SHA-256") | |
| val digest = md.digest(ca.encoded) | |
| val sha256 = digest.joinToString(separator = ":", transform = { "%02x".format(it) }) | |
| val subject = (ca as X509Certificate).subjectDN.name | |
| return "Subject:\n$subject\nSHA256:\n$sha256" | |
| } | |
| fun certInfo(res:Pair<String?, Exception?> ) : String { | |
| return certInfo(res.component1()) | |
| } | |
| } | |
| } |
P.S. I'm neither a security expert nor a proficient Kotlin programmer. If you spot errors or can suggest improvements, let me know.
