Accessing servers with self-signed certificates in Python

As long as I can remember Python was always capable of retrieving web pages from encrypted servers.  In the early days it didn't bother verifying the ssl certificate.  In newer version it does so by default - which is good - and you usually don't have any problems. And if you do it should merit your attention.

However there are situations where this verification breaks things: self-signed certificates. E.g. the ones you use in your local network or as in my case a web cam which actually offers https.  It uses an self-signed certificate - probably the same in all cameras of this type - but hey... beggars can't be choosers.

To access the cam in Firefox you would create a security exception to access the cam, in Python life is not that simple.

The following post shows:

• how to disable the verification
• how to pull the server certificate
• how to use it in Python3
• how to install it in the system

Please note: The following description works on Ubuntu 16.04 LTS. On your distro the directory paths may vary. Change IP addresses, hostnames, filenames, etc. to your needs.

I'm using a small script pulling images from the above mentioned web cam:

import urllib.request
...
hp = urllib.request.urlopen("https://192.168.0.100/pic.jpg")
pic = hp.read()
...

which now results in

urllib.error.URLError: < urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

The following "context" disables the certificate verification

import ssl
...
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

hp = urllib.request.urlopen("https://192.168.0.100/pic.jpg", context=ctx)

That works, but having a certificate verification would be nice. To do that we need the server certificate:

cert = ssl.get_server_certificate( ('192.168.0.100', 443) )
open('/tmp/ipcamera.crt','w').write(cert)

The cert looks like this

-----BEGIN CERTIFICATE-----
MIIClDCCAf2gAwIBAgIJAIMQZ+Ua/bkXMA0GCSqGSIb3DQEBBQUAMGIxCzAJBgNV
...
4XAVFCBZOPwflj9Ug0YNSIgcSfDOxha06C9hwZ0+ZuafkjXv16sGEA==
-----END CERTIFICATE-----


Now you can create a context which uses that cert:

ctx2 = ssl.create_default_context()
ctx2.load_verify_locations("/tmp/ipcamera.crt")

hp = urllib.request.urlopen("https://192.168.0.100", context=ctx2)
...

Which results in:
 
ssl.CertificateError: hostname '192.168.0.100' doesn't match 'IPC'

Well, that didn't work, at least the error message has changed.

Let's have a look at the cert:

>>> ctx2.get_ca_certs()
[{'issuer': ((('countryName', 'ch'),), (('stateOrProvinceName', 'guangdong'),), (('localityName', 'zhenzhen'),), (('organizationName', 'IPCam'),), (('organizationalUnitName', 'IPCam'),), (('commonName', 'IPC'),)), 'notBefore': 'Mar  7 01:24:16 2013 GMT', 'subject': ((('countryName', 'ch'),), (('stateOrProvinceName', 'guangdong'),), (('localityName', 'zhenzhen'),), (('organizationName', 'IPCam'),), (('organizationalUnitName', 'IPCam'),), (('commonName', 'IPC'),)), 'notAfter': 'Feb 23 01:24:16 2063 GMT', 'serialNumber': '831067E51AFDB917', 'version': 3}]

As you can see the commonName for this cert is IPC and we're trying to access the server using the hostname 192.168.0.100. They don't match.

You can fix this in two ways. Either tell Python to ignore the hostname:

ctx3 = ssl.create_default_context()
ctx3.load_verify_locations("/tmp/ipcamera.crt")
ctx3.check_hostname = False

hp = urllib.request.urlopen("https://192.168.0.100", context=ctx3)

or put an entry into /etc/hosts (you need root privileges for that)

192.168.0.100   IPC

System wide integration
Using contexts is fine, but I have to change every piece of code: create the context and use it. It would be nice to have it "simply" work.
For this you need root access. Then you can put the cert into the system wide certificate store and Python will use it like any normal cert - including the one from the Hongkong Post Office :-)

First create the above mentioned entry in /etc/hosts to get the hostname check right.

Then create a the directory /etc/ssl/mycerts and copy ipcamera.crt into it.

The system wide certs are stored in /etc/ssl/certs. In order for your certificate to be found, it must be renamed. Calculate its hash using openssl:

$ openssl x509 -noout -hash -in /etc/ssl/mycerts/ipcamera.crt
ab0cd04d

Now goto /etc/ssl/certs and create the appropriate named link (you must append .0 to the hash).

sudo ln -s ../mycerts/ipcamera.crt ab0cd04d.0

Now it simply works:

w = urllib.request.urlopen("https://IPC")

If there are easier ways to do it, please let me know.



Links:

• https://docs.python.org/2/library/ssl.html
• http://gagravarr.org/writing/openssl-certs/others.shtml#ca-openssl