The 502 Bad Gateway Error

This is a short story of software complexity.

I had a problem with subversion: the use of svn move or svn copy lead to the following error:


I could predict which files would give problems, because those had a "+" sign when I did a SVN status:

The "+" sign means that the add operation is actually a COPY operation, and the history of the original file is copied to the newly added file. In this case, the history of oldname.txt.

Work around

There is a work-around. Simply adding the file, without copying the history log.

However, this will lead to the loss of the history information.

WebDAV

Now, what caused our problem? Deep down, the problem is in mod_ssl. But the link is really not obvious, so let me go step by step.

First of all, let's find out what triggers this "502 Bad Gateway" error. Our subversion repositories use HTTP for the datatransport. Or actually HTTPS because that is more secure. You can even see that in the URL of our repository: https://svn.example.org/svn/repos∞ (An alternative would be to use SSH for data transport, with the URL: svn+ssh://svn.example.org/svn/repos, but let's not go into that detail). To be exact, subversion uses an extension of HTTP, called WebDAV.

Technically, a move or copy operation in subversion is translated to a Webdav COPY operation. The request looks a bit likes this:

Well, actually it is a bit more complex, as the real request is in XML and subversions groups some requests together in single HTTP request to something like COPY /svn/repos/!svn/act/6b2389f7-9223-4eca-b3fa-96cb3da83b78 HTTP/1.1, but you get the general idea.

Apache, the webserver, translates the above request to: "Copy https://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt". It uses the Host parameter svn.example.org and the first request line COPY /svn/repos/oldname.txt HTTP/1.1 to assemble the source URL, https://svn.example.org/svn/repos/oldname.txt∞. However, in some cases, the assembly of the source URL goes wrong, and end up as http://svn.example.org/svn/repos/oldname.txt∞ (note the http instead of https). Apache quickly figures out that it can't move a file http://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt∞, because as far as Apache is concerned http://svn.example.org/∞ and https://svn.example.org/∞ are two entirely different hosts. It will respond with a "502 Bad Gateway" error.

Non-matching source and destination URL

There are multiple causes for a bad translation or non-match between source and destination URL with a HTTP COPY. The two most common causes are:

• Multiple server aliases
• Use of a HTTP reverse proxy server

If a machine listens to multiple names (e.g. www.example.org and svn.example.org), but only one of them is configured in the apache configuration with ServerName www.example.org then Apache will regard this as a two non-matching servers. The solution is to simply add the other names to ServerAlias:

The other common cause is the use of a HTTPS reverse proxy server in front of your HTTP server. The proxy server receives the requests, translates the request and hostname and forwards it to the HTTP server. However, because the proxy server did not translate the Destination field, the HTTP server still sees the HTTPS address and will return the 502 Bad Gateway error.
See for example Subversion behind an Apache Reverse Proxy∞ for information how to fix this.

Mod_ssl says: 443 equals 80

After much strugling, we discovered a third, non so common cause for a non-match between source and destination URL with a HTTP COPY:

• Bad configuration of virtual HTTPS hosts

we find the cause to be in mod_ssl. Somehow, our Apache server believed it was contacted at port 80 using HTTP even though in reality the client contacted it at port 443 using HTTPS. Yes, that's pretty weird, but you can reproduce it yourself with this small snippet of Apache configuration code:



This is a configuration for a sever with two name-based "virtual" hosts: www.example.org and svn.example.org. Both listen to the same IP address on port 443 (which is used for HTTPS). HTTPS is turned on with "SSLEngine on"

As an extra, the server is configured that if you visit https://www.example.org/server-info∞ or https://svn.example.org/server-info∞, you get an page with information about the configuration of the server. You will notice that they say:

URL	servername/port reported by Apache
https://www.example.org/server-info	www.example.org:443
https://svn.example.org/server-info	svn.example.org:80

Now Apache reports the wrong port when requesting https://svn.example.org/server-info∞: Apache thinks it is contacted at port 80, but in reality, it is contacted at port 443. The above configuration caused that problem. In fact, if you would have used this configuration below, there wouldn't have been any problem:


For more information, see Apache bug report #42929∞.

mod_ssl processing of SSL parameters

You may wonder why mod_ssl behaves in this way. For one thing, the second vhost the first (bad) configuration has no "SSLEngine on" directive. So why did Apache server use SSL in the first place when I contacted https://svn.example.org∞? The reason lies in the fact that mod_ssl ignores vhosts! That may seem strange, until you realize that the SSL Handshake takes place before the client sends the Host: header. So before any vhost is determined. To be precise, there can be only one certificate per IP address:port combination. The only IP address:port combination in the above configurations is "*:443". mod_ssl simply looks at the first vhost with that IP address and port, and uses the SSL configuration mentioned there. So effectively, all vhosts with that configuration share these SSL settings:

Virtual Hosts with SSL

Now, an astute reader may wonder how to combine SSL with name-based virtual hosts (www.example.org and svn.example.org that listens on the same IP address and port). After all, one of the features of SSL certificates is that they contain contain a CommonName (CN) field with the hostname of a host: thus either www.example.org or svn.example.org, but not both. However, if my browser requests https://svn.example.org/∞, but gets a certificate with www.example.org as the CommonName, it will complain about an invalid certificate. That is right.

There are two extensions to circumvent the limit of one hostname per IP address:port. The first one is to add a subjectAltName field in your certificate, and the second one is to negotiate the certificate to use during the handshake.

SubjectAltName is the most supported alternative, but is not very scalable, as it requires you to create a new certificate everytime you add a new virtual host to a certain IP address:port combination. However, subjectAltName is widely supported∞. To create a certificate with a subjectAltName, store the certificate configuration file below as ~/servercert.conf and run the following openssl command:




A more recent proposal is to negotiate the certificate to use during the handshake. This is only supported by TLS Handshakes, not by SSL Handshakes. The procedure to do this is called Server Name Indication (SNI). It is at the moment of writing not widely supported, although support seems to be coming.

For more information, see or example Name-based SSL virtual hosts: how to tackle the problem by Kaspar Brand∞ of the Swiss national research network Switch.

Now that you come this far, you can see that our Apache configuration lead to the error in subversion. It's all logical.

Categories

SubversionIssues
CategorySysAdmin