AIR Wiki: The 502 Bad Gateway Error

Most recent edit on 2007-07-20 23:16:40 by FreekDijkstra

Additions:
SubjectAltName is the most supported alternative, but is not very scalable, as it requires you to create a new certificate everytime you add a new virtual host to a certain IP address:port combination. However, subjectAltName is widely supported∞. To create a certificate with a subjectAltName, store the certificate configuration file below as ~/servercert.conf and run the following openssl command:

Deletions:
SubjectAltName is the most supported alternative, but is not very scalable, as it requires you to create a new certificate everytime you add a new virtual host to a certain IP address:port combination. However, [http://wiki.cacert.org/wiki/VhostTaskForce#head-7236c4e2c9932ef42056b3ff6d367053081887de∞ subjectAltName is widely supported]. To create a certificate with a subjectAltName, store the certificate configuration file below as ~/servercert.conf and run the following openssl command:

Edited on 2007-07-20 23:14:23 by FreekDijkstra

Additions:
we find the cause to be in mod_ssl. Somehow, our Apache server believed it was contacted at port 80 using HTTP even though in reality the client contacted it at port 443 using HTTPS. Yes, that's pretty weird, but you can reproduce it yourself with this small snippet of Apache configuration code:

Deletions:
we find the cause to be in mod_ssl. Somehow, our Apache server believed it was contacts to port 80 using HTTP even though in reality we contacted it at port 443 using HTTPS. Yes, that's pretty weird, but you can reproduce it yourself with this small snippet of Apache configuration code:

Edited on 2007-07-20 15:17:25 by FreekDijkstra

Additions:
See for example Subversion behind an Apache Reverse Proxy∞ for information how to fix this.

Deletions:
See for example [http://silmor.de/49∞ Subversion behind an Apache Reverse Proxy] for information how to fix this.

Edited on 2007-07-20 15:16:28 by FreekDijkstra

Additions:

The 502 Bad Gateway Error

Work around

WebDAV

Non-matching source and destination URL

There are multiple causes for a bad translation or non-match between source and destination URL with a HTTP COPY. The two most common causes are:

Multiple server aliases
Use of a HTTP reverse proxy server

If a machine listens to multiple names (e.g. www.example.org and svn.example.org), but only one of them is configured in the apache configuration with ServerName www.example.org then Apache will regard this as a two non-matching servers. The solution is to simply add the other names to ServerAlias:
ServerName www.example.org
ServerAlias svn.example.org
The other common cause is the use of a HTTPS reverse proxy server in front of your HTTP server. The proxy server receives the requests, translates the request and hostname and forwards it to the HTTP server. However, because the proxy server did not translate the Destination field, the HTTP server still sees the HTTPS address and will return the 502 Bad Gateway error.
See for example [http://silmor.de/49∞ Subversion behind an Apache Reverse Proxy] for information how to fix this.

Mod_ssl says: 443 equals 80

After much strugling, we discovered a third, non so common cause for a non-match between source and destination URL with a HTTP COPY:

Bad configuration of virtual HTTPS hosts

we find the cause to be in mod_ssl. Somehow, our Apache server believed it was contacts to port 80 using HTTP even though in reality we contacted it at port 443 using HTTPS. Yes, that's pretty weird, but you can reproduce it yourself with this small snippet of Apache configuration code:

mod_ssl processing of SSL parameters

Virtual Hosts with SSL

Deletions:

The 502 Bad Gateway Error

A typical example of such a bad translation is when you have a HTTPS reverse proxy server in front of your HTTP server. The proxy server receives the requests, translates the request and hostname and forwards it to the HTTP server. However, because the proxy server did not translate the Destination field, the HTTP server still sees the HTTPS address and will return the 502 Bad Gateway error.
However, in our case, the problem was caused elsewhere. After much strugling, we find the cause to be in mod_ssl. Somehow, our Apache server believed it was contacts to port 80 using HTTP even though in reality we contacted it at port 443 using HTTPS. Yes, that's pretty weird, but you can reproduce it yourself with this small snippet of Apache configuration code:

Details of mod_ssl

Virtual Hosts with SSL

Edited on 2007-07-20 09:16:14 by FreekDijkstra

Additions:
For more information, see Apache bug report #42929∞. Now that you come this far, you can see that our Apache configuration lead to the error in subversion. It's all logical.

Deletions:
For more information, see [http://issues.apache.org/bugzilla/show_bug.cgi?id=42929∞ Apache bug report #42929].

Edited on 2007-07-20 09:12:30 by FreekDijkstra

Additions:
Now Apache reports the wrong port when requesting https://svn.example.org/server-info∞: Apache thinks it is contacted at port 80, but in reality, it is contacted at port 443. The above configuration caused that problem. In fact, if you would have used this configuration below, there wouldn't have been any problem:

Deletions:
Now that is very wrong for https://svn.example.org/server-info∞: Apache thinks it is contacted at port 80, but in reality, it is contacted at port 443. The above configuration caused that problem. In fact, if you would have used this configuration below, there wouldn't have been any problem:

Edited on 2007-07-20 09:11:21 by FreekDijkstra

Additions:
This is a short story of software complexity.
Well, actually it is a bit more complex, as the real request is in XML and subversions groups some requests together in single HTTP request to something like COPY /svn/repos/!svn/act/6b2389f7-9223-4eca-b3fa-96cb3da83b78 HTTP/1.1, but you get the general idea.

Deletions:
This is a short story of complexity.
Well, actually it is a bit more complex, as the real details are in XML and subversions groups some requests together in single HTTP request, but you get the general idea.

Edited on 2007-07-20 09:05:36 by FreekDijkstra

Additions:

The 502 Bad Gateway Error

Edited on 2007-07-20 08:48:52 by FreekDijkstra

Additions:
Now, an astute reader may wonder how to combine SSL with name-based virtual hosts (www.example.org and svn.example.org that listens on the same IP address and port). After all, one of the features of SSL certificates is that they contain contain a CommonName (CN) field with the hostname of a host: thus either www.example.org or svn.example.org, but not both. However, if my browser requests https://svn.example.org/∞, but gets a certificate with www.example.org as the CommonName, it will complain about an invalid certificate. That is right. There are two extensions to circumvent the limit of one hostname per IP address:port. The first one is to add a subjectAltName field in your certificate, and the second one is to negotiate the certificate to use during the handshake.
SubjectAltName is the most supported alternative, but is not very scalable, as it requires you to create a new certificate everytime you add a new virtual host to a certain IP address:port combination. However, [http://wiki.cacert.org/wiki/VhostTaskForce#head-7236c4e2c9932ef42056b3ff6d367053081887de∞ subjectAltName is widely supported]. To create a certificate with a subjectAltName, store the certificate configuration file below as ~/servercert.conf and run the following openssl command:
A more recent proposal is to negotiate the certificate to use during the handshake. This is only supported by TLS Handshakes, not by SSL Handshakes. The procedure to do this is called Server Name Indication (SNI). It is at the moment of writing not widely supported, although support seems to be coming.
For more information, see or example Name-based SSL virtual hosts: how to tackle the problem by Kaspar Brand∞ of the Swiss national research network Switch.

Deletions:
Now, an astute reader may wonder how to combine SSL with name-based virtual hosts (www.example.org and svn.example.org that listens on the same IP address and port). After all, one of the features of SSL certificates is that they contain contain a CommonName (CN) field with the hostname of a host: thus either www.example.org or svn.example.org, but not both. However, if my browser requests https://svn.example.org/∞, but gets a certificate with www.example.org as the CommonName, it will complain about an invalid certificate. That is right. However, there are two extensions to circumvent this. The first one is to add a subjectAltName field in your certificate, and the second one is to use
Store the certificate configuration file below as ~/servercert.conf

Edited on 2007-07-20 08:27:29 by FreekDijkstra

Additions:

Virtual Hosts with SSL

Now, an astute reader may wonder how to combine SSL with name-based virtual hosts (www.example.org and svn.example.org that listens on the same IP address and port). After all, one of the features of SSL certificates is that they contain contain a CommonName (CN) field with the hostname of a host: thus either www.example.org or svn.example.org, but not both. However, if my browser requests https://svn.example.org/∞, but gets a certificate with www.example.org as the CommonName, it will complain about an invalid certificate. That is right. However, there are two extensions to circumvent this. The first one is to add a subjectAltName field in your certificate, and the second one is to use

Deletions:
emailAddress = hostmaster@example.org

Edited on 2007-07-20 08:13:53 by FreekDijkstra

Additions:
# Bad configuration of Apache
# Good configuration of Apache
For more information, see [http://issues.apache.org/bugzilla/show_bug.cgi?id=42929∞ Apache bug report #42929].
You may wonder why mod_ssl behaves in this way. For one thing, the second vhost the first (bad) configuration has no "SSLEngine on" directive. So why did Apache server use SSL in the first place when I contacted https://svn.example.org∞? The reason lies in the fact that mod_ssl ignores vhosts! That may seem strange, until you realize that the SSL Handshake takes place before the client sends the Host: header. So before any vhost is determined. To be precise, there can be only one certificate per IP address:port combination. The only IP address:port combination in the above configurations is "*:443". mod_ssl simply looks at the first vhost with that IP address and port, and uses the SSL configuration mentioned there. So effectively, all vhosts with that configuration share these SSL settings:

Deletions:
You may wonder
http://issues.apache.org/bugzilla/show_bug.cgi?id=42929∞

Edited on 2007-07-19 23:00:02 by FreekDijkstra

Additions:
Well, actually it is a bit more complex, as the real details are in XML and subversions groups some requests together in single HTTP request, but you get the general idea.
Apache, the webserver, translates the above request to: "Copy https://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt". It uses the Host parameter svn.example.org and the first request line COPY /svn/repos/oldname.txt HTTP/1.1 to assemble the source URL, https://svn.example.org/svn/repos/oldname.txt∞. However, in some cases, the assembly of the source URL goes wrong, and end up as http://svn.example.org/svn/repos/oldname.txt∞ (note the http instead of https). Apache quickly figures out that it can't move a file http://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt∞, because as far as Apache is concerned http://svn.example.org/∞ and https://svn.example.org/∞ are two entirely different hosts. It will respond with a "502 Bad Gateway" error.
A typical example of such a bad translation is when you have a HTTPS reverse proxy server in front of your HTTP server. The proxy server receives the requests, translates the request and hostname and forwards it to the HTTP server. However, because the proxy server did not translate the Destination field, the HTTP server still sees the HTTPS address and will return the 502 Bad Gateway error.
However, in our case, the problem was caused elsewhere. After much strugling, we find the cause to be in mod_ssl. Somehow, our Apache server believed it was contacts to port 80 using HTTP even though in reality we contacted it at port 443 using HTTPS. Yes, that's pretty weird, but you can reproduce it yourself with this small snippet of Apache configuration code:
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule info_module modules/mod_info.so
User www
Group www
DocumentRoot "/var/www"
<Location /server-info>

SetHandler server-info

</Location>
ErrorLog /var/log/httpd/error_log
Listen *:443
NameVirtualHost *:443
<VirtualHost *:443>

ServerName www.example.org
SSLEngine on
SSLCertificateFile /usr/local/apache2/conf/ssl/servercert.pem
SSLCertificateKeyFile /usr/local/apache2/conf/ssl/serverkey.pem

</VirtualHost>
<VirtualHost *:443>

ServerName svn.example.org

</VirtualHost>
This is a configuration for a sever with two name-based "virtual" hosts: www.example.org and svn.example.org. Both listen to the same IP address on port 443 (which is used for HTTPS). HTTPS is turned on with "SSLEngine on"
As an extra, the server is configured that if you visit https://www.example.org/server-info∞ or https://svn.example.org/server-info∞, you get an page with information about the configuration of the server. You will notice that they say:

URL	servername/port reported by Apache
https://www.example.org/server-info	www.example.org:443
https://svn.example.org/server-info	svn.example.org:80

Now that is very wrong for https://svn.example.org/server-info∞: Apache thinks it is contacted at port 80, but in reality, it is contacted at port 443. The above configuration caused that problem. In fact, if you would have used this configuration below, there wouldn't have been any problem:
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule info_module modules/mod_info.so
User www
Group www
DocumentRoot "/var/www"
<Location /server-info>

SetHandler server-info

</Location>
ErrorLog /var/log/httpd/error_log
Listen *:443
SSLCertificateFile /usr/local/apache2/conf/ssl/servercert.pem
SSLCertificateKeyFile /usr/local/apache2/conf/ssl/serverkey.pem
NameVirtualHost *:443
<VirtualHost *:443>

ServerName www.example.org
SSLEngine on

</VirtualHost>
<VirtualHost *:443>

ServerName svn.example.org
SSLEngine on

</VirtualHost>

Details of mod_ssl

You may wonder
#
# www.example.org certificate configuration file.
#
[ req ]
default_bits = 2048
default_days = 365
default_keyfile = serverkey.pem
prompt = no
encrypt_key = yes
distinguished_name = req_distinguished_name
x509_extensions = req_x509v3_ext
[ req_distinguished_name ]
countryName = AQ
stateOrProvinceName = Antartica
localityName = The Pole
organizationName = Nowhere, inc.
organizationalUnitName = The Middle
commonName = www.example.org
emailAddress = hostmaster@example.org
[ req_x509v3_ext ]
nsCertType = server
subjectAltName = DNS:www.example.org,DNS:svn.example.org
Store the certificate configuration file below as ~/servercert.conf
openssl req -new -x509 -nodes -config ~/servercert.conf \

out /usr/local/apache2/conf/ssl/servercert.pem \
keyout /usr/local/apache2/conf/ssl/serverkey.pem

http://issues.apache.org/bugzilla/show_bug.cgi?id=42929∞

Deletions:
Well, actually it is a bit more complex, as the real details are in XML and subversions groups some requests together in single HTTP request, but you get the general idea. Apache, the webserver, translates the above request to: "Copy https://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt". It uses the Host parameter svn.example.org and the first request line COPY /svn/repos/oldname.txt HTTP/1.1 to assemble the source URL, https://svn.example.org/svn/repos/oldname.txt∞. However, in some cases, the assembly of the source URL goes wrong, and end up as http://svn.example.org/svn/repos/oldname.txt∞ (note the http instead of https). Apache quickly figures out that it can't move a file http://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt∞, because as far as Apache is concerned http://svn.example.org/∞ and https://svn.example.org/∞ are two entirely different hosts. It will respond with a "502 Bad Gateway" error.

Edited on 2007-07-19 22:46:06 by FreekDijkstra

Additions:
Well, actually it is a bit more complex, as the real details are in XML and subversions groups some requests together in single HTTP request, but you get the general idea. Apache, the webserver, translates the above request to: "Copy https://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt". It uses the Host parameter svn.example.org and the first request line COPY /svn/repos/oldname.txt HTTP/1.1 to assemble the source URL, https://svn.example.org/svn/repos/oldname.txt∞. However, in some cases, the assembly of the source URL goes wrong, and end up as http://svn.example.org/svn/repos/oldname.txt∞ (note the http instead of https). Apache quickly figures out that it can't move a file http://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt∞, because as far as Apache is concerned http://svn.example.org/∞ and https://svn.example.org/∞ are two entirely different hosts. It will respond with a "502 Bad Gateway" error.

Deletions:
Well, actually it is a bit more complex, as the real details are in XML and subversions groups some requests together in single HTTP request, but you get the general idea. Apache, the webserver, translates the above request to: "Copy https://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt". It uses the Host parameter svn.example.org and the first request line COPY /svn/repos/oldname.txt HTTP/1.1 to assemble the source URL, https://svn.example.org/svn/repos/oldname.txt∞. However, in some cases, the assembly of the source URL goes wrong, and end up as http://svn.example.org/svn/repos/oldname.txt#∞# (note the http instead of https). Apache quickly figures out that it can't move a file http://svn.example.org/svn/repos/oldname.txt#∞# to https://svn.example.org/svn/repos/newname.txt##∞, because as far as Apache is concerned http://svn.example.org/#∞# and https://svn.example.org/#∞# are two entirely different hosts. It will respond with a "502 Bad Gateway" error.

Edited on 2007-07-19 22:45:30 by FreekDijkstra

Additions:
Well, actually it is a bit more complex, as the real details are in XML and subversions groups some requests together in single HTTP request, but you get the general idea. Apache, the webserver, translates the above request to: "Copy https://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt". It uses the Host parameter svn.example.org and the first request line COPY /svn/repos/oldname.txt HTTP/1.1 to assemble the source URL, https://svn.example.org/svn/repos/oldname.txt∞. However, in some cases, the assembly of the source URL goes wrong, and end up as http://svn.example.org/svn/repos/oldname.txt#∞# (note the http instead of https). Apache quickly figures out that it can't move a file http://svn.example.org/svn/repos/oldname.txt#∞# to https://svn.example.org/svn/repos/newname.txt##∞, because as far as Apache is concerned http://svn.example.org/#∞# and https://svn.example.org/#∞# are two entirely different hosts. It will respond with a "502 Bad Gateway" error.

Deletions:
Well, actually it is a bit more complex, as the real details are in XML and subversions groups some requests together in single HTTP request, but you get the general idea. Apache, the webserver, translates the above request to: "Copy https://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt". It uses the Host parameter svn.example.org and the first request line COPY /svn/repos/oldname.txt HTTP/1.1 to assemble the source URL, [https://svn.example.org/svn/repos/oldname.txt]##∞. However, in some cases, the assembly of the source URL goes wrong, and end up as http://svn.example.org/svn/repos/oldname.txt#∞# (note the http instead of https). Apache quickly figures out that it can't move a file http://svn.example.org/svn/repos/oldname.txt#∞# to https://svn.example.org/svn/repos/newname.txt##∞, because as far as Apache is concerned http://svn.example.org/#∞# and https://svn.example.org/#∞# are two entirely different hosts. It will respond with a "502 Bad Gateway" error.

Edited on 2007-07-19 22:44:52 by FreekDijkstra

Additions:
Well, actually it is a bit more complex, as the real details are in XML and subversions groups some requests together in single HTTP request, but you get the general idea. Apache, the webserver, translates the above request to: "Copy https://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt". It uses the Host parameter svn.example.org and the first request line COPY /svn/repos/oldname.txt HTTP/1.1 to assemble the source URL, [https://svn.example.org/svn/repos/oldname.txt]##∞. However, in some cases, the assembly of the source URL goes wrong, and end up as http://svn.example.org/svn/repos/oldname.txt#∞# (note the http instead of https). Apache quickly figures out that it can't move a file http://svn.example.org/svn/repos/oldname.txt#∞# to https://svn.example.org/svn/repos/newname.txt##∞, because as far as Apache is concerned http://svn.example.org/#∞# and https://svn.example.org/#∞# are two entirely different hosts. It will respond with a "502 Bad Gateway" error.

Deletions:
Well, actually it is a bit more complex, as the real details are in XML and subversions groups some requests together in single HTTP request, but you get the general idea. Apache, the webserver, translates the above request to: "Copy https://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt". It uses the Host parameter svn.example.org and the first request line COPY /svn/repos/oldname.txt HTTP/1.1 to assemble the source URL, https://svn.example.org/svn/repos/oldname.txt##∞. However, in some cases, the assembly of the source URL goes wrong, and end up as http://svn.example.org/svn/repos/oldname.txt#∞# (note the http instead of https). Apache quickly figures out that it can't move a file http://svn.example.org/svn/repos/oldname.txt#∞# to https://svn.example.org/svn/repos/newname.txt##∞, because as far as Apache is concerned http://svn.example.org/#∞# and https://svn.example.org/#∞# are two entirely different hosts. It will respond with a "502 Bad Gateway" error.

Edited on 2007-07-19 22:42:20 by FreekDijkstra

Additions:
The "+" sign means that the add operation is actually a COPY operation, and the history of the original file is copied to the newly added file. In this case, the history of oldname.txt.

Deletions:
The "+" sign means that the add operation is actually a COPY operation, and the history of the original file is copied to the newly added file. In this case, the history of oldname.txt%%.

Oldest known version of this page was edited on 2007-07-19 22:41:46 by FreekDijkstra []

Page view:

This is a short story of complexity.

I had a problem with subversion: the use of svn move or svn copy lead to the following error:

svn: Commit failed (details follow):
svn: COPY of newname.txt: 502 Bad Gateway (https://svn.example.org)

I could predict which files would give problems, because those had a "+" sign when I did a SVN status:

% svn move oldname.txt newname.txt
% svn status
D      oldname.txt
A    + newname.txt

The "+" sign means that the add operation is actually a COPY operation, and the history of the original file is copied to the newly added file. In this case, the history of oldname.txt . There is a work-around. Simply adding the file, without copying the history log. svn revert newname.txt svn add newname.txt However, this will lead to the loss of the history information. Now, what caused our problem? Deep down, the problem is in mod_ssl. But the link is really not obvious, so let me go step by step. First of all, let's find out what triggers this "502 Bad Gateway" error. Our subversion repositories use HTTP for the datatransport. Or actually HTTPS because that is more secure. You can even see that in the URL of our repository: https://svn.example.org/svn/repos (An alternative would be to use SSH for data transport, with the URL: svn+ssh://svn.example.org/svn/repos, but let's not go into that detail). To be exact, subversion uses an extension of HTTP, called WebDAV. Technically, a move or copy operation in subversion is translated to a Webdav COPY operation. The request looks a bit likes this: COPY /svn/repos/oldname.txt HTTP/1.1 Host: svn.example.org Destination: https://svn.example.org/svn/repos/newname.txt∞ %% Well, actually it is a bit more complex, as the real details are in XML and subversions groups some requests together in single HTTP request, but you get the general idea. Apache, the webserver, translates the above request to: "Copy https://svn.example.org/svn/repos/oldname.txt∞ to https://svn.example.org/svn/repos/newname.txt". It uses the Host parametersvn.example.org and the first request line COPY /svn/repos/oldname.txt HTTP/1.1 to assemble the source URL, https://svn.example.org/svn/repos/oldname.txt##∞. However, in some cases, the assembly of the source URL goes wrong, and end up as http://svn.example.org/svn/repos/oldname.txt#∞# (note the http instead of https). Apache quickly figures out that it can't move a file http://svn.example.org/svn/repos/oldname.txt#∞# to https://svn.example.org/svn/repos/newname.txt##∞, because as far as Apache is concerned http://svn.example.org/#∞# and https://svn.example.org/#∞# are two entirely different hosts. It will respond with a "502 Bad Gateway" error.

AIR Wiki : Subversion502BadGateway

The 502 Bad Gateway Error

Work around

WebDAV

Non-matching source and destination URL

Mod_ssl says: 443 equals 80

mod_ssl processing of SSL parameters

Virtual Hosts with SSL

The 502 Bad Gateway Error

Details of mod_ssl

Virtual Hosts with SSL

Categories

The 502 Bad Gateway Error

Virtual Hosts with SSL

Details of mod_ssl