[solved] nginx reverse proxy network error with systemd-networkd

schard · 2018-10-10 11:43:24

On our digital signage systems we use nginx as reverse proxy.
Randomly, after boot, nginx fails to connect to the internet and logs the following error into the journal:

Okt 10 13:37:03 52.1030020 nginx[342]: 2018/10/10 13:37:03 [crit] 356#356: connect() failed (101: Network is unreachable) while resolving, resolver: 8.8.8.8:53

The customized systemd.unit is:

$ cat /usr/lib/systemd/system/application-proxy.service 
[Unit]
Description=Application proxy based on Nginx
After=network.target network-online.target nss-lookup.target
Conflicts=nginx.service

[Service]
Type=forking
PIDFile=/run/application-proxy.pid
PrivateDevices=yes
SyslogLevel=err

ExecStart=/usr/bin/nginx -g 'pid /run/application-proxy.pid; error_log stderr;' -c /etc/application-proxy.conf
ExecReload=/usr/bin/nginx -s reload
KillSignal=SIGQUIT
KillMode=mixed
Restart=on-failure
RestartSec=10

[Install]
WantedBy=graphical.target

The used configurations is:

$ cat /etc/application-proxy.conf 
#  proxy.conf - Proxy HTTP and HTTPS requests for local applications
#
#  Hack for old Adobe Air applications.
#
#  (C) 2017: HOMEINFO - Digitale Informationssysteme GmbH
#
#  Maintainer: Richard Neumann <r dot neumann at homeinfo period de>
#
#####################################################################

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include            nginx/mime.types;
    default_type       application/octet-stream;
    sendfile           on;
    keepalive_timeout  65;

    server {
        listen       127.0.0.1;
        server_name  localhost;
        resolver     8.8.8.8;

	# Handles HTTP requests.
        location ~ ^/proxy/http/(.+)$ {
            proxy_intercept_errors on;
            error_page 301 302 307 = @handle_redirect;
            proxy_pass http://$1$is_args$query_string;
        }

        # Handles HTTPS requests.
        location ~ ^/proxy/https/(.+)$ {
            proxy_intercept_errors on;
            error_page 301 302 307 = @handle_redirect;
            proxy_pass https://$1$is_args$query_string;
        }

        # Handles redirects.
        location @handle_redirect {
            set $saved_redirect_location '$upstream_http_location';
            proxy_pass $saved_redirect_location;
        }
    }
}

I also tried to force a dependency on the respective network services:

$ cat /etc/systemd/system/application-proxy.service.d/override.conf 
[Unit]
After=systemd-networkd.service
After=systemd-resolved.service

But no luck.
Nginx still randomly throws the above error, rendering the reverse proxy useless.
What can I improve / change to ensure, that nginx has the network available?

Update:
Doesn't work either:

$ cat /etc/systemd/system/application-proxy.service.d/override.conf 
[Unit]
After=systemd-networkd-wait-online.service
Requires=systemd-networkd-wait-online.service

Last edited by schard (2018-11-07 11:15:14)

ua4000 · 2018-10-17 16:33:14

I assume you have a resolver already defined in systemd-networkd ?

So you could try to remove "resolver 8.8.8.8" from application-proxy.conf, *if*:

If your config file has static DNS names (not generated), and you are not care about track IP changes without nginx reload, you don't need nginx's resolver. In this case all DNS names will be resolved on startup.
(https://stackoverflow.com/questions/403 … m-in-nginx)

also see here: https://medium.com/@adilatilgan/nginx-r … f3fb94aba5

schard · 2018-10-18 08:51:05

The DNS server is being dynamically retrieved over DHCP and varies between systems.
Deleting the resolver will result in

Okt 18 10:49:56 52.1030020 nginx[11810]: 2018/10/18 10:49:56 [error] 11812#11812: *1 no resolver defined to resolve www.homeinfo.de, client: 127.0.0.1, server: localhost, request: "GET /proxy/h…host: "localhost"
Hint: Some lines were ellipsized, use -l to show in full.

Update 2:
The resolver

resolver     127.0.0.53;

actually works (I was just too stupid to use wget correctly).
So I'll probably migrate all systems to systemd-networkd and systemd-resolved (some legacy devices still run dhcpcd) and go with this solution.
I'll give feedback after some further testing.

Solved:
The aforementioned configuration is stable.

Last edited by schard (2018-11-07 11:15:48)

Arch Linux

#1 2018-10-10 11:43:24

[solved] nginx reverse proxy network error with systemd-networkd

#2 2018-10-17 16:33:14

Re: [solved] nginx reverse proxy network error with systemd-networkd

#3 2018-10-18 08:51:05

Re: [solved] nginx reverse proxy network error with systemd-networkd

Board footer