[SOLVED] Apache Mysteriously Stopping

tony5429 · 2015-12-02 01:28:42

I've been running a simple LAMP server for about a year now with no big issues until a few days ago when Apache started dying on me. Here is a timeline of events, all in UTC time. I find it particulary odd that the restarts happen so close to midnight. Does anyone have any idea what would cause this and how I could fix it? Thanks.

2015-11-20 12:23:27 | system manually rebooted (after a system update), thereby restarting Apache

2015-11-22 00:00:17 | Apache dies, as noted in the below output (collected 5 days later)

[root@myserver ~]# systemctl status httpd
● httpd.service - Apache Web Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Sun 2015-11-22 00:00:17 UTC; 5 days ago
  Process: 18832 ExecStop=/usr/bin/apachectl graceful-stop (code=exited, status=0/SUCCESS)
  Process: 299 ExecStart=/usr/bin/apachectl start -DFOREGROUND (code=killed, signal=HUP)
 Main PID: 299 (code=killed, signal=HUP)

Nov 20 12:23:27 myserver systemd[1]: Started Apache Web Server.
[root@myserver ~]#

2015-11-27 14:38:36 | system manually rebooted (after a system update), thereby restarting Apache

2015-11-30 00:00:21 | Apache dies, as noted in the below output (collected 2 days later)

[root@myserver ~]# systemctl status httpd
● httpd.service - Apache Web Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Mon 2015-11-30 00:00:21 UTC; 2 days ago
  Process: 13845 ExecStop=/usr/bin/apachectl graceful-stop (code=exited, status=0/SUCCESS)
  Process: 296 ExecStart=/usr/bin/apachectl start -DFOREGROUND (code=killed, signal=HUP)
 Main PID: 296 (code=killed, signal=HUP)

Nov 27 14:38:36 myserver systemd[1]: Started Apache Web Server.
[root@myserver ~]#

Last edited by tony5429 (2015-12-17 14:51:58)

Trilby · 2015-12-02 01:33:10

Have you edited the service file, or are you using a user-defined service file? The -DFOREGROUND is not in the original (unless it's been recently added). I'm not familiar with that flag, but if it keeps it running in the foreground it is not surprising that it would get killed by a sighup.

EDIT: nevermind me - I just upgraded my server and see that -DFOREGROUND has been added as a default. Sorry for the noise.

cris9288 · 2015-12-02 03:55:58

Holy shit this has been happening to me recently and it's been driving me crazy. No idea what is causing it, but there may be a clue in my error_log:

[Sun Nov 29 07:51:29.591885 2015] [core:notice] [pid 29945] AH00094: Command line: '/usr/bin/httpd -D FOREGROUND'
[Mon Nov 30 00:00:01.232688 2015] [mpm_prefork:notice] [pid 29945] AH00173: SIGHUP received.  Attempting to restart
[Mon Nov 30 07:11:22.285930 2015] [lbmethod_heartbeat:notice] [pid 16566] AH02282: No slotmem from mod_heartmonitor
[Mon Nov 30 07:11:22.286309 2015] [:notice] [pid 16568] FastCGI: process manager initialized (pid 16568)
[Mon Nov 30 07:11:22.299674 2015] [core:warn] [pid 16566] AH00098: pid file /run/httpd/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Mon Nov 30 07:11:22.301569 2015] [mpm_prefork:notice] [pid 16566] AH00163: Apache/2.4.17 (Unix) OpenSSL/1.0.2d mod_fastcgi/mod_fastcgi-SNAP-0910052141 PHP/5.6.16 configured -- resuming normal operations

You can see that apache does receive a SIGHUP (from mpm_prefork?) I did find this post here:

https://www.ateamsystems.com/tech-blog/ … -midnight/

But I haven't been able to reproduce the grace crash using

 # apachectl graceful

and the crash doesn't happen very frequently. Do you have the mpm_prefork module enabled? I wonder if disabling it will stop crashes altogether.

Last edited by cris9288 (2015-12-02 05:36:54)

anatolik · 2015-12-02 05:45:36

It looks like this problem https://bugs.archlinux.org/task/47110 when logrotate sends SIGHUP. SIGHUP suppose to restart apache daemon but instead it hangs/crashes it.

tony5429 · 2015-12-04 12:00:14

According to the Arch changelog for Apache (https://projects.archlinux.org/svntogit … ges/apache), version 2.4.17-4 says (44 hours ago): "FS#47110 Revert revision 249812 (start service as foreground process)". I just updated and restarted Apache on my server and suspect the issue is solved now; no "-DFOREGROND" tag in the start line anymore.

[root@myserver ~]# systemctl status httpd
● httpd.service - Apache Web Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2015-12-04 11:53:36 UTC; 3min 23s ago
  Process: 288 ExecStart=/usr/bin/apachectl start (code=exited, status=0/SUCCESS)
 Main PID: 413 (httpd)
   CGroup: /system.slice/httpd.service
           ├─413 /usr/bin/httpd -k start                                                                                                                                                        
           ├─414 /usr/bin/httpd -k start                                                                                                                                                        
           ├─415 /usr/bin/httpd -k start                                                                                                                                                        
           ├─416 /usr/bin/httpd -k start                                                                                                                                                        
           ├─417 /usr/bin/httpd -k start                                                                                                                                                        
           └─418 /usr/bin/httpd -k start                                                                                                                                                        
                                                                                                                                                                                                
Dec 04 11:53:02 myserver systemd[1]: Starting Apache Web Server...                                                                                                                           
Dec 04 11:53:20 myserver systemd[1]: httpd.service: PID file /run/httpd/httpd.pid not readable (yet?) after start: No such file or directory                                                 
Dec 04 11:53:36 myserver systemd[1]: Started Apache Web Server.                                                                                                                              
[root@myserver ~]#

Trilby · 2015-12-04 12:16:29

Cool, so I'm not as dumb as I look

Arch Linux

#1 2015-12-02 01:28:42

[SOLVED] Apache Mysteriously Stopping

#2 2015-12-02 01:33:10

Re: [SOLVED] Apache Mysteriously Stopping

#3 2015-12-02 03:55:58

Re: [SOLVED] Apache Mysteriously Stopping

#4 2015-12-02 05:45:36

Re: [SOLVED] Apache Mysteriously Stopping

#5 2015-12-04 12:00:14

Re: [SOLVED] Apache Mysteriously Stopping

#6 2015-12-04 12:16:29

Re: [SOLVED] Apache Mysteriously Stopping

Board footer