HUPing Zombie Parents

A script to reload apache servers with zombie children.

Context:

We’ve encountered a problem where we had zombie CGI processes hanging around. That wouldn’t be a very serious problem per se if there wasn’t a limit on the number of concurrently running CGI processes set in the Apache/mod_fcgi configuration and zombie FCGI processes would block the creation of new FCGI processes.

All this would happen inside docker containers under kubernetes. For an unknown reason, the apache process wouldn’t reap its zombie FCGI children.

Given that the deployed application stack wasn’t recent, debugging it didn’t make much sense.

So I decided to work around the problem and send a HUP to the parent apache process, which would reap it, along with its children and spawn a fresh apache process (that HUP wouldn’t affect the master apache process, and its other workers).

I’ve groomed the net but didn’t find an elegant solution. So here’s mine, it should be generic enough to be applicable over a wider problem space:

#!/bin/bash
#
# Loop and send HUPs to apache2's that are
# parents of zombies

# loop forever
#
while true; do
   # wait 5s between loops
   #
   sleep 5

   zombie_parents=$( ps -eo pid,ppid,state | tail -n +2 | grep Z | awk '{print $2}' )

   for zombie_parent in $zombie_parents; do
      # only send HUP if it is an apache2 process
      #
      if ps -q $zombie_parent -o comm= | grep -q apache2; then
         kill -HUP $zombie_parent
         # sleep another 5s after sending HUP to apache
         sleep 5
      fi
   done
done