I decided to give Apache 2.2’s worker mpm a try (a couple processes that each have many threads rather than the default prefork’s several processes that each handle a single thread), but I was having the Apache root process exit because its children were disappearing. Sometimes this would happen immediately after launching Apache, other times it would happen after I tried to access a web page. I tracked the problem down to being an issue with apache hitting the process limit for a given user. However, modifying limits.conf for the apache user did not solve the problem because the apache server inherits root’s limits on start up rather than referring to limits.conf.

The Initial Problem

After relaunching Apache, its processes would sometimes disappear immediately. Its error log contained (apache2/error_log):

[Sat Jul 11 00:32:42 2009] [notice] Apache/2.2.11 (Unix) configured -- resuming normal operations
[Sat Jul 11 00:32:42 2009] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Sat Jul 11 00:32:44 2009] [alert] No active workers found... Apache is exiting!

I run gentoo’s hardened kernel with grsecurity’s logging active, so I also checked my grsecurity kernel log:

Jul 11 00:32:42 ouroboros grsec: From 71.202.174.186: denied resource overstep by requesting 32 for RLIMIT_NPROC against limit 32 for /usr/sbin/apache2[apache2:14547] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:14543] uid/euid:0/0 gid/egid:0/0

It appears that the number of processes resource limit for a given user applies to threads in Linux, and not just processes. By default, the worker mpm creates two worker processes with up to 25 worker threads each. Each process also has a thread for communicating with the parent process to handle incoming requests. This means there are can be 52 threads created for the apache user. The default limit of 32 processes per user was apparently getting exceeded when the threads were created, leading to the worker processes exiting and disappearing.

Attempting to Use limits.conf

First, I tried to increase the apache user’s limits in limits.conf:

*               hard    nproc           32
apache          hard    nproc           100

However, limits.conf is only referenced by the pam_limits.so module of PAM, meaning that PAM would have to run at some point before or during the start up of the apache server to set its resource limits. Many services on Gentoo Linux use the start-stop-daemon, which supposedly received PAM support in baselayout 1.13, but 1.13 is not in portage. The final version of baselayout 1 is 1.12 on Gentoo, with 2.0 being the next unstable version. Also, the apache2 startup script only calls apache2ctl and never uses start-stop-daemon, meaning that even if this feature were in the stable baselayout, limits.conf is not a normal option

The Final Solution

My final solution was to specify the limit in the configuration file for the startup script. At the bottom of /etc/conf.d/apache2, which is just a shell script file that gets sourced for /etc/init.d/apache2, I added

ulimit -u 100

This line calls bash’s ulimit command and sets the maximum number of processes to 100, which allowed apache to have its threads without crashing.