blog: Setting up nginx to reduce load and memory usage for your Django / Python website.
(Using PHP? I will be writing an nginx + PHP guide shortly. Stay tuned!)
nginx is an HTTP server and mail proxy that has been around for a few years and has been recently getting some attention for its ability to work well with low memory machines, proxy setups, load balancing and easy interfacing with FastCGI and other modules to serve various types of content.
I came into a situation where my 256mb Slice was constantly using swap and having trouble stay afloat. I was using Apache with a mix of mod_python and mod_php to serve content for about 6 websites. You may find yourself in a similar situation, so let’s get started on setting up nginx!
In this tutorial, I will be setting up nginx as a front end to serve plain html and pass Django / Python related requests to a WSGI process from Apache. This will help take a huge load off Apache and allow it to be only used when required.
For this write up, I am assuming you have Apache2 installed already. If you do not, it’s easy to install:
$ sudo aptitude install apache2
And now our essentials:
$ sudo aptitude install nginx
$ sudo aptitude install libapache2-mod-wsgi
It is also recommended to build nginx from source. As of this writing, the Ubuntu repository’s package is a bit out of date but still fully functional. If you are on a Debian system, you should be fine with nginx from the repo. (For this tutorial, I am assuming you installed from the repository, so if you installed from source, your paths may be different.)
All your nginx configuration will be done under the /etc/nginx folder. First thing you will want to open up is /etc/nginx/nginx.conf to understand some basic variables; It should look something like this:
user www-data;
worker_processes 2;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log /var/log/nginx/access.log;
sendfile on;
tcp_nopush on;
keepalive_timeout 65;
tcp_nodelay on;
gzip on;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
- worker_processes is the number of processes to spawn. A worker is similar to a child process in Apache. A simplified way to look at this is to set the number of worker_processes to two(2) or the number of CPU’s your server has. In my case, I have four CPU’s but I am not doing anything heavy with SSL or the like so I keep it as two (2) to keep my load down. Your mileage may vary so you may have to expirement.
- worker_connections is simply the number of connections a single child (worker_process) can handle. So by default you have 1024 * 2 = 2048. That’s a lot! For the average person running some small to medium sized sites this will be sufficient and can be left as default.
- listen is not included in my config but you may want to change the port nginx listens on before deploying and replacing Apache as the front.
- include /etc/nginx/sites-enabled/* is a nice throwback to how Apache does VirtualHosts and will keep us converts comfortable. You place your virtual hosts under /etc/nginx/sites-available/ and then link them when you want to enable, just like you would with Apache
The rest of the configuration is pretty straight forward. You can read more about it from the official nginx wiki
If you made any changes, restart nginx:
$ sudo /etc/init.d/nginx restart
And let’s test if nginx is running correctly:
$ curl 127.0.0.1
Should return a similar page:
<html>
<head>
<title>Welcome to nginx!</title>
</head>
<body bgcolor="white" text="black">
<center><h1>Welcome to nginx!</h1></center>
</body>
</html>
Setup our new site under nginx.
nginx configuration is insanely simple so first, let’s create a new entry for our site under /etc/nginx/sites-available/ and fill in some basic information:
$ sudo vim /etc/nginx/sites-available/mysite.com
server {
listen 199.99.999.999:80; # Modify this to be your servers ip address.
server_name www.mysite.com;
rewrite ^/(.*) http://mysite.com/$1 permanent;
}
server {
listen 199.99.999.999:80;
server_name mysite.com;
access_log /var/log/nginx/mysite.com.access.log;
error_log /var/log/nginx/mysite.com.error.log;
}
Viola, with this simple config our site is now listening for requests. We use the simple rewrite rule to send requests for www.mysite.com to mysite.com. You don’t have to do this, and can just add both variations of your domain to the server_name variable but we’re cool and not fans of the www prefix.
Now, you will notice this is just for static content. It’s a good template to save and use as a base for your other sites but let’s add some functionality to our Django site!
Open up the file again and add the following within your primary server block (the second one!):
location / {
proxy_pass http://127.0.0.1:80/;
include /etc/nginx/proxy.conf;
}
location /static {
root /home/admin/domains/mysite.com;
expires 24h;
}
We’ve got two important entries here and one variable to note:
- In location /, we tell nginx that for this site, send ALL requests under / (so basically, any page on our site) through a proxy_pass to http://127.0.0.1:80. Why port 80 on the local host? We’re going to setup Apache to listen on this port in about 5 minutes!
- expires is a great way to do some super-simple caching. In this case, we cache our /static directory for 24 hours. Of course, you can remove this or change it in a variety of ways
- We use include /etc/nginx/proxy.conf. This keeps nginx.conf clean and sets up some essentials for the proxy. I won’t go into much detail on this, but here’s a basic proxy.conf which you will want to save as the file you included:
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffers 32 4k;
- Finally, we set location /static to point to our media/static directory. This basically tells nginx that for anything under /static, don’t do anything special with it and just serve it. The one key here is to place your static directory’s PARENT dir as the root, and not the static dir itself.
Save your virtual host and enable it, we’re done with nginx here:
$ sudo ln -s /etc/nginx/sites-available/mysite.com /etc/nginx-sites-enabled/mysite.com
$ sudo /etc/init.d/nginx restart
Now, let’s switch Apache to run on the local host
In this case, we want to keep Apache around to serve the python code for our Django site. There are other methods to do this (like FastCGI), but the one we’re using is pretty straight forward and uses the great mod_wsgi module from Apache.
Open up the Apache ports.conf file and edit the following so we’re listening only on the local host.
$ sudo vim /etc/apache2/ports.conf
NameVirtualHost 127.0.0.1:80
Listen 127.0.0.1:80
And restart Apache:
$ sudo apache2ctl graceful
Assuming you had no enabled sites, Apache should have not complained about any ports being taken up. At this point, nginx is running on port 80 for clients (users of our websites) and Apache is running locally on the same port, but is only accessible through instances ran internally. Great!
We’re almost done. As you may remember, we sent requests to Apache but we need to actually do something with those requests. With this in mind, we create a virtual host within Apache with some information to create a WSGI instance. Here’s how the file should look like:
$ sudo vim /etc/apache2/sites-available/mysite.com
<VirtualHost 127.0.0.1:80>
ServerName www.mysite.com
ServerAlias mysite.com
<Directory /home/admin/domains/mysite.com/>
Order deny, allow
Allow from all
</Directory>
LogLevel warn
CustomLog /var/log/apache2/mysite.com.access.log combined
ErrorLog /var/log/apache2/mysite.com.error.log
WSGIDaemonProcess mysite.com user=www-data group=www-data threads=25
WSGIProcessGroup mysite.com
WSGIScriptAlias / /home/admin/domains/mysite.com/mysite.wsgi
</VirtualHost>
This is a lot to deal with but if you’re familiar with Apache it’s pretty basic. We setup a VirtualHost to listen on our localhost, setup some logs, and then create the WSGI Daemon process we will need. The major thing to note here is the WSGIScriptAlias which says for all requests under /, use the mysite.wsgi file to handle them. We’ll get to creating that file in a second.
Enable the virtual host as we are now done here:
$ sudo ln -s /etc/apache2/sites-available/mysite.com /etc/apache2/sites-enabled/mysite.com
And now, let’s create our mysite.wsgi file:
$ vim /home/admin/domains/mysite.com/mysite.wsgi
ALLDIRS = ['/usr/lib/python2.5/site-packages']
# the above directory depends on the location of your python installation.
# if using virtualenv, it will need to match your projects locale.
import os
import sys
import site
prev_sys_path = list(sys.path)
for directory in ALLDIRS:
site.addsitedir(directory)
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
# change this depending on your project.
sys.path.append('/home/admin/domains/mysite.com')
os.environ['PYTHON_EGG_CACHE'] = '/home/admin/.python-eggs'
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
This is a very basic WSGI implementation for a Django app but sets all the necessary environment variables and gets things going. Make sure to modify the paths to match your own projects.
$ sudo apache2ctl graceful
And now, everything is essentially setup. To re-iterate, here’s what we did:
- Installed nginx and mod_wsgi for Apache 2
- Setup nginx to run in place of Apache, and made Apache run only on localhost.
- Told nginx to proxy_pass all requests to mod_wsgi in Apache, except for our /static directory.
- Setup a virtual host within Apache to spawn new WSGI processes.
- Create a mysite.wsgi file in our projects directory.
With everything running, you should notice a huge improvement in memory usage. My 256mb slice went from constantly paging and basically crashing at peak hours to rarely paging during peak hours. Apache is a great server and it has its uses but when you are dealing within a low memory environment, a server like nginx is a great way to take a good portion of the load off Apache.
I hope you enjoyed this guide and that everything made sense. If you are having trouble getting something working, feel free to contact me and I will be glad to help you out or fix any errors within this text.
Useful nginx resources:
