blog: Setting up nginx to reduce load and memory usage for your Django / Python website.

Posted on 06 Apr 2009 under

(Using PHP? I will be writing an nginx + PHP guide shortly. Stay tuned!)

nginx is an HTTP server and mail proxy that has been around for a few years and has been recently getting some attention for its ability to work well with low memory machines, proxy setups, load balancing and easy interfacing with FastCGI and other modules to serve various types of content.

I came into a situation where my 256mb Slice was constantly using swap and having trouble stay afloat. I was using Apache with a mix of mod_python and mod_php to serve content for about 6 websites. You may find yourself in a similar situation, so let’s get started on setting up nginx!

In this tutorial, I will be setting up nginx as a front end to serve plain html and pass Django / Python related requests to a WSGI process from Apache. This will help take a huge load off Apache and allow it to be only used when required.

For this write up, I am assuming you have Apache2 installed already. If you do not, it’s easy to install:

    $ sudo aptitude install apache2

And now our essentials:

    $ sudo aptitude install nginx
    $ sudo aptitude install libapache2-mod-wsgi

It is also recommended to build nginx from source. As of this writing, the Ubuntu repository’s package is a bit out of date but still fully functional. If you are on a Debian system, you should be fine with nginx from the repo. (For this tutorial, I am assuming you installed from the repository, so if you installed from source, your paths may be different.)

All your nginx configuration will be done under the /etc/nginx folder. First thing you will want to open up is /etc/nginx/nginx.conf to understand some basic variables; It should look something like this:


    user www-data;
    worker_processes  2;

    error_log  /var/log/nginx/error.log;
    pid        /var/run/nginx.pid;
    
    events {
        worker_connections  1024;
    }
   
    http {
       include       /etc/nginx/mime.types;
       default_type  application/octet-stream;
     
       access_log  /var/log/nginx/access.log;
    
       sendfile        on;
       tcp_nopush     on;
    
       keepalive_timeout  65;
       tcp_nodelay        on;
     
       gzip  on;
     
       include /etc/nginx/conf.d/*.conf;
       include /etc/nginx/sites-enabled/*;
    }

The rest of the configuration is pretty straight forward. You can read more about it from the official nginx wiki

If you made any changes, restart nginx:

    $  sudo /etc/init.d/nginx restart

And let’s test if nginx is running correctly:

    $ curl 127.0.0.1

Should return a similar page:

    <html>
    <head>
    <title>Welcome to nginx!</title>
    </head>
    <body bgcolor="white" text="black">
    <center><h1>Welcome to nginx!</h1></center>
    </body>
    </html>

Setup our new site under nginx.

nginx configuration is insanely simple so first, let’s create a new entry for our site under /etc/nginx/sites-available/ and fill in some basic information:

    $ sudo vim /etc/nginx/sites-available/mysite.com

    server {
        listen 199.99.999.999:80; # Modify this to be your servers ip address.
        server_name www.mysite.com;
        rewrite ^/(.*) http://mysite.com/$1 permanent;
    }

    server {
        listen 199.99.999.999:80;
        server_name mysite.com;

        access_log /var/log/nginx/mysite.com.access.log;
        error_log /var/log/nginx/mysite.com.error.log;

    }

Viola, with this simple config our site is now listening for requests. We use the simple rewrite rule to send requests for www.mysite.com to mysite.com. You don’t have to do this, and can just add both variations of your domain to the server_name variable but we’re cool and not fans of the www prefix.

Now, you will notice this is just for static content. It’s a good template to save and use as a base for your other sites but let’s add some functionality to our Django site!

Open up the file again and add the following within your primary server block (the second one!):

    location / {
        proxy_pass http://127.0.0.1:80/;
        include /etc/nginx/proxy.conf;
    }

    location /static {
        root    /home/admin/domains/mysite.com;
        expires 24h;
    }

We’ve got two important entries here and one variable to note:

    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    client_max_body_size 10m;
    client_body_buffer_size 128k;
    proxy_connect_timeout 90;
    proxy_send_timeout 90;
    proxy_read_timeout 90;
    proxy_buffers 32 4k;

Save your virtual host and enable it, we’re done with nginx here:

    $ sudo ln -s /etc/nginx/sites-available/mysite.com /etc/nginx-sites-enabled/mysite.com
    $ sudo /etc/init.d/nginx restart

Now, let’s switch Apache to run on the local host

In this case, we want to keep Apache around to serve the python code for our Django site. There are other methods to do this (like FastCGI), but the one we’re using is pretty straight forward and uses the great mod_wsgi module from Apache.

Open up the Apache ports.conf file and edit the following so we’re listening only on the local host.

    $ sudo vim /etc/apache2/ports.conf

    NameVirtualHost 127.0.0.1:80
    Listen 127.0.0.1:80

And restart Apache:

    $ sudo apache2ctl graceful

Assuming you had no enabled sites, Apache should have not complained about any ports being taken up. At this point, nginx is running on port 80 for clients (users of our websites) and Apache is running locally on the same port, but is only accessible through instances ran internally. Great!

We’re almost done. As you may remember, we sent requests to Apache but we need to actually do something with those requests. With this in mind, we create a virtual host within Apache with some information to create a WSGI instance. Here’s how the file should look like:

    $ sudo vim /etc/apache2/sites-available/mysite.com

    <VirtualHost 127.0.0.1:80>
    ServerName www.mysite.com
    ServerAlias mysite.com

    <Directory /home/admin/domains/mysite.com/>
        Order deny, allow
        Allow from all
    </Directory>

    LogLevel warn
    CustomLog /var/log/apache2/mysite.com.access.log combined
    ErrorLog /var/log/apache2/mysite.com.error.log

    WSGIDaemonProcess mysite.com user=www-data group=www-data threads=25
    WSGIProcessGroup mysite.com
    WSGIScriptAlias / /home/admin/domains/mysite.com/mysite.wsgi
    </VirtualHost>

This is a lot to deal with but if you’re familiar with Apache it’s pretty basic. We setup a VirtualHost to listen on our localhost, setup some logs, and then create the WSGI Daemon process we will need. The major thing to note here is the WSGIScriptAlias which says for all requests under /, use the mysite.wsgi file to handle them. We’ll get to creating that file in a second.

Enable the virtual host as we are now done here:

    $ sudo ln -s /etc/apache2/sites-available/mysite.com /etc/apache2/sites-enabled/mysite.com

And now, let’s create our mysite.wsgi file:

    $ vim /home/admin/domains/mysite.com/mysite.wsgi

    ALLDIRS = ['/usr/lib/python2.5/site-packages']
    # the above directory depends on the location of your python installation.
    # if using virtualenv, it will need to match your projects locale.
    import os
    import sys 
    import site

    prev_sys_path = list(sys.path)

    for directory in ALLDIRS:
        site.addsitedir(directory)
        new_sys_path = []
        for item in list(sys.path):
            if item not in prev_sys_path:
                new_sys_path.append(item)
                sys.path.remove(item)
                sys.path[:0] = new_sys_path

    # change this depending on your project.
    sys.path.append('/home/admin/domains/mysite.com')

    os.environ['PYTHON_EGG_CACHE'] = '/home/admin/.python-eggs'
    os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
    import django.core.handlers.wsgi
    application = django.core.handlers.wsgi.WSGIHandler()

This is a very basic WSGI implementation for a Django app but sets all the necessary environment variables and gets things going. Make sure to modify the paths to match your own projects.

    $ sudo apache2ctl graceful

And now, everything is essentially setup. To re-iterate, here’s what we did:

With everything running, you should notice a huge improvement in memory usage. My 256mb slice went from constantly paging and basically crashing at peak hours to rarely paging during peak hours. Apache is a great server and it has its uses but when you are dealing within a low memory environment, a server like nginx is a great way to take a good portion of the load off Apache.

I hope you enjoyed this guide and that everything made sense. If you are having trouble getting something working, feel free to contact me and I will be glad to help you out or fix any errors within this text.

Useful nginx resources:

Others Posts You May Enjoy

Thanks for reading. How about leaving a comment?

blog comments powered by Disqus
Fork me on GitHub