Django 1.8 + GIS (postgis) + PostgreSQL 9.4 + Digital Ocean + Gunicorn + Nginx + Ubuntu 14.04 & Virtualenv. Complete Deployment Guide step by step.

Step 0 : Create droplet with django application image.

Step 1 :

Connect to Digital Ocean server :

ssh root@<ip_address>
  

Step 2 :

As Ubuntu 14.04 comes with postgresql 9.3, remove it completely and install 9.4

Steps are as follows :

sudo apt-get --purge remove postgresql\*
sudo aptitude update
sudo aptitude dist-upgrade
echo "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main" | sudo tee /etc/apt/sources.list.d/pgdg.list
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo aptitude update
## install 9.4
sudo aptitude install --with-recommends postgresql-9.4 postgresql-contrib-9.4 postgresql-server-dev-9.4

  

Step 3 :

Install pip, git , supervisor, postgresql 9.4 etc .

Steps are as follows :

sudo apt-get install python-pip python-dev libpq-dev postgresql-contrib git supervisor
sudo apt-get install python-imaging libjpeg8 libjpeg62-dev libfreetype6 libfreetype6-dev
sudo apt-get install postgresql-9.4-postgis python-psycopg2 postgresql-9.4 postgresql-server-dev-9.4

  

Step 4 :

set up virual env and git repository .

Steps are as follows :

cd /home/django/
sudo pip install virtualenv
virtualenv <prj_name>env
source <prj_name>env/bin/activate
git clone <git repo url> <prj_name>
cd <prj_name>
pip install -r requirements.txt  ## make sure pip install gevent
ln -s /usr/include/freetype2 /usr/include/freetype # if you are using pillow or pip
  

Step 5 :

GEOS, GIS, PROJ, GDAL installation. If you are not intend to use any of this libs, skip this step.

Steps are as follows :

sudo apt-get install binutils libproj-dev gdal-bin
sudo apt-get install python-gdal
  

#make temp directory ourside project directory and download below contents and install. It may take more time .
## GEOS
wget http://download.osgeo.org/geos/geos-3.3.8.tar.bz2
tar xjf geos-3.3.8.tar.bz2
cd geos-3.3.8
./configure
make
sudo make install
sudo ldconfig
cd ..

## PROJ.4
wget http://download.osgeo.org/proj/proj-4.8.0.tar.gz
wget http://download.osgeo.org/proj/proj-datumgrid-1.5.tar.gz
tar xzf proj-4.8.0.tar.gz
cd proj-4.8.0/nad
tar xzf ../../proj-datumgrid-1.5.tar.gz
cd ..
./configure
make
sudo make install
sudo ldconfig
cd ..

## GDAL
wget http://download.osgeo.org/gdal/gdal-1.9.2.tar.gz
tar xzf gdal-1.9.2.tar.gz
cd gdal-1.9.2
./configure
make
sudo make install
sudo ldconfig
cd ..

## PostGIS 
sudo apt-get install libxml2-dev
wget http://download.osgeo.org/postgis/source/postgis-2.1.5.tar.gz
tar xzf postgis-2.1.5.tar.gz
cd postgis-2.1.5
./configure
make
sudo make install
sudo ldconfig
cd ..
###

  

Step 6 :

Allow remote postgres connection if you are going to use pgadmin like tools.

Steps are as follows :

vim /etc/postgresql/9.4/main/postgresql.conf

listen_addresses='localhost'

to

listen_addresses='*'

vim /etc/postgresql/9.4/main/pg_hba.conf

# TYPE DATABASE USER CIDR-ADDRESS  METHOD
host  all  all  0.0.0.0/0 md5

service postgresql restart
  

Step 7 :

Set up db and its postgresql password

Steps are as follows :

sudo su - postgres
psql
ALTER USER postgres with password <something_strong_password>;
CREATE DATABASE <db_name> WITH ENCODING='UTF8' ;
\connect <db_name>;
CREATE EXTENSION postgis;
CREATE EXTENSION postgis_topology;
CREATE EXTENSION postgis_tiger_geocoder;
\q
exit
  

Step 8 :

Migrate Django Models and final changes to settings.py file

Steps are as follows :

# change settings.py to prod db user password
# push all changes to live and pull @ live
python manage.py migrate auth
python manage.py migrate
python manage.py createsuperuser 
python manage.py collectstatic
  

Step 9 :

Nginx , gunicorn, & supervisor changes

Steps are as follows :

## if you have supervisor script for example cronjob then put its short link.
ln -s /home/django/<prj_dir_name>/jobs_supervisor.conf /etc/supervisor/conf.d/jobs_supervisor.conf

# do path changes in /etc/nginx/sites-enabled/django
# do path changes in /etc/init/gunicorn.conf

############## content of gunicorn.conf###############
description "Gunicorn daemon for Django project"

start on (local-filesystems and net-device-up IFACE=eth0)
stop on runlevel [!12345]

# If the process quits unexpectadly trigger a respawn
respawn

setuid django
setgid django
chdir /home/django

exec /home/django/<prj_name>env/bin/gunicorn \
    --name=<prj_name> \
    --pythonpath=<prj_name> \
    --bind=127.0.0.1:9000 \
    --config /etc/gunicorn.d/gunicorn.py \
    <prj_name>.wsgi:application
  

Step 10 :

Above steps has deployed your django project. Note down below quick steps for quick deployment when you make code change to your project.

Steps are as follows :

ssh root@<ip_address>
source /home/django/<prj_name>env/bin/activate
cd /home/django/<prj_name>
git pull origin master
python manage.py collectstatic
python manage.py migrate
cd ..
chown -R django:django <prj_name>
sudo service gunicorn restart
sudo service nginx restart
sudo supervisorctl update
/etc/init.d/supervisor force-reload
  

Docker based Python Project Deployment with NginX, Supervisor, uwsgi on Ubuntu

For Docker installation refer : https://docs.docker.com/installation/ubuntulinux/

Assume your python code is inside /var/app/project/ and server run file is start.py

Note : Create folder/path if don't exists. for example /var/app/run/ which don't exits then create it. similarly for other paths.

Step 1 :

Create one custom supervisord.conf file inside /var/app/supervisor/

Content of file should be :

[unix_http_server]
file=/var/app/run/supervisor.sock
[supervisord]
logfile=/var/app/log/supervisord.log
loglevel=info
pidfile=/var/app/run/supervisord.pid
nodaemon=true
childlogdir=/lib/app/log/supervisor
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/app/run/supervisor.sock
[include]
files = /var/app/supervisor/conf.d/*.conf

Step 2 :

Create "superisor.docker.conf" file inside /var/app/supervisor/conf.d/

Content of file should be :

[program:pyenv]
command = /usr/bin/docker run --rm --name pyenv -v /var/app:/var/app felixonmars/archlinux supervisord -c /var/app/supervisor/supervisord.conf

Step 3 :

make link of file /var/app/supervisor/conf.d/superisor.docker.conf in /etc/supervisor/conf.d/ using :

sudo ln -s /var/app/supervisor/conf.d/superisor.docker.conf /etc/supervisor/conf.d/

Step 4 :

Now onwards , your all supervisor .conf files will be inside /var/app/supervisor/conf.d/

Create project_supervisor_docker.conf file inside /var/app/supervisor/conf.d/ and its content should be :

[program:project]
command = /usr/bin/uwsgi --plugin=python2
    --limit-as=512 --processes=2 --max-request=2000
    --memory-report --enable-threads
    --socket=/var/app/run/uwsgi/project.socket
    --stats=/var/app/run/uwsgi/project.stats
    --logto=/var/app/log/uwsgi/project.log
    --pidfile=/var/app/run/uwsgi/project.pid
    --master --no-orphans --logdate --chmod-socket=660
    --uid=33 --gid=33 --need-app
    --wsgi-file=/var/app/project/start.py
stopsignal=INT

Step 5 :

Create project_master_supervisor_docker.conf inside /var/app/supervisor/conf.d/ and its content :

[program:project_master]
command = /usr/bin/python2 start.py
directory=/var/app/project

Step 6 :

Create Nginx Configuration file for project. Its content should be :

 

upstream push_daemon_project {
    server 127.0.0.1:8284; #Assume push socket on port 8284
}

server {
    listen 80;
    listen [::]:80;

    server_name project.com www.project.com;

    location = /nginx_stub_status {
        stub_status on;
        allow 127.0.0.1;
        deny all;
    }

    location / {
        return 301 https://project.com$request_uri;
    }
}

server {
    listen 443 ssl spdy;
    listen [::]:443 ssl spdy;

    ssl_certificate /var/app/cert/project.com.crt; # certificate ssl
    ssl_certificate_key /var/app/cert/project.com.key; # certificate key ssl

    server_name project.com www.project.com;
    access_log /var/log/nginx/project.access.log;
    error_log /var/log/nginx/project.error.log;

    root /var/app/project/web;  ## template (html) directory

    location /static {
        alias /var/app/project/static;  ## static files
    }

    location = /nginx_stub_status {
        stub_status on;
        allow 127.0.0.1;
        deny all;
    }

    location /websocket {
        proxy_pass http://push_daemon_project;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    location / {
        include uwsgi_params;
        uwsgi_pass unix:///var/app/run/uwsgi/project.socket;
    }
}

Step 7 :

Restart all instances :

sudo chown -R www-data:www-data /var/app/project
sudo service nginx restart
sudo supervisorctl update
sudo /etc/init.d/supervisor force-reload
sudo docker ps -a # you can see docker process id : for ex : 6a2dfdeb0000
sudo docker restart 6a2dfdeb0000

Grunt , bower, nodejs installation & commands

 

sudo apt-get install --reinstall --install-recommends nodejs
sudo npm install -g grunt
sudo npm install -g grunt-cli
sudo npm install -g bower

# Locate package.json and run : 
 sudo npm install
 
# Locate bower.json and run :
 bower install

# Locate .Gruntfile.js and run : 
 "grunt dev --force" in local
 "grunt build --force" in live

* Commands * Git Submodule , merge , git rm reset / undo

  
git clone <SSH-URL> <directory_name>
git submodule init
git submodule update

# merge branch in master
git checkout master
git merge <branch_name>

# Undo "git rm -r ." command.
git reset HEAD

Mongo Useful Commands and steps for Dump , Restore from live with ssh access ( without FTP )

        
# Run in live server 
sudo mongodump --host <host_url> --port 27017 --db <db_name> 
sudo tar -zcvf <compress_file_name_of_dump>.tar.gz dump
# Note down path like
/var/app/dump/<compress_file_name_of_dump>.tar.gz
scp <server_user>@<server_ip_address>:/var/app/dump/<compress_file_name_of_dump>.tar.gz <local_absolute_path_where_dump_will_download>
mongorestore --host localhost --db <db_name> --port 27017 <downloaded_dump_folder_absolute_path>

If you are not able to connect mongo server ...

 
sudo service mongodb stop
sudo rm /var/lib/mongodb/mongod.lock
sudo -u mongodb mongod -f /etc/mongodb.conf --repair
sudo service mongodb start