Thursday, July 2, 2015

Django 1.8 + GIS (postgis) + PostgreSQL 9.4 + Digital Ocean + Gunicorn + Nginx + Ubuntu 14.04 & Virtualenv. Complete Deployment Guide step by step.



Step 0 : Create droplet with django application image.

Step 1 :

Connect to Digital Ocean server :

ssh root@<ip_address>
  

Step 2 :

As Ubuntu 14.04 comes with postgresql 9.3, remove it completely and install 9.4

Steps are as follows :


sudo apt-get --purge remove postgresql\*
sudo aptitude update
sudo aptitude dist-upgrade
echo "deb http://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main" | sudo tee /etc/apt/sources.list.d/pgdg.list
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo aptitude update
## install 9.4
sudo aptitude install --with-recommends postgresql-9.4 postgresql-contrib-9.4 postgresql-server-dev-9.4

  

Step 3 :

Install pip, git , supervisor, postgresql 9.4 etc .

Steps are as follows :


sudo apt-get install python-pip python-dev libpq-dev postgresql-contrib git supervisor
sudo apt-get install python-imaging libjpeg8 libjpeg62-dev libfreetype6 libfreetype6-dev
sudo apt-get install postgresql-9.4-postgis python-psycopg2 postgresql-9.4 postgresql-server-dev-9.4

  

Step 4 :

set up virual env and git repository .

Steps are as follows :

cd /home/django/
sudo pip install virtualenv
virtualenv <prj_name>env
source <prj_name>env/bin/activate
git clone <git repo url> <prj_name>
cd <prj_name>
pip install -r requirements.txt  ## make sure pip install gevent
ln -s /usr/include/freetype2 /usr/include/freetype # if you are using pillow or pip
  

Step 5 :

GEOS, GIS, PROJ, GDAL installation. If you are not intend to use any of this libs, skip this step.

Steps are as follows :

sudo apt-get install binutils libproj-dev gdal-bin
sudo apt-get install python-gdal
  
#make temp directory ourside project directory and download below contents and install. It may take more time .
## GEOS
wget http://download.osgeo.org/geos/geos-3.3.8.tar.bz2
tar xjf geos-3.3.8.tar.bz2
cd geos-3.3.8
./configure
make
sudo make install
sudo ldconfig
cd ..

## PROJ.4
wget http://download.osgeo.org/proj/proj-4.8.0.tar.gz
wget http://download.osgeo.org/proj/proj-datumgrid-1.5.tar.gz
tar xzf proj-4.8.0.tar.gz
cd proj-4.8.0/nad
tar xzf ../../proj-datumgrid-1.5.tar.gz
cd ..
./configure
make
sudo make install
sudo ldconfig
cd ..

## GDAL
wget http://download.osgeo.org/gdal/gdal-1.9.2.tar.gz
tar xzf gdal-1.9.2.tar.gz
cd gdal-1.9.2
./configure
make
sudo make install
sudo ldconfig
cd ..

## PostGIS 
sudo apt-get install libxml2-dev
wget http://download.osgeo.org/postgis/source/postgis-2.1.5.tar.gz
tar xzf postgis-2.1.5.tar.gz
cd postgis-2.1.5
./configure
make
sudo make install
sudo ldconfig
cd ..
###

  

Step 6 :

Allow remote postgres connection if you are going to use pgadmin like tools.

Steps are as follows :

vim /etc/postgresql/9.4/main/postgresql.conf

listen_addresses='localhost'

to

listen_addresses='*'

vim /etc/postgresql/9.4/main/pg_hba.conf

# TYPE DATABASE USER CIDR-ADDRESS  METHOD
host  all  all  0.0.0.0/0 md5

service postgresql restart
  

Step 7 :

Set up db and its postgresql password

Steps are as follows :

sudo su - postgres
psql
ALTER USER postgres with password <something_strong_password>;
CREATE DATABASE <db_name> WITH ENCODING='UTF8' ;
\connect <db_name>;
CREATE EXTENSION postgis;
CREATE EXTENSION postgis_topology;
CREATE EXTENSION postgis_tiger_geocoder;
\q
exit
  

Step 8 :

Migrate Django Models and final changes to settings.py file

Steps are as follows :

# change settings.py to prod db user password
# push all changes to live and pull @ live
python manage.py migrate auth
python manage.py migrate
python manage.py createsuperuser 
python manage.py collectstatic
  

Step 9 :

Nginx , gunicorn, & supervisor changes

Steps are as follows :

## if you have supervisor script for example cronjob then put its short link.
ln -s /home/django/<prj_dir_name>/jobs_supervisor.conf /etc/supervisor/conf.d/jobs_supervisor.conf

# do path changes in /etc/nginx/sites-enabled/django
# do path changes in /etc/init/gunicorn.conf

############## content of gunicorn.conf###############
description "Gunicorn daemon for Django project"

start on (local-filesystems and net-device-up IFACE=eth0)
stop on runlevel [!12345]

# If the process quits unexpectadly trigger a respawn
respawn

setuid django
setgid django
chdir /home/django

exec /home/django/<prj_name>env/bin/gunicorn \
    --name=<prj_name> \
    --pythonpath=<prj_name> \
    --bind=127.0.0.1:9000 \
    --config /etc/gunicorn.d/gunicorn.py \
    <prj_name>.wsgi:application
  

Step 10 :

Above steps has deployed your django project. Note down below quick steps for quick deployment when you make code change to your project.

Steps are as follows :

ssh root@<ip_address>
source /home/django/<prj_name>env/bin/activate
cd /home/django/<prj_name>
git pull origin master
python manage.py collectstatic
python manage.py migrate
cd ..
chown -R django:django <prj_name>
sudo service gunicorn restart
sudo service nginx restart
sudo supervisorctl update
/etc/init.d/supervisor force-reload
  

Monday, June 15, 2015

Docker based Python Project Deployment with NginX, Supervisor, uwsgi on Ubuntu

For Docker installation refer : https://docs.docker.com/installation/ubuntulinux/

Assume your python code is inside /var/app/project/ and server run file is start.py



Note : Create folder/path if don't exists. for example /var/app/run/ which don't exits then create it. similarly for other paths.

Step 1 :

Create one custom supervisord.conf file inside /var/app/supervisor/

Content of file should be :


[unix_http_server]
file=/var/app/run/supervisor.sock
[supervisord]
logfile=/var/app/log/supervisord.log
loglevel=info
pidfile=/var/app/run/supervisord.pid
nodaemon=true
childlogdir=/lib/app/log/supervisor
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/app/run/supervisor.sock
[include]
files = /var/app/supervisor/conf.d/*.conf

Step 2 :

Create "superisor.docker.conf" file inside /var/app/supervisor/conf.d/

Content of file should be :


[program:pyenv]
command = /usr/bin/docker run --rm --name pyenv -v /var/app:/var/app felixonmars/archlinux supervisord -c /var/app/supervisor/supervisord.conf

Step 3 :

make link of file /var/app/supervisor/conf.d/superisor.docker.conf in /etc/supervisor/conf.d/ using :


sudo ln -s /var/app/supervisor/conf.d/superisor.docker.conf /etc/supervisor/conf.d/

Step 4 :

Now onwards , your all supervisor .conf files will be inside /var/app/supervisor/conf.d/

Create project_supervisor_docker.conf file inside /var/app/supervisor/conf.d/ and its content should be :


[program:project]
command = /usr/bin/uwsgi --plugin=python2
    --limit-as=512 --processes=2 --max-request=2000
    --memory-report --enable-threads
    --socket=/var/app/run/uwsgi/project.socket
    --stats=/var/app/run/uwsgi/project.stats
    --logto=/var/app/log/uwsgi/project.log
    --pidfile=/var/app/run/uwsgi/project.pid
    --master --no-orphans --logdate --chmod-socket=660
    --uid=33 --gid=33 --need-app
    --wsgi-file=/var/app/project/start.py
stopsignal=INT

Step 5 :

Create project_master_supervisor_docker.conf inside /var/app/supervisor/conf.d/ and its content :


[program:project_master]
command = /usr/bin/python2 start.py
directory=/var/app/project

Step 6 :

Create Nginx Configuration file for project. Its content should be :

 

upstream push_daemon_project {
    server 127.0.0.1:8284; #Assume push socket on port 8284
}

server {
    listen 80;
    listen [::]:80;

    server_name project.com www.project.com;

    location = /nginx_stub_status {
        stub_status on;
        allow 127.0.0.1;
        deny all;
    }

    location / {
        return 301 https://project.com$request_uri;
    }
}

server {
    listen 443 ssl spdy;
    listen [::]:443 ssl spdy;

    ssl_certificate /var/app/cert/project.com.crt; # certificate ssl
    ssl_certificate_key /var/app/cert/project.com.key; # certificate key ssl

    server_name project.com www.project.com;
    access_log /var/log/nginx/project.access.log;
    error_log /var/log/nginx/project.error.log;

    root /var/app/project/web;  ## template (html) directory

    location /static {
        alias /var/app/project/static;  ## static files
    }

    location = /nginx_stub_status {
        stub_status on;
        allow 127.0.0.1;
        deny all;
    }

    location /websocket {
        proxy_pass http://push_daemon_project;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    location / {
        include uwsgi_params;
        uwsgi_pass unix:///var/app/run/uwsgi/project.socket;
    }
}

Step 7 :

Restart all instances :


sudo chown -R www-data:www-data /var/app/project
sudo service nginx restart
sudo supervisorctl update
sudo /etc/init.d/supervisor force-reload
sudo docker ps -a # you can see docker process id : for ex : 6a2dfdeb0000
sudo docker restart 6a2dfdeb0000


Grunt , bower, nodejs installation & commands

 

sudo apt-get install --reinstall --install-recommends nodejs
sudo npm install -g grunt
sudo npm install -g grunt-cli
sudo npm install -g bower

# Locate package.json and run : 
 sudo npm install
 
# Locate bower.json and run :
 bower install

# Locate .Gruntfile.js and run : 
 "grunt dev --force" in local
 "grunt build --force" in live


* Commands * Git Submodule , merge , git rm reset / undo

  
git clone <SSH-URL> <directory_name>
git submodule init
git submodule update

# merge branch in master
git checkout master
git merge <branch_name>

# Undo "git rm -r ." command.
git reset HEAD

Mongo Useful Commands and steps for Dump , Restore from live with ssh access ( without FTP )

        
# Run in live server 
sudo mongodump --host <host_url> --port 27017 --db <db_name> 
sudo tar -zcvf <compress_file_name_of_dump>.tar.gz dump
# Note down path like
/var/app/dump/<compress_file_name_of_dump>.tar.gz
scp <server_user>@<server_ip_address>:/var/app/dump/<compress_file_name_of_dump>.tar.gz <local_absolute_path_where_dump_will_download>
mongorestore --host localhost --db <db_name> --port 27017 <downloaded_dump_folder_absolute_path>
If you are not able to connect mongo server ...
 
sudo service mongodb stop
sudo rm /var/lib/mongodb/mongod.lock
sudo -u mongodb mongod -f /etc/mongodb.conf --repair
sudo service mongodb start
      

Sunday, June 10, 2012

Django Site in Production with Lighttpd & FastCGI

OS : Ubuntu

Python : 2.7.1

1) Install lighttpd
sudo apt-get, install lighttpd
2) Install django
I want to download it to / var / www installation,
$ Cd / var / www
$ Sudo wget http://media.djangoproject.com/releases/1.3/Django-1.3.1.tar.gz
$ Sudo tar xzvf Django-1.3.1.tar.gz
$ Cd the Django-1.3.1
$ Sudo python setup.py install
3) Installed back to / www
cd / var / www
4) Then we have to check the django whether the installation is successful, we first build a new project with django-admin.py
the sudo django-admin.py startproject the portal
the cd the portal
sudo the chmod + the x the manage.py
sudo python manage.py runserver
Displays the following screen, django has a normal installation
Validating the models ...

0 errors found
Django version 1.3.1, using settings 'portal.settings'
Development server is running at http://127.0.0.1:8000/ with
Quit the server with CONTROL-C.
5) install flup
This is the WSGI Server, before installation, you must first determine the version of the python in the command line input
$ / Var / www / portal $ python
Python 2.7.1 + (r271: 86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2 to
Ubuntu 11.04 Built-in 2.7.1 version? or because I just have the update?? XD So we had to select the development flup-1.0.3 version
$ Cd / var / www
$ Sudo wget http://pypi.python.org/packages/2.7/f/flup/flup-1.0.3.dev_20110405-py2.7.egg # md5 = 
   1fa9a03ade17f88990340884a1113b5a
$ Sudo wget http://peak.telecommunity.com/dist/ez_setup.py
$ Sudo python ez_setup.py flup-1.0.3.dev_20110405-py2.7.egg
6)by adding mysite.fcgi
This code is used to tell the server how to deal with the fastcgi program
#! / Usr / bin / python
import sys, os
# Add a custom Python path.
sys.path.insert (0, / var / www ")
# Switch to the directory of your project. (Optional.)
# Os.chdir ("/ home / user / myproject")
# Set the the DJANGO_SETTINGS_MODULE environment variable.
os.environ ['DJANGO_SETTINGS_MODULE'] = "portal.settings"
from django.core.servers.fastcgi import runfastcgi
runfastcgi (["method = threaded", "daemonize = false"])
Above the red part, be especially careful if you look at the django official website files are written is not the same as the official website of
the paper was wrong, or is the older version of the file XD. I later installed a long time, only to find that there are wrong installation should pay
special attention to Oh! Storage of the above file, enter
$ For python mysite.fcgi
There will be a long list of things. Find what you see

It worked!

This script file is correct.
7) edit the lighttpd config
Start the fastcgi mode
$ Sudo lighttpd-enable-mod Fastcgi
Then here to note in the / etc / lighttpd / you will see to lighttpd.conf this profile do not move, which is the default. lighttpd read this default will
go to read the folder conf-enabled profile without charge. The start fastcgi mode, so read the default
lighttpd would automatically go read the conf-enabled inside the profile. Set fastcgi
$ Vim / etc/lighttpd/conf-enabled/10-fastcgi.conf server.modules + = ("mod_fastcgi",
"Mod_accesslog
)

server.modules + = ("mod_rewrite",
Mod_fastcgi under "
"Mod_accesslog
)
$ SERVER ["socket"] == "172.16.33.10:8000" {
server.document-root = "/ var / www / portal"
the fastcgi.server = (
Fcgi "=> (
"Localhost" => (
The bin-path = "/ var / www / portal / mysite.fcgi
"Socket" => "/ var / www / portal / mysite.sock",
"Check-local" => "the disable"
)
),
)

alias.url = (
"/ Media" => "/ usr/local/lib/python2.7/dist-packages/django/contrib/admin/media /",
)

url.rewrite-once = (
"^ (/ Media. *) $" => "$ 1"
^ / The favicon.ico $ "=>" / media / the favicon.ico
"^ (/ *) $" => "/ Mysite.fcgi $ 1"
)
}
8)Establish runfastcgi the script
This script is mainly through the manage.py the command the specified runfcgi parameter operation, this script will be related to things in
order to facilitate the start Fastcgi the server into a script.
    #! / Bin / bash
    # Replace these three settings.
    PROJDIR = "/ var / www / portal"
    PIDFILE = "$ PROJDIR / mysite.pid"
    SOCKET = "$ PROJDIR / mysite.sock"


    cd $ PROJDIR
    if [-f $ PIDFILE]; then
    kill `cat - $ PIDFILE`
    rm-f - $ PIDFILE
    fi


    the exec / usr / bin / env - \
    PYTHONPATH is = ".. / Python: .." \
    . / Manage.py runfcgi socket = $ SOCKET pidfile = $ PIDFILE 
9)To remember to to modify runfastcgi execute permissions.
$ Sudo chmod + x runfastcgi
10)Restart lighttpd
$ / Etc / init.d / the lighttpd
11 ) implementation of runfastcgi
$. / Runfastcgi
12 )enter the URL in the browser http:// [yourIP]: 8000
You will able to see start page of your django site.
13 )File permissions
drwxr--xr-x 2 www-data root 4096 2011-09-22 19:42 the portal

portal folder inside the file are as follows:
- Rw-r - r - 1 www-data root 0 2011-09-22 19:16 __ init__.py
-Rw-r - r - 1 www-data root 124 2011-09-22 19:17 __ init__.pyc
-Rwxrwxrwx 1 www-data root 503 2011-09-22 19:16 the manage.py
-Rwxrwxrwx 1 www-data root 401 2011-09-22 7:22 p.m. mysite.fcgi
-Rwxrwxrwx 1 www-data root 334 2011-09-22 19:31 runfastcgi
-Rw-r - r - 1 www-data root 5031 2011-09-22 19:16 your settings.py
-Rw-r - r - 1 www-data root 2652 2011-09-22 19:17 settings.pyc
-Rw-r - r - 1 www-data root 565 2011-09-22 19:16 The urls.py
-Rw-r - r - 1 www-data www-data 261 2011-09-22 19:42 urls.pyc
13 )use the django when the Admin can not find staic resource (CSS, image, js )?
Modify the lighttpd config. alias.url = ("/ static / admin /" => "/ usr/local/lib/python2.7/dist-packages/django/contrib/admin/media /")

[Reference]
http://blog.finalevil.com/2011/09/how-to-use-django-on-lighttpd-with.html

Sunday, June 3, 2012

Django 1.3 Language Translation

Translating Templates with Django

Below is steps for translating one Django websites into Norwegian.

Settings

As I'm only providing translation for Norwegian, I set the following in my settings.py file:
LANGUAGE_CODE = 'en-gb'

LANGUAGES = (
  ('nb', 'Norwegian Bokmal'),
  ('nn', 'Norwegian Nynorsk'),
  ('en-gb', 'English'),
)
This defaults to English, but limits the list of languages on offer. I also added LocaleMiddleware to my MIDDLEWARE_CLASSES:
MIDDLEWARE_CLASSES = (
    'django.middleware.csrf.CsrfViewMiddleware',                      
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.locale.LocaleMiddleware',
   ...
)

Mark Up Template

Open up the template to translate, and add the trans tag around the text you want translating. So,
<h2>Part 9 - Three Toms and Rolls</h2>
<p>This section contains the first exercises containing snare drum rolls.</p>
<h2>Part 10 - Ride Cymbal and Cut Common</h2>
<p>This part introduces the ride cymbal to the drum kit</p>
becomes
<h2>{% trans "Part" %} 9 - {% trans "Three Toms and Rolls" %}</h2>
<p>{% trans "This section contains the first exercises containing snare drum rolls." %}</p>
<h2>{% trans "Part" %} 10 - {% trans "Ride Cymbal and Cut Common" %}</h2>
<p>{% trans "This part introduces the ride cymbal to the drum kit" %}</p>
Add {% load i18n %} at the top of the template too.

Run make-messages.py

Once this is done, you need to run a script which will create message files. You need to run dango-admin.py makemessages in the same directory as your settings file, after creating a locale directory
me@coderpriyu:~/web/progperc/site$ export PYTHONPATH=~/web/site
me@coderpriyu:~/web/progperc/site$ ~/web/progperc/django/bin/django-admin.py makemessages -l nb
Error: This script should be run from the Django SVN tree or your project or app tree. If you did indeed 
run it from the SVN checkout or your project or application, maybe you are just missing the conf/locale 
(in the django tree) or locale (for project and application) directory? It is not created automatically, you 
have to create it by hand if you want to enable i18n for your project or application.
This error means that we need to create a locale directory in the directory that contains settings.py:
me@coderpriyu:~/web/progperc/site$ mkdir locale
me@coderpriyu:~/web/progperc/site$ ~/web/progperc/django/bin/django-admin.py makemessages -l nb
processing language nb
Error: errors happened while running xgettext on coverage.py
/bin/sh: xgettext: not found
This error occurs because we don't have the xgettext command installed. I'm on Ubuntu 11.04, so sudo apt-get install gettext should do the trick.

Success!:
me@coderpriyu:~/web/progperc/site$ ~/web/progperc/django/bin/django-admin.py makemessages -l nb
processing language nb
I now have a file, in ~/web/site/locale/nb/LC_MESSAGES called django.po, with contents similar to the following:
# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2011-06-17 11:10+0100\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"Language: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"

#: volume/templates/volume/Volume2.html:8
#: volume/templates/volume/Volume2.html:18
msgid "Part"
msgstr ""

#: volume/templates/volume/Volume2.html:8
msgid "Three Toms and Rolls"
msgstr ""

#: volume/templates/volume/Volume2.html:9
msgid ""
"This section contains the first exercises containing snare drum rolls.  "
msgstr ""

#: volume/templates/volume/Volume2.html:18
msgid "Ride Cymbal and Cut Common"
msgstr ""

#: volume/templates/volume/Volume2.html:19
msgid ""
"This part introduces the ride cymbal to the drum kit."
msgstr ""

Add translation

Add your translation into the msgstr section (Warning! Google translate Norwegian ahead!)
#: volume/templates/volume/Volume2.html:8
msgid "Three Toms and Rolls"
msgstr "Tre Toms og Rolls"

#: volume/templates/volume/Volume2.html:9
msgid ""
"This section contains the first exercises containing snare drum rolls.  "
msgstr "Denne delen inneholder de første øvelsene inneholder skarptromme ruller."

#: volume/templates/volume/Volume2.html:18
msgid "Ride Cymbal and Cut Common"
msgstr "Ride Cymbal og Cut Common"

#: volume/templates/volume/Volume2.html:19
msgid ""
"This part introduces the ride cymbal to the drum kit."
msgstr "Denne delen introduserer ride cymbal til trommesett."

Compile Language File

Once we've put our translations in, we need to compile the .po file into a .mo file.
me@coderpriyu:~/web/progperc/site$ ~/web/progperc/django/bin/django-admin.py compilemessages
processing file django.po in /home/coderpriyu/web/site/locale/nb/LC_MESSAGES
You should now be able to run up your app, change your locale in the browser, and see the translated text.

Tuesday, April 17, 2012

Django Initial data dumping and loading

"manage.py dumpdata" for dumping data to json file.

$./ manage.py dumpdata app.myModel1 app.myModel2 <relative_path_to_app>/fixtures/initial_data.json
   --indent 4

"manage.py loaddata" for loading data from json file.

$ ./manage.py  loaddata <relative_path_to_app>/fixtures/initial_data.json

Wednesday, April 4, 2012

SVN vs GIT

In the world of programming and engineering, it is necessary to keep track of changes during the application development process. This allows developers and programmers revert unwanted changes in applications under the development process. 

Version Control Systems (VCS) are stand-alone web 2.0 applications that have made the job easier by providing systemized management of multiple revisions of the same unit of information, whether it is a simple document or a digital document, such as source codes of applications or blueprints of electronic models. VCS allows developers to work collaboratively on the same application from different locations using repositories.

The two most famous types of version control systems are:
1-     Centralized version control system - subversion (SVN)
2-     Distributed or decentralized version control system - Git

In the case of centralized version control systems, there is a single central repository and all of the changes that are made to the documents are saved in that repository. There is a client server approach in the case of CVCS, where a single repository is stored on the server that clients can also sync up to.

In case of distributed or decentralized VCS, there is a peer-to-peer approach that clients can synchronize with by exchanging patches from peer to peer. Clients can make changes in the repositories and those changes will be local to them unless they synchronize with someone else. Depending on the requirements, Git also offers a centralized repository. In other terms, each peer has a bona-fide repository which is the working copy of codebase. DVCS provides independent as well as canonical repository.

SVN vs. Git
Both VCS and DVCS are highly appreciated by their users, but I have found that DVCS has some advantages over VCS. Let’s look at the concept of VCS and DVCS in detail by comparing the available systems of both types – SVN and Git.

  • With Git, clients can commit changes to their localized repositories as new revisions while being offline. However, SVN does not provide this facility as user must be online in order to push to the repository from the working copy.
  • SVN help is more organized and to the point while Git provides more than what is actually required. There is some time wasted since it is difficult to get a quick reference from Git’s search.
  • Git’s complete copy of the data is stored locally in the client’s system so it is extremely fast when compared to SVN. With Git there is no time wasted when waiting for network response time, but with SVN it takes longer because all of the data is stored in a centralized repository.
  • There is a smaller chance of data being lost in Git because data copies are stored locally in clients systems. The number of backups available is the same as the number of users on any repository. With SVN, if there is a data loss in the central repository, it will be gone forever.
  • The Git repository has an efficient memory because the data’s file format is compressed, which is not the case with SVN. Also, there are always two copies of a file in the working directory of SVN. One copy is used for storing the actual work while the other copy contains the information used to aid operations (status and commit). Git has a small index file to store the info related to a particular file. When there are a lot of documents, there is a huge impact on disk space in the SVN compared with Git.
  • Git DVCS is based on the concept of branching. The working directory of a developer is itself a branch. In Git, we can easily view the working directories of developers while they are modifying two or more unrelated files at the same time as different branches stemming from the same common base revision of the project. With SVN, there is almost no concept of branching
  • In Git a large number of users can commit or push data to the same repository. If someone wants to push work in a Git repository, then there is no need to worry about data lost or immediate merging of others changes because commits are not sequential in Git like SVN.
  • Git allows its users to have control over the merging of data in synchronized repositories. Merges are always pulled by someone and nobody can push to commit merges in someone else’s repository. The facility to merge data is also there in SVN, but it is somewhat incomplete. SVN merge records seem to miss some of the important details that Git keeps track of.
  • Git keeps track of contents while SVN keeps record of files. Because Git keeps track of contents, whenever there is even a small change in content it tracks it as a separate change. Because of this, the history of a single file in Git is split.
  • Git will not allow you to checkout a subdirectory. Instead, the user will have to checkout the whole repository.  In SVN, checkouts at subdirectory level are possible.

By looking at an overview of the features of Git and SVN, we can see that Git is preferable in most circumstances. Developers, researchers, engineers and other users of VCS have more inclination towards Git, so I predict that Git will be the future of revision control.

References:
“Git's Major Features over Subversion” retrieved Dec 15, 2008 from “http://git.or.cz/gitwiki/GitSvnComparsion


Tuesday, April 3, 2012

Virtual Environment for Django and python with version specified.

Python2.6 is the default Python version on Ubuntu 10.04. Now you may still want to run your Django websites with Python2.5. A nice way to do this is by creating virtual environments to handle $PYTHONPATH and avoid conflicts among different versions. Such a tool already exists: Virtualenvwrapper.

First, to install Virtualenvwrapper, we will need easy_install:
$ sudo apt-get install python-setuptools
$ sudo easy_install virtualenv
$ sudo easy_install virtualenvwrapper
$ mkdir ~/.virtualenvs
We just need to add the following lines to .bashrc.
export WORKON_HOME=$HOME/.virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
And run it.
$ source .bashrc
We then create a virtual environment, here named django with Python2.5. Python2.6 is the default version on Ubuntu 10.04.

To create virtual environment 
    here -p specifies the package
$ mkvirtualenv -p python2.5 django
To use virtual environment named "django"
  bracket contains the current virtual environment
$ workon django
To close virtual environment named "django"
$ (django)...:$ deactivate
We are ready to go with Python 2.5. workon sets the current virtual environment and deactivate exits from it.

We can now add some additional Python paths on our virtual environment, for example:
$ workon django
$ (django)...:$ add2virtualenv /home/user/additional_python_path
$ (django)...:$ add2virtualenv
Usage: add2virtualenv dir [dir ...]
Existing paths:
home/user/additional_python_path
$ (django)...:$
Finally here are some other useful commands:
$ mkvirtualenv django2
$ rmvirtualenv django2
$ workon django
$ (django)...:$ deactivate
$ workon
django
Now The most important thing is "How to use virtualenv in
production ???"

For that you will need to add it's path to your PYTHONPATH.
For example if your env had for path "/home/www/my_project/env/", the path to add would be:
/home/www/env/lib/python2.7/site-packages/
You can set this up in many different ways, but if your are generating your fcgi or uwsgi interface through manage.py, simply add the following at the very top of your manage.py (before the rest):
import os
my_virtualenv_path
= "/home/www/my_project/env/lib/python2.7/site-packages/"
# Add it to your PYTHONPATH
os
.path.append(my_virtualenv_path)
You can adopt this to whatever your setup is, just in case you could also do the following in the shell :
export PYTHONPATH:$PYTHONPATH:/home/www/my_project/env/lib/python2.7/site-packages/
You will also need to add the directory of where your settings.py file is located to the PYTHONPATH, so django will be able to discover it, just proceed in a similar manner to do so.

Tuesday, December 20, 2011

Scribd documents on Your page ( API ) - Django / python


In Django to display your Scribd documents in your page , first you need to install scribd python package . To install it refer this link

after installing it in your views write this def (add url accordingly).
def scribd(request):
    extra_context = {}
    extra_context['formName'] = 'scribd'   
    data = []
    import scribd       
    scribd.api_key = '#Your api key'
    scribd.api_secret = '#Your secret key'
    api_user = scribd.api_user   
    all_documents = api_user.all()
    for ad in all_documents:       
        temp = []
        document = ad
        document.load()   
        attrsDict = document.get_attributes()               
        temp.append(document.get_scribd_url())
        temp.append(attrsDict['thumbnail_url'])
        temp.append(document.get_download_url())     
        data.append(temp)
       
    extra_context['data'] = data
    return render_to_response('scribd.html', extra_context,
 context_instance=RequestContext(request))

In your scribd.html file............

My Documents on scribd:
< table border='1' >  
   {% for d in data %}               
        < tr>   
            < td width='82px'><a href='{{d.0}}'
 style='color:blue;font-weight: normal;'>
Doc {{forloop.counter}}:</a></td>
            <td><img width='' src='{{d.1}}' 
alt='doc{{forloop.counter}}'/>
</td>
            <td><a href='{{d.2}}' style='color:blue;
font-weight: normal;'>
Download doc {{forloop.counter}}
</a></td>
        </tr>       
   {% endfor %}
   </table> 

Live Demo
http://coderpriyu.alwaysdata.net/scribd/


Thats it....................

Tuesday, December 6, 2011

Database Connection Pool Solution for Django + Mysql


The solution:

An alternative DATABASE_ENGINE for django that leverages core django mysql code with minimal overlap and sqlalchemy for connection pooling. Also, a pretty ‘DRY’ stub to mix in your own pooling if desired, or adapt to use postgres instead of mysql.

1. Install sqlalchemy library: http://www.sqlalchemy.org/download.html
I used “Latest 0.5? myself.
2. Grab: mysql_pool.tgz

3. Make mysql_pool reachable by your project
3a. Unpack in your python’s “site-packages” directory
3b. Or: unpack somewhere in your project directory, and edit the “base.py” file in mysql_pool to fix the import lines containing uw.udjango.db.engine to be the new location mysql_pool.

4. Edit your settings.py to change DATABASE_ENGINE
4a. If 3a, set to uw.udjango.db.engine.mysql_pool
4b. If 3b, set to yourproject.whatever.mysql_pool

5. Edit your settings.py to add these (required) tuning settings:
  DBPOOL_WAIT_TIMEOUT = 28800  # your mysql db’s server side inactive connection kill time
# discernable by ’show GLOBAL variables;’ in mysql, look for ‘wait_timeout’, changeable if desired
DBPOOL_SIZE = 20 # the maintained number of dbconnections, over this returned conns are destroyed
DBPOOL_MAX = 100 # the max allow connections, period
DBPOOL_INTERNAL_CONN_TIMEOUT = 10 # how long to wait for mysql to give you a connection

That should do it!

The explanation:

I’ve been reading up on the various was being proposed to make some kind of persistent connection re-use part of core django. The are a couple of options I like, especially having a core database engine choice of sqlalchemy. A lot of implications there though, and I didn’t want to tackle them all.

At first I figured I’d write my own, and stubbed out the code to do that. Since I’m using mysql, I copied it’s engine tree in django.db.backends into my project and began working on it. Quickly I realized I didn’t want to rewrite, or maintain duplicates of, the whole tree. I only cared about two calls really: self.connection = Database.connect() and self.connection.close(). Modifying those would be a way to plug in pooling of my own. Also, in the process I looked to SQLAlchemy’s pool code as an example of Python implementation, and it seemed pretty good. Eventually I’d like to add some more subtlety to the pool grow/shrink strategy.

I started riffing on the solution described on Ed Menendez’s site, and merging in some of my ideas to make it more DRY.

Basically, all the classes in mysql_pool use class naming tricks to extend-without-modification and assume the identity and characteristics of the core mysql engine. That means there is no code in mysql_pool to maintain other than the “base.py” file, and even that one is able to use the same short cut for most of its classes, the exception being DatabaseWrapper, which is updated to use SQLAlchemy’s pool.

Also, I added settings.py level control of SQLAlchemy’s QueuePool, which is the pool type mysql_pool is forced to use.

Other links of note on related topics:

Proposal: user-friendly API for multi-database support

Monday, December 5, 2011

Django: How to add Google +1 button to your website.



What is Google +1 button?

It's new Google social button. It's much similar to Facebook "Like" button. While available during one month it earned popularity that could compete with Facebook's social button. Corporation of Good knows what to do, so you probably want to have one on your website inline with Facebook's Like...

Add Google +1 button to your site:

- Open http://www.google.com/webmasters/+1/button/index.html and generate own button for your website.

Google proposes HTML code like this:

<html>
  <head>
    <title>+1 demo: Basic page</title>
    <link rel="canonical" href="http://www.example.com" />
    <script type="text/javascript" src="https://apis.google.com/js/plusone.js">
    </script>
  </head>
  <body>
    <g:plusone></g:plusone>
  </body>
</html>

It's quite simple, as you can see. And that's much pretty it. Oh no wait...

- Add some variable to your template like "google_target_url" like this:


#pass variable to template with google +1 target url
return render_to_response("template.html", 
                         {"google_target_url":google_target_url,})

Finally you can add property "href" (similar to common a href=" " pattern) to target +1 action to custom url. In general you can now generate page with multiple +1 buttons on it. For e.g.: one button under one blogpost or image...

<g:plusone href="http://www.example.com/custom_page1/" size="standard" count="false"></g:plusone>


Django - compressing CSS/JS files with django-compressor



There are 2 main usual tasks with web project's deployment and about .css and .js files.

First one - size minimization. But there are lot's of utilities helping you to compress CSS files. Delete unused spaces, comments and so on... Second one - Version control. For e. g. When you've updated the script on your deployment server, but user's browser uses old one until user manually hits 'Refresh'.

That's where Static files compressor comes in to mind.



Upon selecting among available one's found top "google" results:
- django-compress
- django-compressor
- webassets

Project uses 'django.contrib.staticfiles', so django-compress was not compatible... It does not support Django's static files gently. Webassets s a good lib. but project has a huge amount of different static. Maybe it's ok for e small project, but when you need to specify/change a 100's javascript in python module for certain templates... Nothing good comes in mind. And imho TEMPLATES... That's where thing's like those should live.

So Django-compressor was chosen using those criteria. And, to mention, project is actively developed, supported and documented. Anyway adding it to your django project is quite simple, as most good apps.

What it does, it turns this:

<script type="text/javascript" src="{{ MEDIA_URL }}public/js
/jquery-1.4.1.min.js"></script>
<script type="text/javascript" src="{{ MEDIA_URL }}banners/js/
jquery.cycle.all.js"></script>
<script type="text/javascript" src="{{ MEDIA_URL }}js/gb/greybox.js"
></script>
<script type="text/javascript">jQuery.browser.msie6 = 
jQuery.browser.msie &&  parseInt(jQuery.browser.version) 
== 6 &&  
!window["XMLHttpRequest"];
if (!jQuery.browser.msie6) {
 $(document).ready(function(){
   $("div#body_wrapper").wrap('<div id="body_wrapper_shadow_right"
><div id="body_wrapper_shadow_left">'+
'</div></div>');
 });
</script>

Into this:

<script type="text/javascript" src="/static/CACHE/js/8dd1a2872443.js" 
charset="utf-8"></script>

The install is quite simple and you need to follow this manual.
Then configure it. Basically you hava all set for it to function. Except for me having a bit wired way of storing static files. I had to remap default COMPRESS_ROOT and COMPRESS_URL variables. It's quite easily done in settings.py. And it has lot's of other settings, that I did not require.

Anyhow this is it. Now you can wrap all of your code with handy templatetags that will do all the work for you. it may look somehow like this:

{% load compress %}

{% compress css %}
<link rel="stylesheet" href="/static/css/one.css" type="text/css" 
charset="utf-8">
<style type="text/css">p { border:5px solid green;}</style>
<link rel="stylesheet" href="/static/css/two.css" type="text/css" 
charset="utf-8">
{% endcompress %}

or for .js files:

{% load compress %}

{% compress js %}
<script src="/static/js/one.js" type="text/javascript" charset="utf-8"
></script>
<script type="text/javascript" charset="utf-8">obj.value = "value"; 
</script>
{% endcompress %}

It will generate a custom .js/.css files with all of your compressed static at one file in your media/static dir that can be cashed. They will have links, like "/media/CACHE/css/105dsb963311.css"and put your newly generated script/style there. 

This is it. Use this in your projects. It economies your server's time/traffic and helps you to update scripts at all user's clients in their browsers when you deploy a new version.

Saturday, December 3, 2011

Django Tips & Features before getting Started



1. Dont hardcode MEDIA_ROOT and TEMPLATE_DIRS use 
        os.path.realpath(os.path.dirname)__file__)) 
hardcoding will cause problem while moving from dev->test->live

2. Dont hardcode static files in templates. use MEDIA_URL
EG: 
In future if you change your cdn, you will have to change it only in the settings
To get the context variable use django context processor. Its very easy to write and this makes MEDIA_URL and any other variable you like available across all the templates 

3. Use the url function to define the urls. Dont hardcode urls. Use reverse() in views.py and {% url %} tag in templates

4. Thirdparty app django-command-extension
This app is very useful. It gives some other useful commands  like
        -> shell_plus -  extension of shell. It will autoload all enabled django  models.
-> sqldiff - gives you the sql diff between the code and the DB
        -> show_urls - displays the url routes that are defined in the project
        -> graph_models - creates a GraphViz dot file 
        -> clean_pyc - removes all the pyc files    (Eclipse helois pydev makes it very easy...)
        -> reset_db - Resets a database

5. use pdb to debug django projects

6. dont copy paste html across templates. use {% extends %}, {% include %} and other builtin templatetags. write your own custom templatetags if necessary (they are very easy to write)

7. understand django middleware. Use this if you want to do some something before the request is processed / after the response is generated etc. - again they are very easy to write

8. use django forms - they are really really powerful 

9. use django test client for unit testing. It acts a dummy web browser. Its very useful to simulate GET and POST request