Virtualenv is a great Python tool for isolating dependencies when developing a new project. And virtualenvwrapper is the convenient shell script that should be part of virtualenv by default, IMO. virtualenwrapper lets you do things like:
mkvirtualenv --no-site-packages newproject workon newproject
… and you’re done. Anything you install via pip, after that, will be confined to your virtualenv. Then when deployment time comes, you can go pip freeze > requirements.txt, your users can go pip install -r requirements.txt and all is neat and tidy with the world.
If you are writing a web app your users are probably your web server(s). Then Fabric comes into the mix. Fabric is designed to make deploying your web project a one-liner. It’s pretty thrilling to use too. Here is a typical fabric command:
def deploy_version(version):
"Specify a specific version to be made live"
require('path')
env.version = version
with cd('%(path)s' % env):
run('rm releases/previous')
run('mv releases/current releases/previous')
run('ln -s $(version) releases/current' % env)
restart_webserver()
Nice, no? These slides are a nice intro to fabric as well.
The reason that ‘with cd’ context manager is needed is because Fabric doesn’t keep ‘state’. Apparently each run command is living in its own ssh session, just about. This is a problem if you need to source files, as when using virtualenv. This is hinted at (here, here, here, here) but not really explained clearly anywhere.
To get it working, this is what I ended up doing:
(on my dev machine)
.bash_profile
if [ $USER == blaugher ]; then
export WORKON_HOME=/home/blaugher/virtualenvs
source /usr/local/bin/virtualenvwrapper.sh
fi
fabfile.py
def setup():
"""
Setup a fresh virtualenv as well as a few useful directories, then run
a full deployment
"""
sudo('aptitude install -y python-setuptools apache2 libapache2-mod-wsgi')
sudo('easy_install pip')
sudo('pip install virtualenv')
sudo('pip install virtualenvwrapper')
put('.bash_profile', '~/.bash_profile')
run('mkdir -p %(workon_home)s' % env)
with settings(warn_only=True):
# just in case it already exists, let's ditch it
run('rmvirtualenv %(project_name)s' % env)
run('mkvirtualenv --no-site-packages %(project_name)s' % env)
# [... plus other useful stuff.]
def install_requirements():
"Install the required packages from the requirements file using pip"
with cd('%(path)s' % env):
run('workon %(project_name)s && pip install -r ./releases/%(release)s/requirements.txt' % env)
So, that’s reasonably nice. The .bash_profile is so-written because of this bug – somehow one of the virtualenvwrapper files ends up being owned by root, which causes an IOError for non-root users. You could change the shell command for when you run sudo but it would be pretty tedious. sudo and run have an option of shell (boolean) and in the env you can set the shell command to be used (by default it is /bin/bash -l -c) but there is no easy way to specify different shell commands for run vs sudo commands.
Virtualenvwrapper recommends to add the ‘export’ and ‘source’ lines to your .bashrc. By adding them to .bash_profile instead they will be executed for login shells – like our /bin/bash/ -l, i.e. for all fabric commands, and we don’t have to explicitly source any file in fabric. I thought this was a neat side-step of that problem. I’m not sure what the other implications of .bashrc vs .bash_profile are.
My next problem was a call like this:
run('mkvirtualenv %(project_name)s --no-site-packages' % env)
Fabric was complaining that it was getting back a return code of 1. This seemed odd as it looked like it was working. Even running it by hand still looked good:
blaugher@tardis:~$ mkvirtualenv qwerty --no-site-packages New python executable in qwerty/bin/python Installing distribute.................................. .......................................................... ................................................................ .....................done.
But not:
blaugher@tardis:~$ echo $? 1
I had joined the mailing list and was preparing to write up my problem when I spotted this earlier reply:
Try reversing the order of env3 and —no-site-packages on the command line. mkvirtualenv expects the environment name to be the last argument.
!!!
Sure enough –
blaugher@tardis:~$ mkvirtualenv --no-site-packages qwerty New python executable in qwerty/bin/python Installing distribute........................................................ ...................................................... ...................................................................done. virtualenvwrapper.user_scripts Creating /home/blaugher/virtualenvs/qwerty/bin/predeactivate virtualenvwrapper.user_scripts Creating /home/blaugher/virtualenvs/qwerty/bin/postdeactivate virtualenvwrapper.user_scripts Creating /home/blaugher/virtualenvs/qwerty/bin/preactivate virtualenvwrapper.user_scripts Creating /home/blaugher/virtualenvs/qwerty/bin/postactivate virtualenvwrapper.user_scripts Creating /home/blaugher/virtualenvs/qwerty/bin/get_env_details (qwerty)blaugher@tardis:~$
That’s a little too subtle for my liking. Although note that as well as the extra user_scripts stuff, this time that the virtualenv is actually activated, as it is supposed to be (the “(qwerty)” prefix tells you you are working in a virtualenv).
A couple of other points. If Django is in your requirements.txt file rather than manually installed on your server, you will want to make sure your Django-specific Python calls look something like run('workon %(project_name)s && python manage.py syncdb' % env).
Making ‘workon’ into something more generically useful is obviously not too difficult. This example fabfile adds a method to do it. Apparently an upcoming version of fabric might have a command like prefix, which would work very well.
Finally using virtualenvwrapper means all your virtualenvs are collected together in their own directory, separate to the projects you use them for. Docs on virtualenv alone tend to suggest nesting the project code within the virtualenv itself, but I prefer the approach of virtualenvwrapper.
The moral of the story is, all your problems are already solved, and all you need to do is locate your answers, out there, somewhere. :)
