python virtualenv with node environment via nodeenv

Virtualenvwrapper is one of the most useful tools a python user should be familiar with.  Built on top of virtualenv, it helps us avoid a lot of redundant “code administrative” work and simplifies our 3rd party python package isolation.

NodeJS has a similar isolation mechanism and a number of such isolation tools available.

I prefer to use the nodeenv tool because it saves me the trouble of typing out long “source” commands (much like virtualenvwrapper helps me avoid typing in long “source” commands provided by the virtualenv tool).  Also, the integration with virtualenvwrapper implies that I don’t have to waste time remembering new commands to manage my isolated nodejs environment.

So here’s how it’s done. Really efficiently! (assuming of course you have installed your python virtualenvwrapper and virtualenv tools properly):

$ mkvirtualenv myproject1
$ pip install nodeenv  # This installs the nodeenv package into our new python virtualenv so named "myproject1" above
$ nodeenv -p  # This commands installs nodejs and adds new shell functions to our virtualenvwrapper shell functions
* Install node.js (0.10.12) ..
* Appending nodeenv settings to /Users/calvin/.virtualenvs/myproject1/bin/activate
$ deactivate; workon project1   # Deactivate and re-activate to ensure we load in the updated shell functions and environment

Once this is done, we are all set with project1 being our nodejs AND python isolated environment.  When we use the `npm install -g` command, we install our npm packages into our node virtual env.

$ npm install -g yo grunt-cli bower

$ yo angular
[?] Would you like to include Twitter Bootstrap?: No
[?] Would you like to include angular-resource.js?: Yes
[?] Would you like to include angular-cookies.js?: Yes
[?] Would you like to include angular-sanitize.js?: Yes

This installs the `yo`, `grunt-cli` and `bower` tools directly into our node virtual environment and using `yo angular` we easily scaffold out a new angularjs project.

A quick peek at our ~/.virtualenvs/myproject1 directory shows us the directory structure that includes python packages and npm packages.


$ cd ~/.virtualenvs/myproject1

$ tree -L 2
.
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate_this.py
│   ├── bower -> ../lib/node_modules/bower/bin/bower
│   ├── easy_install
│   ├── easy_install-2.7
│   ├── get_env_details
│   ├── grunt -> ../lib/node_modules/grunt-cli/bin/grunt
│   ├── istanbul -> ../lib/node_modules/istanbul/lib/cli.js
│   ├── karma -> ../lib/node_modules/karma/bin/karma
│   ├── node
│   ├── nodeenv
│   ├── npm -> ../lib/node_modules/npm/bin/npm-cli.js
│   ├── pip
│   ├── pip-2.7
│   ├── postactivate
│   ├── postdeactivate
│   ├── preactivate
│   ├── predeactivate
│   ├── python
│   ├── python2 -> python
│   ├── python2.7 -> python
│   └── yo -> ../lib/node_modules/yo/bin/yo
├── include
│   └── python2.7 -> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
├── install.cfg
├── lib
│   ├── dtrace
│   ├── node_modules
│   └── python2.7
├── share
│   └── man
├── src
│   └── node-v0.10.12
└── tags

That’s it.

Installing pyscopg2 in virtualenv against postgresql92, instead of postgresql91

In my previous post, I explained how we can run postgresql92 and postgresql91 concurrently.  However, that post does not explain how we can ensure that the psycopg2 we install in our project’s python virtual environment will build against postgresql92 server.

When psycopg2 is installed (via `pip install psycopg2` for instance), psycopg2 depends on the presence of `pg_config` to locate the currently active postgresql version.

If we do `pg_config –sharedir`, we will see

calvin$ pg_config --sharedir
/opt/local/share/postgresql91

which tells us that if we run `pip install virtualenv`, we will essentially be building psycopg2 against postgresql91.

Fortunately for us, using macports, it’s trivial and completely painless to switch our postgresql server version.


calvin$ sudo port select --list postgresql
Password:
Available versions for postgresql:
 none
 postgresql91 (active)
 postgresql92

calvin$ sudo port select --set postgresql postgresql92
Selecting 'postgresql92' for 'postgresql' succeeded. 'postgresql92' is now active.

calvin$ sudo port select --list postgresql
Available versions for postgresql:
 none
 postgresql91
 postgresql92 (active)

Now, running `pip install psycopg2` in our python virtual environment will correctly build our psycopg2 against our postgresql 9.2 server.

calvin$ pip install psycopg2
Downloading/unpacking psycopg2
 Downloading psycopg2-2.4.5.tar.gz (719kB): 719kB downloaded
 Running setup.py egg_info for package psycopg2

 no previously-included directories found matching 'doc/src/_build'
Installing collected packages: psycopg2
 Running setup.py install for psycopg2
 building 'psycopg2._psycopg' extension
 /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -O2 -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.4.5 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x090202 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I. -I/opt/local/include/postgresql92 -I/opt/local/include/postgresql92/server -c psycopg/psycopgmodule.c -o build/temp.macosx-10.7-x86_64-2.7/psycopg/psycopgmodule.o
 /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -O2 -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.4.5 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x090202 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I. -I/opt/local/include/postgresql92 -I/opt/local/include/postgresql92/server -c psycopg/green.c -o build/temp.macosx-10.7-x86_64-2.7/psycopg/green.o
 /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -O2 -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.4.5 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x090202 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I. -I/opt/local/include/postgresql92 -I/opt/local/include/postgresql92/server -c psycopg/pqpath.c -o build/temp.macosx-10.7-x86_64-2.7/psycopg/pqpath.o
 /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -O2 -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.4.5 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x090202 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I. -I/opt/local/include/postgresql92 -I/opt/local/include/postgresql92/server -c psycopg/utils.c -o build/temp.macosx-10.7-x86_64-2.7/psycopg/utils.o
 /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -O2 -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.4.5 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x090202 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I. -I/opt/local/include/postgresql92 -I/opt/local/include/postgresql92/server -c psycopg/bytes_format.c -o build/temp.macosx-10.7-x86_64-2.7/psycopg/bytes_format.o

Q.E.D.

Next! :-)

geodjango: django tests break when creating new postgis test databases

The new postgis 2.0 library (easily installed via `sudo port install postgis2 +postgresql92` if you are using macports as mentioned in my previous post) helps us convert a standard postgresql database into a spatially-aware database via the simple `CREATE EXTENSION postgis;` command:

$  psql92 -p 5433 -U postgres -d your_db

psql92 (9.2.1)
Type "help" for help.
your_db=#  CREATE EXTENSION postgis;

Unfortunately, when we run our django tests, `test_your_db`, which is newly created by django’s test runner whenever the test begins, will not be postgis-enabled. This leads immediately to test failure when our code fails to recognize the Geometry fields in our project’s GeoDjango models.

To get around this problem, we will need to specifically set up a `template_postgis` template database, much like what we did before postgis 2.0 (reference django docs https://docs.djangoproject.com/en/dev/ref/contrib/gis/install/postgis/#creating-a-spatial-database-template-for-earlier-versions).  But because I am running an instance of postgresql 9.1 server and an instance of 9.2 server concurrently, I will need to modify my commands a little.

Like this:-

calvin$ psql92 -p 5433 -U postgres
psql92 (9.2.1)
Type "help" for help.

postgres=# CREATE DATABASE template_postgis ENCODING='utf-8';
CREATE DATABASE
postgres=# UPDATE pg_database SET datistemplate='true' WHERE datname='template_postgis';
UPDATE 1
postgres=# \q

calvin$ POSTGIS_SQL_PATH=/opt/local/share/postgresql92/contrib/postgis-2.0

calvin$ psql92 -p 5433 -U postgres -d template_postgis -f $POSTGIS_SQL_PATH/postgis.sql

calvin$ psql92 -p 5433 -U postgres -d template_postgis -f $POSTGIS_SQL_PATH/spatial_ref_sys.sql

calvin$ psql92 -p 5433 -U postgres -d template_postgis -c "GRANT ALL ON geometry_columns TO PUBLIC;"
GRANT

calvin$ psql92 -p 5433 -U postgres -d template_postgis -c "GRANT ALL ON geography_columns TO PUBLIC;"
GRANT

calvin$ psql92 -p 5433 -U postgres -d template_postgis -c "GRANT ALL ON spatial_ref_sys TO PUBLIC;"
GRANT

With this, running `./manage.py test` with your geodjango-based application will work perfectly fine because django’s default test runner will look for the template_postgis template database to instantiate our now spatially-aware test database.