python virtualenv with node environment via nodeenv

Virtualenvwrapper is one of the most useful tools a python user should be familiar with.  Built on top of virtualenv, it helps us avoid a lot of redundant “code administrative” work and simplifies our 3rd party python package isolation.

NodeJS has a similar isolation mechanism and a number of such isolation tools available.

I prefer to use the nodeenv tool because it saves me the trouble of typing out long “source” commands (much like virtualenvwrapper helps me avoid typing in long “source” commands provided by the virtualenv tool).  Also, the integration with virtualenvwrapper implies that I don’t have to waste time remembering new commands to manage my isolated nodejs environment.

So here’s how it’s done. Really efficiently! (assuming of course you have installed your python virtualenvwrapper and virtualenv tools properly):

$ mkvirtualenv myproject1
$ pip install nodeenv  # This installs the nodeenv package into our new python virtualenv so named "myproject1" above
$ nodeenv -p  # This commands installs nodejs and adds new shell functions to our virtualenvwrapper shell functions
* Install node.js (0.10.12) ..
* Appending nodeenv settings to /Users/calvin/.virtualenvs/myproject1/bin/activate
$ deactivate; workon project1   # Deactivate and re-activate to ensure we load in the updated shell functions and environment

Once this is done, we are all set with project1 being our nodejs AND python isolated environment.  When we use the `npm install -g` command, we install our npm packages into our node virtual env.

$ npm install -g yo grunt-cli bower

$ yo angular
[?] Would you like to include Twitter Bootstrap?: No
[?] Would you like to include angular-resource.js?: Yes
[?] Would you like to include angular-cookies.js?: Yes
[?] Would you like to include angular-sanitize.js?: Yes

This installs the `yo`, `grunt-cli` and `bower` tools directly into our node virtual environment and using `yo angular` we easily scaffold out a new angularjs project.

A quick peek at our ~/.virtualenvs/myproject1 directory shows us the directory structure that includes python packages and npm packages.


$ cd ~/.virtualenvs/myproject1

$ tree -L 2
.
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate_this.py
│   ├── bower -> ../lib/node_modules/bower/bin/bower
│   ├── easy_install
│   ├── easy_install-2.7
│   ├── get_env_details
│   ├── grunt -> ../lib/node_modules/grunt-cli/bin/grunt
│   ├── istanbul -> ../lib/node_modules/istanbul/lib/cli.js
│   ├── karma -> ../lib/node_modules/karma/bin/karma
│   ├── node
│   ├── nodeenv
│   ├── npm -> ../lib/node_modules/npm/bin/npm-cli.js
│   ├── pip
│   ├── pip-2.7
│   ├── postactivate
│   ├── postdeactivate
│   ├── preactivate
│   ├── predeactivate
│   ├── python
│   ├── python2 -> python
│   ├── python2.7 -> python
│   └── yo -> ../lib/node_modules/yo/bin/yo
├── include
│   └── python2.7 -> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
├── install.cfg
├── lib
│   ├── dtrace
│   ├── node_modules
│   └── python2.7
├── share
│   └── man
├── src
│   └── node-v0.10.12
└── tags

That’s it.

Installing pyscopg2 in virtualenv against postgresql92, instead of postgresql91

In my previous post, I explained how we can run postgresql92 and postgresql91 concurrently.  However, that post does not explain how we can ensure that the psycopg2 we install in our project’s python virtual environment will build against postgresql92 server.

When psycopg2 is installed (via `pip install psycopg2` for instance), psycopg2 depends on the presence of `pg_config` to locate the currently active postgresql version.

If we do `pg_config –sharedir`, we will see

calvin$ pg_config --sharedir
/opt/local/share/postgresql91

which tells us that if we run `pip install virtualenv`, we will essentially be building psycopg2 against postgresql91.

Fortunately for us, using macports, it’s trivial and completely painless to switch our postgresql server version.


calvin$ sudo port select --list postgresql
Password:
Available versions for postgresql:
 none
 postgresql91 (active)
 postgresql92

calvin$ sudo port select --set postgresql postgresql92
Selecting 'postgresql92' for 'postgresql' succeeded. 'postgresql92' is now active.

calvin$ sudo port select --list postgresql
Available versions for postgresql:
 none
 postgresql91
 postgresql92 (active)

Now, running `pip install psycopg2` in our python virtual environment will correctly build our psycopg2 against our postgresql 9.2 server.

calvin$ pip install psycopg2
Downloading/unpacking psycopg2
 Downloading psycopg2-2.4.5.tar.gz (719kB): 719kB downloaded
 Running setup.py egg_info for package psycopg2

 no previously-included directories found matching 'doc/src/_build'
Installing collected packages: psycopg2
 Running setup.py install for psycopg2
 building 'psycopg2._psycopg' extension
 /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -O2 -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.4.5 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x090202 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I. -I/opt/local/include/postgresql92 -I/opt/local/include/postgresql92/server -c psycopg/psycopgmodule.c -o build/temp.macosx-10.7-x86_64-2.7/psycopg/psycopgmodule.o
 /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -O2 -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.4.5 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x090202 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I. -I/opt/local/include/postgresql92 -I/opt/local/include/postgresql92/server -c psycopg/green.c -o build/temp.macosx-10.7-x86_64-2.7/psycopg/green.o
 /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -O2 -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.4.5 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x090202 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I. -I/opt/local/include/postgresql92 -I/opt/local/include/postgresql92/server -c psycopg/pqpath.c -o build/temp.macosx-10.7-x86_64-2.7/psycopg/pqpath.o
 /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -O2 -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.4.5 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x090202 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I. -I/opt/local/include/postgresql92 -I/opt/local/include/postgresql92/server -c psycopg/utils.c -o build/temp.macosx-10.7-x86_64-2.7/psycopg/utils.o
 /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -O2 -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.4.5 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x090202 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I. -I/opt/local/include/postgresql92 -I/opt/local/include/postgresql92/server -c psycopg/bytes_format.c -o build/temp.macosx-10.7-x86_64-2.7/psycopg/bytes_format.o

Q.E.D.

Next! :-)

IPython on steroids with qt console, via Macports and in Virtualenv

Getting iPython installed in a virtualenv is straightforward.

The gotcha for new pythonistas not familiar with iPython is that it requires gnu readline to display colors, provide autocomplete functionality and a host of other usability features. Unfortunately, on Mac OSX, getting readline via `pip install readline` will not quite work.  This is due to a Mac OSX-specific PYTHONPATH problem.  With `pip installs readline`, it will never be imported, because readline.so goes in site-packages, which ends up behind the libedit system one, located in lib-dynload (yes, OSX Python path order is very odd).

To solve this problem, we will, instead, run this after we have run `pip install readline`:-

easy_install -a readline

The `-a` option means `–always-copy` which copies all needed packages into our installation directory.  Executing this command while in our virtualenv ensures that the readline package gets placed correctly in our virtualenv directory.  Checking:-

(python-for-scientists)Calvins-MacBook-Pro.local ttys009 Sat Nov 03 12:36:36 |~/work/python-for-scientists|
calvin$ ls -la ~/.virtualenvs/python-for-scientists/lib/python2.7/site-packages/
total 16
drwxr-xr-x 14 calvin staff 476 Nov 3 12:27 .
drwxr-xr-x 51 calvin staff 1734 Nov 2 15:21 ..
drwxr-xr-x 19 calvin staff 646 Nov 3 12:27 IPython
drwxr-xr-x 10 calvin staff 340 Nov 2 15:03 distribute-0.6.28-py2.7.egg
-rw-r--r-- 1 calvin staff 285 Nov 2 15:36 easy-install.pth
drwxr-xr-x 10 calvin staff 340 Nov 3 12:27 ipython-0.13.1-py2.7.egg-info
drwxr-xr-x 38 calvin staff 1292 Nov 2 15:22 numpy
drwxr-xr-x 7 calvin staff 238 Nov 2 15:22 numpy-1.6.2-py2.7.egg-info
drwxr-xr-x 4 calvin staff 136 Nov 2 15:03 pip-1.2.1-py2.7.egg
drwxr-xr-x 3 calvin staff 102 Nov 2 15:03 readline
drwxr-xr-x 6 calvin staff 204 Nov 2 15:36 readline-6.2.4.1-py2.7-macosx-10.7-x86_64.egg
drwxr-xr-x 38 calvin staff 1292 Nov 2 16:00 scipy
drwxr-xr-x 7 calvin staff 238 Nov 2 16:00 scipy-0.11.0-py2.7.egg-info
-rw-r--r-- 1 calvin staff 30 Nov 2 15:03 setuptools.pth

Now, in our virtual env, our iPython shell should work perfectly, with all the features afforded by the gnu readline library.

Moving on, we now want the power of iPython on in a Qt4-powered GUI console.  This will prove to be a little tricky on Mac OSX with Macports.

1.  Qt4 on Mac OSX

The first thing we need to do is to install the qt4-mac package.  Straightforward via MacPorts:-

calvin$ sudo port -v install qt4-mac

2. Python bindings to Qt4

The problem begins when we try to install a python binding library for Qt4. We have to possible options from PyPi

Attempting `pip install PyQt` in your virtual env fails out right. Because PyQt package on pypi does not include a setup.py.

Going the PySide route requires us to have cmake, which can easily be solved by installing cmake system-wide with `sudo port install cmake`.  However, that still fails to help us get `pip install pyside` working right.  We will encounter an error like this:-

error: Failed to locate the Python library /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/libpython2.7.so.1
 Complete output from command /Users/calvin/.virtualenvs/python-for-scientists/bin/python -c "import setuptools;__file__='/Users/calvin/.virtualenvs/python-for-scientists/build/pyside/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/kl/_52jng9s6sl2knv_0jds9w140000gn/T/pip-7zqGiu-record/install-record.txt --single-version-externally-managed --install-headers /Users/calvin/.virtualenvs/python-for-scientists/bin/../include/site/python2.7:
 Removing /Users/calvin/.virtualenvs/python-for-scientists/build/pyside/PySide

To solve this problem, we will instead rely on the Macports version of PyQt.

calvin$ port search pyqt4
py-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

py24-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

py25-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

py26-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

py27-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

py31-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

py32-pyqt4 @4.9.4 (python, devel)
 PyQt4 is a set of Python bindings for the Qt4 toolkit

3. System-wide iPython qtconsole

So,

calvin$ sudo port -v install py27-pyqt4

---> Computing dependencies for py27-pyqt4.
---> Fetching archive for py27-pyqt4
---> py27-pyqt4-4.9.4_0.darwin_11.x86_64.tbz2 doesn't seem to exist in /opt/local/var/macports/incoming/verified
---> Attempting to fetch py27-pyqt4-4.9.4_0.darwin_11.x86_64.tbz2 from http://packages.macports.org/py27-pyqt4
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
---> Attempting to fetch py27-pyqt4-4.9.4_0.darwin_11.x86_64.tbz2 from http://mse.uk.packages.macports.org/sites/packages.macports.org/py27-pyqt4
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
---> Attempting to fetch py27-pyqt4-4.9.4_0.darwin_11.x86_64.tbz2 from http://lil.fr.packages.macports.org/py27-pyqt4
 % Total % Received % Xferd Average Speed Time Time Time Current
 Dload Upload Total Spent Left Speed
 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
---> Fetching distfiles for py27-pyqt4
---> Verifying checksum(s) for py27-pyqt4
---> Checksumming PyQt-mac-gpl-4.9.4.tar.gz
---> Extracting py27-pyqt4
---> Extracting PyQt-mac-gpl-4.9.4.tar.gz
---> Applying patches to py27-pyqt4
---> Applying patch-configure.py
patching file configure.py
---> Applying patch-fix-qt_apps_dir.diff
patching file examples/demos/qtdemo/menumanager.py
patching file examples/designer/plugins/plugins.py
---> Configuring py27-pyqt4
Determining the layout of your Qt installation...

...

Qt v4.8.3 free edition is being used.
Qt is built as a framework.
SIP 4.13.3 is being used.
The Qt header files are in /opt/local/include.
The shared Qt frameworks are in /opt/local/Library/Frameworks.
The Qt binaries are in /opt/local/bin.
The Qt mkspecs directory is in /opt/local/share/qt4.
These PyQt modules will be built: QtCore, QtGui, QtHelp, QtMultimedia,
QtNetwork, QtDeclarative, QtOpenGL, QtScript, QtScriptTools, QtSql, QtSvg,
QtTest, QtWebKit, QtXml, QtXmlPatterns, QtDesigner.
The PyQt Python package will be installed in
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages.
PyQt is being built with generated docstrings.
PyQt is being built with 'protected' redefined as 'public'.
The Designer plugin will be installed in /opt/local/share/qt4/plugins/designer.
The PyQt .sip files will be installed in /opt/local/share/py27-sip/PyQt4.
pyuic4, pyrcc4 and pylupdate4 will be installed in
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin.
Generating the C++ source for the QtCore module...

...

x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Loader/qobjectcreator.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/__init__.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/compiler.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/indenter.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/misc.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/proxy_metaclass.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/qobjectcreator.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4/uic/Compiler/qtproxies.py
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pylupdate4
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pyrcc4
x ./opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pyuic4
x ./opt/local/bin/pylupdate4-2.7
x ./opt/local/bin/pyrcc4-2.7
x ./opt/local/bin/pyuic4-2.7
---> Cleaning py27-pyqt4
---> Removing work directory for py27-pyqt4
---> Updating database of binaries: 100.0%
---> Scanning binaries for linking errors: 100.0%
---> No broken files found.

With this done, and with a system-wide zmq/py27-zmq installed (via `sudo port -v install zmq py27-zmq` command), our system-wide ipython will work beautifully with `ipython qtconsole` outside a python virtual env.

4. iPython qtconsole in virtualenv

However, if we attempt to do the same in an isolated python virtual env, we will encounter this error triggered when qtconsoleapp.py in our virtualenv iPython gets called:-


File "/Users/calvin/.virtualenvs/python-for-scientists/lib/python2.7/site-packages/IPython/frontend/qt/console/qtconsoleapp.py", line 56, in <module>
 from IPython.external.qt import QtCore, QtGui
 File "/Users/calvin/.virtualenvs/python-for-scientists/lib/python2.7/site-packages/IPython/external/qt.py", line 43, in <module>
 raise ImportError('Cannot import PySide >= 1.0.3 or PyQt4 >= 4.7')
ImportError: Cannot import PySide >= 1.0.3 or PyQt4 >= 4.7

which of course, is to be expected.

So, we aren’t satisfied yet because we cannot run our ipython qtconsole within an isolated python virtual env.  Our isolated python virtualenv will not be able to locate the system-wide py27-qt4 that we have installed above; and we also cannot depend on `pip install pyqt` or `pip install pyside` commands inside the virtual env, we will need to resort to a little bit of bash scripting trickery – i.e. symlink to our installed system-wide libraries (PyQt4 and its dependency SIP) to solve this problem from our virtualenv location.

And here’s our little bash script to achieve this:-

Once we run `./symlink_pyqt4_and_sip.sh` while in our virtual env, PyQt4 and SIP will now be available to us!

5. Finishing up: ipython qtconsole in virtualenv!

Quick check:-


calvin$ ls -la $VIRTUAL_ENV/lib/python2.7/site-packages
total 32
drwxr-xr-x 12 calvin staff 408 Nov 3 13:59 .
drwxr-xr-x 51 calvin staff 1734 Oct 31 14:07 ..
drwxr-xr-x 19 calvin staff 646 Oct 31 10:54 IPython
lrwxr-xr-x 1 calvin staff 93 Nov 3 13:59 PyQt4 -> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/PyQt4
drwxr-xr-x 10 calvin staff 340 Oct 31 10:13 distribute-0.6.28-py2.7.egg
-rw-r--r-- 1 calvin staff 285 Oct 31 10:15 easy-install.pth
drwxr-xr-x 10 calvin staff 340 Oct 31 10:15 ipython-0.13.1-py2.7.egg-info
drwxr-xr-x 4 calvin staff 136 Oct 31 10:13 pip-1.2.1-py2.7.egg
drwxr-xr-x 3 calvin staff 102 Oct 31 10:13 readline
drwxr-xr-x 6 calvin staff 204 Oct 31 10:15 readline-6.2.4.1-py2.7-macosx-10.7-x86_64.egg
-rw-r--r-- 1 calvin staff 30 Oct 31 10:13 setuptools.pth
lrwxr-xr-x 1 calvin staff 94 Nov 3 13:59 sip.so -> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sip.so

Yup.  As you can see, PyQt4 and sip.so are correctly symlinked from our virtualenv directory to the system-wide one.

Recall that iPython qt console requires a couple more dependencies; but now we can get them via pip (and not macports) – `pip install pygments pyzmq` (Note that zmq library itself should already be installed system-wide but the python bindings pyzmq – different name compared to our macports’ py27-zmq – needs to be installed within our virtualenv).

Once done, running `ipython qtconsole` will work beautifully, as expected.