virtualenv with numpy and scipy on Mac OSX

Getting numpy, scipy and associated libraries and their corresponding C/C++ libraries installed correctly on virtualenv (with the help of virtualenv wrapper), Mac OSX can be a little troublesome.

Here’s a step-by-step as a reference for my colleagues and fellow pythonista peers…

1. Get our virtual environment up-and-running

$ mkvirtualenv -p python2.7 --distribute python-for-scientists
Running virtualenv with interpreter /opt/local/bin/python2.7
New python executable in python-for-scientists/bin/python
Installing distribute..........................................................................................................................................................................................................done.
Installing pip................done.
virtualenvwrapper.user_scripts creating /Users/calvin/.virtualenvs/python-for-scientists/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/calvin/.virtualenvs/python-for-scientists/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/calvin/.virtualenvs/python-for-scientists/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/calvin/.virtualenvs/python-for-scientists/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/calvin/.virtualenvs/python-for-scientists/bin/get_env_details

 2. Personal Convention: working directory for project-specific python files

As a personal convention, I like to synchronize my project’s root directory name with my virtual environment name. In this case, “python-for-scientists”.

$ mkdir -p ~/work/python-for-scientists
$ cd ~/work/python-for-scientists

And the fun begins…

3.  NumPy: pip install numpy

To install numpy in our python virtual environment, it’s a simple as running `pip install numpy`.

But if this is your first time getting your feet wet with coding, be aware that on Mac OSX, you will need to install XCode and associated SDKs/compilers provided by https://developer.apple.com/.  Typically, once we have gotten that installed, the clang and clang++ compilers will be available on our system.  Checking…

(python-for-scientists)55.64.131.216.client.dyn.strong-sf15.reliablehosting.com ttys004 Fri Nov 02 15:18:57 |~/work/python-for-scientists|
calvin$ which clang++
/usr/bin/clang++
(python-for-scientists)55.64.131.216.client.dyn.strong-sf15.reliablehosting.com ttys004 Fri Nov 02 15:19:16 |~/work/python-for-scientists|
calvin$ clang --version
Apple clang version 4.0 (tags/Apple/clang-421.0.60) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin11.4.2
Thread model: posix
(python-for-scientists)55.64.131.216.client.dyn.strong-sf15.reliablehosting.com ttys004 Fri Nov 02 15:19:19 |~/work/python-for-scientists|
calvin$ clang++ --version
Apple clang version 4.0 (tags/Apple/clang-421.0.60) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin11.4.2
Thread model: posix

Now, numpy can be downloaded and installed as expected:-

calvin$ pip install numpy

If numpy and all its c modules get downloaded and compiled correctly, we should see something similar at the end.

clang: numpy/random/mtrand/randomkit.c
clang: numpy/random/mtrand/mtrand.c
/usr/bin/clang -bundle -undefined dynamic_lookup -isysroot / -L/opt/local/lib build/temp.macosx-10.7-x86_64-2.7/numpy/random/mtrand/mtrand.o build/temp.macosx-10.7-x86_64-2.7/numpy/random/mtrand/randomkit.o build/temp.macosx-10.7-x86_64-2.7/numpy/random/mtrand/initarray.o build/temp.macosx-10.7-x86_64-2.7/numpy/random/mtrand/distributions.o -Lbuild/temp.macosx-10.7-x86_64-2.7 -o build/lib.macosx-10.7-x86_64-2.7/numpy/random/mtrand.so

Creating build/scripts.macosx-10.7-x86_64-2.7/f2py
 adding 'build/scripts.macosx-10.7-x86_64-2.7/f2py' to scripts
 changing mode of build/scripts.macosx-10.7-x86_64-2.7/f2py from 644 to 755

 changing mode of /Users/calvin/.virtualenvs/python-for-scientists/bin/f2py to 755
Successfully installed numpy
Cleaning up...

Notice also that numpy include f2py, which is an excellent program for scientists or engineers that would like to integrate their python program with existing fortran based programs.

Checking that we have installed numpy correctly:-

calvin$ python -c "import numpy; print numpy.version.version"
1.6.2

 4. SciPy on Mac OSX

That was easy. Here comes the complication. Getting SciPy to install correctly on Mac OSX is a little more troublesome.  We will need a proper fortran compiler installed first before pip install scipy would work.

This is the link where I got my gfortran compiler - http://gcc.gnu.org/wiki/GFortranBinaries#MacOS

Download the appropriate gfortran binary for your Mac OSX version.  You can use this if you are running Mac OSX Lion – http://quatramaran.ens.fr/~coudert/gfortran/gfortran-4.6.2-x86_64-Lion.dmg.

I am in an adventurous mood, so I decided to go for the high performance computing (HPC) for Mac OSX version instead.  That version can be found on http://hpc.sourceforge.net/, specifically here - http://prdownloads.sourceforge.net/hpc/gfortran-lion.tar.gz?download.

Once you have gotten your gfortran compiler installed correctly, you can compile your Fortran programs too, if that matters to you as an engineer/scientist. :-)

Checking:-

(python-for-scientists)55.64.131.216.client.dyn.strong-sf15.reliablehosting.com ttys004 Fri Nov 02 15:38:45 |~/work/python-for-scientists|
calvin$ gfortran --version
GNU Fortran (GCC) 4.8.0 20120603 (experimental)
Copyright (C) 2012 Free Software Foundation, Inc.

GNU Fortran comes with NO WARRANTY, to the extent permitted by law.
You may redistribute copies of GNU Fortran
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING

With f2py from NumPy mentioned above, we now have the power of Fortran and Python in our hands!

Moving on, now that we have got gfortran sorted, we want to get SciPy installed.  So let’s get cracking…

calvin$ pip install scipy

...
...
/usr/bin/clang -bundle -undefined dynamic_lookup -isysroot / -L/opt/local/lib build/temp.macosx-10.7-x86_64-2.7/scipy/ndimage/src/nd_image.o build/temp.macosx-10.7-x86_64-2.7/scipy/ndimage/src/ni_filters.o build/temp.macosx-10.7-x86_64-2.7/scipy/ndimage/src/ni_fourier.o build/temp.macosx-10.7-x86_64-2.7/scipy/ndimage/src/ni_interpolation.o build/temp.macosx-10.7-x86_64-2.7/scipy/ndimage/src/ni_measure.o build/temp.macosx-10.7-x86_64-2.7/scipy/ndimage/src/ni_morphology.o build/temp.macosx-10.7-x86_64-2.7/scipy/ndimage/src/ni_support.o -Lbuild/temp.macosx-10.7-x86_64-2.7 -o build/lib.macosx-10.7-x86_64-2.7/scipy/ndimage/_nd_image.so

 warning: no files found matching 'f2py.py'
 no previously-included directories found matching 'scipy/special/tests/data/boost'
Successfully installed scipy
Cleaning up...

If all goes well, we should see the “successfully installed scipy” message at the end of it all.

Now that we have gotten this out of the way, we can check

calvin$ python -c "import scipy; print scipy.version.version"
0.11.0

5. What’s Next?

So this is pretty much there is to it for our basic set-up to begin powerful numerical analysis and exploratory data science with NumPy and SciPy.

In my next post, I will run through iPython install (which is trivial but with a small readline hiccup though) and Qt4 integration with iPython (which is a little messy for virtualenv in Mac OSX) giving us a Qt4-based GUI iPython shell that comes with wonderful functionalities like rendering embedded charts and visualization inline in iPython.

Powerful stuff.

  • Guest

    A heads-up: the links to GFortran binaries at gcc.gnu.org all failed to load, so I used a [GNU Fortran installer for OS X on this 'tools for R' page](http://r.research.att.com/tools/).

  • DaveEveritt

    A heads-up: the links to GFortran binaries at gcc.gnu.org all failed to load, so I used a GNU Fortran installer for OS X on this ‘tools for R’ page: http://r.research.att.com/tools/.

  • yerfatma

    For the lazy (like me), homebrew will install gfortran for you with no muss.