Python in CentOS 7

There are several options for using Python in CentOS 7.  In most cases, you'll want to use one of our custom-built installations:

/usr/local/bin/python2.6
/usr/local/bin/python2.7
/usr/local/bin/python3.4
/usr/local/bin/python3.5

These are symlinks to their full installation trees in /usr/local/stow/.  For example:

/usr/local/bin/python2.6 --> /usr/local/stow/python-2.6.9/bin/python2.6
/usr/local/bin/python2.7 --> /usr/local/stow/python-2.7.9/bin/python2.7
/usr/local/bin/python3.4 --> /usr/local/stow/python-3.4.3/bin/python3.4
/usr/local/bin/python3.5 --> /usr/local/stow/python-3.5.2/bin/python3.5

Alternatively, you can load each version of Python's respective module (see Module System):

module load python-2.6
module load python-2.7
module load python-3.4
module load python-3.5

To understand what effect running a module file has, see it's corresponding module file in /usr/local/etc/modulefiles/.

We've configured our CentOS 7 system to be compatible with either usage of /usr/local/bin symlinks, or with module files.  You may use whichever you prefer or find more convenient.

The most significant advantage to using a module file is that the command "python" becomes associated with the appropriate binary.  For example, /usr/local/bin contains "python#.#" for each specific version.  But if you have pre-existing code that uses, for example, "#!/usr/bin/env python", you will want "python" to point to the right version.

Observe the effect of the module files:

[user123@box999 ~]# which python
/usr/bin/python

Now load the module for Python 3.4.3, and try again:

[user123@box999 ~]# module load python-3.4
[user123@box999 ~]# which python
/usr/local/stow/python-3.4.3/bin/python

What about /usr/bin/python?

CentOS 7 comes with /usr/bin/python (version 2.7).  This version relies on locally installed packages, most of which also come from RPMs, some of which are usually somewhat outdated.  We've observed in our previous operating system, RHEL6, that most users want cutting edge Python packages.  In RHEL6, we used a hybrid type of package management system where we installed custom packages to a network location and then instructed users on an individual-user basis to modify their PYTHONPATH variable to utilize our custom packages with the system version of Python.  We decided to make a clean break from this practice in CentOS 7, and simply install our own version of Python 2.7 (/usr/local/bin/python2.7).  This removes the need to fiddle with PYTHONPATH.

While users are encouraged to use our versions of Python detailed above, there is absolutely nothing stopping you from using /usr/bin/python.  You just might find it's available packages to be a bit bare and underwhelming in comparison to what we've made available in our versions.

CIMS Package Installations

We have made an effort to pre-populate each of our versions of Python with the most commonly used packages such as numpy, scipy, etc.  There are two ways to get a list of what we have installed for each version: either use the links in /usr/local/bin, or load the appropriate module and use "pip":

[user123@box999 ~]# pip2.7 freeze
ad3==2.0.2
AddOns==0.7
amqp==1.4.6
... 

[user123@box999 ~]# module load python-2.7
[user123@box999 ~]# pip freeze
ad3==2.0.2
AddOns==0.7
amqp==1.4.6
... 

User Package Installations

Python packages are usually very lightweight, and can be installed to a user's home directory with very little effort.  Any user has the freedom to do package installations on a per-user level without needing root privileges.

Either use pip, or download and install from source.

[user123@box999 ~]# pip3.4 install beautifulsoup4 --user -I

Note that the capital I argument is necessary if you attempt to install your own package for a package that we already have some version of.  -I says "ignore the fact that this package is already installed".

At this point, the package you've installed will be located at /home/user123/.local/lib/python3.4/site-packages.

Conversely, if you know where to obtain the source for a Python package that is not available from PyPI (pip's repository), download it, extract it, and install it in a similar manner:

[user123@box999 ~]# wget https://pypi.python.org/packages/source/j/jycat/jycat-0.0.17.tar.gz#md5=8a9bf3efdbc8a59f4dcdd2f08423c52d
[user123@box999 ~]# tar -xvf jycat-0.0.17.tar.gz 
[user123@box999 ~]# cd jycat-0.0.17/
[user123@box999 ~]# python2.7 setup.py install --user

In this example, jycat will be installed to /home/user123/.local/lib/python2.7/site-packages.

For packages installed in this manner, you should be able to immediately begin importing and using them without any other modifications:

[user123@box999 ~]# python2.7 -c 'import jycat'

virtualenv

An even better practice than installing your own packages is to simply clone our Python environment into your own "virtual environment", using the Python tool "virtualenv".  Google "why use virtualenv" for information about why you should be using virtualenv as well as more detailed guides about how to use virtualenv.

The short version is that an individual Python project could have specific version dependencies.  For example, if you write a program using Package ABC from our CIMS global packages, and then at some later date we update Package ABC, and that update either changes or removes existing functionality, your program will likely stop working properly.  Therefore it makes sense to isolate each Python project into it's own environment, using only the exact package versions that your project needs.

To get started with virtualenv, decide which version of Python you want to use, and simply pass the virtualenv program a new virtual environment name:

[user123@box999 ~]# cd /scratch/python_projects
[user123@box999 python_projects]# virtualenv3.4 new_proj1

As simple as that, you now have a brand new and clean Python 3.4 installation which contains ZERO packages (besides the defaults and built-ins).  To begin using it, you need to "activate" it:

[user123@box999 python_projects]# source new_proj1/bin/activate

Now, "python", "pip", etc. will all point to your new Python installation.

(new_proj1)[user123@box999 python_projects]# which pip
/scratch/python_projects/new_proj1/bin/pip
(new_proj1)[user123@box999 python_projects]# which python
/scratch/python_projects/new_proj1/bin/python

And you can confirm that you have no packages installed with "pip freeze".  This means you have a completely blank slate to do anything you want without requiring root privileges.  If your project will rely on an old version of a package, simply install the old version as needed.  This, for example, will install beautifulsoup4 4.1.0 as opposed to the most recent 4.4.0 version:

(new_proj1)[user123@box999 python_projects]# pip install beautifulsoup4==4.1.0

When you're done using your virtual environment:

(new_proj1)[user123@box999 python_projects]# deactivate
[user123@box999 python_projects]# 

Package Requests

If you don't want to manage your own packages and believe that you know of a package that will be of benefit to the CIMS community, please let us know at helpdesk@cims.nyu.edu.