Python in CentOS 7
* refers to any available minor version of the major version (Python 2 and Python 3 are major). For example, in python 3.* , the * could be substituted for 6 for Python 3.6, and so on.
There are several options for using Python in CentOS 7. In most cases, you'll want to use one of our custom-built installations:
/usr/local/bin/python3.*
These are symlinks to their full installation trees in /usr/local/stow/. For example:
/usr/local/bin/python3.* --> ../stow/python-3.*/bin/python3.*
Alternatively, you can load each version of Python's respective module (see Module System):
module load python-3.*
To understand what effect running a module file has, see it's corresponding module file in /usr/local/etc/modulefiles/.
We've configured our CentOS 7 system to be compatible with either usage of /usr/local/bin symlinks, or with module files. You may use whichever you prefer or find more convenient.
The most significant advantage to using a module file is that the command "python" becomes associated with the appropriate binary. For example, /usr/local/bin contains "python#.#" for each specific version. But if you have pre-existing code that uses, for example, "#!/usr/bin/env python", you will want "python" to point to the right version.
Observe the effect of the module files:
[user123@box999 ~]# which python
/usr/bin/python
Now load the module for Python 3.*, and try again (note for python 3, you will need to use the "python3" command):
[user123@box999 ~]# module load python-3.*[user123@box999 ~]# python3 -V
[user123@box999 ~]# which python3
/usr/local/stow/python-3.*/bin/python
Python 3.*.*
What about /usr/bin/python?
CentOS 7 comes with /usr/bin/python (version 2.7). This version relies on locally installed packages, most of which also come from RPMs, some of which are usually somewhat outdated. We've observed in our previous operating system, RHEL6, that most users want cutting edge Python packages. In RHEL6, we used a hybrid type of package management system where we installed custom packages to a network location and then instructed users on an individual-user basis to modify their PYTHONPATH variable to utilize our custom packages with the system version of Python.
While users are encouraged to use our versions of Python detailed above, there is absolutely nothing stopping you from using /usr/bin/python. You just might find it's available packages to be a bit bare and underwhelming in comparison to what we've made available in our versions.
CIMS Package Installations
We have made an effort to pre-populate each of our versions of Python with the most commonly used packages such as numpy, scipy, etc. There are two ways to get a list of what we have installed for each version: either use the links in /usr/local/bin, or load the appropriate module and use "pip":
[user123@box999 ~]# pip3.* freeze
ad3==2.0.2
AddOns==0.7
amqp==1.4.6
...
[user123@box999 ~]# module load python-3.*
[user123@box999 ~]# pip3 freeze
ad3==2.0.2
AddOns==0.7
amqp==1.4.6
...
User Package Installations
Python packages are usually very lightweight, and can be installed to a user's home directory with very little effort. Any user has the freedom to do package installations on a per-user level without needing root privileges.
Either use pip, or download and install from source.
[user123@box999 ~]# pip3.* install beautifulsoup4 --user -I
Note that the capital I argument is necessary if you attempt to install your own package for a package that we already have some version of. -I says "ignore the fact that this package is already installed".
At this point, the package you've installed will be located at /home/user123/.local/lib/python3.*/site-packages.
Conversely, if you know where to obtain the source for a Python package that is not available from PyPI (pip's repository), download it, extract it, and install it in a similar manner:
[user123@box999 ~]# wget https://pypi.python.org/packages/source/j/jycat/jycat-0.0.17.tar.gz#md5=8a9bf3efdbc8a59f4dcdd2f08423c52d
[user123@box999 ~]# tar -xvf jycat-0.0.17.tar.gz
[user123@box999 ~]# cd jycat-0.0.17/
[user123@box999 ~]# python3.* setup.py install --user
In this example, jycat will be installed to /home/user123/.local/lib/python3.6/site-packages.
For packages installed in this manner, you should be able to immediately begin importing and using them without any other modifications:
[user123@box999 ~]# python3.* -c 'import jycat'
virtualenv
An even better practice than installing your own packages is to simply clone our Python environment into your own "virtual environment", using the Python tool "virtualenv". Google "why use virtualenv" for information about why you should be using virtualenv as well as more detailed guides about how to use virtualenv.
The short version is that an individual Python project could have specific version dependencies. For example, if you write a program using Package ABC from our CIMS global packages, and then at some later date we update Package ABC, and that update either changes or removes existing functionality, your program will likely stop working properly. Therefore it makes sense to isolate each Python project into it's own environment, using only the exact package versions that your project needs.
To get started with virtualenv, decide which version of Python you want to use, and simply pass the virtualenv program a new virtual environment name:
[user123@box999 ~]# cd /scratch/python_projects
[user123@box999 python_projects]# virtualenv3.6 new_proj1
As simple as that, you now have a brand new and clean Python 3.6 installation which contains ZERO packages (besides the defaults and built-ins). To begin using it, you need to "activate" it:
[user123@box999 python_projects]# source new_proj1/bin/activate
Now, "python", "pip", etc. will all point to your new Python installation.
(new_proj1)[user123@box999 python_projects]# which pip
/scratch/python_projects/new_proj1/bin/pip
(new_proj1)[user123@box999 python_projects]# which python
/scratch/python_projects/new_proj1/bin/python
(new_proj1)[user123@box999 python_projects]# pip install beautifulsoup4==4.1.0
When you're done using your virtual environment:
(new_proj1)[user123@box999 python_projects]# deactivate
[user123@box999 python_projects]#
Jupyter Notebook
If you wish to run a Jupyter notebook, you may do so without any special setup by first loading our python-3 module, like so:
[user123@box999 ~] module load python-3
[user123@box999 ~] jupyter notebook
If you want to use Jupyter Notebook with your own choice of packages, you can set up a virtualenv as in the above section, but make sure to run pip install jupyter while the virtualenv is active to install the necessary packages.
Running Jupyter Notebook remotely
In order to run a Jupyter Notebook on a Courant compute server remotely, you might try a series of commands like so, replacing "12345" with an unused port number between 10000 and 49151 and "snappy1" with any compute server that you wish to use:
[user123@home ~] ssh -4 -L 12345:localhost:12345 user123@access.cims.nyu.edu
[user123@access ~] ssh -4 -L 12345:localhost:12345 snappy1.cims.nyu.edu
[user123@snappy1 ~] module load python-3
[user123@snappy1 ~] jupyter notebook --no-browser --port 12345
Then, you should be able to use Jupyter Notebook by copying one of the URLs with a token (beginning with "localhost", for example) that appears into your browser on your local machine.
If you want to find an unused port number, one option is to use trial and error, selecting a random port between 10000 and 49151. If you attempt to use an already-used port, you will see error messages such as "Address already in use" or "cannot listen to port". You may also search for an unused port number by examining the output of the following commands and making sure to choose one that is not listed:
[user123@home ~] ssh user123@access.cims.nyu.edu
[user123@access ~] netstat -4tuln | grep LISTEN
The following line would indicate, for example, that port 12345 is already in use.
tcp 0 0 127.0.0.1:12345 0.0.0.0:* LISTEN
Package Requests
If you don't want to manage your own packages and believe that you know of a package that will be of benefit to the CIMS community, please let us know at helpdesk@cims.nyu.edu.