Python in Red Hat Enterprise Linux 9 (RHEL9)

Please note that the asterisk (*) refers to any available minor version of the major version (Python 2 and Python 3 are major). For example, in python 3.* , the * could be substituted for 12 for Python 3.12, and so on.

Related Pages: 

 

There are several options for using Python in Rat Hat Enterprise Linux 9 (RHEL9).  In most cases, you'll want to use one of our custom-built installations, located at the following path:
/usr/local/bin/python3.*

Each of these installations is a symlink to their full installation trees in /usr/local/stow/.  For example:

/usr/local/bin/python3.* --> ../stow/python-3.*/bin/python3.*

Alternatively, you can load each version of Python's respective module (see Module System):

module load python-3.*

To understand what effect running a module file has, see it's corresponding module file in /usr/local/etc/modulefiles/.

We've configured our RHEL9 system to be compatible with either usage of /usr/local/bin symlinks, or with module files.  You may use whichever you prefer or find more convenient.

The most significant advantage to using a module file is that the command "python" becomes associated with the appropriate binary.  For example, /usr/local/bin contains "python#.#" for each specific version.  But if you have pre-existing code that uses, for example, "#!/usr/bin/env python", you will want "python" to point to the right version.

Before loading a module file, observe which Python interpreter is currently active: 

[user123@box999 ~]# which python
/usr/bin/python

Now load the module for Python 3.*, and try again (note for python 3, you will need to use the "python3" command):

[user123@box999 ~]# module load python-3.*
[user123@box999 ~]# which python3
/usr/local/stow/python-3.*/bin/python
[user123@box999 ~]# python3 -V
Python 3.*.*

Please note that on Courant RHEL9, /usr/bin/python and /usr/bin/python3 refer to the system Python. Loading a module updates your PATH so python/python3 use the selected CIMS Python instead. 

What about /usr/bin/python?

RHEL9 includes a system Python interpreter located at /usr/bin/python (currently Python 3.9.x). This interpreter is primarily intended for operating system components and system utilities. This interpreter's packages are managed through RPM, and are generally selected for long-term stability rather than the latest feature releases.

On RHEL9, we provide parallel Python installations rather than modifying the system Python environment (as we did so with previous operating systems like RHEL6). 

While you are free to use /usr/bin/python, we have found that most users prefer our Courant-provided Python installations (via /usr/local/bin or the module system) for development work. These versions of Python include a broader and more up-to-date collection of commonly used packages, while also allowing for greater flexibility for installing additional packages. 

CIMS Package Installations

We have made an effort to pre-populate each of our versions of Python with the most commonly used packages such as numpy, scipy, etc.  There are two ways to get a list of what we have installed for each version: either use the links in /usr/local/bin, or load the appropriate module and use "pip":

[user123@box999 ~]# pip3.* freeze
ad3==2.0.2
AddOns==0.7
amqp==1.4.6
... 

[user123@box999 ~]# module load python-3.*
[user123@box999 ~]# pip3 freeze
ad3==2.0.2
AddOns==0.7
amqp==1.4.6
... 

User Package Installations

Python packages are usually very lightweight, and can be installed to a user's home directory with very little effort.  Any user has the freedom to do package installations on a per-user level without needing root privileges.

Either use pip, or download and install from source.

[user123@box999 ~]# pip3.* install beautifulsoup4 --user -I

Note that the capital I argument (which stands for "Ignore") is necessary if you attempt to install your own package for a package that we already have some version of.  -I says "ignore the fact that this package is already installed".

At this point, the package you've installed will be located at /home/user123/.local/lib/python3.*/site-packages.

Conversely, if you know where to obtain the source for a Python package that is not available from PyPI (pip's repository), then you can download it, extract it, and install it in a similar manner:

[user123@box999 ~]# wget https://pypi.python.org/packages/source/j/jycat/jycat-0.0.17.tar.gz#md5=8a9bf3efdbc8a59f4dcdd2f08423c52d
[user123@box999 ~]# tar -xvf jycat-0.0.17.tar.gz 
[user123@box999 ~]# cd jycat-0.0.17/
[user123@box999 ~]# python3.* setup.py install --user

In this example, jycat will be installed to /home/user123/.local/lib/python3.6/site-packages.

For packages installed in this manner, you should be able to immediately begin importing and using them without any other modifications:

[user123@box999 ~]# python3.* -c 'import jycat'

Python Virtual Environments (venv)

An even better practice than installing your own packages directly into your home directory is to isolate each project into its own Python virtual environment, using Python's built-in venv module.

If you are unfamiliar with virutal envinroments, we would highly recommend searching for "why use Python virtual environments" for background and best practices.

The short version: individual Python projects often depend on specific package versions and dependencies. For example, let's say you're writing a program using Package ABC from our global CIMS Python installation. At some point in the future, however, we update Package ABC. This updated version of Package ABC  happens to either change or remove existing functionality, resulting in your program no longer functioning properly. Alternatively, if you were to isolate each Python project into its own virtual environment, then you can control exactly which package versions are installed and utilized. 

Creating a Virtual Environment on RHEL9

To get started with creating a virtual environment, first decide which version of Python you would like to use. 

If you're using a CIMS-provided Python module, you can load it with the following command: 

[user123@box999 ~]# module load python-3.12

Next, create a new virtual environment, using the venv command. In this example we are making a new virtual environment named "new_proj1": 

[user123@box999 ~]# cd /scratch/python_projects
[user123@box999 python_projects]# python3 -m venv new_proj1

Just like that, we have craeted a clean Python environment containing only the standard library and built-in packages.

To begin using this new vitual environment, you need to activate it: 

[user123@box999 python_projects]# source new_proj1/bin/activate

Once your virtual environment has been activated, "python", "pip", and related tools will all now point to your new Python virtual environment: 

(new_proj1)[user123@box999 python_projects]# which pip
/scratch/python_projects/new_proj1/bin/pip
(new_proj1)[user123@box999 python_projects]# which python
/scratch/python_projects/new_proj1/bin/python

Furthermore, you can also confirm that no additional packages are installed with pip freeze. 
 
You now have a completed isolated environment, and can install exactly the package versions your project requires -- without the need for root privileges. If your project relies on a specific, older version of a package, then simply install the old version as needed.  This, for example, will install beautifulsoup4 4.1.0 as opposed to the more recent 4.4.0 version:

(new_proj1)[user123@box999 python_projects]# pip install beautifulsoup4==4.1.0

When you're done using your virtual environment, make sure to deactivate it accordingly: 

(new_proj1)[user123@box999 python_projects]# deactivate
 

Jupyter Notebook / JupyterLab

If you wish to run a Jupyter on a CIMS RHEL9 systems, you may do so without any special setup by first loading a Python module:

[user123@box999 ~] module load python-3.12

Then lauch either Jupyter lab or jupyter notebook, depending on your interface preferences: 
[user123@box999 ~] jupyter notebook

[user123@box999 ~] jupyter lab

If you want to use Jupyter with your own choice of project-specific packages, first create and activate a virtual enivornment as described in the previous section:

[user123@box999 ~] module load python-3.12

[user123@box999 ~] python3 -m venv my_env 

[user123@box999 ~] source my_env/bin/activate

Once your vitual environment has been activated, make sure to then install Jupyter inside the virtual environment itself:

[user123@box999 ~] pip install --upgrade pip

[user123@box999 ~] pip install jupyterlab 

Finally, launch Jupyter from within the activated environment: 

[user123@box999 ~] jupyter lab

By launching Jupyter in this manner, you are ensuring that Jupyter uses only the packages that have already been installed within this virtual environment. 

 

Running Jupyter Remotely (SSH Port Forwarding) 

To run Jupyter on a Coourant computer server remotely and access it from your local machine, you may use SSH port forwarding. 

Choose an unused port number between 10000 and 49151 (for example, 12345). 

From your local machine, ssh first to the Courant access server:

[user123@home ~] ssh -4 -L 12345:localhost:12345 user123@access.cims.nyu.edu

Once connected to access, you can then ssh to any other Courant compute server. For example, here is how you would connect to crunchy5: 

[user123@access ~] ssh -4 -L 12345:localhost:12345  crunchy5.cims.nyu.edu

Once on the compute server, load up the Python module of your choice, and then launch Jupyter: 


[user123@crunchy5 ~] module load python-3.12
[user123@crunchy5 ~] jupyter lab --no-browser --port 12345

At this point, Jupyter will display a URL containing a security token (beginning with http://localhost:12345/?token=...). Copy this provided URL into a web browser on your local machine in order to access the notebook. 

Finding an Unused Port

If you want to find an unused port number, one option is to use trial and error, selecting a random port between 10000 and 49151. If you attempt to use a port that is already in use, you may see errors such as "Address already in use" or "cannot listen to port."

To check which ports are currently listening on a remote machine, you may also search for an unused port number by examining the output of the following commands, making sure to choose a port number that is not listed:

[user123@home ~] ssh user123@access.cims.nyu.edu
[user123@access ~] netstat -4tuln | grep LISTEN

The output of these commands will provide a list of port numbers, so you would just need to select a port number that does not appear in the output. The following line would indicate, for example, that port 12345 is already in use.

tcp 0 0 127.0.0.1:12345 0.0.0.0:* LISTEN

Package Requests

If you don't want to manage your own packages and believe that you know of a package that will be of benefit to the CIMS community, please let us know at helpdesk@cims.nyu.edu.