Anaconda Python module with ready set of scientific packages

The Python Lmod modules provide a convenient environment for working with different Python versions (3.9.18, 3.10.13, 3.11.7), each equipped with pre-installed packages. These modules come pre-configured with both conda and pip environments, allowing users to seamlessly install additional packages using pip.

Loading Python Lmod Modules

To utilize the Python Lmod modules, load the desired version (for instance 3.10.3) with the following command:

module load fg-python/3.10.13

Package Management

• Listing Installed Packages: To view the packages installed in the current Python environment, use:

  pip list

• Checking Package Availability: Confirm if a specific package is available in the environment:

  pip show <package name>

• Interactive Python Session: Open an interactive Python session effortlessly:

  python

• Installing Additional Packages: Install a new package using pip:

  pip install --trusted-host pypiserver.app.finngen.fi -i https://pypiserver.app.finngen.fi/simple/ <package name>

Jupyter Integration

After loading the Python Lmod module, you can easily launch Jupyter Lab for a more interactive environment:

jupyter lab

Customizing Jupyter Environment: You can bind mount directories you need to serve notebooks from by exporting the SINGULARITY_COMMAND_OPTS. For example, to bind mount the finngen/red directory, you could use the following command:

export SINGULARITY_COMMAND_OPTS="-B /finngen/red"

Afterwards, you could start Jupyter Lab from that location by doing:

jupyter lab --notebook-dir="/finngen/red"

Troubleshooting: If, after loading a Python module and installing packages as described above, you find that you cannot load the installed packages in your Jupyter notebook, you may need to install an IPython kernel for the activated module. This kernel can then be used in your notebooks, allowing you to load the installed packages.

To create the kernel, run the following command:

python -m ipykernel install [--user] [--name <machine-readable-name>] [--display-name <"User Friendly Name">]

For example:

python -m ipykernel install --user --name python310

This will create a new kernel named python310 that you can select in your Jupyter notebooks. This kernel should now have access to all the packages you installed inside the module you loaded. For more information, please visit the IPython documentation on kernel installation.

Click here to visit the site with the full Jupyter official documentation.

BigQuery Integration

The following BigQuery packages are pre-installed:

• Google Cloud BigQuery

• Pandas_GBQ

  Example command to check BigQuery data in the terminal:

  python -c 'from google.cloud import bigquery; import pandas_gbq; client = bigquery.Client(); query = """ SELECT FINNGENID FROM finngen-production-library.sandbox_tools_r10.finngen_r10_minimum_v1 LIMIT 10 """; df = pandas_gbq.read_gbq(query, project_id = client.project); print(df) '

  The above command retrieves 10 FINNGENIDs from the DF10 release minimum table in BigQuery.

Visualization Packages

The Python Lmod modules include the following visualization packages:

• Plotly

• Matplotlib

• Seaborn

• UpSetPlot

More visualization packages can be added based on your requirements.

Working in Virtual Environments

To activate virtual environments, shell into the container using alias commands that you can see when you do:

module spider <module name>

To create a virtual environment inside the fg-python/3.10.13 Lmod module for instance, follow these steps:

1. Load the Python module:

   module load fg-python/3.10.13

2. Shell into the environment:

   fg-python-shell

3. Create a virtual environment named my-venv:

   python3 -m venv --system-site-packages my-venv

4. Activate the virtual environment:

   source my-venv/bin/activate

5. Install packages within the virtual environment:

   pip install --trusted-host pypiserver.app.finngen.fi -i https://pypiserver.app.finngen.fi/simple/ <package name>