Topics Map > Computing > linux
Topics Map > Computing > research
Environment Modules
- What is Environment Modules?
- Requirements
- How To use EM?
- Advanced Usage
- Other Things to Note
- Summary of Demonstrated Commands
- A Simple Example
- Frequently Asked Questions
- Where to Get More Information
- Requesting Additional Software
What is Environment Modules?
Environment Modules(EM) is a tool that simplifies modifying your shell environment. Each module defines the changes to be made to the user's shell environment(PATHs, environment variables, aliases, etc). Modules can be loaded and unloaded to enable and disable software packages or specific versions of software packages.
Through EM you almost instantly switch between different versions of a software package or library. You can have multiple modules loaded at any one time. In fact, a module may depend on another module and load it automatically. You can also define "collections" of modules and save them so you can later load the entire collection.
Requirements
The only requirement is that you use the Bash shell which the default for most users. If you are using CSH, Tcsh, or another shell please contact the Math Department IT staff and we'll work with you to determine the best way forward.
How To use EM?
The commands: ml and module
The ml command acts as a shortcut to the module command thus it supports all the command line switches and module sub-commands that are supported by module. We'll use the ml command in all following examples as it is shorter and supports a few additional features.
Available Modules
To list all available modules, simply run ml avail:
youser@server:~$ ml avail
-------- /usr/local/modules/spack/linux-ubuntu22.04-x86_64_v4 --------
cuda/11.8.0-isnefyp cuda/12.2.1-zl7h6i4
-------- /usr/local/modules/spack/linux-ubuntu22.04-x86_64_v3 --------
cmake/3.27.9-rnw2hs4
hdf5/1.12.2-intel-oneapi-mpi-2021.10.0-7mkj6fo
hdf5/1.12.2-openmpi-4.1.6-ld6t6q6
intel-mpi-benchmarks/2021.3-intel-oneapi-mpi-2021.10.0-x2j7lky
intel-oneapi-advisor/2023.2.0-zoniz6v
intel-oneapi-ccl/2021.10.0-zrwnlfc
intel-oneapi-compilers/2023.2.1-5kjoz6w
intel-oneapi-dal/2023.2.0-djgwkoa
intel-oneapi-dnn/2023.2.0-xrrpilw
intel-oneapi-dpct/2023.2.0-rccee66
intel-oneapi-dpl/2022.2.0-6dblupq
intel-oneapi-inspector/2023.2.0-sum2iln
intel-oneapi-ipp/2021.9.0-jdp5om7
intel-oneapi-ippcp/2021.8.0-s64hsrv
intel-oneapi-itac/2021.10.0-dna22g4
intel-oneapi-mkl/2024.0.0-ssypz4h
intel-oneapi-mpi/2021.10.0-wc44zcn
intel-oneapi-tbb/2021.10.0-zogj5qg
intel-oneapi-vtune/2023.2.0-tearzzx
llvm/11.0.1-tz55ulq
llvm/18.1.3-ot4cwl2
netcdf-c/4.8.1-openmpi-4.1.6-3aaesni
netcdf-cxx4/4.3.1-openmpi-4.1.6-qngnhkh
netcdf-fortran/4.5.4-openmpi-4.1.6-2elfh3k
opencl-headers/3.0-zu5gojz
openmpi/4.1.6-kwr5cw7
-------- /usr/local/modules/spack/linux-ubuntu22.04-x86_64_v2 --------
cmake/3.27.9-7hjre6l
hdf5/1.12.2-intel-oneapi-mpi-2021.12.1-zydlnpy
hdf5/1.12.2-openmpi-4.1.6-3d4g3le
intel-mpi-benchmarks/2021.7-intel-oneapi-mpi-2021.12.1-umbl3y3
intel-oneapi-advisor/2024.1.0-vujvcf2
intel-oneapi-ccl/2021.12.0-eqtxkkl
intel-oneapi-compilers/2024.1.0-fcfvbip
intel-oneapi-dal/2024.2.0-ew72mav
intel-oneapi-dnn/2024.1.1-mhiknl5
intel-oneapi-dpct/2024.1.0-t4qh6a3
intel-oneapi-dpl/2022.5.0-iwapqg4
intel-oneapi-inspector/2024.1.0-flpjdy3
intel-oneapi-ipp/2021.11.0-6jpjwvq
intel-oneapi-ippcp/2021.11.0-zrstrey
intel-oneapi-itac/2022.1.0-57in7fn
intel-oneapi-mkl/2024.0.0-r3zxuyu
intel-oneapi-mpi/2021.12.1-rrlz45q
intel-oneapi-tbb/2021.12.0-wbazjso
intel-oneapi-vtune/2024.1.0-w3f5wtc
llvm/11.0.1-cwvc2hm
llvm/18.1.3-ga4boei
netcdf-c/4.8.1-openmpi-5.0.3-zbdxzmn
netcdf-cxx4/4.3.1-openmpi-5.0.3-ohzlxi2
netcdf-fortran/4.5.4-openmpi-5.0.3-6qeufom
openmpi/5.0.3-dqx4r63
openssl/3.0.2-zo23scx
---------------------- /usr/local/modules/main -----------------------
bertini/1.6 magma/2.28-2-auto magma/latest-auto
bertini/2.0-alpha6 magma/2.28-2-avx maple/2021
bertini/2.0-HEAD magma/2.28-2-avx2 maple/2023
dedalus/3.0.2 magma/2.28-4-auto maple/2024
dedalus/latest magma/2.28-4-avx mathematica/13.2
gap/4.12.0 magma/2.28-4-avx2 mathematica/13.3
gap/4.12.2 magma/2.28-4-avx2-cuda11 mathematica/14.1
gap/4.13.1 magma/2.28-6-auto matlab/2023b
julia/1.6.7 magma/2.28-6-avx matlab/2024a
julia/1.9.4 magma/2.28-6-avx2 phc/2.4.74
julia/1.10.1 magma/2.28-6-avx2-cuda11 phc/2.4.89
julia/1.10.4 magma/2.28-11-auto phc/2.4.90
julia/latest magma/2.28-11-avx sage/10.0
julia/lts magma/2.28-11-avx2 sage/10.3
lean/4.8.0 magma/2.28-11-avx2-cuda11
------------------- /usr/local/Modules/modulefiles -------------------
dot module-git module-info modules null use.own
Tip: You can abbreviate 'avail' as 'av'
Deciphering module lists
Modules are grouped into folders by the operating system and CPU architecture they are for. Example:
-------- /usr/local/modules/spack/linux-ubuntu22.04-x86_64_v3 --------
cmake/3.27.9-rnw2hs4
hdf5/1.12.2-intel-oneapi-mpi-2021.10.0-7mkj6fo
hdf5/1.12.2-openmpi-4.1.6-ld6t6q6
intel-mpi-benchmarks/2021.3-intel-oneapi-mpi-2021.10.0-x2j7lky
...
In the above example the modules are for:
- Linux
- of the Ubuntu type at version 22.04
- running on an x86 64 bit processor
- of the v3 architecture type
Other modules you'll see have their hardware requirements in their name. This is currently limited to our Magma programs:
magma/2.28-6-auto - this one auto-detects which one of the following to use
magma/2.28-2-avx - For CPUs that support the AVX SIMD instruction
magma/2.28-2-avx2 - For CPUs that support the AVX2 SIMD instruction
magma/2.28-2-avx2-cuda11 - For CPUs that support the AVX2 SIMD instruction and systems with GPUs and CUDA11 installed
And further still, modules can list the compiler and libraries that were used when they were compiled. For example "intel-mpi-benchmarks/2021.3-intel-oneapi-mpi-2021.10.0-x2j7lky" is:
- The intel-mpi-benchmarks software package
- Version 2021.3
- Linked with the intel-oneapi-mpi library version 2021.10.0
- The 7 character string at the end of the module names above are the first 7 characters of a hash generated from the parameters used to compile the package. It just ensures uniqueness. If the hash changes then you know the package has been updated in some way.
Loading a Module and Listing Loaded Modules
To load a module just list it's full name (name/version) on the ml command. Multiple modules can be loaded this way:
youser@server:~$ ml julia/1.10.1 boost/1.84.0
To list loaded modules just run ml with no arguments:
youser@server:~$ ml
Currently Loaded Modulefiles:
1) julia/1.10.1 2) boost/1.84.0
Module Information
To get some information on a module
youser@server:~$ ml whatis julia/latest
-------------------------------------- /usr/local/modules/main --------------------------------------
julia/latest: Julia v1.10.1
More information may be available using the help command:
youser@server:~$ ml help bertini/2.0-HEAD
-------------------------------------------------------------------
Module Specific Help for /usr/local/modules/main/bertini/2.0-HEAD:
Bertini 2.0-HEAD
Latest code from the B2 GIT repository
Not recommended for general use.
-------------------------------------------------------------------
Tip: You can abbreviate 'whatis' to 'wh'
Tip: At any time you can use the TAB key to auto-complete the ml command. Double-tapping TAB will show you the current possible completions. Try: ml magma<TAB><TAB> <TAB><TAB>
Unloading a Module
youser@server:~$ ml -boost/1.84.0
youser@server:~$ ml
Currently Loaded Modulefiles:
1) julia/1.10.1
Swapping a module
To switch modules in one command:
youser@server:~$ ml swap julia/1.10.1 julia/lts
youser@server:~$ ml
Currently Loaded Modulefiles:
1) julia/lts
Or in this case we are swapping a module of the same name for a different version so just give it the new version:
youser@server:~$ ml swap julia/lts
Removing all modules
youser@server:~$ ml purge
Fixing Problems
There may be times when your shell environment gets changed in a way that causes problems. For example an environment variable set by loading a module is changed and this causes the module to not function properly. You can reload all modules using the reload command to fix this:
youser@server:~$ echo $GAP_DIR
youser@server:~$ ml gap/4.12.0
youser@server:~$ echo $GAP_DIR
/usr/local/Gap/gap-4.12.0
youser@server:~$ unset GAP_DIR # We just broke it!
youser@server:~$ echo $GAP_DIR # Oh noes!
youser@server:~$ ml reload # reload to the rescue
youser@server:~$ echo $GAP_DIR # Yay!
/usr/local/Gap/gap-4.12.0
Advanced Usage
Creating Collections for Later Use
Lets say you have some projects that each use a number of modules. In order to set up your shell environment to work in one project or to switch between them projects requires some EM commands that you need to remember. You can bundle the requirements into "collections" and recall those with a single command.
Create a Collection
youser@server:~$ ml boost/1.84.0 fftw/3.3.10-avx2 intel-oneapi julia/1.10.1
youser@server:~$ ml
Currently Loaded Modulefiles:
1) boost/1.84.0 2) fftw/3.3.10-avx2 3) intel-oneapi 4) julia/1.10.1
youser@server:~$ ml save project1
youser@server:~$ ml purge
youser@server:~$ ml
Load a Collection
youser@server:~$ ml restore project1
Loading boost/1.84.0
Loading fftw/3.3.10-avx2
Loading intel-oneapi
Loading julia/1.10.1
List Collections
youser@server:~$ ml savelist
Named collection list:
1) project1
Delete a Collection
youser@server:~$ ml saverm project1
youser@server:~$ ml savelist
No named collection.
Using Modules in Scripts
If you want to activate modules in Bash scripts you can just add the required commands to the script. Take note: If you load modules on the command line and then run a shell script those modules will continue to be loaded for that script as your shell environment at the command line is inherited by the shell script.
#!/bin/bash
ml maple/2023
# The rest of the script...
Other Things to Note
- You can only load one version of a module at a time
- Some modules cannot be loaded at the same time because their software conflicts. This is usually due to Conda-based installations.
- Some modules will work only on certain servers because there are CPU and/or GPU requirements. These requirements are usually made clear in the module name.
Summary of Demonstrated Commands
| Command | Use |
| ml | List loaded modules |
| ml avail | List available modules |
| ml <module name/version> ... | Load modules |
| ml -<module name/version> ... | Unload modules |
| ml whatis <module name/version> | Display brief module information |
| ml help <module name/version> | Display module help information |
| ml swap <module name> OR ml swap <module name/version> <module name/version> |
Swap a loaded module with another one |
| ml purge | Unload all loaded modules |
| ml reload | Reload all modules |
| ml save <collection name> | Create a collection of modules |
| ml savelist | Lists saved collections |
| ml restore <collection name> | Load a collection of modules |
| ml saverm <collection name> | Delete a collection |
A Simple Example
You have done some work with Magma 2.28-2-avx2 previously and after modifying your code you tested it again on 2.28-11-avx2 and you get different results. After verifying that your code should be correct you wanted to test it on the version of Magma you first used and on an intermediate version. You would do this:
$ ml magma/2.28-2-avx2
$ magma your-code.magma
# results here
$ ml swap magma magma/2.28-6-avx2
$ magma your-code.magma
# results here
Frequently Asked Questions
- When will Environment Modules(EM) be available?
- It's available right now.
- Why should I use EM?
- If you are going to be working on the Math Department's research servers after summer of 2025 EM will be the only (easy) means of using some software.
- Can I create my own modules for software in my home folder?
- Yes! While it is an advanced topic you can look to this guide for a starting point: https://researchcomputing.princeton.edu/support/knowledge-base/custom-modules
Where to Get More Information
Command help: module help
Manual page(advanced): man module
Documentation:
Web pages:
- Github page: https://github.com/cea-hpc/modules
- Home page: https://modules.sourceforge.net/
- All documentation: https://modules.readthedocs.io/en/latest/
Requesting Additional Software
Contact the Math Department IT staff if there is any software(including libraries, build tools, languages, etc) you would like installed on the servers: mathitstaff@g-groups.wisc.edu