diff --git a/content/applications/user_software/using_anaconda_package_manager.md b/content/applications/user_software/using_anaconda_package_manager.md index b8851b4e3753d684b1058813cf24ccda96fa6fd1..b8ab2e7b3de5df9b3a2a8abb748b7067a93bf89d 100644 --- a/content/applications/user_software/using_anaconda_package_manager.md +++ b/content/applications/user_software/using_anaconda_package_manager.md @@ -21,9 +21,12 @@ straightforward. - [Creating custom Anaconda Environments](#creating-custom-anaconda-environments) - [Using /common for environments](#using-common-for-environments) - [Adding and Removing Packages from an Existing Environment](#adding-and-removing-packages-from-an-existing-environment) +- [Moving and Recreating Existing Environment](#moving-and-recreating-existing-environment) +- [Remove Unused Anaconda Packages and Caches](#remove-unused-anaconda-packages-and-caches) - [Creating custom GPU Anaconda Environment](#creating-custom-gpu-anaconda-environment) - [Creating custom MPI Anaconda Environment](#creating-custom-mpi-anaconda-environment) - [Using an Anaconda Environment in a Jupyter Notebook](#using-an-anaconda-environment-in-a-jupyter-notebook) +- [Using Mamba](#using-mamba) ### Using Anaconda @@ -203,6 +206,71 @@ conda remove -n mynumpy --all {{< /highlight >}} {{% /panel %}} +### Moving and Recreating Existing Environment + +Sometimes conda environments need to be moved (e.g., from `$HOME` to `$COMMON` in order to reduce used space in `$HOME`) +or recreated (e.g., when shared with someone). This is done using `environment.yml` file as shown below. + +{{% panel theme="info" header="Activate the conda environment to export" %}} +{{< highlight bash >}} +conda activate mynumpy +{{< /highlight >}} +{{% /panel %}} + +Then export the active conda environment to file `environment.yml`. + +{{% panel theme="info" header="Export conda environment" %}} +{{< highlight bash >}} +conda env export > environment.yml +{{< /highlight >}} +{{% /panel %}} + +Next, deactivate the conda environment. + +{{% panel theme="info" header="Exit current environment" %}} +{{< highlight bash >}} +conda deactivate +{{< /highlight >}} +{{% /panel %}} + +The file `environment.yml` contains both `pip` and `conda` packages installed in the activated environment. +This file can now be shared or used to recreate the conda environment elsewhere. + +The exported environment can be recreated in `$COMMON` with: + +{{% panel theme="info" header="Recreate conda environment in $COMMON" %}} +{{< highlight bash >}} +conda env create -p $COMMON/mynumpy -f environment.yml +{{< /highlight >}} +{{% /panel %}} + +After the conda environment has been exported or recreated, if needed, the original conda environment can be removed. + +{{% panel theme="info" header="Remove conda environment" %}} +{{< highlight bash >}} +conda remove -n mynumpy --all +{{< /highlight >}} +{{% /panel %}} + +### Remove Unused Anaconda Packages and Caches + +By default, conda environments are installed in the user’s home directory at `~/.conda/envs`. +conda caches and package tarballs are stored in `~/.conda/` as well. +For larger or many conda environments, the size of the directory `~/.conda/` can easily reach the `$HOME` space quota limit of 20GB per user. + +In addition to [Moving and Recreating Existing Environment](#moving-and-recreating-existing-environment), one can remove unused conda packages and caches. + +{{% panel theme="info" header="Remove unused conda packages and caches" %}} +{{< highlight bash >}} +conda clean --all +{{< /highlight >}} +{{% /panel %}} + +{{% notice info %}} +Please note that this command will only remove index cache, and unused cache packages and tarballs and will not +affect nor break the current conda environments you have. +{{% /notice %}} + ### Creating Custom GPU Anaconda Environment We provide GPU versions of various frameworks such as `tensorflow`, `keras`, `theano`, via [modules](../../modules). @@ -357,3 +425,22 @@ Jupyter Notebook. To do so, follow the steps below, replacing corner. {{< figure src="/images/24151931.png" height="400" class="img-border">}} +### Using Mamba +[Mamba](https://mamba.readthedocs.io/en/latest/) is an alternative to Conda that is in general faster and performs better at resolving dependencies in conda environments. +Mamba is available as part of the `anaconda` modules on Swan. + +Mamba can be used by simply replacing `conda` with `mamba` in all `conda` commands provided here. + +{{% panel theme="info" header="Load the Anaconda module to start using Mamba" %}} +{{< highlight bash >}} +module load anaconda +{{< /highlight >}} +{{% /panel %}} + +To create a new environment called 'mynumpy' and install NumPy version 1.17, along with any required dependencies, the command is: + +{{% panel theme="info" header="Create a new environment by providing a name and package specification" %}} +{{< highlight bash >}} +mamba create -n mynumpy numpy=1.17 +{{< /highlight >}} +{{% /panel %}}