Skip to content
Snippets Groups Projects

Update conda notes

Merged Natasha Pavlovikj requested to merge conda_cleanup into master
1 file
+ 87
0
Compare changes
  • Side-by-side
  • Inline
@@ -21,9 +21,12 @@ straightforward.
@@ -21,9 +21,12 @@ straightforward.
- [Creating custom Anaconda Environments](#creating-custom-anaconda-environments)
- [Creating custom Anaconda Environments](#creating-custom-anaconda-environments)
- [Using /common for environments](#using-common-for-environments)
- [Using /common for environments](#using-common-for-environments)
- [Adding and Removing Packages from an Existing Environment](#adding-and-removing-packages-from-an-existing-environment)
- [Adding and Removing Packages from an Existing Environment](#adding-and-removing-packages-from-an-existing-environment)
 
- [Moving and Recreating Existing Environment](#moving-and-recreating-existing-environment)
 
- [Remove Unused Anaconda Packages and Caches](#remove-unused-anaconda-packages-and-caches)
- [Creating custom GPU Anaconda Environment](#creating-custom-gpu-anaconda-environment)
- [Creating custom GPU Anaconda Environment](#creating-custom-gpu-anaconda-environment)
- [Creating custom MPI Anaconda Environment](#creating-custom-mpi-anaconda-environment)
- [Creating custom MPI Anaconda Environment](#creating-custom-mpi-anaconda-environment)
- [Using an Anaconda Environment in a Jupyter Notebook](#using-an-anaconda-environment-in-a-jupyter-notebook)
- [Using an Anaconda Environment in a Jupyter Notebook](#using-an-anaconda-environment-in-a-jupyter-notebook)
 
- [Using Mamba](#using-mamba)
### Using Anaconda
### Using Anaconda
@@ -203,6 +206,71 @@ conda remove -n mynumpy --all
@@ -203,6 +206,71 @@ conda remove -n mynumpy --all
{{< /highlight >}}
{{< /highlight >}}
{{% /panel %}}
{{% /panel %}}
 
### Moving and Recreating Existing Environment
 
 
Sometimes conda environments need to be moved (e.g., from `$HOME` to `$COMMON` in order to reduce used space in `$HOME`)
 
or recreated (e.g., when shared with someone). This is done using `environment.yml` file as shown below.
 
 
{{% panel theme="info" header="Activate the conda environment to export" %}}
 
{{< highlight bash >}}
 
conda activate mynumpy
 
{{< /highlight >}}
 
{{% /panel %}}
 
 
Then export the active conda environment to file `environment.yml`.
 
 
{{% panel theme="info" header="Export conda environment" %}}
 
{{< highlight bash >}}
 
conda env export > environment.yml
 
{{< /highlight >}}
 
{{% /panel %}}
 
 
Next, deactivate the conda environment.
 
 
{{% panel theme="info" header="Exit current environment" %}}
 
{{< highlight bash >}}
 
conda deactivate
 
{{< /highlight >}}
 
{{% /panel %}}
 
 
The file `environment.yml` contains both `pip` and `conda` packages installed in the activated environment.
 
This file can now be shared or used to recreate the conda environment elsewhere.
 
 
The exported environment can be recreated in `$COMMON` with:
 
 
{{% panel theme="info" header="Recreate conda environment in $COMMON" %}}
 
{{< highlight bash >}}
 
conda env create -p $COMMON/mynumpy -f environment.yml
 
{{< /highlight >}}
 
{{% /panel %}}
 
 
After the conda environment has been exported or recreated, if needed, the original conda environment can be removed.
 
 
{{% panel theme="info" header="Remove conda environment" %}}
 
{{< highlight bash >}}
 
conda remove -n mynumpy --all
 
{{< /highlight >}}
 
{{% /panel %}}
 
 
### Remove Unused Anaconda Packages and Caches
 
 
By default, conda environments are installed in the user’s home directory at `~/.conda/envs`.
 
conda caches and package tarballs are stored in `~/.conda/` as well.
 
For larger or many conda environments, the size of the directory `~/.conda/` can easily reach the `$HOME` space quota limit of 20GB per user.
 
 
In addition to [Moving and Recreating Existing Environment](#moving-and-recreating-existing-environment), one can remove unused conda packages and caches.
 
 
{{% panel theme="info" header="Remove unused conda packages and caches" %}}
 
{{< highlight bash >}}
 
conda clean --all
 
{{< /highlight >}}
 
{{% /panel %}}
 
 
{{% notice info %}}
 
Please note that this command will only remove index cache, and unused cache packages and tarballs and will not
 
affect nor break the current conda environments you have.
 
{{% /notice %}}
 
### Creating Custom GPU Anaconda Environment
### Creating Custom GPU Anaconda Environment
We provide GPU versions of various frameworks such as `tensorflow`, `keras`, `theano`, via [modules](../../modules).
We provide GPU versions of various frameworks such as `tensorflow`, `keras`, `theano`, via [modules](../../modules).
@@ -357,3 +425,22 @@ Jupyter Notebook. To do so, follow the steps below, replacing
@@ -357,3 +425,22 @@ Jupyter Notebook. To do so, follow the steps below, replacing
corner.
corner.
{{< figure src="/images/24151931.png" height="400" class="img-border">}}
{{< figure src="/images/24151931.png" height="400" class="img-border">}}
 
### Using Mamba
 
[Mamba](https://mamba.readthedocs.io/en/latest/) is an alternative to Conda that is in general faster and performs better at resolving dependencies in conda environments.
 
Mamba is available as part of the `anaconda` modules on Swan.
 
 
Mamba can be used by simply replacing `conda` with `mamba` in all `conda` commands provided here.
 
 
{{% panel theme="info" header="Load the Anaconda module to start using Mamba" %}}
 
{{< highlight bash >}}
 
module load anaconda
 
{{< /highlight >}}
 
{{% /panel %}}
 
 
To create a new environment called 'mynumpy' and install NumPy version 1.17, along with any required dependencies, the command is:
 
 
{{% panel theme="info" header="Create a new environment by providing a name and package specification" %}}
 
{{< highlight bash >}}
 
mamba create -n mynumpy numpy=1.17
 
{{< /highlight >}}
 
{{% /panel %}}
Loading