using_anaconda_package_manager.md 10.5 KB
Newer Older
Adam Caprez's avatar
Adam Caprez committed
1
2
3
4
+++
title = "Using Anaconda Package Manager"
description = "How to use the Anaconda Package Manager on HCC resources."
+++
5

Adam Caprez's avatar
Adam Caprez committed
6
7
[Anaconda](https://www.anaconda.com/what-is-anaconda),
from [Anaconda, Inc](https://www.anaconda.com)
8
9
10
11
is a completely free enterprise-ready distribution for large-scale data
processing, predictive analytics, and scientific computing. It includes
over 195 of the most popular Python packages for science, math,
engineering, and data analysis. **It also offers the ability to easily
Adam Caprez's avatar
Adam Caprez committed
12
create custom _environments_ by mixing and matching different versions
13
14
15
16
17
of Python and/or R and other packages into isolated environments that
individual users are free to create.**  Anaconda includes the `conda`
package and environment manager to make managing these environments
straightforward.

Adam Caprez's avatar
Adam Caprez committed
18
- [Using Anaconda](#using-anaconda)
Natasha Pavlovikj's avatar
i    
Natasha Pavlovikj committed
19
20
- [Creating custom Anaconda Environment](#creating-custom-anaconda-environment)
- [Creating custom GPU Anaconda Environment](#creating-custom-gpu-anaconda-environment)
Adam Caprez's avatar
Adam Caprez committed
21
22
- [Adding Packages to an Existing Environment](#adding-packages-to-an-existing-environment)
- [Using an Anaconda Environment in a Jupyter Notebook on Crane](#using-an-anaconda-environment-in-a-jupyter-notebook-on-crane)
23

Adam Caprez's avatar
Adam Caprez committed
24
### Using Anaconda
25
26
27
28
29

While the standard methods of installing packages via `pip`
and `easy_install` work with Anaconda, the preferred method is using
the `conda` command.  

Adam Caprez's avatar
Adam Caprez committed
30
{{% notice info %}}
31
Full documentation on using Conda is available
Adam Caprez's avatar
Adam Caprez committed
32
at http://conda.pydata.org/docs/
33

Adam Caprez's avatar
Adam Caprez committed
34
35
A [cheatsheet](/attachments/11635089.pdf) is also provided.
{{% /notice %}}
36
37
38
39
40
41
42
43

A few examples of the basic commands are provided here.  For a full
explanation of all of Anaconda/Conda's capabilities, see the
documentation linked above. 

Anaconda is provided through the `anaconda` module on HCC machines.  To
begin using it, load the Anaconda module.

Adam Caprez's avatar
Adam Caprez committed
44
45
{{% panel theme="info" header="Load the Anaconda module to start using Conda" %}}
{{< highlight bash >}}
46
module load anaconda
Adam Caprez's avatar
Adam Caprez committed
47
48
{{< /highlight >}}
{{% /panel %}}
49

Adam Caprez's avatar
Adam Caprez committed
50
To display general information about Conda/Anaconda, use the `info` subcommand.
51

Adam Caprez's avatar
Adam Caprez committed
52
53
{{% panel theme="info" header="Display general information about Conda/Anaconda" %}}
{{< highlight bash >}}
54
conda info
Adam Caprez's avatar
Adam Caprez committed
55
56
{{< /highlight >}}
{{% /panel %}}
57
58
59
60
61
62

Conda allows the easy creation of isolated, custom environments with
packages and versions of your choosing.  To show all currently available
environments, and which is active, use the `info `subcommand with the
`-e` option.

Adam Caprez's avatar
Adam Caprez committed
63
64
{{% panel theme="info" header="List available environments" %}}
{{< highlight bash >}}
65
conda info -e
Adam Caprez's avatar
Adam Caprez committed
66
67
{{< /highlight >}}
{{% /panel %}}
68
69
70

The active environment will be marked with an asterisk (\*) character.

Adam Caprez's avatar
Adam Caprez committed
71
72
The `list` command will show all packages installed
in the currently active environment.
73

Adam Caprez's avatar
Adam Caprez committed
74
75
{{% panel theme="info" header="List installed packages in current environment" %}}
{{< highlight bash >}}
76
conda list
Adam Caprez's avatar
Adam Caprez committed
77
78
{{< /highlight >}}
{{% /panel %}}
79

Adam Caprez's avatar
Adam Caprez committed
80
To find the names of packages, use the `search` subcommand.
81

Adam Caprez's avatar
Adam Caprez committed
82
83
{{% panel theme="info" header="Search for packages" %}}
{{< highlight bash >}}
84
conda search numpy
Adam Caprez's avatar
Adam Caprez committed
85
86
{{< /highlight >}}
{{% /panel %}}
87
88
89
90
91

If the package is available, this will also display available package
versions and compatible Python versions the package may be installed
under.

Natasha Pavlovikj's avatar
i    
Natasha Pavlovikj committed
92
93
### Creating Custom Anaconda Environment

94
95
96
97
98
The `create` command is used to create a new environment.  It requires
at a minimum a name for the environment, and at least one package to
install.  For example, suppose we wish to create a new environment, and
need version 1.8 of NumPy.

Adam Caprez's avatar
Adam Caprez committed
99
100
{{% panel theme="info" header="Create a new environment by providing a name and package specification" %}}
{{< highlight bash >}}
101
conda create -n mynumpy numpy=1.8 
Adam Caprez's avatar
Adam Caprez committed
102
103
{{< /highlight >}}
{{% /panel %}}
104
105
106
107

This will create a new environment called 'mynumpy' and installed NumPy
version 1.8, along with any required dependencies.  

Adam Caprez's avatar
Adam Caprez committed
108
To use the environment, we must first *activate* it.
109

Adam Caprez's avatar
Adam Caprez committed
110
111
{{% panel theme="info" header="Activate environment" %}}
{{< highlight bash >}}
112
source activate mynumpy
Adam Caprez's avatar
Adam Caprez committed
113
114
{{< /highlight >}}
{{% /panel %}}
115
116
117
118

Our new environment is now active, and we can use it.  The shell prompt
will change to indicate this as well (this can be disable if desired).

Natasha Pavlovikj's avatar
i    
Natasha Pavlovikj committed
119
120
121
122
123
124
125
126
127
128
129
130
### Creating Custom GPU Anaconda Environment

We provide GPU versions of various frameworks such as `tensorflow`, `keras`, `theano`, via [modules](../module_commands). However, sometimes you may need additional libraries or packages that are not available as part of these modules. In this case, you will need to create your own GPU Anaconda environment.

To do this, you need to first clone one of our GPU modules to a new Anaconda environment, and then install the desired packages in this new environment.

The reason for this is that the GPU modules we support are built using the specific CUDA drivers our GPU nodes have. If you just create custom GPU environment without cloning the module, your code will not utilize the GPUs.


For example, if you want to use `tensorflow` with additional packages, first do:
{{% panel theme="info" header="Cloning GPU module to a new Anaconda environment" %}}
{{< highlight bash >}}
131
module load tensorflow-gpu/py36/1.12 anaconda
Natasha Pavlovikj's avatar
i    
Natasha Pavlovikj committed
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
conda create -n tensorflow-gpu-1.12-custom --clone $CONDA_DEFAULT_ENV
module purge
{{< /highlight >}}
{{% /panel %}}

This will create a new `tensorflow-gpu-1.12-custom` environment in your home directory that is a copy of the `tensorflow-gpu` module. Then, you can install the additional packages you need in this environment.
{{% panel theme="info" header="Install new packages in the currently active environment" %}}
{{< highlight bash >}}
module load anaconda
source activate tensorflow-gpu-1.12-custom
conda install <packages>
{{< /highlight >}}
{{% /panel %}}

Next, whenever you want to use this custom GPU Anaconda environment, you need to add these two lines in your submit script:
{{< highlight bash >}}
module load anaconda
source activate tensorflow-gpu-1.12-custom
{{< /highlight >}}

{{% notice info %}}
If you have custom GPU Anaconda environment please only use the two lines from above and **DO NOT** load the module you have cloned earlier. Using `module load tensorflow-gpu/py36/1.12` and `source activate tensorflow-gpu-1.12-custom` in the same script is **wrong** and may give you various errors and incorrect results.
{{% /notice %}}


Adam Caprez's avatar
Adam Caprez committed
157
### Adding Packages to an Existing Environment
158
159
160
161
162
163

To install additional packages in an environment, use the `install`
subcommand.  Suppose we want to install iPython in our 'mynumpy'
environment.  While the environment is active, use `install `with no
additional arguments.  

Adam Caprez's avatar
Adam Caprez committed
164
165
{{% panel theme="info" header="Install a new package in the currently active environment" %}}
{{< highlight bash >}}
166
conda install ipython
Adam Caprez's avatar
Adam Caprez committed
167
168
{{< /highlight >}}
{{% /panel %}}
169
170
171
172

If you aren't currently in the environment you wish to install the
package in, add the `-n `option to specify the name.

Adam Caprez's avatar
Adam Caprez committed
173
174
{{% panel theme="info" header="Install new packages in a specified environment" %}}
{{< highlight bash >}}
175
conda install -n mynumpy ipython
Adam Caprez's avatar
Adam Caprez committed
176
177
{{< /highlight >}}
{{% /panel %}}
178

Adam Caprez's avatar
Adam Caprez committed
179
The `remove` subcommand to uninstall a package functions similarly.
180

Adam Caprez's avatar
Adam Caprez committed
181
182
{{% panel theme="info" header="Remove package from currently active environment" %}}
{{< highlight bash >}}
183
conda remove ipython
Adam Caprez's avatar
Adam Caprez committed
184
185
{{< /highlight >}}
{{% /panel %}}
186

Adam Caprez's avatar
Adam Caprez committed
187
188
{{% panel theme="info" header="Remove package from environment specified by name" %}}
{{< highlight bash >}}
189
conda remove -n mynumpy ipython
Adam Caprez's avatar
Adam Caprez committed
190
191
{{< /highlight >}}
{{% /panel %}}
192

Adam Caprez's avatar
Adam Caprez committed
193
To exit an environment, we *deactivate* it.
194

Adam Caprez's avatar
Adam Caprez committed
195
196
{{% panel theme="info" header="Exit current environment" %}}
{{< highlight bash >}}
197
source deactivate
Adam Caprez's avatar
Adam Caprez committed
198
199
{{< /highlight >}}
{{% /panel %}}
200
201
202
203

Finally, to completely remove an environment, add the `--all `option
to `remove`.

Adam Caprez's avatar
Adam Caprez committed
204
205
{{% panel theme="info" header="Completely remove an environment" %}}
{{< highlight bash >}}
206
conda remove -n mynumpy --all
Adam Caprez's avatar
Adam Caprez committed
207
208
{{< /highlight >}}
{{% /panel %}}
209

Adam Caprez's avatar
Adam Caprez committed
210
### Using an Anaconda Environment in a Jupyter Notebook on Crane
211
212

It is not difficult to make an Anaconda environment available to a
Adam Caprez's avatar
Adam Caprez committed
213
214
Jupyter Notebook. To do so, follow the steps below, replacing
`myenv` with the name of the Python or R environment you wish to use:
215
216

1.  Stop any running Jupyter Notebooks and ensure you are logged out of
Adam Caprez's avatar
Adam Caprez committed
217
    the JupyterHub instance at https://crane.unl.edu
218
219
220
221
222
223
224
225
    1.  If you are not logged out, please click the Control Panel button
        located in the top right corner.
    2.  Click the "Stop My Server" Button to terminate the Jupyter
        server.
    3.  Click the logout button in the top right corner.  
          
2.  Using the command-line environment, load the target conda
    environment:
Adam Caprez's avatar
Adam Caprez committed
226
    {{< highlight bash >}}source activate myenv{{< /highlight >}}
227
228
229
230
231
232

3.  Install the Jupyter kernel and add the environment:

    1.  For a **Python** conda environment, install the IPykernel
        package, and then the kernel specification:

Adam Caprez's avatar
Adam Caprez committed
233
        {{< highlight bash >}}
234
235
236
237
238
        # Install ipykernel
        conda install ipykernel

        # Install the kernel specification
        python -m ipykernel install --user --name "$CONDA_DEFAULT_ENV" --display-name "Python ($CONDA_DEFAULT_ENV)"
Adam Caprez's avatar
Adam Caprez committed
239
        {{< /highlight >}}
240
241
242
243

    2.  For an **R** conda environment, install the jupyter\_client and
        IRkernel packages, and then the kernel specification:

Adam Caprez's avatar
Adam Caprez committed
244
        {{< highlight bash >}}
245
246
247
248
249
250
251
252
253
        # Install PNG support for R, the R kernel for Jupyter, and the Jupyter client
        conda install r-png
        conda install r-irkernel jupyter_client

        # Install jupyter_client 5.2.3 from anaconda channel for bug workaround
        conda install -c anaconda jupyter_client

        # Install the kernel specification
        R -e "IRkernel::installspec(name = '$CONDA_DEFAULT_ENV', displayname = 'R ($CONDA_DEFAULT_ENV)', user = TRUE)"
Adam Caprez's avatar
Adam Caprez committed
254
        {{< /highlight >}}
255
256

4.  Once you have the environment set up, deactivate it:
Adam Caprez's avatar
Adam Caprez committed
257
    {{< highlight bash >}}source deactivate{{< /highlight >}}
258
259
260
261

5.  To make your conda environments accessible from the worker nodes,
    enter the following commands:

Adam Caprez's avatar
Adam Caprez committed
262
    {{< highlight bash >}}
263
264
265
    mkdir -p $WORK/.jupyter
    mv ~/.local/share/jupyter/kernels $WORK/.jupyter
    ln -s $WORK/.jupyter/kernels ~/.local/share/jupyter/kernels
Adam Caprez's avatar
Adam Caprez committed
266
    {{< /highlight >}}
267

Caughlin Bohn's avatar
Caughlin Bohn committed
268
	{{% notice note %}}
Adam Caprez's avatar
Adam Caprez committed
269
270
271
272
**Note**: Step 5 only needs to be done once. Any future created
environments will automatically be accessible from SLURM notebooks
once this is done.
{{% /notice %}}
273

Adam Caprez's avatar
Adam Caprez committed
274
6.  Login to JupyterHub at https://crane.unl.edu
275
    and create a new notebook using the environment by selecting the
Adam Caprez's avatar
Adam Caprez committed
276
    correct entry in the `New` dropdown menu in the top right
277
    corner.  
Adam Caprez's avatar
Adam Caprez committed
278
    {{< figure src="/images/24151931.png" height="400" class="img-border">}}