using_anaconda_package_manager.md 8.25 KB
Newer Older
Carrie A Brown's avatar
Carrie A Brown committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
+++
title = "Using Anaconda Package Manager"
description = "How to use the Anaconda Package Manager on HCC resources."
weight=10
+++

[Anaconda](https://www.anaconda.com/what-is-anaconda),
from [Anaconda, Inc](https://www.anaconda.com)
is a completely free enterprise-ready distribution for large-scale data
processing, predictive analytics, and scientific computing. It includes
over 195 of the most popular Python packages for science, math,
engineering, and data analysis. **It also offers the ability to easily
create custom _environments_ by mixing and matching different versions
of Python and/or R and other packages into isolated environments that
individual users are free to create.**  Anaconda includes the `conda`
package and environment manager to make managing these environments
straightforward.

- [Using Anaconda](#using-anaconda)
- [Installing Packages](#installing-packages)
- [Adding Packages to an Existing Environment](#adding-packages-to-an-existing-environment)
- [Using an Anaconda Environment in a Jupyter Notebook on Crane](#using-an-anaconda-environment-in-a-jupyter-notebook-on-crane)

### Using Anaconda

While the standard methods of installing packages via `pip`
and `easy_install` work with Anaconda, the preferred method is using
the `conda` command.  

{{% notice info %}}
Full documentation on using Conda is available
at http://conda.pydata.org/docs/

A [cheatsheet](/attachments/11635089.pdf) is also provided.
{{% /notice %}}

A few examples of the basic commands are provided here.  For a full
explanation of all of Anaconda/Conda's capabilities, see the
documentation linked above. 

Anaconda is provided through the `anaconda` module on HCC machines.  To
begin using it, load the Anaconda module.

{{% panel theme="info" header="Load the Anaconda module to start using Conda" %}}
{{< highlight bash >}}
module load anaconda
{{< /highlight >}}
{{% /panel %}}

To display general information about Conda/Anaconda, use the `info` subcommand.

{{% panel theme="info" header="Display general information about Conda/Anaconda" %}}
{{< highlight bash >}}
conda info
{{< /highlight >}}
{{% /panel %}}

Conda allows the easy creation of isolated, custom environments with
packages and versions of your choosing.  To show all currently available
environments, and which is active, use the `info `subcommand with the
`-e` option.

{{% panel theme="info" header="List available environments" %}}
{{< highlight bash >}}
conda info -e
{{< /highlight >}}
{{% /panel %}}

The active environment will be marked with an asterisk (\*) character.

The `list` command will show all packages installed
in the currently active environment.

{{% panel theme="info" header="List installed packages in current environment" %}}
{{< highlight bash >}}
conda list
{{< /highlight >}}
{{% /panel %}}

### Installing Packages

To find the names of packages, use the `search` subcommand.

{{% panel theme="info" header="Search for packages" %}}
{{< highlight bash >}}
conda search numpy
{{< /highlight >}}
{{% /panel %}}

If the package is available, this will also display available package
versions and compatible Python versions the package may be installed
under.

The `create` command is used to create a new environment.  It requires
at a minimum a name for the environment, and at least one package to
install.  For example, suppose we wish to create a new environment, and
need version 1.8 of NumPy.

{{% panel theme="info" header="Create a new environment by providing a name and package specification" %}}
{{< highlight bash >}}
conda create -n mynumpy numpy=1.8 
{{< /highlight >}}
{{% /panel %}}

This will create a new environment called 'mynumpy' and installed NumPy
version 1.8, along with any required dependencies.  

To use the environment, we must first *activate* it.

{{% panel theme="info" header="Activate environment" %}}
{{< highlight bash >}}
source activate mynumpy
{{< /highlight >}}
{{% /panel %}}

Our new environment is now active, and we can use it.  The shell prompt
will change to indicate this as well (this can be disable if desired).

### Adding Packages to an Existing Environment

To install additional packages in an environment, use the `install`
subcommand.  Suppose we want to install iPython in our 'mynumpy'
environment.  While the environment is active, use `install `with no
additional arguments.  

{{% panel theme="info" header="Install a new package in the currently active environment" %}}
{{< highlight bash >}}
conda install ipython
{{< /highlight >}}
{{% /panel %}}

If you aren't currently in the environment you wish to install the
package in, add the `-n `option to specify the name.

{{% panel theme="info" header="Install new packages in a specified environment" %}}
{{< highlight bash >}}
conda install -n mynumpy ipython
{{< /highlight >}}
{{% /panel %}}

The `remove` subcommand to uninstall a package functions similarly.

{{% panel theme="info" header="Remove package from currently active environment" %}}
{{< highlight bash >}}
conda remove ipython
{{< /highlight >}}
{{% /panel %}}

{{% panel theme="info" header="Remove package from environment specified by name" %}}
{{< highlight bash >}}
conda remove -n mynumpy ipython
{{< /highlight >}}
{{% /panel %}}

To exit an environment, we *deactivate* it.

{{% panel theme="info" header="Exit current environment" %}}
{{< highlight bash >}}
source deactivate
{{< /highlight >}}
{{% /panel %}}

Finally, to completely remove an environment, add the `--all `option
to `remove`.

{{% panel theme="info" header="Completely remove an environment" %}}
{{< highlight bash >}}
conda remove -n mynumpy --all
{{< /highlight >}}
{{% /panel %}}

### Using an Anaconda Environment in a Jupyter Notebook on Crane

It is not difficult to make an Anaconda environment available to a
Jupyter Notebook. To do so, follow the steps below, replacing
`myenv` with the name of the Python or R environment you wish to use:

1.  Stop any running Jupyter Notebooks and ensure you are logged out of
    the JupyterHub instance at https://crane.unl.edu
    1.  If you are not logged out, please click the Control Panel button
        located in the top right corner.
    2.  Click the "Stop My Server" Button to terminate the Jupyter
        server.
    3.  Click the logout button in the top right corner.  
          
2.  Using the command-line environment, load the target conda
    environment:
    {{< highlight bash >}}source activate myenv{{< /highlight >}}

3.  Install the Jupyter kernel and add the environment:

    1.  For a **Python** conda environment, install the IPykernel
        package, and then the kernel specification:

        {{< highlight bash >}}
        # Install ipykernel
        conda install ipykernel

        # Install the kernel specification
        python -m ipykernel install --user --name "$CONDA_DEFAULT_ENV" --display-name "Python ($CONDA_DEFAULT_ENV)"
        {{< /highlight >}}

    2.  For an **R** conda environment, install the jupyter\_client and
        IRkernel packages, and then the kernel specification:

        {{< highlight bash >}}
        # Install PNG support for R, the R kernel for Jupyter, and the Jupyter client
        conda install r-png
        conda install r-irkernel jupyter_client

        # Install jupyter_client 5.2.3 from anaconda channel for bug workaround
        conda install -c anaconda jupyter_client

        # Install the kernel specification
        R -e "IRkernel::installspec(name = '$CONDA_DEFAULT_ENV', displayname = 'R ($CONDA_DEFAULT_ENV)', user = TRUE)"
        {{< /highlight >}}

4.  Once you have the environment set up, deactivate it:
    {{< highlight bash >}}source deactivate{{< /highlight >}}

5.  To make your conda environments accessible from the worker nodes,
    enter the following commands:

    {{< highlight bash >}}
    mkdir -p $WORK/.jupyter
    mv ~/.local/share/jupyter/kernels $WORK/.jupyter
    ln -s $WORK/.jupyter/kernels ~/.local/share/jupyter/kernels
    {{< /highlight >}}

{{% notice note %}}
**Note**: Step 5 only needs to be done once. Any future created
environments will automatically be accessible from SLURM notebooks
once this is done.
{{% /notice %}}

6.  Login to JupyterHub at https://crane.unl.edu
    and create a new notebook using the environment by selecting the
    correct entry in the `New` dropdown menu in the top right
    corner.  
    {{< figure src="/images/24151931.png" height="400" class="img-border">}}