using_distributed_environment_modules_on_osg.md

title = "Using Distributed Environment Modules on OSG"
description = "Using Distributed Environment Modules on OSG"
weight=40

Many commonly used software packages and libraries are provided on the OSG through the module command.  OSG modules are made available through the OSG Application Software Installation Service (OASIS). The set of modules provided on OSG can differ from those on the HCC clusters.  To switch to the OSG modules environment on an HCC machine:

{{< highlight bash >}} [apple@login.crane~]$ source osg_oasis_init {{< /highlight >}}

Use the module avail command to see what software and libraries are available:

{{< highlight bash >}} [apple@login.crane~]$ module avail ------------------- /cvmfs/oasis.opensciencegrid.org/osg/modules/modulefiles/Core --------------------

abyss/2.0.2 gnome_libs/1.0 pegasus/4.7.1 ant/1.9.4 gnuplot/4.6.5 pegasus/4.7.3 ANTS/1.9.4 graphviz/2.38.0 pegasus/4.7.4 (D) ANTS/2.1.0 (D) grass/6.4.4 phenix/1.10 apr/1.5.1 gromacs/4.6.5 poppler/0.24.1 (D) aprutil/1.5.3 gromacs/5.0.0 (D) poppler/0.32 arc-lite/2015 gromacs/5.0.5.cuda povray/3.7 atlas/3.10.1 gromacs/5.0.5 proj/4.9.1 atlas/3.10.2 (D) gromacs/5.1.2-cuda proot/2014 autodock/4.2.6 gsl/1.16 protobuf/2.5 {{< /highlight >}}   Loading modules is done with the module load command: 

{{< highlight bash >}} [apple@login.crane~]$ module load python/2.7 {{< /highlight >}}

There are two things required in order to use modules in your HTCondor job.

1. Create a wrapper script for the job.  This script will be the executable for your job and will load the module before running the main application.  

2. Include the following requirements in the HTCondor submission script:

   {{< highlight batch >}}Requirements = (HAS_MODULES =?= TRUE){{< /highlight >}}

   or 

   {{< highlight batch >}}Requirements = [Other requirements ] && (HAS_MODULES =?= TRUE){{< /highlight >}}

A simple example using modules on OSG

The following example will demonstrate how to use modules on OSG with an R script that implements a Monte-Carlo estimation of Pi (mcpi.R).

First, create a file called mcpi.R:

{{% panel theme="info" header="mcpi.R" %}}{{< highlight R >}} montecarloPi <- function(trials) { count = 0 for(i in 1:trials) { if((runif(1,0,1)^2 + runif(1,0,1)^2)<1) { count = count + 1 } } return((count*4)/trials) }

montecarloPi(1000) {{< /highlight >}}{{% /panel %}}

Next, create a wrapper script called R-wrapper.sh to load the required modules (R and libgfortran), and execute the R script:

{{% panel theme="info" header="R-wrapper.sh" %}}{{< highlight bash >}} #!/bin/bash

EXPECTED_ARGS=1

if [ $# -ne $EXPECTED_ARGS ]; then echo "Usage: R-wrapper.sh file.R" exit 1 else module load R module load libgfortran Rscript $1 fi {{< /highlight >}}{{% /panel %}}

This script takes the name of the R script (mcpi.R) as it's argument and executes it in batch mode (using the Rscript command) after loading the R and libgfortran modules.

Make the script executable:

{{< highlight bash >}}[apple@login.crane~]$ chmod a+x R-script.sh{{< /highlight >}}

Finally, create the HTCondor submit script, R.submit:

{{% panel theme="info" header="R.submit" %}}{{< highlight batch >}} universe = vanilla log = mcpi.log.(Cluster).(Process) error = mcpi.err.(Cluster).(Process) output = mcpi.out.(Cluster).(Process) executable = R-wrapper.sh transfer_input_files = mcpi.R arguments = mcpi.R

Requirements = (HAS_MODULES =?= TRUE) queue 100 {{< /highlight >}}{{% /panel %}}

This script will queue 100 identical jobs to estimate the value of Pi. Notice that the wrapper script is transferred automatically with the job because it is listed as the executable.  However, the R script (mcpi.R) must be listed after transfer_input_files in order to be transferred with the job.

Submit the jobs with the condor_submit command: 

{{< highlight bash >}}[apple@login.crane~]$ condor_submit R.submit{{< /highlight >}}

Check on the status of your jobs with condor_q:

{{< highlight bash >}}[apple@login.crane~]$ condor_q{{< /highlight >}}

When your jobs have completed, find the average estimate for Pi from all 100 jobs:

{{< highlight bash >}} [apple@login.crane~]$ grep "[1]" mcpi.out.* | awk '{sum += $2} END { print "Average =", sum/NR}' Average = 3.13821 {{< /highlight >}}