The Runtime environment defines the information available at execution time for Executable plugin. The information can be accessed within the plugin’s script to support its activity.

Default Dependencies


By default, all the Executable Plugins have a dependency against the resource GOBYWEB_SERVER_SIDE. This resource includes a set of basic tools used by the execution engine itself to handle the job environment. Options declared in the resource are available as dynamic variables.

Among the others, the resource makes available two useful dynamic environment variables:

RESOURCES_GOBYWEB_SERVER_SIDE_QUEUE_WRITER

This variable is exported by resource GOBYWEB_SERVER_SIDE. It refers to an utility that can be used to report information on the status of the current job:

     ${RESOURCES_GOBYWEB_SERVER_SIDE_QUEUE_WRITER} \
            --tag ${TAG} \
            --status ${JOB_PART_DIFF_EXP_STATUS} \
            --description "Start discover-sequence-variations for part # ${CURRENT_PART}." \
            --index ${CURRENT_PART} \
            --job-type job-part

RESOURCES_FETCH_URL_SCRIPT

This variable is available because of a transitive dependency on the resource FETCH_URL declared in the GOBYWEB_SERVER_SIDE’s configuration file. It refers the script that can be used to fetch any resource from a remote URL. This is a sample usage of the script:

    # Download a resource:

  • ${RESOURCES_FETCH_URL_SCRIPT} http://sourceforge.net/projects/samtools/files/samtools/0.1.18/samtools-0.1.18.tar.bz2/download samtools-0.1.18.tar.bz2
  • bunzip2 samtools-${VERSION}.tar.bz2

The script is especially useful for artifacts, where packages to install are often downloaded from remote locations.

Transitive Dependencies


Dependencies among plugins are transitive. This means that if a plugin A depends on B and B has a dependency on C, plugin A can also see dynamic variables from C, has C files available in its JOB_DIR, has C artifacts installed (if C is a resource) and so on.

Dynamic Environment Variables


Dynamic variables are the main source of information available to plugins at execution time. They provide information about input parameters, files, resources and runtime information.

A dynamic variable is an environment variable created on the fly starting from a plugin configuration.

The name of a variable follows this pattern:

(PLUGIN_TYPE)_(PLUGIN_ID)_(SOURCE_ID)

where:

  • PLUGIN_TYPE is one among RESOURCES, PLUGINS_ALIGNER, PLUGINS_ALIGNMENT_ANALYSIS, PLUGINS_TASK, depending on the type of the plugin that declared the source.
  • PLUGIN_ID is the ID of the plugin that declared the source.
  • SOURCE_ID is the ID of the element to which the variable value is associated.

Here are some examples to explain how to access this variables:

  • resource plugin with ID=SAMTOOLS declares a file with ID=EXEC_PATH, the path of the file will be available (to executable plugins depending on the resource) as value of the environment variable RESOURCES_SAMTOOLS_EXEC_PATH
  • aligner plugin with ID=BWA_BAM declares an option with ID=ALIGNER_OPTIONS, the value assigned to this option is available in the environment variable PLUGINS_ALIGNER_BWA_BAM_ALIGNER_OPTIONS

Static Environment Variables


When developing an Executable plugin or preparing an Artifact installation, a certain number of environment variables are available for being used in their respective scripts (script.sh and install.sh).

ALIGNER_OPTIONS

ALIGNER_OPTIONS contains options set by the caller, such as options to support spliced alignments, etc.

AMBIGUITY_THRESHOLD

AMBIGUITY_THRESHOLD is obtained from the plugin option mechanism, if available.

BASENAME

The basename to assign to the output alignment.

BISULFITE_OPTION

BISULFITE_OPTION is equals to “true” when the reads have been bisulfite converted, “false” otherwise.

CURRENT_PART

CURRENT_PART contains 1 if the job is not parallelized or the value of the part processed by the current sub-job.

END_POSITION

END_POSITION contains byte offset where to stop reading into the Goby compact-reads sample file.

ENTRIES_DIRECTORY

ENTRIES_DIRECTORY is the folder where entries files of the input alignments are available.

ENTRIES_FILES

ENTRIES_FILES is a space-separated list with all the entries files of the input alignments.

JOB_DIR

JOB_DIR contains the absolute path of the directory where plugin files are copied before the job execution. Files declared in the plugin configuration can be referred in the script.sh as follows:

# Refer a plugin file:
     ${JOB_DIR}/filename

Files stored here are maintained after the plugin execution ends. The job directory folder will remain on the cluster shared filesystem until it is removed (typically administrators install crontab scripts that prune jobs older than one week or so). Keeping the job folder for this period of time around helps troubleshoot job execution problems.

JOB_ID

JOB_ID contains the unique identifier assigned by the execution engine to the job or a job part (in case of parallelized jobs).

JOB_NAME

JOB_NAME contains the name assigned by the execution engine to the job or a job part (in case of parallelized jobs).

JOB_PART_DIFF_EXP_STATUS

JOB_PART_DIFF_EXP_STATUS can be used to report back information about the execution status of the current job. For example:

     ${RESOURCES_GOBYWEB_SERVER_SIDE_QUEUE_WRITER} \
       --tag ${TAG} \
       --status ${JOB_PART_DIFF_EXP_STATUS} \
       --description "Start discover-sequence-variations for part # ${CURRENT_PART}." \
       --index ${CURRENT_PART} \
       --job-type job-part

JOB_PART_FAILED_STATUS

JOB_PART_DIFF_EXP_STATUS contains a failed code. It can be used to report back problems about the execution of the current job. For example:

     ${RESOURCES_GOBYWEB_SERVER_SIDE_QUEUE_WRITER} \
       --tag ${TAG} \
       --status ${JOB_PART_DIFF_EXP_STATUS} \
       --description "Start discover-sequence-variations for part # ${CURRENT_PART}." \
       --index ${CURRENT_PART} \
       --job-type job-part

JOB_SCRIPT

JOB_SCRIPT contains the absolute path of the current executed script.

ORGANISM

ORGANISM is the type of organism of the input alignments or samples.

PAIRED_END_ALIGNMENT

PAIRED_END_ALIGNMENT is equals to “true” when the reads are paired-end, “false” otherwise.

PAIRED_END_DIRECTIONS

PAIRED_END_DIRECTIONS is the paired read directions, as specified on the user interface for the sample.

READS

READS contains the complete Goby compact-reads sample file

RESULT_FILE_EXTENSION

The extension of the expected results produced by the job. It depends on the produces*FormatOutput element declared in the executable plugin’s configuration file.

START_POSITION

START_POSITION contains byte offset where to start reading into the Goby compact-reads sample file.

SLICING_PLAN_FILENAME

SLICING_PLAN_FILENAME contains the absolute path of the file with the slicing plan.

TAG

TAG contains the unique identified assigned to the job by the PluginsSDK.

TMPDIR

TMPDIR contains the absolute path of the working directory where a plugin is executed.
Files stored here are deleted after the plugin execution ends.

Built-in functions


When developing an Executable plugin, a certain number of functions are available in the script.sh.

dieUponError

dieUponError allows to check the returned code of the previous command and reports an error message to the submission engine in case of failure. Sample usage:

    # Check an exit code and report an error:

  • READ_FILES_LIST=`${FILESET_COMMAND} –fetch INPUT_READS.READS_FILE`
  • dieUponError “Failed to fetch INPUT_READS.READS_FILE”

Calling this function also causes the immediate halt of the job execution.