Anatomy of a Plugin


Each plugin lives in a dedicated directory under the plugins root/plugins/ directory. By convention, the name of the directory is the identifier of the plugin or the identifier plus the version.

The minimal requirement of a plugin is its description that must be provided in a configuration file named config.xml. The schema of the file is defined in the plugins.xsd available in the plugins root/schemas directory. Executable plugins must also provide a Bash shell script named script.sh with an implementation of the functions required to that specific type of plugins (for further details, see in each plugin’s page). Resource plugins must also provide a Bash shell script named install.sh when installing artifacts.

Finally, plugins may have an arbitrary number of files associated that are used by the plugin itself or by other plugins depending on it. Such files have to be declared in the plugin configuration file and are moved along with the plugin before its execution.

Steps for creating a Plugin


Here are suggested steps that you could follow to develop a new GobyWeb plugin:

  1. Create a new plugin directory under the plugins root/plugins/(PLUGIN TYPE)/MY_PLUGIN
  2. Write a plugin configuration file under that directory (this file must be called config.xml), and validate it against plugins root/schemas/plugins.xsd. Various XML editors support validation against an XML schema. If you are not sure which editor to use, we would suggest trying Intelij IDEA community edition (free and open-source).
  3. If the plugin is a Resource and declares artifacts, you must provide an artifact.sh script under the plugin directory. See artifacts for template and details about this script.
  4. If the plugin is an Executable plugin, create a script.sh file to implement the plugin’s behavior (see each plugin’s page for details)
  5. Use the command line interface to test the plugin (register an instance of the fileset, submit the executable plugin as job, submit a resource for installation)

Before writing a new plugin, check the existing GitHub GobyWeb plugin repository to discover which plugins are already available and can be reused in the context of the new plugin. Resource plugins are likely to be of interest since you can import them in your new plugin and reuse their functionality. Also consider contributing your plugins back to GitHub to enrich the repository with new plugins when they could be of general use. You can do this by forking the GobyWeb plugin repository and creating a pull request with your changes.

Checking the plugin environment


When writing the script.sh to implement the plugin’s behavior, it is useful to know which runtime environment the plugin will have access to once submitted for execution as job. To do that, the Plugins-SDK offers the command plugins-view-job-env. This command takes as input the ID and Version of the new plugin (resource or executable plugin) and prints out the list of environment variables visible to the plugin.

Here is a sample usage with an executable plugin:

plugins-view-job-env \
--plugins-dir PLUGINS_ROOT_LOCATION \
--job BWA_GOBY_ARTIFACT:1.2

And this is the result of the execution:

Plugin [BWA_GOBY_ARTIFACT, 1.2] has access to the following environment variables:
PLUGINS_ALIGNER_BWA_GOBY_ARTIFACT_ALL_OTHER_OPTIONS
PLUGINS_ALIGNER_BWA_GOBY_ARTIFACT_AMBIGUITY_THRESHOLD
PLUGINS_ALIGNER_BWA_GOBY_ARTIFACT_FILES_SCRIPT
PLUGINS_ALIGNER_BWA_GOBY_ARTIFACT_MAXIMUM_NUMBER_GAP_EXTENSIONS
PLUGINS_ALIGNER_BWA_GOBY_ARTIFACT_MAXIMUM_NUMBER_GAP_OPENS
RESOURCES_BASH_LIBRARY_CALCULATE_MD5
RESOURCES_BASH_LIBRARY_MAPS_IN_BASH3
RESOURCES_BWA_WITH_GOBY_ARTIFACT_INSTALL
RESOURCES_ENSEMBL_GENOMES_INSTALL
RESOURCES_FAI_INDEXED_GENOMES_INSTALL
RESOURCES_FETCH_URL_SCRIPT
RESOURCES_FETCH_URL_SCRIPT_PATTERN
RESOURCES_GOBYWEB_SERVER_SIDE_ARTIFACTS_MANAGER_JAR
RESOURCES_GOBYWEB_SERVER_SIDE_ARTIFACTS_SCRIPT
RESOURCES_GOBYWEB_SERVER_SIDE_DEPENDENCIES_JAR
RESOURCES_GOBYWEB_SERVER_SIDE_FILESET_JAR
RESOURCES_GOBYWEB_SERVER_SIDE_GLOBAL_GOBY_JAR
RESOURCES_GOBYWEB_SERVER_SIDE_ICB_GROOVY_SUPPORT_JAR
RESOURCES_GOBYWEB_SERVER_SIDE_LOG4J_PROPERTIES
RESOURCES_GOBYWEB_SERVER_SIDE_PROCESS_SAMPLES
RESOURCES_GOBYWEB_SERVER_SIDE_QUEUE_WRITER
RESOURCES_GOBYWEB_SERVER_SIDE_QUEUE_WRITER_WRAPPER
RESOURCES_GOBYWEB_SERVER_SIDE_TSV_VCF_TO_SQLITE
RESOURCES_GOBY_CPP_API_INSTALL
RESOURCES_GROOVY_EXECUTABLE
RESOURCES_GROOVY_INSTALL
RESOURCES_PROTOBUF_CPP_INSTALL
RESOURCES_SAMTOOLS_EXEC_PATH
RESOURCES_SAMTOOLS_INSTALL

 

This output means that, for instance, the variable ${RESOURCES_SAMTOOLS_INSTALL} will be defined when the plugin is executed.

Here is a sample usage with a resource plugin:

plugins-view-job-env \
--plugins-dir PLUGINS_ROOT_LOCATION \
--resource ANNOTATE_VCF:1.1

And this is the result of the execution:

Plugin [ANNOTATE_VCF, 1.1] has access to the following environment variables:
RESOURCES_ANNOTATE_VCF_EXEC_PATH
RESOURCES_BASH_LIBRARY_CALCULATE_MD5
RESOURCES_BASH_LIBRARY_MAPS_IN_BASH3
RESOURCES_ENSEMBL_ANNOTATIONS_BIOMART_SCRIPT
RESOURCES_ENSEMBL_ANNOTATIONS_INSTALL
RESOURCES_ENSEMBL_API_INSTALL
RESOURCES_ENSEMBL_API_SETUP
RESOURCES_ENSEMBL_GENOMES_INSTALL
RESOURCES_FETCH_URL_SCRIPT
RESOURCES_FETCH_URL_SCRIPT_PATTERN
RESOURCES_GOBYWEB_SERVER_SIDE_ARTIFACTS_MANAGER_JAR
RESOURCES_GOBYWEB_SERVER_SIDE_ARTIFACTS_SCRIPT
RESOURCES_GOBYWEB_SERVER_SIDE_DEPENDENCIES_JAR
RESOURCES_GOBYWEB_SERVER_SIDE_FILESET_JAR
RESOURCES_GOBYWEB_SERVER_SIDE_GLOBAL_GOBY_JAR
RESOURCES_GOBYWEB_SERVER_SIDE_ICB_GROOVY_SUPPORT_JAR
RESOURCES_GOBYWEB_SERVER_SIDE_LOG4J_PROPERTIES
RESOURCES_GOBYWEB_SERVER_SIDE_PROCESS_SAMPLES
RESOURCES_GOBYWEB_SERVER_SIDE_QUEUE_WRITER
RESOURCES_GOBYWEB_SERVER_SIDE_QUEUE_WRITER_WRAPPER
RESOURCES_GOBYWEB_SERVER_SIDE_TSV_VCF_TO_SQLITE
RESOURCES_GROOVY_EXECUTABLE
RESOURCES_GROOVY_INSTALL
RESOURCES_SAM_JDK_INSTALL
RESOURCES_TABIX_BGZIP_EXEC_PATH
RESOURCES_TABIX_EXEC_PATH
RESOURCES_TABIX_INSTALL
RESOURCES_VARIANT_EFFECT_PREDICTOR_INSTALL
RESOURCES_VARIANT_EFFECT_PREDICTOR_SCRIPT
RESOURCES_VCF_TOOLS_INSTALL