2.3.2 [pushed to our new GobyWeb CTSC instance on Sept 17 2014]
- Improved integration with the Plugins SDK.
- Fixed progress bar for running alignments.
- Various fixes and optimizations on job results access and management.
- Fixed deletion of gobyweb objects with no fileset associated.
- Fixed sharing of gobyweb objects among users.
- Fixed handling on multiple version of the same artifact in the repository.
- Manage data on a remote cluster.
- Submit jobs using the Plugins SDK 2.4.0.
- Model data as Filesets.
- Implement access to Lucene database with an HTTP index.
- Configuration has been substantially simplified, thanks to the clean architecture of the SDK.
- Fixed a problem with the Google Chart API that prevented sequence variations plots to be displayed in the web app.
- Advance dependency on Artifacts to 2.2.3:
- Simplified reporting in log files after artifact installation. Log only shows output for steps of the installation that failed, or indicates success.
- Fixed a bug that caused exports for artifacts with related version numbers to be generated.
- Exports are now only generated for artifacts listed in an artifact installation request.
- Delete temporary artifact installation directories after the installation succeeds or fails.
- Advance dependency on Plugins to 2.0.2:
- Add ability to detect when plugin files are declared but do not exist in the plugin directory.
- Add dependency on Grails Standalone plugin. This will help experiment with a standalone GobyWeb jar file for easier installations.
- Expose pathTransformationService for use in pre-deployment scripts.
- Change push script to pull plugins from git or rsync to remote plugin root.
- Set BWA_GOBY_NUM_THREADS directly in plugin script.
- Add new configuration setting for loading plugin configuration from an arbitrary location.
- Remove plugins root from GobyWeb project.
- Default configuration keeps log files for 30 days.
- Export READS_PLATFORM as environment variable to alignment executions.
- Introduce artifacts for plugins (depend on Artifacts 2.1).
- Plugins System separated from gobyweb: new plugins registry and configurations hierarchy (depend on Plugins 2.0.)
- Distribute plugin configurations separately from the Web Application.
- Multiple Alignments now use an SQL query to list alignments.
- Batch upload procedure with and without TSV files was not working properly because of some missing auto-binding properties in the Sample class.
- Solved some potential NPEs when listing Analyses with no feedback from their correspondent jobs.
2.0 [pushed to our GobyWeb CTSC instance on Mar 21 2013]
- Changes in the technological foundations: migration to Grails 2.1.3.
- New and simplified configuration of the application.
- Improved data access performance and scalability through optimized SQL queries.
- New Data Migration Tool for importing data from previous GobyWeb version.
- Runtime support for moving data across different GobyWeb deployments (absolute paths are no longer stored in the database).
- Extensive tracing of all the user activities.
- New Garbage Collector for uploaded files. Expiration time can be controlled with a new parameter fileExpirationTimeInDays.
- Statistics on storage consumption available from administrator accounts.
- Solved an issue with TSV-based uploads (TSV file was considered a reads file too).
1.9.2 [pushed to our GobyWeb CTSC instance on Nov 28 2012]
- Automatic detection if GobyWeb is running in an embedded or standard web server. This avoid to list recognized servers’ names.
- Solved a null pointer exception for just registered users’ statistics
- Read-only users can now view their alignments packaged for download.
- Write type of analysis rather than “diffExpLongType” on web pages.
- License GobyWeb plugins under the LGPL3 license. We make this change to make it possible for others to contribute new GobyWeb plugins, and to facilitate the inspection, reuse and extension the initial set of plugins distributed with GobyWeb. We plan to host source code for GobyWeb plugins at GitHub before.
- Added a mechanism to periodically calculate data usage for sets of users. A configuration parameter (intervalForStatisticsInHours, set in webapp/conf/Config.groovy) allows to configure the interval in between two statistics’ calculations. Usage data can be viewed for all users by administrator accounts. Individual users can see how much data their account holds (break-down by samples, alignments and group comparison analyses).
- Defined Lucene index view of the results for EdgeR plugin.
- Increase JVM memory for Bismark plugin (we found one dataset where one chunk ran out of memory with the previous configuration).
- Changed Bismark plugin to support mapping reads longer than 60bp (bismark requires all bases on one line in the fastq file so we changed the way fastq files are formatted before calling Bismark).
1.9 [pushed to our GobyWeb CTSC instance on Nov 8 2012]
- Sample upload now supports a tab-delimited file with information about how to combine input files into samples. This new import mechanism makes it possible to concatenate several reads file into one sample,
or to group input files into pairs.
- We have completely rewritten the table viewer (results of group comparison) for performance on very large tables and to make it possible to filter out groups of columns more easily. Table indexing is now performed with Lucene and performance of multi column filters is dramatically improved.
- Added a mechanism that makes it possible for plugins to obtain information from the domain objects available to the web app. Using this mechanism, plugins can customize the job start directory before deployment to the grid. See the DIFF_EXP_DESEQ plugin as an example, in config.xml:
<script phase=”pre-deployment” language=”groovy” filename=”sampleToGroups.groovy”/>
- Added a pathogen detection plugin (contributed by Zachary Zeno during his summer internship in the lab, uses Minia or Trinity for contig assembly from short reads, Minia is recommended)
- Added a Bismark aligner plugin (contributed by Zachary Zeno during his summer internship in the lab)
- Added an EdgeR differential expression plugin (contributed by Nyasha Chambwe)
- Added a STAR aligner plugin (see Dobin, Alex et al. STAR: ultrafast universal RNA-seq aligner. Bioinformatics 2012)
- Added a splicing usage differential expression plugin to measure differences in splice site usage across groups of samples (statistics estimated with DESeq or EdgeR), compatible with either GSNAP or STAR aligners.
- Ability to mark users as read only. Read only users can view data, but cannot upload data or submit
- In the group comparison analysis wizard, show the number of alignments in each group.
1.8 [pushed to our GobyWeb CTSC instance on Sept 6 2012]
- Resource plugins can now require other resources as pre-requisite to their installation.
- New option in browse “View cluster queue” to display the running, queued jobs on the cluster.
- Add empirical p-value estimation for base-level plugin. Add DMR attribute that identifies the last base of a DMR of a fixed size with more than n significant sites.
- Display gobyVersion for diffExp
- Collect plugin versions when creating a new DiffExp or AlignJob. Displayed on the respective show screens.
- Changes to support importing multiple tables for browsing.
- The GobyWeb version number now follows the format 1.x.y, following the Goby convention (and the GobyWeb version numbers in changelog). This makes it possible to compute if a version is more recent and is needed to help maintain backward compatibility. About page now should show both the version number and the compilation time.
- Version.txt must be updated manually before a web binary release.
Release 1.7.1 includes a plugin for methylation estimation over annotated regions of the genome and bug fixes for problems found after 1.7 was deployed. This is the first public release of GobyWeb with the plugin infrastructure.
GobyWeb 1.7 is a major release with lots of new features (released Jan 9 2012 to our CTSC community). We have added a brand new plugin mechanism to make it easier to integrate new aligners and analysis tools with GobyWeb. New plugins can be added at run-time and the user interface adapts as needed.
- We parallelized count calculations for Goby and DESeq statistics. You should see major speed improvements for exon analysis.
- Add a mechanism for resource plugins. Alignment and analysis plugins can require resources (by id and version). Only plugins for which all resources are available in the instance of GobyWeb install. The files that a resource plugin are copied to the job directory of alignment/analysis plugins that use the resource. The latest version of a resource is configured it if matches the minimal version number requested by the client plugin. By convention, resource plugins should be described under the plugins/resources directory, and named with ID_VERSION (all uppercase).
- Added support for plugins for alignment and aligment analyses (e.g., diff exp, seqvar)
- New methylation plugin. We introduce the ability to reuse plugin scripts from other plugins. In this case, the methylation plugin imports the SEQ_VAR_GOBY script.sh file as its script. The feature also makes it possible to name plugin scripts something other than script.sh, and to import other files as needed from different plugins.
This version supports single end and paired-end files uploaded in the .fastq.gz.tar format. This format is being delivered by the WMC epigenomic core facility and consists of tar files containing .fastq.gz files. GobyWeb will automatically recognize files that end in the extension .fastq.gz.tar and convert such files to the compact-reads format for further processing. The upload supports paired-end input when the fastq.gz file match the pattern *R0_???* *R1_???* where ? is a digit. For instance, if a tar file contains a_R0_000.fastq.gz and a_R1_000.fastq.gz, the two fastq files are assembled to reconstitute the pairs. Please note that since the epigenomics core provides both reads passing filters and reading not passing filters, we remove all reads not passing quality filters (to this end we assume the reads are produced by the Illumina 1.8+ pipeline and follow the procedure described at http://cancan.cshl.edu/labmembers/gordon/fastq_illumina_filter/, keeping reads without :Y: in the FASTQ read-id line).
This version was slightly modified from version 1.6 to facilitate distribution and installation of a binary distribution. It has been packaged on November 9th 2011 (release tag 20111109).
Version released on Oct 29 1011 to our CTSC investigators. New features and bug fixes include:
- Performs read pre-processing on the grid. This provides much better throughput when importing several read files for projects with billions of reads.
- Convert tab delimited or VCF format to SQLLite tables on the grid. Group comparison now only show the job completed status when the data is readily available and viewable in table format. Several imports can also proceed in parallel, as supported by compute grid capacity.
- Read import now provides status updates in a way similar to alignment and group comparison jobs.
- Multi-download was broken in version 1.5 when we migrated the application to a new application server. Multi-download creation is fixed.
- The type of columns in TSV files is now determined automatically and this type is used when importing the table. This enables to search columns for differential expression results in the Table viewer more efficiently.
- Spliced alignment with GSNAP for RNA-Seq data
- Calling differential allelic expression in RNA-Seq data (with Goby)
- Calling CNVs with significant coverage differences between groups of samples in DNA-Seq data
- Calling genotypes in DNA-Seq (and download VCF files, which can be viewed in IGV)
- Comparing genotypes across groups of samples (e.g., allelic tests at sites covered with reads, or discovery or new variations or mutations)
- Estimating methylation rates in bisulfite converted samples in Methyl-Seq or RRBS data and finding sites differentially methylated between groups of samples (methylation rates are written to VCF and can be viewed natively in IGV).
This version was released to our CTSC investigators in the Spring 2011 and added these features:
This first semi-public release of GobyWeb was made available to investigators of our CTSC at WMC, MSKCC, HSS and Hunter College in January 2011. The version focused on RNA-Seq data analysis and had support for the BWA and Last aligners.