Our project roadmaps are maintained on Google Drive and editable by members of the lab. A roadmap is only provided as an indication of where the project is heading. Please be aware that our priorities may change and the roadmap be revised accordingly at any time. We learn from each iteration of the languages and adjust the roadmap according to this experience.

REQUIREMENTS

  1. Simplify notations to facilitate naive use and documentation. Provide auto-completion and visual aids.
  2. Make cardinality of tasks and data explicit. One should not have to wonder how many times (0, 1 or n) a task will be executed, or how to change cardinality.
  3. Distinguish between libraries of processes, and a workflow. Nextflow mixes the two at the moment, which makes it hard to reuse processes defined in another workflow.

TODO

  1. Expose the cluster name in the cluster config. Make the name a parameters for all the cluster-* commands.
  2. Install packages on the cluster nodes with ansible to enable parallel installation on nodes of the cluster (needed for cluster with tens of nodes).
  3. Implement a way to link the process folder in the console output to the location (open the terminal, finder…). This will save time during training and allow to explain other aspects in more details.
  4. Implement a way for a process to indicate the resources it needs to run on a SGE cluster (similar to GobyWeb). These requirements will be then translated to SGE directives with a process scope (http://www.nextflow.io/docs/latest/process.html#process-clusteroptions).

DONE:

Released 1.5.0

  1. Support provisioning of a cluster in the cloud with elasticluster.
  2. Support running workflows in the cloud

Released 1.4.1

  1. Add intention to duplicate channel output into several new channels.
  2. Add similar intention to duplicate workflow inputs.

Released 1.4.0

  1. Migrated to MPS 3.3.

Released 1.3.1

  1. Make sure icons are visible when running as a plugin.
  2. Make sure absolute paths can be navigated with interactive docker path. Should always work irrespective of configuration for working dir.
  3. Support HiDPI icons.
  4. Fixed typesystem for MapFunction
  5. Added intention to create a Dockerfile from a configured NyoshScript.

Released 1.3

  1. Production mode story: Frozen production environment for clinical applications or fully reproducible runs for publication. Install GobyWeb resources inside the image, so that a container can start immediately with installed resources. Provide easy switch from development to production environment.

Released 1.2

  1. Developed image to install GobyWeb resources and artifacts before a Process starts
  2. Develop Process and Workflow for Training sessions. Focus on generating RNA-Seq counts with Kallisto.
  3. Fix any problem encountered in previous step.
  4. Write documentation for new features introduced in 1.2
  5. Support docker containers (merge docker branch)
  6. Add ability to refer to resources and artifacts from within the RichTextScript.
  7. Add support for running simple BASH script in a Docker container (to help quick debugging).

Released 1.1

  1. Add a direct dependency against the logger plugin in the build script.
  2. Implement most functions supported by Nextflow.io (e.g., collectFiles, functions with closures, etc). [Jason]
  1. Implement constants (e.g., variables that do not create channels, but whose value is always the same and can be used directly inside a nextflow.io process). [Jason]
  2. Implement command line parameters, a variation on constants, where initializer is taken from values on the command line.
  3. Make one or a few short videos demonstrating how to create processes and the advantage of the workbench when developing workflows.
  4. Address GitHub issues
  5. Release 1.1
  6. Write Execution and Configuration chapters of the documentation booklet. [Manuele]
  7. Merge 1.0 changes back into master

Released 1.0

  1. Write documentation booklet [explain basics of using the workbench, how to run a workflow, how to configure the environment, explain how to extend the language, such as when adding new functions].
  2. Fixes to apply to the preview release (build #177)
  3. collate function needs a textgen.
  4. Various bug fixes, see 1.0 branch commit log.
  5. Support running via ssh on a remote host where nextflow is installed.
  6. Implement mechanism to set value of output in a process. Using expressions with input values, variables from baselanguage script. Should use a syntax a bit easier to follow than nextflow’s (e.g., type name(=[expression to calculate value to output]).
  7. Refactor channel initializer concept hierarchy to follow simpler design (discussed Monday July 27). This will remove a lot of unecessary code and make the editor experience more consistent for the end-user.
  8. Implement tuples.
  9. Detect and avoid cycles in the workflow. For possible starting points, see: (Jason wrote: I made it such that there can be circular dependencies, which should be found by the user. Cycles are not possible with the model and an algorithm was not needed)  
    1. http://stackoverflow.com/questions/261573/best-algorithm-for-detecting-cycles-in-a-directed-graph
    2. http://www.cs.nyu.edu/courses/summer04/G22.1170-001/6a-Graphs-More.pdf

 

  1. Refactored Channel concept to ProcessInputOutput. Substantially simplified the concept hierarchy. Removed behavior methods used in TextGen and use polymorhism when appropriate to determine type compatibility and perform TextGen output for process input/output.
  2. Implement epilog/reporting constructs (that generate to .subscribe blocks).
  3. Model processes, with input, output and scripts.
  4. Model process inputs
  5. Model process outputs
  6. Implement scripts as BASH text with RichText (http://voelter.de/data/pub/mpm13.pdf).
  7. Model explicit cardinality of ChannelTypes (list, tuple, single values).
  8. Model workflow I/O (partially done)
  9. Make it possible to use auto-completion to refer workflow parameters from a Process Script.
    1. keep in mind that a process is defined always outside a workflow
    2. a process could model ValueFromAWorkflow similarly to ValueFromAChannel (make common super-concept that implements IWord)
    3. when a process is instantiated in the workflow, entries for parameters expected from the process must be automatically added to the workflow input args section
    4. if a param is deleted, a typechecking rule should highlight the problem in the editor in the ProcessRef node with an appropriate error message
  10. Define operators that can be applied to a Channel instance
    1. ignore for the moment the compatibility with the ChannelType, all operators can be applied to any channel
    2. more than one operator can be applied to a channel
  11. Execute the same test pipeline we wrote to evaluate Nextflow (submit, analyze, combine)
  12. Model workflow configuration (where it will be executed, how, and so on)
  13. Must be able to submit / execute the test pipeline on the OGE cluster (must update to Java 7)
  14. Make explicit how many times a process ref will be executed in the workflow (according to its expected ChannelTypes and their operators, e.g. toList() makes a process ref to be executed only once).
    1. Show this cardinality in the workflow editor (0, 1 or N)
  15. Validate connected channels through ChannelType’s compatibility
  16. Run directly from MPS [on local machine]
  17. Implement typesystem for workflows and processes